path: root/meta/classes/ccache.bbclass
Commit message (Collapse)AuthorAgeFilesLines
* ccache.bbclass: Set CCACHE_CONFIGPATH to fix race issuesRobert Yang2019-01-141-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | Fixed race issues when parallel build: ccache: error: /path/to/ccache/i586-poky-linux/mmc-utils/ccache.conf: No such file or directory ccache: error: /path/to/ccache/i586-poky-linux/mmc-utils/ccache.conf: No such file or directory This is because we set CCACHE_DIR for earch recipe, and ccache will create a ccache.conf for each CCACHE_DIR when CCACHE_CONFIGPATH is not set, but there might be a race issue in parallel build: ccache gcc file1.c ccache gcc file2.c If the two ccache processes use fopen(path, "w") to create ccache.conf at the same time, the error would happen. Set CCACHE_CONFIGPATH to meta/conf/ccache.conf can fix the problem, and we can add other configs to the file when needed. And also set cache_dir_levels to 1 (default is 2) since each recipe has a cache dir, thus we don't have too many files in one dir. (From OE-Core rev: 2abbc4d0cd571e82ed6188d3b2d84b4cd6be25e8) Signed-off-by: Robert Yang <> Signed-off-by: Richard Purdie <>
* ccache.bbclass: Add task do_cleanccacheRobert Yang2019-01-141-0/+14
| | | | | | | (From OE-Core rev: 36cead66fbadd8c3827aec4b67ea124ee3c2ff94) Signed-off-by: Robert Yang <> Signed-off-by: Richard Purdie <>
* ccache.bbclass: Make it can be shared between different buildsRobert Yang2019-01-141-1/+21
| | | | | | | | | | | | CCACHE_BASEDIR: ccache removes this from file path, so that hashes will be the same in different build dirs. CCACHE_TOP_DIR: Set it to a shared location for different builds. (From OE-Core rev: 35d7fe73bba15de16d2eb0a4b12ef03b57b23306) Signed-off-by: Robert Yang <> Signed-off-by: Richard Purdie <>
* ccache.bbclass: Refactor it to make it more reliableRobert Yang2019-01-141-3/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The previous ccache.bbclass has the following problems: - It uses host's ccache for native recipes, but this may not work on some hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always build failures (m4-native failed at do_configure, and others will also be failed if I disable CCACHE for m4-native) - native/nativesdk/cross/crosssdk recipes use host's ccache, but target uses ccache-native, this may confuse user. - The target recipes may use both host's ccache and ccache-native, this may cause unexpected problems and be hard to debug. This is because ccache-native is in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when rebuild target recipes, and then it would use hosttools/ccache, but the previous ccache files were generated by ccache-native. - Target recipes can't use ccache when no ccache is installed on the host: CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}" After refactored: All types recipes (native, target and others) will use ccache-native except ccache-native itself, host's cache won't be used any more. It is more reliable now, which will work everywhere when ccache-native can be built. And now we need use "CCACHE_DISABLE = '1'" to disable ccache for the recipe rather than "CCACHE = ''" since we set CCACHE in anonymous function, and d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we can't check whether CCACHE is set or not in anonymous function since it is always set. Use CCACHE_DISABLE to disable it would be more clear. (From OE-Core rev: b25271b65262f70d849a4861da216c9be6c54d53) Signed-off-by: Robert Yang <> Signed-off-by: Richard Purdie <>
* ccache: let ccache create CCACHE_DIR, reducing hash changesRoss Burton2017-07-171-3/+0
| | | | | | | (From OE-Core rev: 408c5c0f942fa4a7b4df6aacf336d685037ca76c) Signed-off-by: Ross Burton <> Signed-off-by: Richard Purdie <>
* ccache: don't export CCACHE_DISABLE globallyRoss Burton2017-07-171-1/+0
| | | | | | | | | | | | | | | CCACHE_DISABLE was added to bitbake.conf in oe-core dd2bab (June 2012) because autogen-native exports HOME=/dev/null during the build, which is then used by a host ccache to construct the path to it's cache (/dev/null/.ccache) and this fails. However we now always export CCACHE_DIR to solve the same problem in a more efficient way so CCACHE_DISABLE can be deleted. (From OE-Core rev: ef33a3138384667f819688141086102e6e83ec44) Signed-off-by: Ross Burton <> Signed-off-by: Richard Purdie <>
* ccache: Use MULTIMACH_TARGET_SYS not HOST_SYSRichard Purdie2017-03-311-1/+1
| | | | | | | | | I suspect this was a typo and that TARGET_SYS makes more sense here. Its also the only remaining user of MULTIMACH_HOST_SYS in OE-Core. Change it. (From OE-Core rev: fd51900f203ae997b0f606f94ab87c12e37696c0) Signed-off-by: Richard Purdie <>
* ccache: Disable CCACHE_HASHDIR by defaultMike Crowe2017-03-171-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | As of ccache-3.3, ccache tries to ensure that the paths in the debug information are always correct. It does this by including the current directory in the hash if debug output is enabled. It includes support for detecting remapping via a single -fdebug-prefix-map argument uses the remapped directory in the hash instead. The DEBUG_PREFIX_MAP in bitbake.conf remaps the source directory, target sysroot and native sysroot separately which results in multiple -fdebug-prefix-map arguments. Although ccache passes all these arguments through to the compiler, it only enables the special behaviour described above if the last one matches the current directory. (See ) Even if ccache did correctly honour each of the remapping arguments, the hashes would still be different every time ${PV} or ${PR} change because the default DEBUG_PREFIX_MAP contains maps to paths including them. So it seems that for ccache to be of any use with this configuration, CCACHE_NOHASHDIR needs to be set. (From OE-Core rev: fb7a5cdcff19bb44a25a51e20de0440c1ebcc057) Signed-off-by: Mike Crowe <> Helped-by: Ross Burton <> Signed-off-by: Ross Burton <> Signed-off-by: Richard Purdie <>
* Switch to Recipe Specific SysrootsRichard Purdie2017-01-231-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch is comparatively large and invasive. It does only do one thing, switching the system to build using recipe specific sysroots and where changes could be isolated from it, that has been done. With the current single sysroot approach, its possible for software to find things which aren't in their dependencies. This leads to a determinism problem and is a growing issue in several of the market segments where OE makes sense. The way to solve this problem for OE is to have seperate sysroots for each recipe and these will only contain the dependencies for that recipe. Its worth noting that this is not task specific sysroots and that OE's dependencies do vary enormously by task. This did result in some implementation challenges. There is nothing stopping the implementation of task specific sysroots at some later point based on this work but that as deemed a bridge too far right now. Implementation details: * Rather than installing the sysroot artefacts into a combined sysroots, they are now placed in TMPDIR/sysroot-components/PACKAGE_ARCH/PN. * WORKDIR/recipe-sysroot and WORKDIR/recipe-sysroot-native are built by hardlinking in files from the sysroot-component trees. These new directories are known as RECIPE_SYSROOT and RECIPE_SYSROOT_NATIVE. * This construction is primarily done by a new do_prepare_recipe_sysroot task which runs before do_configure and consists of a call to the extend_recipe_sysroot function. * Other tasks need things in the sysroot before/after this, e.g. do_patch needs quilt-native and do_package_write_deb needs dpkg-native. The code therefore inspects the dependencies for each task and adds extend_recipe_sysroot as a prefunc if it has populate_sysroot dependencies. * We have to do a search/replace 'fixme' operation on the files installed into the sysroot to change hardcoded paths into the correct ones. We create a fixmepath file in the component directory which lists the files which need this operation. * Some files have "postinstall" commands which need to run against them, e.g. gdk-pixbuf each time a new loader is added. These are handled by adding files in bindir with the name prefixed by "postinst-" and are run in each sysroot as its created if they're present. This did mean most sstate postinstalls have to be rewritten but there shouldn't be many of them. * Since a recipe can have multiple tasks and these tasks can run against each other at the same time we have to have a lock when we perform write operations against the sysroot. We also have to maintain manifests of what we install against a task checksum of the dependency. If the checksum changes, we remove its files and then add the new ones. * The autotools logic for filtering the view of m4 files is no longer needed (and was the model for the way extend_recipe_sysroot works). * For autotools, we used to build a combined m4 macros directory which had both the native and target m4 files. We can no longer do this so we use the target sysroot as the default and add the native sysroot as an extra backup include path. If we don't do this, we'd have to build target pkg-config before we could built anything using pkg-config for example (ditto gettext). Such dependencies would be painful so we haven't required that. * PKDDATA_DIR was moved out the sysroot and works as before using sstate to build a hybrid copy for each machine. The paths therefore changed, the behaviour did not. * The ccache class had to be reworked to function with rss. * The TCBOOTSTRAP sysroot for compiler bootstrap is no longer needed but the -initial data does have to be filtered out from the main recipe sysroots. Putting "-initial" in a normal recipe name therefore remains a bad idea. * The logic in insane needed tweaks to deal with the new path layout, as did the debug source file extraction code in package.bbclass. * The logic in sstate.bbclass had to be rewritten since it previously only performed search and replace on extracted sstate and we now need this to happen even if the compiled path was "correct". This in theory could cause a mild performance issue but since the sysroot data was the main data that needed this and we'd have to do it there regardless with rss, I've opted just to change the way the class for everything. The built output used to build the sstate output is now retained and installed rather than deleted. * The search and replace logic used in sstate objects also seemed weak/incorrect and didn't hold up against testing. This has been rewritten too. There are some assumptions made about paths, we save the 'proper' search and replace operations to fixmepath.cmd but then ignore this. What is here works but is a little hardcoded and an area for future improvement. * In order to work with eSDK we need a way to build something that looks like the old style sysroot. "bitbake build-sysroots" will construct such a sysroot based on everything in the components directory that matches the current MACHINE. It will allow transition of external tools and can built target or native variants or both. It also supports a clean task. I'd suggest not relying on this for anything other than transitional purposes though. To see XXX in that sysroot, you'd have to have built that in a previous bitbake invocation. * pseudo is run out of its components directory. This is fine as its statically linked. * The hacks for wayland to see allarch dependencies in the multilib case are no longer needed and can be dropped. * wic needed more extensive changes to work with rss and the fixes are in a separate commit series * Various oe-selftest tweaks were needed since tests did assume the location to binaries and the combined sysroot in several cases. * Most missing dependencies this work found have been sent out as separate patches as they were found but a few tweaks are still included here. * A late addition is that extend_recipe_sysroot became multilib aware and able to populate multilib sysroots. I had hoped not to have to add that complexity but the meta-environment recipe forced my hand. That implementation can probably be neater but this is on the list of things to cleanup later at this point. In summary, the impact people will likely see after this change: * Recipes may fail with missing dependencies, particularly native tools like gettext-native, glib-2.0-native and libxml2.0-native. Some hosts have these installed and will mask these errors * Any recipe/class using SSTATEPOSTINSTFUNCS will need that code rewriting into a postinst * There was a separate patch series dealing with roots postinst native dependency issues. Any postinst which expects native tools at rootfs time will need to mark that dependency with PACKAGE_WRITE_DEPS. There could well be other issues. This has been tested repeatedly against our autobuilders and oe-selftest and issues found have been fixed. We believe at least OE-Core is in good shape but that doesn't mean we've found all the issues. Also, the logging is a bit chatty at the moment. It does help if something goes wrong and goes to the task logfiles, not the console so I've intentionally left this like that for now. We can turn it down easily enough in due course. (From OE-Core rev: 809746f56df4b91af014bf6a3f28997d6698ac78) Signed-off-by: Richard Purdie <>
* meta: remove True option to getVar callsJoshua Lock2016-12-161-1/+1
| | | | | | | | | | | | | getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f) Signed-off-by: Joshua Lock <> Signed-off-by: Ross Burton <> Signed-off-by: Richard Purdie <>
* ccache.bbclass: don't remove CCACHE_DIR as part of do_cleanAndre McCurdy2016-08-171-2/+0
| | | | | | | | | | | | | | | | | | Removing the ccache directory as part of do_clean is unnecessarily conservative and defeats many of the benefits of ccache. The original justification for this behaviour was to avoid confusion in the corner case that the ccache directory becomes corrupted. However the standard approach for dealing with such highly unlikely corner cases (ie manually removing tmp) would also recover from corruption of the ccache directories, without the negative impact of defeating ccache during normal development. (From OE-Core rev: 6ae6680ad8d51eff756dcb6500fca2530e3e3e73) Signed-off-by: Andre McCurdy <> Signed-off-by: Ross Burton <> Signed-off-by: Richard Purdie <>
* meta: Don't use deprecated bitbake APIRichard Purdie2013-09-011-1/+1
| | | | | | | | | | These have been deprecated for a long time, convert the remaining references to the correct modules and prepare for removal of the compatibility support from bitbake. (From OE-Core rev: 6a39835af2b2b3c7797fe05479341d71a3f3aaf6) Signed-off-by: Richard Purdie <>
* ccache: allow CCACHE_DIR to be overridenWenzong Fan2013-07-311-1/+1
| | | | | | | | | | | Use weaker assignment for CCACHE_DIR to allow users to override it if desired. (From OE-Core rev: 61e864e2d020c820cf90807b48babee3b24f9446) Signed-off-by: Wenzong Fan <> Signed-off-by: Saul Wold <> Signed-off-by: Richard Purdie <>
* ccache: Ensure CCACHE_DIR is cleaned by do_clean taskRichard Purdie2012-07-101-0/+2
| | | | | | | | | | This ensures a rebuild does really build from scratch when ccache is in use. [YOCTO #2696] (From OE-Core rev: ddf52d0161096b089fad8f3ace69b6515d7b7226) Signed-off-by: Richard Purdie <>
* bitbake.conf: disable ccache explicitly if it is not enabledRobert Yang2012-06-281-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | The autogen-native built error on FC17: ccache: failed to create /dev/null/.ccache This is because the default gcc command of FC17 is a symlink to ccache, so the ccache will always be used regardless to the setting of CCACHE, ccache uses $HOME/.ccache as the CACHE_DIR by default, but autogen set HOME=/dev/null, so the error happens. Disable ccache explicitly if it is not enabled would fix the problem, otherwise it would always use ccache regardless to the setting of CCACHE on Fedora 17. The ccache 3.1.7 has a bug, it would always create $CCCHE_DIR/.ccache even CCACHE_DISABLE=1. Unset CCACHE_DISABLE in ccache.bbclass, since ccache only checks whether there is a CCACHE_DISABLE in the environment or not, it doesn't care about its value, so we need unset it explicitly when enable ccache. [YOCTO #2554] (From OE-Core rev: dd2bab9b6a973d8086dfb6282e781fd79d30b05a) Signed-off-by: Robert Yang <> Signed-off-by: Richard Purdie <>
* ccache: Separate out into its own classRichard Purdie2012-05-301-0/+5
Currently, ccache is used if it is present. When building from scratch it gives no performance improvement and creates a ton of empty directories even when its not in use. This change moves ccache support to a bbclass file which the user can choose to enable. This should make builds more determinstic and make it easier/clearer to the end user when its being used and when it is not. (From OE-Core rev: 2acf8da4f13c175ea818b9514677b7059de1e3e2) Signed-off-by: Richard Purdie <>