summaryrefslogtreecommitdiffstats
path: root/meta/classes/reproducible_build.bbclass
Commit message (Collapse)AuthorAgeFilesLines
* reproducible_build: Work around caching issuesRichard Purdie2021-10-041-4/+7
| | | | | | | | | | | | | | | SOURCE_DATE_EPOCH can be expanded early in the parsing process before the class extensions are applied. This can mean the directory pointed to for the SDE can be incorrect until later in parsing. Cache the file name in the cached value and allow it to dynamically update. This isn't ideal but avoding expansion of the variable likely isn't possible and I'm not sure how else to handle this. This works around the issue until a better solution can be found. (From OE-Core rev: 44dc97cd1223e4d2b635669627ec5f796838d42d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible_build/package_XXX: Ensure SDE task is in dependency chainRichard Purdie2021-09-231-0/+2
| | | | | | (From OE-Core rev: 938e4dfb68a465f61cb754b2cd28d0728616b5a9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible_build: Remove BUILD_REPRODUCIBLE_BINARIES checkingMark Hatle2021-09-111-19/+34
| | | | | | | | | | | | | | | | | | | Previously if BUILD_REPRODUCIBLE_BINARIES was set to 0, the system would fall back and select the default epoch (April 2011), but still perform the reproducible build actions. This resulted in binaries that had an unusually old date. Simplify the functions and remove the anonymous python as no longer necessary. Also improve the documentation to better explain what the class is doing and how a recipe can override the behavior if necessary. (From OE-Core rev: 1976013b026cfba94de32a13e994d92d7e9e39e5) Signed-off-by: Mark Hatle <mark.hatle@xilinx.com> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Convert to new override syntaxRichard Purdie2021-08-021-1/+1
| | | | | | | | | | | | This is the result of automated script conversion: scripts/contrib/convert-overrides.py <oe-core directory> converting the metadata to use ":" as the override character instead of "_". (From OE-Core rev: 42344347be29f0997cc2f7636d9603b1fe1875ae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/reproducible_build: Use atomic rename for SDE fileJoshua Watt2021-06-031-7/+6
| | | | | | | | | | | | | | | | | If an existing source date epoch file was found during do_unpack, it was deleted and a new one would be written in its place. This causes a race with check-before-use code in get_source_date_epoch_value. Resolve the problem by making do_unpack write the new source date epoch to a temporary file, then do an atomic rename to ensure it's always present, and change the check-before-use code to use a EAFP exception instead of checking for file existence. [YOCTO #14384] (From OE-Core rev: 0b5e3b33187bf78a2d62cc886463e4b27d6bd228) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/lib/scripts: Use bb.utils.rename() instead of os.rename()Devendra Tewari2021-05-061-1/+1
| | | | | | | | | | | | | | | | | | Incremental build in Docker fails with: OSError: [Errno 18] Invalid cross-device link when source and destination are on different overlay filesystems. Rather than adding fallback code to every call site, use a new wrapper in bitbake which detects this case and falls back to shutil.move which is slower but will handtle the overlay docker filesystems correctly. [YOCTO #14301] (From OE-Core rev: 656a65b2b84e7d529b89cf5de7eb838f902d84a2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible_build.bbclass: Enable -Wdate-timeKhem Raj2021-05-011-0/+3
| | | | | | | | | | | | This will help identifying packages using __TIME__, __DATE__ and __TIMESTAMP__ macros in code, its a warning from the compiler and the packages which use -Werror will break the build with this and where they don't we will atleast have a warning in the build logs (From OE-Core rev: 20335cd89001f5fb159f5f1b0c3bd5e40b8b2fb5) Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible: Improve SOURCE_DATE_EPOCH_FALLBACK handlingRichard Purdie2021-02-241-4/+0
| | | | | | | | | | | | | | Ensure the fallback value if used is written to the SDE file and hence stored in sstate, reducing any confusion within the code over '0' values. Bump the HASHEQUIV_VERSION since we've had a ton of trouble with ensuring this rolls out correctly on the autobuilder so others may too, take a clean slate for it. (From OE-Core rev: 53ffffa32b76330835287dfc05d4e4796841af08) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible_builds: SOURCE_DATE_EPOCH should not be 0Jan-Simon Möller2021-02-241-4/+15
| | | | | | | | | | | | | | | | | | A SOURCE_DATE_EPOCH of 0 might be misinterpreted by namely rpm as no SDE. e.g.: char *srcdate = getenv(SOURCE_DATE_EPOCH); if (srcdate && rpmExpandNumeric(%{?clamp_mtime_to_source_date_epoch})) Solve this by: - providing a SOURCE_DATE_EPOCH_FALLBACK variable != 0 - changing defaults to use it - using SOURCE_DATE_EPOCH_FALLBACK also when reading 0 out of sstate-cache (From OE-Core rev: 9a1dde74e794362399193dc3f81c9685a83d0776) Signed-off-by: Jan-Simon Möller <jsmoeller@linuxfoundation.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/reproducible: Move to library codeJoshua Watt2020-07-221-87/+3
| | | | | | | | | | Moves most of the python code used for dealing with the source date epoch to library code. (From OE-Core rev: a7ede90955bc0c8bec1cbb3cab498ef2583b2f4e) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducibile_build: Fix SDE file generation when unpack rerunsRichard Purdie2020-03-081-4/+5
| | | | | | | | | | | | | | | Currently, if an existing TMPDIR is rebuilt, do_fetch/do_unpack can rerun but SDE would remain unchanged. This leads to different results compared to a fresh build. An example change which triggered this is: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=cb4e69e6346a9fbeebf83a5d5397cacbd41d48b5 Instead, delete any existing SDE and recalculate if we're reunning. Also rename and drop the do_ prefix since these are for tasks, not functions. (From OE-Core rev: 5d579fc2fe71637fc6e071aa66542befa39ac8bb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/reproducible_build: Read SDE file laterJoshua Watt2020-02-031-9/+31
| | | | | | | | | | | | | | | | | Defers the resolution of the SOURCE_DATE_EPOCH until the variable needs to be actually realized with a value. The previous method of loading the value in anonymous python had issues because it could occur before other anonymous python functions that affect the location of the epoch file, such as when a recipe uses AUTOINC/AUTOREV or allarch.bbclass. Also adds more logging to help diagnose issues in the future. [YOCTO #13763] (From OE-Core rev: b3313a10a3eb93f0a3710a35de0404fb49cd6202) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/reproducible_build: Create SDE destinationJoshua Watt2019-10-021-0/+1
| | | | | | | | | Creates the source data epoch file destination directory when restoring from sstate (From OE-Core rev: f5259a62d444dfc221a9ba592dbca35564f794ac) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/reproducible_build: Move SDE deploy to another directoryJoshua Watt2019-09-271-2/+10
| | | | | | | | | | | | | | | | | | | | | | | The deployment of the source date epoch file had a race condition where any task attempting to read from the file would race with creation of the sstate archive for the do_deploy_source_date_epoch task. The creation of the sstate archive requires moving the directory to a temporary location, then moving it back. This means that the file disappears for a short period of time, which will cause a failure if any other task is running and trying to open the file to get the current source date epoch. The solution is to copy the source date epoch file to a separate directory when deploying so the file never disappears. When the file is restored from sstate, it is moved to the correct location after being extracted. [YOCTO #13501] (From OE-Core rev: ac27d12fe5480e9b8cc93de6a32bf9631c52d7f4) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible: Don't look for youngest file when no source tarballDouglas Royds2018-11-231-5/+8
| | | | | | | | | | | | | | | | | | | Some packages (eg. init-ifupdown) take their source files entirely from openembedded-core, that is, they download no source tarball. These recipes either don't use S at all (ie. it is empty at unpack time), or they set S = WORKDIR (as in init-ifupdown). Looking at the file timestamps in the WORKDIR causes a non-reproducible SOURCE_DATE_EPOCH, as files taken from file:// URIs do not have reproducible timestamps. If S == WORKDIR, we are better to assume that there is no source tarball, and to fall back to a fixed timestamp for the SOURCE_DATE_EPOCH. This makes the init-ifupdown build reproducible. (From OE-Core rev: d395bad0179037eb5d0fa4d921985c87ae13f3a4) Signed-off-by: Douglas Royds <douglas.royds@taitradio.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible: Refactor: Break out fixed_source_date_epoch() functionDouglas Royds2018-11-231-4/+5
| | | | | | | (From OE-Core rev: 4eb6def4fe82959c2a348142b9eada27d3354aef) Signed-off-by: Douglas Royds <douglas.royds@taitradio.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible_build: update do_deploy_source_date_epoch commmentDouglas Royds2018-11-071-1/+4
| | | | | | | | | | | | Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE. If none of the existing mechanisms are suitable, replace the do_deploy_source_date_epoch task with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE. (From OE-Core rev: e7b891b76954c784f5a93bd0a1c91315673ce40d) Signed-off-by: Douglas Royds <douglas.royds@taitradio.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible: Consistent debug loggingDouglas Royds2018-09-211-8/+11
| | | | | | | | | | | | | | | Log both the source of the source_date_epoch and the resulting source_date_epoch in all cases. Also, now that we are determining the source_date_epoch successfully for both yocto and non-yocto kernels, remove the inherits_class('kernel') exception. We will log a failure to find a source_date_epoch for kernels as well. (From OE-Core rev: 48fe0e83435f9fb53b1e5b37d5d9f1c2caccd22d) Signed-off-by: Douglas Royds <douglas.royds@taitradio.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible: Don't enforce existence of ${S} dirDouglas Royds2018-09-211-5/+1
| | | | | | | | | | Unnecessary. (From OE-Core rev: 75d5a9d72566aca7a9f08bea21523b3c66bddeae) Signed-off-by: Douglas Royds <douglas.royds@taitradio.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible: Find the git repo in WORKDIR/git or S firstDouglas Royds2018-09-211-12/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Change the search regime for find_git_folder(): 1. WORKDIR/git: This is the default git fetcher unpack path 2. ${S} 3. Go looking for .git/ under the WORKDIR as a last resort. linux-yocto: We had an existing (silent) defect. The linux-yocto recipes all specify two git SRC_URIs, one for the kernel source itself, the other for the kmeta data (config fragments and friends). find_git_folder() was finding the git checkout for the kmeta data, but due to a typo in the git log -1 --pretty=%ct line, we were (silently) reading the source_date_epoch from the ${S} directory = STAGING_KERNEL_DIR, which is empty. If your build/ happened to be inside a git checkout, git would walk up the directory tree, and silently read the commit timestamp from this other git checkout. The correct path to read the git commit timestamp from is the "gitpath", being that found by find_git_folder(), though this function was incorrectly finding the kmeta data checkout, not the kernel source tree. Non-kernel git recipes: The default git fetcher clones and checks out the sources at WORKDIR/git/ regardless of the setting of S (unless subpath or destsuffix is set). find_git_folder() now looks for the WORKDIR/git/.git/ directory first. Non-yocto linux kernels: Kernel recipes that don't inherit kernel-yocto should always set S = ${WORKDIR}/git, so that when base_do_unpack_append() in kernel.bbclass moves the checkout down to the STAGING_KERNEL_DIR and symlinks it as WORKDIR/git, the build can still work by following the symlink. We were previously failing to follow the symlink in the os.walk(), but we now look first for WORKDIR/git/.git/, and find it due to the symlink. If none of the above mechanisms work for finding the git checkout, perhaps there was a subpath or destsuffix specified in the SRC_URI. We go looking for the git checkout under the WORKDIR as a last resort. (From OE-Core rev: b0ddb141d36853447f85ecaac07dbc9c5779627f) Signed-off-by: Douglas Royds <douglas.royds@taitradio.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible: Rename ambiguous "path" var to sourcedir or workdir as appropriateDouglas Royds2018-09-211-18/+19
| | | | | | | | (From OE-Core rev: 01358d6a89623a38e66969daa431d2eecb1ce8a2) Signed-off-by: Douglas Royds <douglas.royds@taitradio.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible: Refactor: get_source_date_epoch_from_youngest_file()Douglas Royds2018-09-211-33/+40
| | | | | | | | (From OE-Core rev: 2c8baf6c50e5a266a1e9ac939415bd481668e17e) Signed-off-by: Douglas Royds <douglas.royds@taitradio.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible: Tidy class descriptionDouglas Royds2018-09-211-27/+22
| | | | | | | | | | Tidy whitespace, delete commented-out lines (From OE-Core rev: 254eb5a2a12fd6b5cf7427818ea9fa886ea407a6) Signed-off-by: Douglas Royds <douglas.royds@taitradio.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/reproducible_build: Avoid dereferencing symlinksJoshua Watt2018-08-081-2/+2
| | | | | | | | | | | | Using os.path.getmtime() will dereference symbolic links in an attempt to get the last modified time. This can cause errors if the target doesn't exist, or worse map to some absolute build host path which would make a build not reproducible. (From OE-Core rev: fae23c72288068f90e2f357a8abf1384850c02ed) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible_build.bbclass: support for binary reproducibilityJuro Bystricky2018-03-251-0/+150
Setup environment for builds requiring binary reproducibility. Determine and export SOURCE_DATE_EPOCH per each recipe. This is a crucial step to achieve binary reproducibility. The value for this variable (timestamp) is obtained after source code for a recipe has been unpacked, but before it is patched. If the code sources come from a GIT repo, we get the timestamp from the top commit. (GIT repo does not preserve file mktime timestamps). Otherwise, if GIT repo is not present, we try to get mtime from known files such as NEWS, ChangeLog, etc. If this also fails, we go through all files and get the timestamp from the youngest one. We create an individual timestamp for each recipe. The timestamp is stored in the file '__source_date_epoch.txt' (in the folder source-date-epoch_). Later on, each task reads this file and sets the exported value of SOURCE_DATE_EPOCH to the value found in the file. Uasge: INHERIT += "reproducible_build" [YOCTO#11178] [YOCTO#11179] (From OE-Core rev: cc438ac7711dedbe05d654e99af9316c9215b02e) Signed-off-by: Juro Bystricky <juro.bystricky@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-15 04:33:41 +0000