summaryrefslogtreecommitdiffstats
path: root/meta/lib/oeqa/selftest/cases/reproducible.py
Commit message (Collapse)AuthorAgeFilesLines
* reproduce: Fix exclusion list for rpmRichard Purdie2021-02-261-0/+1
| | | | | | | | | | rpm uses "XXX-VERSION" compared to deb/ipk which use "XXX_VERSION" which breaks the go package exclusion. Work around with both go_ and go- for now. (From OE-Core rev: 2af535282833dddb58f24ec5b360230c47e6f6b1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: Remove exclusions for recipes which now reproduceRichard Purdie2021-02-241-8/+2
| | | | | | | | | | git, libproxy, libcap-ng(-python), babeltrace2, rsync and groff have their reproducibility issues fixed, remove from the exclusion list. Also fix whitespace. (From OE-Core rev: 044eda9c70ddda195b1c0ce1d228904392431582) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Revert "oe-selftests: add rpm to reproducible build selftest"Richard Purdie2021-02-171-1/+1
| | | | | | | | This reverts commit 84d3a90557444d8cd83d780b1c3f2b278ab07af2. This shouldn't have been merged, its not ready yet. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftests: add rpm to reproducible build selftestJan-Simon Möller2021-02-171-1/+1
| | | | | | | | | | | | | | rpm packaging has shown good numbers wrt reproducible builds as well when testing core-image-minimal locally: Reproducibility summary for rpm: same=4671 different=0 missing=0 total=4671 Thus enable it in the autobuilder to be more widely tested. (From OE-Core rev: 84d3a90557444d8cd83d780b1c3f2b278ab07af2) Signed-off-by: Jan-Simon Möller <jsmoeller@linuxfoundation.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* acpica: Fix reproducibility issuesJoshua Watt2021-02-171-1/+0
| | | | | | | | | | Fixes reproducibility when build acpica yacc sources by instructing bison on how to do the file prefix mapping (From OE-Core rev: d13a1df7bd173fca3fc2cdf889d5eb487e35d6fd) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: Update exclusionsRichard Purdie2021-02-161-4/+6
| | | | | | | | | Add ruby-ri-docs and meson back as not reproducible, remove watchdog, xorg-minimal-fonts and xmlto as issues fixed. (From OE-Core rev: 5e209a7455a231342cded99985b4a6caff424e4f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: reproducible: Add more loggingJoshua Watt2021-02-161-0/+3
| | | | | | | | | | | The reproducible build tests can take a long time, so having more logging messages at various points in the build can help debug where the build is taking a long time. (From OE-Core rev: 6b792afe8759d62af8e713b86dad8f6721961a05) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: reproducible: Fix SSTATE_MIRRORS variableJoshua Watt2021-02-161-1/+1
| | | | | | | | | | The SSTATE_MIRRORS variable was misspelled, which allowed the "clean" test build to pull from the mirror. (From OE-Core rev: e42497bd84d0bb370a9f7b0448bff29f01fd1b0c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: Remove no longer needed exclusionsRichard Purdie2021-02-131-8/+0
| | | | | | | | | It appears these exclusions are no longer needed with master, drop them and improve our reproducibilty metrics. (From OE-Core rev: c71c984a9cd6d130ece08153d7d92bb33c7ec444) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: Sort the unused exclusion listRichard Purdie2021-02-131-1/+1
| | | | | | | | This makes comparisions between lists easier. (From OE-Core rev: d2c52125d1cdc06c7e08d507ca68f3e4612a4314) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: track unusued entries in the exclusion listAlexander Kanavin2021-02-131-4/+10
| | | | | | | | | This helps with trimming down the list, and towards 100% reproducibility :) (From OE-Core rev: da7a173d7a01524229c8515326465968a845e96f) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: remove spirv-tools-dev from exclusion listJose Quaresma2021-02-081-1/+0
| | | | | | | (From OE-Core rev: ecb156fa391b29c6b317abb7bb126a36d709be6a) Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: Add useradd-staticids to reproducible builds testsRichard Purdie2020-12-301-0/+4
| | | | | | | | | Its surprising we've made it this far without this, add in some specific uid/gid settings to ensure these don't cause false positives to the tests. (From OE-Core rev: 77397bace64dfd03e704400fea918d93f00d0bf7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: add an exclusion list for items that are not yet ↵Alexander Kanavin2020-12-151-3/+79
| | | | | | | | | | | | | | | reproducible Hopefully over time this list will be reduced to an empty one. Non-reproducible excluded packages are not given to diffoscope and do not cause a failure, but still saved side-by-side with non-reproducible failing ones to make investigation easier. (From OE-Core rev: 406bd0d48d8f90e2c836f7d3e204f21d5f13c833) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/reproducible: enable world reproducibility testAlexander Kanavin2020-12-151-1/+3
| | | | | | | | | | Add systemd and pam distro features, and commercial license flag to include more recipes into the world set. (From OE-Core rev: ed8f94942c07784f9f6eef5c00b75aa9b398ea5e) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: reproducible: Fix test not producing diffsJoshua Watt2020-09-021-1/+27
| | | | | | | | | | | | | | | | Diffoscope changed the --exclude-directory-metadata option to require an argument. Add a test to validate that diffoscope is functioning as expected to ensure that future upgrades do not unintentionally break the reproducibility tests. [YOCTO #14025] (From OE-Core rev: ea8fbcb7978ce48d7a9a83143d09402329535f86) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: reproducible: Include jquery in resultsJoshua Watt2020-02-191-3/+10
| | | | | | | | | | Includes jquery in the output directory to make the diffoscope output easier to navigate (From OE-Core rev: ff7c386455848e497ef40f9fee2be87d5171e203) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: reproducible: Run diffoscope on saved outputJoshua Watt2020-02-141-7/+17
| | | | | | | | | | | | If there are differing packages and they are being saved for review, automatically run diffoscope on them and include the output in the saved output. The output is currently done in HTML format since these are typically published on a webpage by the autobuilder. (From OE-Core rev: 6e1d5e8b58f0940ba6dfd99536159dd974e6f24c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/reproducible: Improve test output and ensure deb+ipk comparedRichard Purdie2020-02-061-2/+7
| | | | | | | | | | | | | Adding newline characters between the packages in the failure output massively improves readability. Also ensure to output ipk failures when there are deb failures by calling self.fail() at the end, else sometimes only partial differences are returned. (From OE-Core rev: 6e2e0480852177db75a6108d77c99c92c4e9950f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: reproducible: Do not strip packagesJoshua Watt2020-02-031-0/+1
| | | | | | | | | | | | | | Do not strip packages when testing reproducible builds. In some cases, stripped data differs between builds, but then gets removed. However, the contents affect the generation of the GCC build-id, which then differs in the resulting ELF files, even though the data that caused this is no longer there. Inhibit stripping so that their causes can be evaluated. (From OE-Core rev: 0e23c6faf8169a333ae29064553705b5d7d6b589) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible: Allow configuration of saved debug outputRichard Purdie2020-02-021-1/+12
| | | | | | | | | | | | | | If OEQA_DEBUGGING_SAVED_OUTPUT is set in the environment, use this location to store reproducibile build failure output. This aids debugging on the YP autobuilder in particular. Use a date in the directory name to make it easier to find failure output. Also clean up empty directories as they're unnecessary distracting noise. (From OE-Core rev: 138ed4aa96ec5069f9f7fd02994a42452dbccc2d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/reproducible: Fix unset save_dir variableRichard Purdie2020-01-221-5/+5
| | | | | | | | Previous refactoring broke the case where save_dir was set. Fix this. (From OE-Core rev: 9bfa04aa1f1a0b38ffa51f0211f3a46a90e17b0a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/reproducible: Fix extra data reportingJoshua Watt2020-01-221-5/+5
| | | | | | | | | | A typo was preventing the extra data about the reproducible build from being reported in the test results (From OE-Core rev: 7c653337360dd5e49809b930f22a999c1d6e09ee) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/reproducible: Add flag for building from sstateJoshua Watt2020-01-221-31/+35
| | | | | | | | | | | Adds a flag to control if the reproducible QA test should allow building from sstate or not. Building from sstate may not be reproducible depending on how the sstate is populated. (From OE-Core rev: c24b71e9ec2e518bd24da20ddbb1406910b1b92e) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: reproducible: Test core-image-sato and core-image-full-cmdlineJoshua Watt2019-12-181-1/+1
| | | | | | | | | | Adds core-image-sato and core-image-full-cmdline to the list of images that the reproducible build test builds. (From OE-Core rev: d6c62cc0305ab082e990041da104fedb5798e0e4) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: reproducible: Add option to capture bad packagesJoshua Watt2019-11-071-0/+20
| | | | | | | | | | | Adds an option that can be used to copy the offending packages to a temp directory for later evaluation. This is useful on the Autobuilder to investigate failures. (From OE-Core rev: 54b29bae78d1711074fbd18f0350ef0b83b555d1) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/reproducible: test ipkgs tooRoss Burton2019-09-271-1/+1
| | | | | | | | | | | | Now that opkg-build can build reproducible ipkgs, we can also add those to the test case. [ YOCTO #13513 ] (From OE-Core rev: 3806d38e6194a42534381993fcfcc3ff39550a13) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest: Use extraresults on self instead of self.tcNathan Rossi2019-09-071-8/+5
| | | | | | | | | | | | | In order to take advantage of multiprocess execution of tests the extraresults must be passed through the TestResult. With changes to how oeqa/core handles test cases the extraresults attribute of the testcase is passed to the TestResult, with passing across process boundaries handled automatically. (From OE-Core rev: 6a1b0c2003a0b4a1983f9494440e6ea02dc25585) Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: reproducible: Preserve sstate mirror for first buildJoshua Watt2019-09-061-1/+1
| | | | | | | | | | | | | | | | | | Preserves the SSTATE_MIRROR variable for the first build in the reproducible test. This patch is intended to test the theory that using the Yocto autobuilder sstate mirror will still be reproducible. The autobuilder always does clean builds and never rebuilds recipes that were built previously, thus building with the mirror but not sharing tmpdir and sstate with previous builds should be reproducible. There is no guarantee that all sstate caches are populated from clean builds so this patch cannot be submitted to master, and I'm not bothering sending it to the mailing list. (From OE-Core rev: 6b8e0077339a89cb01aa40c1b367a4e41a638892) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: reproducible: Do two clean buildsJoshua Watt2019-08-301-20/+28
| | | | | | | | | | | | | Perform two clean builds without sstate instead of one partial rebuild with sstate and one clean build without. There are some classes of reproducibility issues that this solves, and while we would like to resolve them in the long term the direction to do so is not currently clear. (From OE-Core rev: e97c529bfa4e1d0038ea44f15ee3298003daf981) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Enable reproducible build testJoshua Watt2019-08-301-1/+0
| | | | | | | | | [YOCTO #13323] (From OE-Core rev: 3fc08155b72f711c48f9a667539305eea869b5e9) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: reproducible: Record packages in test resultsJoshua Watt2019-08-301-0/+10
| | | | | | | | | | | | | Records the results of the reproducibility analysis in the JSON test results file. This makes it easier to do post-test analysis on the packages. [YOCTO #13324] (From OE-Core rev: 6c357c8eee322163da96df676a54ff70a0bdc911) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: reproducible: Use subTest for packagesJoshua Watt2019-08-301-10/+11
| | | | | | | | | | | Runs each package class reproducibility test in a separate sub-test. This allows the other sub tests to still run in the event that one fails. (From OE-Core rev: f0095e6c1b915e1e6ded111f4bf77fff0362e29a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: reproducible: Cleanup reproducible buildJoshua Watt2019-08-301-3/+10
| | | | | | | | | | | | Cleans up the output from the reproducible build before building to ensure consistent results. Note that the output put is purposely left after around after the build so that non-reproducible packages can be diffed. (From OE-Core rev: f67bb34c8e7950779920b696f8502dbaed9eded5) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Add reproducible build selftestJoshua Watt2019-06-101-0/+160
Adds an initial test for reproducible builds to the OE selftest. This initial test builds core-image-minimal using sstate, then does a clean build without sstate in another build directory, and finally does a binary comparison of the resulting package files between the two builds. The test is currently always skipped since it doesn't pass yet, but it can easily be enabled locally (From OE-Core rev: 2e591bdf93ec9e59b900562263dfe8e72b163baa) Signed-off-by: Joshua Watt <JPEWHacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-20 08:15:14 +0000