summaryrefslogtreecommitdiffstats
path: root/scripts/oe-build-perf-test
Commit message (Collapse)AuthorAgeFilesLines
* oe-build-perf-test: pack all buildstat in one fileMarkus Lehtonen2017-03-271-0/+1
| | | | | | | | | | | | | | | | Write out all buildstats into one big json file, instead of using multiple per-measurement files. Individual buildstats will be indexed using "<test_name>.<measurement_name>" as the key. Also, changes the per-testcase working directories into temporary directories that will be removed after test execution as there are no more per-testcase data files to store permanently. [YOCTO #10582] (From OE-Core rev: a7f2e8915db379021f3409ca640de5d3b054a830) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: drop --commit-resultsMarkus Lehtonen2017-02-151-104/+0
| | | | | | | | | | | | | | | Drop support for committing results into Git repository. The functionality was not directly related to testing and feels unnecessary complication of the script. The functionality has been moved into a separate oe-git-archive script. [YOCTO #10582] (From OE-Core rev: 4de387c0cfcb6b58760c6b6e150474abe82bfe4c) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: remove unused imports and fix indentMarkus Lehtonen2017-01-231-1/+0
| | | | | | | | | | [YOCTO #10590] (From OE-Core rev: 0b1892fa9165407a156609ff7cb3708e21bacd8c) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: save test metadata in a separate fileMarkus Lehtonen2017-01-231-4/+114
| | | | | | | | | | | | | | | The patch introduces a new metadata (.json or .xml) file in the output directory. All test meta data, e.g. git revision information and tester host information is now stored there. The JSON report format is slightly changed as the metadata is not present in results.json anymore. [YOCTO #10590] (From OE-Core rev: 2036c646019660e32f1bc277fdec0cdbff0afdd4) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: enable xml reportingMarkus Lehtonen2017-01-231-0/+6
| | | | | | | | | | | | | Add --xml command line option to oe-build-perf-test script for producing a test report in JUnit XML format instead of JSON. [YOCTO #10590] (From OE-Core rev: 21ae1c491b93675254b7733640662b566ed76f98) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: return 2 if some tests failedMarkus Lehtonen2016-10-011-1/+1
| | | | | | | | | | | | Add a new return value '2' that indicates that some tests failed but there were no fatal errors (i.e. configuration mistakes or bugs in the tests themselves). (From OE-Core rev: 194e95f3f068456f30c0e971eb8e6e775279427c) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: fix log file pathMarkus Lehtonen2016-09-061-1/+1
| | | | | | | | | | The --log-file command line argument was slightly broken as {out_dir} string replacement was not working as expected. (From OE-Core rev: fc62f54e3d788cc79fd27664f05db7efccef23ab) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: rename log file and implement --log-fileMarkus Lehtonen2016-09-031-1/+4
| | | | | | | | | | | | | Rename the (main) log file of the oe-build-perf-test script from 'output.log' to 'oe-build-perf-test.log'. Also, add a new command line option --log-file which makes it possible to use an alternative log file name/path, if needed. Note that the file name/path is relative to the output directory. (From OE-Core rev: 4909fae1a6d1d068b33252088b41b8d82d1a836c) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: update globalres and git even if tests failedMarkus Lehtonen2016-09-031-6/+6
| | | | | | | | | | | Write globalres log file and commit results to Git even if some tests failed. Now that tests do not depend on each other there should be no risk of bogus results caused by test failures. (From OE-Core rev: 8036975b268fe209476e230555006facd3cbda71) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: add {git_commit_count} keyword for --commit-results-tagMarkus Lehtonen2016-08-251-1/+1
| | | | | | | | | | | Makes it possible to create easily sortable tags. Also, the default tag format is updated to use the new keyword. (From OE-Core rev: e3161654d75dfc3b059c519205b38b26e3ffb215) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: new {tag_num} keyword for --commit-results-tagMarkus Lehtonen2016-08-251-1/+1
| | | | | | | | | | | | | | | | | | | | | | This makes it possible to create numbered tags, where the "basename" of the tag is the same and the only difference is an (automatically) increasing index number. This is useful if you do multiple test runs on the same commit. For example, using: --commit-results-tag {tester_host}/{git_commit}/{tag_num} would give you tags something like: myhost/decb3119dffd3fd38b800bebc1e510f9217a152e/0 myhost/decb3119dffd3fd38b800bebc1e510f9217a152e/1 ... The default tag format is updated to use this new keyword in order to prevent unintentional tag name clashes. (From OE-Core rev: cf2aba16338a147f81802f48d2e24a96c7133548) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: tag results committed to GitMarkus Lehtonen2016-08-251-1/+5
| | | | | | | | | | | | | | | | | | | | Create a Git tag when committing results to a Git repository. This patch also implements --commit-results-tag command line option for controlling the tag name. The value is a format string where the following fields may be used: - {git_branch} - target branch being tested - {git_commit} - target commit being tested - {tester_host} - hostname of the tester machine Tagging can be disabled by giving an empty string to --commit-results-tag. The option has no effect if --commit-results is not defined. (From OE-Core rev: 60059ff5b81d6ba9ba344161d51d1290559ac2df) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: pre-check Git repo when using --commit-resultsMarkus Lehtonen2016-08-251-0/+27
| | | | | | | | | | | | | | Do a pre-check on the path that is specified with --commit-results before running any tests. The script will create and/or initialize a fresh Git repository if the given directory does not exist or if it is an empty directory. It fails if it finds a non-empty directory that is not a Git repository. (From OE-Core rev: 759357a3bdbe75a3409b9e58979ab8b45d9b6ae8) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: implement --commit-results-branchMarkus Lehtonen2016-08-251-1/+5
| | | | | | | | | | | | | | | | A new command line option for defining the branch where results are commited. The value is actually a format string accepting two field names: - {git_branch} expands to the name of the target branch being tested - {tester_host} expands to the hostname of the tester machine The option has no effect if --commit-results is not used. (From OE-Core rev: b54b63395ec632748a57a702812c8a9a07af35ab) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: support committing results data to GitMarkus Lehtonen2016-08-251-0/+5
| | | | | | | | | | | | Implement a new command line option '--commit-results' which commits the test results data into a Git repository. The given path must be an existing initialized local Git repository. (From OE-Core rev: b6f635513ca971402e7a970acc2168fb5d4a9476) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: use absolute paths in cmdline argsMarkus Lehtonen2016-08-251-1/+3
| | | | | | | | | | This is safer as the current working directory may change. (From OE-Core rev: 4b7bf7860713581ba351599fe32817ba24e8f8d0) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: implement --run-tests optionMarkus Lehtonen2016-08-251-1/+6
| | | | | | | | | | | | | Makes it possible to run only a subset of tests. NOTE: The tests currently have (unwritten) dependencies on each other so use this option with care. Mainly for debugging. (From OE-Core rev: be4373be54e5b84f951771b0e75140f212838020) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: align log message format with testrunner outputMarkus Lehtonen2016-08-201-2/+4
| | | | | | | | | | The previous attempt on this was a bit erroneous, dropping time stamps completely although only the timestamp format should've been changed. (From OE-Core rev: bafcff95e2b5e0b9a8c76ce46a62667bf6f49b00) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: simplify stderr log formatMarkus Lehtonen2016-08-171-3/+2
| | | | | | | | | | | | Remove timestamps from the stderr log in order to make the console output more readable, i.e. more in line with the output from unittest runner. (From OE-Core rev: d28eeeabde9b4b7160a273445023a44fd50e29ab) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: set-up file logging as early as possibleMarkus Lehtonen2016-08-171-3/+4
| | | | | | | | | | So that the log file would not miss any records. (From OE-Core rev: 9ce6e20ce239067896dc65f09e3fef1173293065) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: suppress logger output when tests are being runMarkus Lehtonen2016-08-171-1/+10
| | | | | | | | | | | | | Prevent logger from writing to stderr when the tests are being run by the TestRunner. During this time the logger output is only written to the log file. This way the console output from the script is cleaner and not mixed with possible logger records. (From OE-Core rev: 36f58b5172d4e2e182aa447fb3ec4d1ac9f6820d) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: use new unittest based frameworkMarkus Lehtonen2016-08-171-6/+13
| | | | | | | | | | | Convert scripts/oe-build-perf-test to be compatible with the new Python unittest based buildperf test framework. (From OE-Core rev: 249d99cd7ec00b3227c194eb4b9b21ea4dcb7315) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa.buildperf: add BuildPerfTestResult classMarkus Lehtonen2016-08-171-0/+10
| | | | | | | | | | | | | The new class is derived from unittest.TextTestResult class. It is actually implemented by modifying the old BuildPerfTestRunner class which, in turn, is replaced by a totally new simple implementation derived from unittest.TestRunner. (From OE-Core rev: 89eb37ef1ef8d5deb87fd55c9ea7b2cfa2681b07) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: add --out-dir command line argumentMarkus Lehtonen2016-07-011-1/+3
| | | | | | | | | | The new option defines the output directory for the test result data. (From OE-Core rev: bc865b5cb5a9a76048ee9c55a29f5e1a926bb543) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: enable lockingMarkus Lehtonen2016-07-011-0/+23
| | | | | | | | | | Makes it possible to guard that multiple tests are not run in parallel. (From OE-Core rev: 181e92e7a1bccf678b3eb1bf547608a142784f97) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: implement --globalres-file optionMarkus Lehtonen2016-07-011-0/+5
| | | | | | | | | | | | Using this option the script appends test results into a 'global results file'. A CSV-formatted output of the results. This option is to provide compatibility with the old build-perf-test.sh. (From OE-Core rev: e9f18e63220e452f2b0c878998e57d944ae83980) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: enable logging into fileMarkus Lehtonen2016-07-011-1/+15
| | | | | | | | (From OE-Core rev: d5bbcdaf332b2a15382672d6a40b2d2b807b9dde) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa.buildperf: implement BuildPerfTestRunner classMarkus Lehtonen2016-07-011-2/+8
| | | | | | | | | | | | The new class is responsible for actually running the tests and processing their results. This commit also adds a decorator function for adding new tests. No automatic test discovery, at least yet. (From OE-Core rev: bf90aecb7e150d6bfac7240286c797b79d26528b) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa.buildperf: functionality to drop kernel cachesMarkus Lehtonen2016-07-011-0/+4
| | | | | | | | | | | | | | | | | | Add a new utility class for dropping Linux kernel caches. It uses sudo and tee to write to the drop_caches file. Checking if the user has the permissions to drop caches (without a password) is done by trying to writing an invalid value to the drop_caches file. This way, we will find if writing (with tee) is possible but not really dropping caches, yet. User can avoid giving the password by adding something like: <user> ALL = NOPASSWD: /usr/bin/tee /proc/sys/vm/drop_caches to the system sudoers file. (From OE-Core rev: c9cb248429ced50c96d11ba5361c272d4c9b9323) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-build-perf-test: add pre-run sanity checkMarkus Lehtonen2016-07-011-0/+28
| | | | | | | | | | | The script will be required to be run in an initialized bitbake build environment. (From OE-Core rev: 1bce7b10283255a4498d11ead920c1f3b1dec4de) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts: introduce oe-build-perf-testMarkus Lehtonen2016-07-011-0/+51
Initial wireframe for re-writing build-perf-test.sh in Python. (From OE-Core rev: 764eb2d011305b84501cc183531a2a5353b0b5ab) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-04-27 19:46:13 +0000