summaryrefslogtreecommitdiffstats
path: root/scripts/oe-selftest
Commit message (Collapse)AuthorAgeFilesLines
* oe-selftest: Error if known problem variables are setRichard Purdie2017-04-131-0/+8
| | | | | | | | | | | | | | | Setting SANITY_TESTED_DISTROS or PRSERV_HOST are known to break oe-selftest. Rather than have the user experience this, refuse to execute unless the environment is correct. Ideally we'd try and unset these but that is a more invasive change and this at least makes people aware of the problem. [YOCTO #11292] (From OE-Core rev: 0c9b981e88c76da316e76f17e6da3a03b87c5008) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest: automatically add the meta-selftest layerBenjamin Esquivel2017-01-231-2/+10
| | | | | | | | | | | | adding the meta-selftest layer is a small bump you always encounter when you want to run selftest. Adding an automatic procedure to include the layer if it is not yet present. (From OE-Core rev: b5000dd7b1d74d5dfede869ffa75ed87f5a8553c) Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: Error if the user has buildhistory enabledRichard Purdie2017-01-201-0/+4
| | | | | | | | | | | | | buildhistory interfers with various tests so error if the user has it enabled, hence avoiding time spent running the tests only for them to fail. For example, if there is an sstate cache and something comes from that cache, you can get a "version went backwards" warning from buildhistory which would then change the outcome of a test. Its safer/easier to disable it. (From OE-Core rev: 941ec7c50c30052e346e72fef2920135dba89a21) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa.utils.metadata: allow storing any bitbake config variablesMarkus Lehtonen2017-01-191-2/+2
| | | | | | | | | | | | | | | | | | | Make it possible to store any bitbake config variables in the metadata. Config values will be stored under a new config element in the xml report: <config> <variable name="MACHINE">qemux86</variable> </config> The value of MACHINE is moved there instead of having a dedicated <machine> element. [YOCTO #10590] (From OE-Core rev: 6e7e6e37664b0a86111272f5f6f4a4e1d0f23302) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa.utils.metadata: rename 'revision' to 'commit'Markus Lehtonen2017-01-191-1/+1
| | | | | | | | | | | | | | Revision is a bit vague and could point to a tag, for example. Git commit objects are unambiguous and persistent so be explicit that the element should contain git commit hash. [YOCTO #10590] (From OE-Core rev: a6521b89e86ddba6bd646dd3974d7b0390323b56) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa.utils.metadata: drop 'unknown' git data elementsMarkus Lehtonen2017-01-191-2/+2
| | | | | | | | | | | | | It's better just to not have the xml elements than to have elements with faux data. One could have git branch named 'unknown', for example. [YOCTO #10590] (From OE-Core rev: fce531c21f5e56d0f416b3405a0b0fc5ba567679) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa.utils.metadata: re-organise distro informationMarkus Lehtonen2017-01-191-2/+2
| | | | | | | | | | | | | Use the same format, based on /etc/os-release, as for host distro information. [YOCTO #10590] (From OE-Core rev: 0156ef46ccf5334ee72f0202f1089249c62af37b) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: fix typoChen Qi2017-01-161-1/+1
| | | | | | | | | | Change 'agains' to 'against'. (From OE-Core rev: b72dba9cadb1ae0ee3feb184af1c5bb4a99f70e3) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: fix behaviour if oe-selftest.log is a dangling symlinkPaul Eggleton2017-01-091-1/+2
| | | | | | | | | | | | | If you delete the log file that the oe-selftest.log symlink points to but not the symlink itself, because we were using os.path.exists() here the code assumed that the symlink didn't exist when in fact it still did. Use os.path.lexists() instead. (From OE-Core rev: 263af91a0efd21e041ecdb0c40f9b2d4e735f67d) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: Improve BUILDDIR environment handlingRichard Purdie2017-01-051-6/+13
| | | | | | | | | | | | | | | | Its possible something (like bitbake/tinfoil2) may mess around with the environment and using the enviroment as a global variable store isn't particularly nice anyway. This patch changes the BUILDDIR usages so that the environment isn't used as a global store and a global variable is used instead. Whilst that is still not perfect, it does avoid the current double and triple backtraces we're seeing where tinfoil2/bitbake has trampled the enviroment leading to failures of failures making debugging even harder. (From OE-Core rev: 689b676bbf2f1a5fadb04aeb41d5e68e35356545) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: import git module only when neededEd Bartosh2016-12-201-1/+1
| | | | | | | | | | | | | | | | | git module is not included into standard Python library and therefore causes import errors on the systems where PythonGit is not installed. As git module only used in the code implementing --repository functionality it's better to import git only in the scope that requires it. [YOCTO #10821] (From OE-Core rev: 66be32c1a075201d6ee0e9b9e10b84e6a2ace745) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: Add option to submit test result to a git repository.Mariano Lopez2016-12-131-0/+102
| | | | | | | | | | | | | | | | | | | | | | This new option allows to commit the result to a git repository, along with the results it will add a metadata file for information of the current selftest run, such as: hostname, machine, distro, distro version, host version, and layers. This implementation will have a branch per different hostname, testing branch, and machine. To use this feature use: oe-selftest <options> --repository <repository_link> [YOCTO #9954] (From OE-Core rev: 758e4b7e619f9aaa79e3910b5539ff8fb2d1064a) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: enforce en_US.UTF-8 localeMaciej Borzecki2016-11-231-0/+3
| | | | | | | | | | | Replicate bitbake and eforce en_US.UTF-8 locale so that ouptut of locale-aware tools remains stable. (From OE-Core rev: 17cd2cb99d3610fd77595ff116b2168188c250cd) Signed-off-by: Maciej Borzecki <maciej.borzecki@rndity.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: fix handling of test cases without ID in --list-tests-byMaciej Borzecki2016-11-151-4/+9
| | | | | | | | | | | | | | | | | | | | | | | | Running `oe-selftest --list-tests-by module wic` will produce the following backtrace: Traceback (most recent call last): File "<snip>/poky/scripts/oe-selftest", line 668, in <module> ret = main() File "<snip>/poky/scripts/oe-selftest", line 486, in main list_testsuite_by(criteria, keyword) File "<snip>/poky/scripts/oe-selftest", line 340, in list_testsuite_by ts = sorted([ (tc.tcid, tc.tctag, tc.tcname, tc.tcclass, tc.tcmodule) for tc in get_testsuite_by(criteria, keyword) ]) TypeError: unorderable types: int() < NoneType() The root cause is that a test case does not necessarily have an ID assigned, hence its value is None. Since Python 3 does not allow comparison of heterogeneous types, TypeError is raised. (From OE-Core rev: 71c6790689e2cbd3c4e882335b3b03e635ad46ed) Signed-off-by: Maciej Borzecki <maciej.borzecki@rndity.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: check for coverage version before starting testsHumberto Ibarra2016-09-151-0/+4
| | | | | | | | | | | python coverage versions lower than 4.x have problems with some distros. Adding the 4.x version as requirement to continue with coverage tracking. [YOCTO #10207] (From OE-Core rev: a378b817504986173c4b0984a28aead247589b3f) Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: simplifying log filenamesBenjamin Esquivel2016-08-011-1/+1
| | | | | | | | | | | | | | | | | avoiding characters like ':' and making a clearer separation of the fields that compose the filename. Changing from: oe-selftest-2016-07-20_16:05:27.log to: oe-selftest-20160720-160527.log (From OE-Core rev: e7b2362d723b5dcabb440cd513380bfe8a0badb2) Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: export test results via xmlrunnerBenjamin Esquivel2016-08-011-3/+30
| | | | | | | | | | | | | | | | | | | if available, use the xmlrunner for exporting the test results to a dir named the same than the log where the text results are stored. this means creating a dir with the name of the log (without the .log) and dumping there the xml files that indicate the results of each of the tests. if xmlrunner is not available then it will behave the same as before, no xml exports. [YOCTO#9682] (From OE-Core rev: d51f9dd34d759c77b9e7050405cbb6a88a578f73) Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: print errors when failed to find testRobert Yang2016-07-211-2/+10
| | | | | | | | | | | | | For example: $ oe-selftest --run-tests-by name hello world 2016-07-12 00:33:28,678 - selftest - ERROR - Failed to find test: hello 2016-07-12 00:33:28,679 - selftest - ERROR - Failed to find test: world (From OE-Core rev: 665a0f93bde0d61e0c7ceab072ca3f1f22b2f700) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts: python3: decode subprocess outputEd Bartosh2016-06-031-1/+1
| | | | | | | | | | | stdeout and stderr content returned by subprocess API has different types in Python 3(bytes) and Python 2(string). Decoding it to 'utf-8' makes it unicode on both pythons. (From OE-Core rev: ae4d36375683b6cfd48af25bfca70b2e77f7ade5) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/lib: Complete transition to python3Richard Purdie2016-06-021-4/+6
| | | | | | | | | This patch contains all the other misc pieces of the transition to python3 which didn't make sense to be broken into individual patches. (From OE-Core rev: fcd6b38bab8517d83e1ed48eef1bca9a9a190f57) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/lib: Update to explictly create lists where neededRichard Purdie2016-06-021-1/+1
| | | | | | | | | Iterators now return views, not lists in python3. Where we need lists, handle this explicitly. (From OE-Core rev: caebd862bac7eed725e0f0321bf50793671b5312) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Remove inadvertant python3 specific changeRichard Purdie2016-05-221-1/+1
| | | | | | (From OE-Core rev: 1bfe160fa1050c189364e2dec5d018f3ac91a670) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/selftest/scripts: Use print function for python3 compatibilityRichard Purdie2016-05-211-25/+24
| | | | | | | | | | | | Used print function instead of print statement to make the code work in python 3. [Changes from both Ed and Richard] (From OE-Core rev: ced1995694c394d92cb82fb9c25a33dc027a3b69) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Improve listing by reducing unneeded spacingHumberto Ibarra2016-05-191-5/+5
| | | | | | | | | | | | | | | | | | | | | The --list-tests option assumes a terminal of 150 characters wide, which is clearly wrong. The output for this command is messy and hard to understand for lower widths. Every command should look good in a 80 characters terminal. Unfortunately, this can't be done at the moment. The bad naming of testcases have made the test names incredibly long. This patch reduces spacing between columns and shows the tests names in a concise and understandable way. The format is even the same one required for running a testcase. Once the testcase naming improves, this output will look even better. [Yocto #9534] (From OE-Core rev: 53da74155942febd520836cabf3aa727c53ce5ca) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: Correct the usage examplesPeter Kjellerstedt2016-04-181-4/+4
| | | | | | | (From OE-Core rev: 7f5d150e4f9270e586c8a279ee64561c54689ffc) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: avoid the creation of coverage file when coverage not ↵Humberto Ibarra2016-04-061-14/+14
| | | | | | | | | | | | | | | | | | | installed Coverage subprocessing file is being created even when coverage is not installed, which causes errors of "module not found" to be send to the oe-selftest output. This patch adds indent to the block of code creating this coverage file, so it can only be executed when coverage is actually installed. [Yocto #9334] (From OE-Core rev: 03d1711350895e941a476bd2d2a6cd389be07509) Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: remove coverage file if any coverage option is givenHumberto Ibarra2016-04-061-15/+16
| | | | | | | | | | | | | | | Coverage temporal file for sub-processing is being removed only when the --coverage option was found. This is wrong since the file is created when any coverage option (source, include or omit) is given, even if --coverage is not one of them. This patch makes sure to remove the file if any coverage option was given. (From OE-Core rev: 7157fd3bfbf2d5ae07fd4e726884baf7138bb24f) Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: remove unneeded coverage warningHumberto Ibarra2016-04-061-3/+1
| | | | | | | | | | | | There is a message that warns the user about enabling subprocessing for coverage to work. After the fix for Yocto #8930, this task is done automatically, so the warning is not needed anymore. (From OE-Core rev: 1ccfdb31ea8a50e695d368c4fc78185254aac763) Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts, lib: Don't limit traceback lengths to arbitrary valuesRichard Purdie2016-03-311-1/+1
| | | | | | | | | | | | | | | | There appears to have been a lot of copy and pasting of the code which prints tracebacks upon failure and limits the stack trace to 5 entries. This obscures the real error and is very confusing to the user it look me an age to work out why some tracebacks weren't useful. This patch removes the limit, making tracebacks much more useful for debugging. [YOCTO #9230] (From OE-Core rev: 6069175e9bb97ace100bb5e99b6104d33163a3a2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts:/oe-selftest: Use timestamp instead of test names in coverage data fileHumberto Ibarra2016-03-281-4/+9
| | | | | | | | | | | | | | | | | | | | | This fixes the problem by changing the name to the coverage data file, using the timestamp as an identifier. The name for the coverage data file is constructed based on the tests ran; this has created a couple of issues so far, affecting coverage report. If --run-tests-by option is given, the data file name won't have any identifier, causing following runs to overwrite themselves. On the other hand, if too many tests are given, the file name exceeds linux limits and fails to store the coverage data all together. [Yocto #9253] (From OE-Core rev: 7c0c9d1411e04935675f4c4be8768723c1d09b50) Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: Fixed --list-tests-by tag optionDaniel Istrate2016-03-201-10/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Commit 35be67951305950ba797dc2efddbc7d88fc0556a broke the --list-tests-by tag option. This patch fixes that. Having a module in lib/oeqa/selftest named testmodule: class TestClass(oeSelfTest): @tag(feature='tag1') def test_func1(self): pass @tag(feature=('tag1', 'tag2')) def test_func2(self): pass @tag(feature=('tag2', 'tag3')) def test_func3(self): pass @tag(feature=('tag1', 'tag2', 'tag3')) def test_func4(self): pass $ oe-selftest --list-tests-by tag tag1 ID TAG(s) NAME CLASS MODULE ---- ---------------- ---------- --------- -------- tag1 test_func1 TestClass testmodule tag1, tag2 test_func2 TestClass testmodule tag1, tag2, tag3 test_func4 TestClass testmodule ______________________________ Filtering by: tag Looking for: tag1 Total found: 3 $ oe-selftest --list-tests-by tag tag1 tag2 ID TAG(s) NAME CLASS MODULE ---- ---------------- ---------- --------- -------- tag1 test_func1 TestClass testmodule tag1, tag2 test_func2 TestClass testmodule tag1, tag2, tag3 test_func4 TestClass testmodule tag2, tag3 test_func3 TestClass testmodule ______________________________ Filtering by: tag Looking for: tag1, tag2 Total found: 4 $ oe-selftest --list-tests-by tag tag* ID TAG(s) NAME CLASS MODULE ---- ---------------- ---------- --------- -------- tag1 test_func1 TestClass testmodule tag1, tag2 test_func2 TestClass testmodule tag1, tag2, tag3 test_func4 TestClass testmodule tag2, tag3 test_func3 TestClass testmodule ______________________________ Filtering by: tag Looking for: tag* Total found: 4 (From OE-Core rev: 28c1cffacf341ad64a5b68d8a0176f92b49135c0) Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Add short names to most common optionsHumberto Ibarra2016-03-121-4/+4
| | | | | | | | | | | | | | Add short names to most common options in oe-selftest. The options changed were --run-tests, --run-all-tests, --list-tests and --list-modules. [Yocto #9079] (From OE-Core rev: 05f699e3c2847f0c64118df8ffd43bf2f95ed4d1) Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Add search expression matching to run/list optionsHumberto Ibarra2016-03-121-64/+44
| | | | | | | | | | | | | | | | The oe-selftest script required an exact matching for the parameters passed to its run-tests-by and list-tests-by options. Many tests can be retrieved here and filtering is a must. This patch add this filtering functionality by enabling the use of wildcards such as "*". [Yocto #8916] (From OE-Core rev: 01a242645cc12daed3f4da30d8c49813642d75e6) Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: Add support for lib/oeqa/selftest subdirectoriesCostin Constantin2016-02-281-25/+28
| | | | | | | | | | | | | | This patch adds functionality to allow creating subdirectories inside lib/oeqa/selftest for all layers present in BBLAYERS. Like this, test cases can be grouped into organized directories. Addresses [YOCTO #7865] (From OE-Core rev: 445b84456659ebb355149f6b0dca543b0bb2679c) Signed-off-by: Costin Constantin <costin.c.constantin@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Use site.USER_SITE to run coverage configuration code ↵Humberto Ibarra2016-02-281-12/+34
| | | | | | | | | | | | | | | | | | | | | | for sub-process Coverage in oe-selftest currently requires to create or modify a sitecustomize.py file according the coverage tool setup instructions (http://coverage.readthedocs.org/). This file has to be located in the system's python folder, which is not a good solution since this folder is not accesible to non-privileged users. The best solution so far is to create this file in the home directory. This is implemented by creating the temporal file in the user site default folder. [Yocto #8930] (From OE-Core rev: 3f9b1658d745b536eff1017b2c74a9dff46b6f4a) Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Add filtering to the coverage data gathered by oe-selftestHumberto Ibarra2016-02-281-7/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When --coverage is used, oe-selftest gathers coverage data from the testcases executed. The command lacks a way of filtering which files to gather coverage data from. This patch adds three options to specify which files should be considered. The --coverage-source option specifies folders, while --coverage-include and --coverage-omit specify patterns to have an extra level of filtering. Some examples: 1. oe-selftest --run-all-tests --coverage Gathers coverage data from the default poky folders 2. oe-selftest --run-all-tests --coverage --coverage-include /home/me/poky/scripts/* Gathers coverage data only for the files located under '/home/me/poky/scripts' 3. oe-selftest --run-all-tests -coverage --coverage-omit /home/me/poky/meta* Gathers coverage data. Files inside all the folders starting with 'meta' under '/home/me/poky' are omited 4. oe-selftest --run-all-tests --coverage --coverage-source /home/me/poky/bitbake Gathers coverage data only from files inside the folder: '/home/me/poky/bitbake' [Yocto #8920] (From OE-Core rev: 923481c7d8c09ed9b03109cf4debcc6b07845c59) Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Allow to run tests on random/all MACHINEsDaniel Istrate2016-01-071-9/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add an option for random MACHINE into oe-selftest: --machine [random/all] 1. random: will set a random MACHINE for each test 2. all: will run tests for all machines Custom machine sets only weak default values (??=) for MACHINE in machine.inc. This let test cases that require a specific MACHINE to be able to override it, using (?= or =). e.g.: oe-selftest --run-tests signing --machine random --> will run all tests switching MACHINE randomly for each test oe-selftest --run-tests signing --machine all --> for each machine will run all tests oe-selftest --run-all-tests --machine random Also update oeqa/selftest/base.py to accomodate this feature. Fix for [YOCTO #5880]. (From OE-Core rev: 4a9c3653eecd9a3c1b45bab5e16c8d679c55f8bd) Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: New option --list-testsDaniel Istrate2016-01-071-0/+21
| | | | | | | | | | | | This option will list all available tests in a comprehensive manner. Fix for [YOCTO #8868] (From OE-Core rev: 8b9ea3ee39bceef7ee6faeef64f33bd9dbad2e4b) Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: Improved --list-classes when determining test namesDaniel Istrate2016-01-071-1/+1
| | | | | | | | | | | | | | | | | --list-classes does a weak validation when determining test names: (if method.startswith("test_") which could report any class attribute that starts with 'test_' as a valid test case. This fix checks that the class attribute that starts with 'test_' is also callable (is a method). Fix for [YOCTO #8862] (From OE-Core rev: 175810503d5596370cf7d840539ebdf35cf30278) Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Remove extra coverage data added to unittestsHumberto Ibarra2016-01-071-69/+103
| | | | | | | | | | | | | | | | | | | Coverage data tracking initiates too early, causing coverage data from the oe-selftest environment setting to be added to each run. Even when no tests are run oe-selftest reports around 24% of coverage due to this extra data. Change the custom resultclass used by the TextTestRunner to one generated from the command arguments. The generated class processes coverage when needed, running coverage setup just before the first testcase is run and reporting after the last one finished. [Yocto #8846] (From OE-Core rev: d66a4c5cb77f745e973daf34b84724f91549f391) Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Add support for selftest log with timestampCostin Constantin2015-12-281-1/+6
| | | | | | | | | | | | Each time oe-selftest runs, the oe-selftest.log file is overwritten. This patch solves it by adding time stamp to each selftest log file and doing a symlink named as oe-selftest.log to the last one created. (From OE-Core rev: da8097480ad70e7a75608d733c63c3ae5a337974) Signed-off-by: Costin Constantin <costin.c.constantin@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts: print usage in argparse-using scripts when a command-line error occursPaul Eggleton2015-12-281-1/+2
| | | | | | | | | | | | | | For scripts that use Python's standard argparse module to parse command-line arguments, create a subclass which will show the usage the usage information when a command-line parsing error occurs. The most common case would be when the script is run with no arguments; at least then the user immediately gets to see what arguments they might need to pass instead of just an error message. (From OE-Core rev: d62fe7c9bc2df6a4464440a3cae0539074bf99aa) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts: oe-selftest Added new features.Daniel Istrate2015-12-121-2/+216
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | [YOCTO #8750] Allow oe-selftest to run custom test suites based on different criteria 1. Can run custom lists of tests based on different criteria: --run-tests-by <name|class|module|id|tag> <list of tests|classes|modules|ids|tags> eg: --run-tests-by module imagefeatures signing recipetool --run-tests-by id 1377 1273 935 --run-tests-by tag wic sstate bitbake 2. Can list tests based on different criteria: --list-tests-by <name|class|module|id|tag> <list of tests|classes|modules|ids|tags> eg: --list-tests-by module imagefeatures signing recipetool --list-tests-by id 1377 1273 935 --list-tests-by tag wic sstate bitbake 3. Can list all tags that have been set to test cases: --list-tags The list of tags should be kept as minimal as possible. This helps preview the tags used so far. To take advantage of the 'tag' feature: - add @tag(feature=<>) to testcases eg: @tag(feature='signing') for a single tag @tag(feature=(('signing', 'sstate')) or @tag(feature=['signing', 'sstate']) for multiple tags (From OE-Core rev: 2d3a6d22e155911e39e4b7e323317f4a7cb1cb95) Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: Enable code coverage on unit testsLeonardo Sandoval2015-12-011-0/+61
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Enable code coverage through the library 'python coverage'. In case the environment variable COVERAGE_PROCESS_START is present (one of the requisites for measuring sub-processes; the second one is including some coverage statements into the python sitecustomize.py file) it will be taken into account, otherwise it is exported with value '.coveragerc'. The latter value is a configuration file (also automatically created) with some default settings. Once tests are executed, a coverage report is shown on the log and the coverage output data is stored with name '.coverage.<args>' where '<args>' is the name of the unit tests executed or 'all_tests' when running with --run-all-tests. This output data can be latter used for better reporting using the same tool (coverage). As briefly indicate before, measuring sub-process implies setting the env variable COVERAGE_PROCESS_START (done automatically by the oe-selftest code with this patch if not already set) and creating a sitecustomize.py as explained on [1]. If either one of these is missing, complete coverage will be incomplete. Current measurements for 'oe-selftest --run-all-tests' indicate that current coverage is around 42 % taking into account BBLAYERS, bitbake and scripts folders. More details on [2], indicating the coverage per file/module. This tasks has been done together with Humberto Ibarra <humberto.ibarra.lopez@linux.intel.com> [YOCTO #8679] [1] http://coverage.readthedocs.org/en/latest/subprocess.html [2] https://bugzilla.yoctoproject.org/attachment.cgi?id=2854 (From OE-Core rev: b3feee2cefbbd98b66dc395b651f47c5028c80a0) Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: add scripts/lib and bitbake/lib to pathPaul Eggleton2015-07-271-1/+4
| | | | | | | | | | In particular, this allows us to use code from bitbake's bb module (such as tinfoil). (From OE-Core rev: 5ca91cd0b56eb2da80c781fb46ae0b600c3f1eb9) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: add libdirs from BBPATH to sys.pathChristopher Larson2015-07-271-0/+7
| | | | | | | | | | | | This ensures that oeqa.selftest.* from layers are found. [YOCTO #7625] (From OE-Core rev: 182b8ed9f26ed8b6a8eb6dcaec82db9aca14b010) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: obey oeqa.selftest.__path__Christopher Larson2015-07-271-5/+6
| | | | | | | | | | | | | This ensures that all paths that hold selftest tests will be checked (oeqa.selftest is a namespace package). [YOCTO #7625] (From OE-Core rev: 3c60cbced7b101ee52ce4a0a0bce542fd38f1821) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Added mechanism for including/removing bblayers.incDaniel Istrate2015-07-081-1/+15
| | | | | | | | | | | | When oe-selftest starts it includes bblayers.inc into bblayers.conf When oe-selftest ends it deletes bblayers.inc and the included line from bblayers.conf (From OE-Core rev: cf7bf27f565b34fdcd2caa25aaef068c0970965e) Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: timestamp the test runsRoss Burton2015-06-271-1/+12
| | | | | | | | | | The selftest can take a couple of hours to run, so add a custom result class to timestamp the output to make it easy to spot any slow tests. (From OE-Core rev: cce89c44948ee66ad0abb491be57e270038270e4) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: add command line option to list test classesAlexandru DAMIAN2015-02-081-0/+20
| | | | | | | | | | | | | | | While trying to discover what tests are available, I felt the need to be able to list all individual tests so I can run specific tests. This patch adds the "--list-classes" command line option that lists the unit test classes and methods available. (From OE-Core rev: f0ba400a398a172f9ebf953bb3a26e5d911a17d6) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-17 01:29:51 +0000