summaryrefslogtreecommitdiffstats
path: root/scripts/oe-selftest
Commit message (Collapse)AuthorAgeFilesLines
* oe-selftest: Add support for lib/oeqa/selftest subdirectoriesCostin Constantin2016-02-281-25/+28
| | | | | | | | | | | | | | This patch adds functionality to allow creating subdirectories inside lib/oeqa/selftest for all layers present in BBLAYERS. Like this, test cases can be grouped into organized directories. Addresses [YOCTO #7865] (From OE-Core rev: 445b84456659ebb355149f6b0dca543b0bb2679c) Signed-off-by: Costin Constantin <costin.c.constantin@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Use site.USER_SITE to run coverage configuration code ↵Humberto Ibarra2016-02-281-12/+34
| | | | | | | | | | | | | | | | | | | | | | for sub-process Coverage in oe-selftest currently requires to create or modify a sitecustomize.py file according the coverage tool setup instructions (http://coverage.readthedocs.org/). This file has to be located in the system's python folder, which is not a good solution since this folder is not accesible to non-privileged users. The best solution so far is to create this file in the home directory. This is implemented by creating the temporal file in the user site default folder. [Yocto #8930] (From OE-Core rev: 3f9b1658d745b536eff1017b2c74a9dff46b6f4a) Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Add filtering to the coverage data gathered by oe-selftestHumberto Ibarra2016-02-281-7/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When --coverage is used, oe-selftest gathers coverage data from the testcases executed. The command lacks a way of filtering which files to gather coverage data from. This patch adds three options to specify which files should be considered. The --coverage-source option specifies folders, while --coverage-include and --coverage-omit specify patterns to have an extra level of filtering. Some examples: 1. oe-selftest --run-all-tests --coverage Gathers coverage data from the default poky folders 2. oe-selftest --run-all-tests --coverage --coverage-include /home/me/poky/scripts/* Gathers coverage data only for the files located under '/home/me/poky/scripts' 3. oe-selftest --run-all-tests -coverage --coverage-omit /home/me/poky/meta* Gathers coverage data. Files inside all the folders starting with 'meta' under '/home/me/poky' are omited 4. oe-selftest --run-all-tests --coverage --coverage-source /home/me/poky/bitbake Gathers coverage data only from files inside the folder: '/home/me/poky/bitbake' [Yocto #8920] (From OE-Core rev: 923481c7d8c09ed9b03109cf4debcc6b07845c59) Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Allow to run tests on random/all MACHINEsDaniel Istrate2016-01-071-9/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add an option for random MACHINE into oe-selftest: --machine [random/all] 1. random: will set a random MACHINE for each test 2. all: will run tests for all machines Custom machine sets only weak default values (??=) for MACHINE in machine.inc. This let test cases that require a specific MACHINE to be able to override it, using (?= or =). e.g.: oe-selftest --run-tests signing --machine random --> will run all tests switching MACHINE randomly for each test oe-selftest --run-tests signing --machine all --> for each machine will run all tests oe-selftest --run-all-tests --machine random Also update oeqa/selftest/base.py to accomodate this feature. Fix for [YOCTO #5880]. (From OE-Core rev: 4a9c3653eecd9a3c1b45bab5e16c8d679c55f8bd) Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: New option --list-testsDaniel Istrate2016-01-071-0/+21
| | | | | | | | | | | | This option will list all available tests in a comprehensive manner. Fix for [YOCTO #8868] (From OE-Core rev: 8b9ea3ee39bceef7ee6faeef64f33bd9dbad2e4b) Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: Improved --list-classes when determining test namesDaniel Istrate2016-01-071-1/+1
| | | | | | | | | | | | | | | | | --list-classes does a weak validation when determining test names: (if method.startswith("test_") which could report any class attribute that starts with 'test_' as a valid test case. This fix checks that the class attribute that starts with 'test_' is also callable (is a method). Fix for [YOCTO #8862] (From OE-Core rev: 175810503d5596370cf7d840539ebdf35cf30278) Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Remove extra coverage data added to unittestsHumberto Ibarra2016-01-071-69/+103
| | | | | | | | | | | | | | | | | | | Coverage data tracking initiates too early, causing coverage data from the oe-selftest environment setting to be added to each run. Even when no tests are run oe-selftest reports around 24% of coverage due to this extra data. Change the custom resultclass used by the TextTestRunner to one generated from the command arguments. The generated class processes coverage when needed, running coverage setup just before the first testcase is run and reporting after the last one finished. [Yocto #8846] (From OE-Core rev: d66a4c5cb77f745e973daf34b84724f91549f391) Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Add support for selftest log with timestampCostin Constantin2015-12-281-1/+6
| | | | | | | | | | | | Each time oe-selftest runs, the oe-selftest.log file is overwritten. This patch solves it by adding time stamp to each selftest log file and doing a symlink named as oe-selftest.log to the last one created. (From OE-Core rev: da8097480ad70e7a75608d733c63c3ae5a337974) Signed-off-by: Costin Constantin <costin.c.constantin@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts: print usage in argparse-using scripts when a command-line error occursPaul Eggleton2015-12-281-1/+2
| | | | | | | | | | | | | | For scripts that use Python's standard argparse module to parse command-line arguments, create a subclass which will show the usage the usage information when a command-line parsing error occurs. The most common case would be when the script is run with no arguments; at least then the user immediately gets to see what arguments they might need to pass instead of just an error message. (From OE-Core rev: d62fe7c9bc2df6a4464440a3cae0539074bf99aa) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts: oe-selftest Added new features.Daniel Istrate2015-12-121-2/+216
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | [YOCTO #8750] Allow oe-selftest to run custom test suites based on different criteria 1. Can run custom lists of tests based on different criteria: --run-tests-by <name|class|module|id|tag> <list of tests|classes|modules|ids|tags> eg: --run-tests-by module imagefeatures signing recipetool --run-tests-by id 1377 1273 935 --run-tests-by tag wic sstate bitbake 2. Can list tests based on different criteria: --list-tests-by <name|class|module|id|tag> <list of tests|classes|modules|ids|tags> eg: --list-tests-by module imagefeatures signing recipetool --list-tests-by id 1377 1273 935 --list-tests-by tag wic sstate bitbake 3. Can list all tags that have been set to test cases: --list-tags The list of tags should be kept as minimal as possible. This helps preview the tags used so far. To take advantage of the 'tag' feature: - add @tag(feature=<>) to testcases eg: @tag(feature='signing') for a single tag @tag(feature=(('signing', 'sstate')) or @tag(feature=['signing', 'sstate']) for multiple tags (From OE-Core rev: 2d3a6d22e155911e39e4b7e323317f4a7cb1cb95) Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: Enable code coverage on unit testsLeonardo Sandoval2015-12-011-0/+61
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Enable code coverage through the library 'python coverage'. In case the environment variable COVERAGE_PROCESS_START is present (one of the requisites for measuring sub-processes; the second one is including some coverage statements into the python sitecustomize.py file) it will be taken into account, otherwise it is exported with value '.coveragerc'. The latter value is a configuration file (also automatically created) with some default settings. Once tests are executed, a coverage report is shown on the log and the coverage output data is stored with name '.coverage.<args>' where '<args>' is the name of the unit tests executed or 'all_tests' when running with --run-all-tests. This output data can be latter used for better reporting using the same tool (coverage). As briefly indicate before, measuring sub-process implies setting the env variable COVERAGE_PROCESS_START (done automatically by the oe-selftest code with this patch if not already set) and creating a sitecustomize.py as explained on [1]. If either one of these is missing, complete coverage will be incomplete. Current measurements for 'oe-selftest --run-all-tests' indicate that current coverage is around 42 % taking into account BBLAYERS, bitbake and scripts folders. More details on [2], indicating the coverage per file/module. This tasks has been done together with Humberto Ibarra <humberto.ibarra.lopez@linux.intel.com> [YOCTO #8679] [1] http://coverage.readthedocs.org/en/latest/subprocess.html [2] https://bugzilla.yoctoproject.org/attachment.cgi?id=2854 (From OE-Core rev: b3feee2cefbbd98b66dc395b651f47c5028c80a0) Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: add scripts/lib and bitbake/lib to pathPaul Eggleton2015-07-271-1/+4
| | | | | | | | | | In particular, this allows us to use code from bitbake's bb module (such as tinfoil). (From OE-Core rev: 5ca91cd0b56eb2da80c781fb46ae0b600c3f1eb9) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: add libdirs from BBPATH to sys.pathChristopher Larson2015-07-271-0/+7
| | | | | | | | | | | | This ensures that oeqa.selftest.* from layers are found. [YOCTO #7625] (From OE-Core rev: 182b8ed9f26ed8b6a8eb6dcaec82db9aca14b010) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: obey oeqa.selftest.__path__Christopher Larson2015-07-271-5/+6
| | | | | | | | | | | | | This ensures that all paths that hold selftest tests will be checked (oeqa.selftest is a namespace package). [YOCTO #7625] (From OE-Core rev: 3c60cbced7b101ee52ce4a0a0bce542fd38f1821) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: Added mechanism for including/removing bblayers.incDaniel Istrate2015-07-081-1/+15
| | | | | | | | | | | | When oe-selftest starts it includes bblayers.inc into bblayers.conf When oe-selftest ends it deletes bblayers.inc and the included line from bblayers.conf (From OE-Core rev: cf7bf27f565b34fdcd2caa25aaef068c0970965e) Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: timestamp the test runsRoss Burton2015-06-271-1/+12
| | | | | | | | | | The selftest can take a couple of hours to run, so add a custom result class to timestamp the output to make it easy to spot any slow tests. (From OE-Core rev: cce89c44948ee66ad0abb491be57e270038270e4) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: add command line option to list test classesAlexandru DAMIAN2015-02-081-0/+20
| | | | | | | | | | | | | | | While trying to discover what tests are available, I felt the need to be able to list all individual tests so I can run specific tests. This patch adds the "--list-classes" command line option that lists the unit test classes and methods available. (From OE-Core rev: f0ba400a398a172f9ebf953bb3a26e5d911a17d6) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: fix startup backtraceAlexandru DAMIAN2015-02-081-1/+3
| | | | | | | | | | | Fixing a backtrace dump that happens if the script is started without sourcing the oe-init-build-env first. (From OE-Core rev: 0d769abcab272f41d74ed4d7915d26c7c309253a) Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: add command-line parsing and optionsCorneliu Stoicescu2014-07-191-28/+53
| | | | | | | | | [YOCTO #6453] (From OE-Core rev: d9291390d56cae9c9d0f5e44d5e7293260dad077) Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: return based on the test resultsStefan Stanacar2014-02-091-2/+4
| | | | | | | | | | Regardless if the tests passed or not the script returned 0, which isn't what one would expect. (From OE-Core rev: c38f943c7fbb1fc077c875099dce8f73f41043b9) Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oeqa/selftest/base, scripts/oe-selftest: fix wrong remove path and do a ↵Stefan Stanacar2013-12-051-0/+12
| | | | | | | | | | | | | | | | complete cleanup at the end The script should clean-up all the .inc files that might have been created by tests regardless of the outcome or if the script is interrupted. (currently the last test will leave a conf/selftest.inc around, even if it's not included anywhere) Also fix delete_recipeinc to actually delete what's supposed to. (From OE-Core rev: 6008745c56800e0f5f01a756be0701cebd9de4ae) Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-selftest: script to run builds as unittest against bitbake or ↵Stefan Stanacar2013-12-031-0/+148
various scripts The purpose of oe-selftest is to run unittest modules added from meta/lib/oeqa/selftest, which are tests against bitbake tools. Right now the script it's useful for simple tests like: - "bitbake --someoption, change some metadata, bitbake X, check something" type scenarios (PR service, error output, etc) - or "bitbake-layers <...>" type scripts and yocto-bsp tools. This commit also adds some helper modules that the tests will use and a base class. Also, most of the tests will have a dependency on a meta-selftest layer which contains specially modified recipes/bbappends/include files for the purpose of the tests. The tests themselves will usually write to ".inc" files from the layer or in conf/selftest.inc (which is added as an include in local.conf at the start and removed at the end) It's a simple matter or sourcing the enviroment, adding the meta-selftest layer to bblayers.conf and running: oe-selftest to get some results. It would finish faster if at least a core-image-minimal was built before. [ YOCTO #4740 ] (From OE-Core rev: 41a4f8fb005328d3a631a9036ceb6dcf75754410) Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-16 14:20:39 +0000