summaryrefslogtreecommitdiffstats
path: root/meta/lib/oeqa/oetest.py
Commit message (Collapse)AuthorAgeFilesLines
* oeqa: fix hasPackage, add hasPackageMatchRoss Burton2017-02-101-7/+21
| | | | | | | | | | | | | | | | | hasPackage() was looking for the string provided as an RE substring in the manifest, which resulted in a large number of false positives (i.e. libgtkfoo would match "gtk+"). Rewrite the manifest loader to parse the files into a proper data structure, change hasPackage to do full string matches, and add hasPackageMatch which does RE substring matches. (From OE-Core rev: b9409863af71899e02275439949e3f4cdfaf2d0f) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Sona Sarmadi <sona.sarmadi@enea.com>
* oetest: make console output more verboseEd Bartosh2016-04-181-7/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Added output of test runner to bitbake console output. bitbake <image> -c testimage now reports test progress to the console: NOTE: Executing RunQueue Tasks ... NOTE: recipe core-image-lsb-1.0-r0: task do_testimage: Started core-image-lsb-1.0-r0 do_testimage: test_ping (oeqa.runtime.ping.PingTest) ... ok core-image-lsb-1.0-r0 do_testimage: test_ssh (oeqa.runtime.ssh.SshTest) ... ok ... ---------------------------------------------------------------------- core-image-lsb-1.0-r0 do_testimage: Ran 38 tests in 785.100s core-image-lsb-1.0-r0 do_testimage: core-image-lsb-1.0-r0 do_testimage: OK (skipped=1) Adding more output to the console should also prevent autobuilder to kill long running tests as ab assumes that test is stuck if no console output produced by it for a long time. [YOCTO #8238] (From OE-Core rev: 06aa68138ec67623892221fbe9516bbb3795b48e) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runexported.py: Fix exported testMariano Lopez2016-04-121-1/+5
| | | | | | | | | | | | | With the changes introduced to test the eSDK the runexported test failed during the execution. This change fix runexported test in the least invasive way, because of the release cycle. (From OE-Core rev: da0abb9679cb1fd639859a2fdbd82101d0a81259) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/testsdk: Pass tcname to SDK and SDKExt contextsAníbal Limón2016-02-281-3/+4
| | | | | | | | | | | tcname is needed for eSDK update testcase will be used for publish it and then try to update (From OE-Core rev: a75944a63482597be88ff0f3ce55025647b78e2c) Signed-off-by: Aníbal Limón <limon.anibal@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/testsdk: Move the removal of bitbake PATH to eSDK context onlyAníbal Limón2016-02-281-1/+11
| | | | | | | | | | | | | | The removal of bitbake and scripts PATH is only needed by eSDK tests so move to eSDK context only. This also it's a support for eSDK update test because it needs to execute oe-publish-sdk from scripts. (From OE-Core rev: 4ae0c84568f39661722cbceba8ddab22cffe5003) Signed-off-by: Aníbal Limón <limon.anibal@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/oetest: Fix compatibility SDK tests using eSDK.Aníbal Limón2016-02-101-3/+10
| | | | | | | | | | | | | | | | | When run SDK tests over eSDK we need to use SDKExtTestContext instead of SDKTestContext because if we use SDKTestContext search for SDK manifest and depends on the SDK manifest generation so populate_sdk needs to be executed. Adds a compatibility mode flag to SDKExtTestContext for search tests over sdk module instead of sdkext module and change testsdk calls to comply with this new param. (From OE-Core rev: ebe743235b383b17225553d84bf8e6f80d364dcd) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/populate_sdk_ext: Add SDK_EXT_TARGET_MANIFEST and SDK_EXT_HOST_MANIFESTAníbal Limón2016-02-101-2/+9
| | | | | | | | | | | | | Extensible SDK needs to point to the correct manifest so add SDK_EXT_TARGET_MANIFEST and SDK_EXT_HOST_MANIFEST variables. oeqa/oetest.py: Fix SDKExtTestContext for load the correct manifests. (From OE-Core rev: 2310d5ad03531b7e1f9572c12c83c2fedc0291c9) Signed-off-by: Aníbal Limón <limon.anibal@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testsdkext: Add skeleton for support Extensible SDK tests.Aníbal Limón2016-02-101-1/+11
| | | | | | | | | | | | | | oeqa/sdkext: Add module and __init__.py will contain eSDK tests. classes/testsdk: Add support for run eSDK tests. oeqa/oetest: Create oeSDKExtTest for now only inherit oeSDKTest, modified SDKExtTestContext now inherit SDKTestContext and set sdkext filesdir for store data fixtures. (From OE-Core rev: f3781544a5c077610498a6b7dc5244ee4c5bc6df) Signed-off-by: Aníbal Limón <limon.anibal@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/oetest: oeSDKTest when run a command redirect env output to nullAníbal Limón2016-02-101-1/+1
| | | | | | | | | | | | | | | | | | | Some tests are failing because the eSDK env load script shows a help message on the output so redirect this to /dev/null. There was a discussion with Paul Eggleton [1], he don't want to have env variables to change the behaviour of what the env script needs to print so redirect the output in oeSDK.run() method. I didn't agree because it can hides another messages like error ones and with the variable you can control what prints or not. [1] http://lists.openembedded.org/pipermail/openembedded-core/2016-February/116744.html (From OE-Core rev: 7defb10d10861501947b4a686510e957c72313e6) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oetest.py/TestContext: Move loadTests and runTests inside it.Aníbal Limón2016-02-101-100/+97
| | | | | | | | | | | | Method's for loadTests and runTests make sense to define inside TestContext because it can be different around Image, SDK, SDKExt. (From OE-Core rev: 03af7b99e3ce36ce3e29dc31e33d2cc74eb14849) Signed-off-by: Aníbal Limón <limon.anibal@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage/testsdk: Move get test suites routine inside TestContext.Aníbal Limón2016-02-101-84/+108
| | | | | | | | | | | In order to provide better abstraction move functions to get the test suite inside the TestContext. (From OE-Core rev: 552285688441330440bd1bc138aeaea87549024a) Signed-off-by: Aníbal Limón <limon.anibal@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage/testsdk: Modularize TestContext.Aníbal Limón2016-02-101-0/+63
| | | | | | | | | | | | Move anonymous duplicated class TestContext from testimage/testsdk to oeqa/oetest now we have two new classes ImageTestContext and SDKTestContext with common code in TestContext class. (From OE-Core rev: 593f2fdf6ee94c5f91761a669048da3598cbe3fa) Signed-off-by: Aníbal Limón <limon.anibal@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* get_test_suites: Add sdkext type for load test suites.Aníbal Limón2016-02-101-0/+2
| | | | | | | | (From OE-Core rev: 01a420dc9c0c04c793bbf5824db7c2b9d7c9de9b) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: Modularize helper functions for get test lists.Aníbal Limón2016-02-101-0/+74
| | | | | | | | | | | | | Test lists functions can be used in other parts so modularize it and move to oeqa/oetest.py library. Testimage class was updated to meet the new sign of the functions. (From OE-Core rev: c9f771533af70e7ccb1e7064e58926cfaee7367a) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oetest: Add tearDownLocal classMariano Lopez2015-10-191-0/+6
| | | | | | | | | | | | | | | | | | | | In the current state there are some runtime test that don't run the tearDown method fromm oeRuntimeTest class because the tearDown class is overwritten in the child class. This change adds tearDownLocal method in oeRuntimeTest class that will run after tearDown. This method can be overwritten in the child classes to implement specific test functionality. [YOCTO #8465] (From OE-Core rev: b5917f8032d6965596868b2fe01da4e0682e2804) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oetest: Change logic of a failed testMariano Lopez2015-10-011-13/+13
| | | | | | | | | | | | | | | | | | | | | Currently the logic to check if a test failed was to check for an exception in the thread, but some decorators used in the syslog runtime test would generate and handle exceptions; this will mess with the current check logic and will dump the host and the target as if the test failed. This patch changes the check logic to verify if the test that just happend is in the failure test list and dump the host and target accordingly. [YOCTO #8406] (From OE-Core rev: 8b97d9320b989023c62db8246f5d8d2126c3723e) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/oetest: Fix SDK command executionRichard Purdie2015-09-281-1/+1
| | | | | | | | | | | | The SDK environment wasn't being sourced when running these test commands, meaning in some cases the cross compiler was being tested, not the SDK tools. This is clearly not the intent so fix this by ensuring the SDK environment is present. This fixes test failures in multilib SDKs. (From OE-Core rev: 39c56f7c280cfe6bf40ea9145f3f7060b565bc62) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/oetest: Remove bb as requirement for oetest.Lucian Musat2015-09-241-8/+21
| | | | | | | | | | | In order for the test export runner to work oetest needs to be separated from bitbake environment. There is no need to use bb import here so we can use a logger instead. (From OE-Core rev: 22bd8a600823306d7c5965a9a69b8e8888993513) Signed-off-by: Lucian Musat <george.l.musat@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/testimage: Enhance -v switch in testimageLucian Musat2015-09-231-0/+13
| | | | | | | | | | | | | When testimage is run with -v switch now individual test progress can be seen directly in bitbake console. [YOCTO #6841] (From OE-Core rev: 27138b2eeafa8b81f3020ad9d8a55263e636288c) Signed-off-by: Lucian Musat <george.l.musat@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/decorators: Added decorator to restart the DUT in case of test hang.Lucian Musat2015-09-181-0/+6
| | | | | | | | | | | | | | Once the DUT is hanged during testing, currently all the following test cases have to wait for default timeout to exit. Using this decorator the user can choose a timeout at case by case basis and what happens when the timeout is reached by overwriting the self.target.restart method. [YOCTO #7853] (From OE-Core rev: ce9a7501ea29ddba61ef7c297223b3f7eca5a2a1) Signed-off-by: Lucian Musat <george.l.musat@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: handle SIGTERM to conclude runqemuMariano Lopez2015-09-091-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | In the current state if a SIGTERM is sent to the testimage worker, the worker will exit but runqemu and qemu won't exit and the processes need to be killed manually to free the bitbake lock. This allows to catch the SIGTERM signal in testimage, this way it is possible to stop runqemu and qemu and allow to free the bitbake lock. Also this allows to skip the rest of the tests when running the tests in qemu or real hardware. This also solves minimal breaks in the setup of the runtime test when checking if qemu is alive. [YOCTO #8239] (From OE-Core rev: 2694d2f17d597b44fcc7aed5f6836081fa88a6b3) Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com> Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Test failure/cleanup improvementsRichard Purdie2015-09-061-0/+3
| | | | | | | | | | | | | | | | | Currently, if qemu segfaults, the tests merrily continue trying to execute which takes time for them to timeout and is a bit silly. Worse, no logs about the segfault are shown to the user, its silent! This patch tries to unravel the tangled web of issues and ensures that we: * install a SIGCHLD handler which tells the user qemu exited * check if qemu is running, if it isn't fail the test outright * don't leave processes behind in sshcontrol which would hold bitbake.lock and block shutdown (From OE-Core rev: 5c04b1ca1e989f569d5755a646734d01a0c56cae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: filter proper test cases by tagszjh2015-09-041-1/+27
| | | | | | | | | | | If a test case is decorate by oeqa.utils.decorators.tag, this case will by add a tag, testrunner will filter these tags by TEST_SUITES_TAGS [YOCTO #7849] (From OE-Core rev: 085589b1018ba4d950baf7bcfb499be02c1b29fc) Signed-off-by: zjh <junhuix.zhang@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* runexported: Make it compatible with host dumpMariano Lopez2015-09-031-2/+1
| | | | | | | | | | | | | | Currently it is not possible to run a exported test, but this patch will allow to use the HosDumper class when running a exported test, otherwise the HostDumper class will break runexpored test. [YOCTO #8118] (From OE-Core rev: 25814dbdd12db0499e4d81876873158e1b8a4fcc) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* qemurunner: Added host dumps when there are errorsMariano Lopez2015-09-031-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | This adds an instance of HostDumper to qemurunner, with this instance now is possible to get dumps from the host when there is an error. This adds dump points in the next cases: - runqemu exits before seeing qemu pid - Fail to get qemu process arguments - Not reach login banner before timeout - qemu pid never appears This also modifies the constructors of BaseDumper, HostDumper and TargetDumper, they don't require the datastore anymore, but the feature to replace datastore variables has been lost (never used) [YOCTO #8118] (From OE-Core rev: b0af40fb76cd5035696e9d8a44f815f64214d23a) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oetest: Fix regresion when testing real hardwareMariano Lopez2015-08-301-5/+7
| | | | | | | | | | | | | | This fix the regresion introduced in commit 9c72c1a5aa0b49d3895bbefee7a264adfcc6f4ca when testing with real hardware. This regression happens when a test in real hardware fails. [YOCTO #8203] (From OE-Core rev: e63889cc70041ada022c2ebe789b569f9e44dbd6) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* dump: Created new classes for dump host and targetMariano Lopez2015-08-301-46/+6
| | | | | | | | | | | | | | | | | | | | | | | | It makes sense to separate the dump commands from the oeRuntimeTest class, this way it can be used in all the test context. These are the changes included in this patch: - Created classes: BaseDumper, HostDumper, TargetDumper - Create an instance of HostDumper in imagetest.bbclass and add it to TestContext class, this way any class that have access to the TestContext would be able to dump logs from the host - Create an instance of TargetDumper in QemuTarget class after get the runner, this way it is accessible during the tests. [YOCTO #8118] (From OE-Core rev: ad10af6be343b5425fde43055263b0744c161cb3) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: Run commands in target and host when test failsMariano Lopez2015-08-301-15/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch modify three files altought two of them are minimal modifications. This version includes the changes proposed by Paul. testimage.bbclass: Create new vars for easy modification of the dump directory and commands to be run on host and target when a test fails TESTIMAGE_DUMP_DIR: Directory to save the dumps testimage_dump_target: Commands to run on target testimage_dump_host: Commands to run on host oetest.py: - Allow to use the vars defined in testimage class - Now able to run commands in the host and dump the results - Fix an issue with the condition where to run the dump commands (Before it run the commands every test after a failure, now it runs the commands only in tests that failed) - Fix the output to stdout [YOCTO #8118] (From OE-Core rev: 26fe645457633f90bb5ddbb12f5f7b9ca4a06cc5) (From OE-Core rev: 7b4fbbf979ed22434b8e3f83ae145139bb0d9fc7) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/oetest.py: add better package search for hasPackage()Costin Constantin2015-08-261-3/+3
| | | | | | | | | | | | | Modified hasPackage() to split the content of pacakage manifest file in containing lines and search at the begining of each line the existance of the needed pkg. [YOCTO #8170] (From OE-Core rev: f07045fcae859c902434062d1725f1348f42d1dd) Signed-off-by: Costin Constantin <costin.c.constantin@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oetest.py: Don't wait to write dump filesMariano Lopez2015-08-161-17/+15
| | | | | | | | | | | | | | This allows to write the dump files immediately after get the data from the target. Before this, it would run all the commands and write the files. The old behavior could cause no log written at all if the serial console gets stuck. (From OE-Core rev: 73c98d38e94d3b1407620c134f3b00dcd9d6132c) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oetest.py: Added method tearDown for oeRuntimeTestMariano Lopez2015-08-161-0/+37
| | | | | | | | | | | | | | | | The tearDown method is triggered when a tests ends it doesn't matter if fails or succeeds. Inside this method added an evalution to check if fails and then run some commands in the target to get the data for later debugging. [YOCTO #8118] (From OE-Core rev: 8bbfef69828d9b053e2a33dfa9d8318d9572cf6b) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oeqa: Add sdk tests for gcc/perl/pythonRichard Purdie2015-04-291-0/+10
| | | | | | | | | Enhance testing of the generated SDK tarballs by adding tests for gcc/perl/python based on the existing runtime tests. (From OE-Core rev: 18160403427b2aab4207c939312fb0981c3f2d1b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: sort modules based on dependenciesPatrick Ohly2015-04-101-3/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | TEST_SUITES="auto" is useful to run all suitable tests without having to hard-code the list. However, it did not take test dependencies into account, which can be an issue for tests which really depend on some other test to run first. To fix this, modules get loaded in the order determined by TESTS_SUITES, but then get re-ordered based on dependencies derived from @skipUnlessPassed before running them. The original order is used to break ties when there are no dependencies, so reordering only occurs when really necessary. @skipUnlessPassed gets extended such that it makes the test name a method depends on available for inspection by the test loader in oetest.py. Unfortunately Python's unittest offers no API to inspect tests in a TestSuite, so the code has to rely on implementation details to find all tests. The worst that can happen when the implementation changes is that tests are not found and reordering does not happen. (From OE-Core rev: 6e4543a35836c572b23b9f8162b19d1e038d3ed2) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/oetest.py: use bb loggingPatrick Ohly2015-04-101-2/+3
| | | | | | | | | | | | | | | Plain print has two drawbacks: - the output is handled by different processes, causing the message about found tests to appear randomly after the result of the initial tests - the output is not part of the bitbake console output Affects image testing with testimage.bbclass. (From OE-Core rev: 4b6f87a82ece230b7296f1577d4092ef559af1d4) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Added package installer to oetest to aid in future automatic install ↵Lucian Musat2014-09-101-0/+7
| | | | | | | | | | of packages (From OE-Core rev: e0af8a3c751f938faf7deb60b51f42450ae58e27) Signed-off-by: Lucian Musat <georgex.l.musat@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/oetest.py: enable sdk tests to use hasFeature and hasPackage methods.Corneliu Stoicescu2014-08-271-2/+2
| | | | | | | | | In order to use the hasFeature and hasPackage methods, we need to make oeSDKTest extend oeTest and also set the test context (tc) attribute in the oeTest class when loading the tests. (From OE-Core rev: 54436aeed5ac5e0da0779919f8524a0603e19c6a) Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/oetest.py: enable sdk testingCorneliu Stoicescu2014-08-181-11/+18
| | | | | | | | | | | | - add support for sdk tests in the loadTests and runTests methods - add new oeSDKTest test object NOTE: Original patch made by: Richard Purdie <richard.purdie@linuxfoundation.org> (From OE-Core rev: 6c147e5c82b39773e135ca828b37905cbb31be3b) Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/rutime: Added testcase decorators for automated runtime tests. Also ↵Lucian Musat2014-07-251-0/+2
| | | | | | | | | added LogResults decorator for oeTest class in oetest.py (From OE-Core rev: 95b83084487d0712362ade8ac487999c3274bb96) Signed-off-by: Lucian Musat <georgex.l.musat@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: Refactor test skipping decorators to use the unittest result objectLucian Musat2014-07-251-17/+0
| | | | | | | | | | | | In order to make the test skipping decorators independent of the oeTest object we rely on the unittest result object to construct skip, fail and error lists used by these decorators. Created a new object getResults that analyses upper frames and retrieves the unittest result object instance, then return a list of failed, skipped and error tests. Also removed the oetest import from decorators.py because it was no longer required. (From OE-Core rev: 4d2d201158236bd4c72546cf8db88681ff921b11) Signed-off-by: Lucian Musat <georgex.l.musat@intel.com> Signed-off-by: Saul Wold <sgw@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: add ability to export testsStefan Stanacar2014-02-091-11/+6
| | | | | | | | | | | | | | | | | | | | | Add the ability to export the tests so that they can run independently of the build system, as is required if you want to be able to hand the test execution off to a scheduler. Booting/deployment of the target is still handled by the build system, as before, only the execution of the tests happens outside of the build system. Tests exported are the ones defined in TEST_SUITES. No tests have been changed as interesting parts of the data store have been exported and tests can continue to query them as before. Small adjustments were made for a couple of oeqa modules though. [YOCTO #5613] (From OE-Core rev: 155dd52e0f707e06f50756584a50f744ba6b7844) Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com> Signed-off-by: Saul Wold <sgw@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime: drop restartTarget methodStefan Stanacar2014-01-291-4/+0
| | | | | | | | | | | | | | | Restart is a property of a specific target, not of a test class, should a test really need to restart the target the direct method should be called. Also some tests used this to enforce more ram, which makes sense only for qemu targets only (and the inital reason this was needed isn't valid anymore, qemu machines had the default ram size bumped a while ago). (From OE-Core rev: 333a4326082e500bdbcd323af37e183e74adf617) Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage: use the new targetcontrol.py module for running testsStefan Stanacar2013-12-031-10/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | This patch makes the necessary changes for using the targetcontrol.py module so that one can run the same tests on a qemu instance or a remote machine based on the value of TEST_TARGET variable: "qemu" or "simpleremote". The default value is "qemu" which starts a qemu instance and it's the with what we currently have. With "simpleremote", the remote machine must be up with network and ssh and you need to set TEST_TARGET_IP with the IP address of the remote machine (it can still be a qemu instance that was manually started). Basically testimage.bbclass now does something along the lines of: - load tests -> deploy (prepare) / start target -> run tests. There were a couple of changes necessary for tests and also some cleanups/renames that were needed to adjust this change. (use ip everywhere when refering to target and server_ip when refering to host/build machine) Also two unnecessary and unsed methods were dropped from sshcontrol. [ YOCTO #5554 ] (From OE-Core rev: a7820350fa3271d78ed7476e02f4aef593be1125) Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oeqa: use the new manifest file for determining if a package is installedStefan Stanacar2013-12-031-10/+4
| | | | | | | | | | | | | | | | Use the new manifest file instead of ${WORKDIR}/installed_pkgs.txt for determining if an image has a certain package, because installed_pkgs.txt goes away with rm_work enabled. We can't use the IMAGE_MANIFEST var for the file path because that relies on IMAGE_NAME which changes at every run (because of date), so we use the link which points to the last one built. [ YOCTO #5072 ] (From OE-Core rev: f57c83fc33583c140f668946f3f3e79b960aa9ee) Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oeqa: add oeTest superclassStefan Stanacar2013-11-051-19/+21
| | | | | | | | | | | | | | Add oeTest superclass, make oeRuntimeTest inherit that and make the necesarry adjustments. Tests in lib/oeqa/runtime don't change, they still inherit oeRuntimeTest. We should do this because oetest.py in the future can be a base module for more stuff than oeRuntimeTest. (From OE-Core rev: cd4ed41a070bd52c446ac3df8337f17ab9421145) Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com> Signed-off-by: Saul Wold <sgw@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/oetest: oeRuntimeTest: enable long messagesMihai Lindner2013-09-061-0/+2
| | | | | | | | | | Set longMessage to True for all tests derived from oeRuntimeTest, in order to have somewhat info on assertions with cryptic or no messages. (From OE-Core rev: e5196ebd6c652431a3b80cdc9f60b752401314f4) Signed-off-by: Mihai Lindner <mihaix.lindner@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oeqa: change behaviour for unskippable testsStefan Stanacar2013-08-261-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a test module wants to be skipped because it doesn't apply to the image but it was nevertheless a required test (one in TEST_SUITES), we issued an warning that it was a required test and went on with running the module. Usually all tests in the module failed (e.g gcc tests on a non-sdk image), but this allowed us to know that something went wrong with the image (some package/feature didn't make it). However, instead of just issuing an warning and running the tests it's better to throw an exception. The traceback will tell us what's wrong, and we don't run every single test method. Output will look like this: --snip-- | NOTE: Test modules ['oeqa.runtime.ping', 'oeqa.runtime.ssh', 'oeqa.runtime.gcc'] | NOTE: Found 5 tests | test_ping (oeqa.runtime.ping.PingTest) ... ok | test_ssh (oeqa.runtime.ssh.SshTest) ... ok | ERROR | | ====================================================================== | ERROR: setUpModule (oeqa.runtime.gcc) | ---------------------------------------------------------------------- | Traceback (most recent call last): | File "/mnt/back/yocto/poky/meta/lib/oeqa/runtime/gcc.py", line 8, in setUpModule | skipModule("Image doesn't have tools-sdk in IMAGE_FEATURES") | File "/mnt/back/yocto/poky/meta/lib/oeqa/oetest.py", line 108, in skipModule | "\nor the image really doesn't have the requred feature/package when it should." % (modname, reason)) | Exception: | Test gcc wants to be skipped. | Reason is: Image doesn't have tools-sdk in IMAGE_FEATURES | Test was required in TEST_SUITES, so either the condition for skipping is wrong | or the image really doesn't have the requred feature/package when it should. | | ---------------------------------------------------------------------- | Ran 2 tests in 1.036s | | FAILED (errors=1) | NOTE: Sending SIGTERM to runqemu --snip-- (From OE-Core rev: fd51cecf8b258d9f839a0ecebde69d09f75dc468) Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com> Signed-off-by: Saul Wold <sgw@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oeqa: make it possible to restart the targetStefan Stanacar2013-08-261-0/+7
| | | | | | | | | | | | | Tweak QemuRunner so we can actually restart the qemu target in a test (if we want more memory for example). Also add a restart method to the base test class so that tests can use it. (From OE-Core rev: 9de7fe11967576f4a8b24e653c6b9a02e5f6d85b) Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com> Signed-off-by: Saul Wold <sgw@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage.bbclass, lib/oeqa: add headers and commentsStefan Stanacar2013-07-161-0/+9
| | | | | | | | | | | Adds some comments to testimage.bbclass and the files it calls, just to give an ideea of what it does. (From OE-Core rev: 8bbb7116cf02466dfc59a17dc7bb51287aeea55b) Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com> Signed-off-by: Saul Wold <sgw@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oeqa/oetest.py: provide a ps command for all testsStefan Stanacar2013-07-161-3/+2
| | | | | | | | | | | | Many tests will use 'ps' but we need to know if it's busybox or standard ps. Drop the existing check from the connman test. (From OE-Core rev: 1515d33d2c5b7275a3ac20e07c1db1d8273de796) Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com> Signed-off-by: Saul Wold <sgw@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oeqa/oetest.py: base module for all runtime unittestsStefan Stanacar2013-07-091-0/+101
This module contains the base class for all runtime tests and some helper methods. (From OE-Core rev: 7765c27705e90381975fb2b89ea2181287517761) Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com> Signed-off-by: Radu Moisan <radu.moisan@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-01 00:03:29 +0000