summaryrefslogtreecommitdiffstats
path: root/meta/lib/oeqa
Commit message (Collapse)AuthorAgeFilesLines
* selftest/package: Correct format arguments in test_gdb_hardlink_debugOla x Nilsson2018-12-131-1/+1
| | | | | | | (From OE-Core rev: 7c2c9b876334cc7357f818c3b3744fc97987e38f) Signed-off-by: Ola x Nilsson <olani@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/sdk/galculator: rewrite to use new helpersRoss Burton2018-12-081-20/+27
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/sdk/python: fix version typoRoss Burton2018-12-081-1/+1
| | | | | | | (From OE-Core rev: 581b43d7b3566624e6b17d516755d8e7a5142ebf) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/sdk/assimp: use helpersRoss Burton2018-12-081-28/+1
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/sdk/case: add fundamental helper methodsRoss Burton2018-12-081-0/+32
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/runcmd: Increase timeout deltaRichard Purdie2018-12-081-2/+2
| | | | | | | | | Expecting 1s accuracy on a 2s timeout on a heavily loaded system has proven to be unreliable. Update this to a 5s timeout with a 3s delta which should be achievable. (From OE-Core rev: 5feecb639d49d72d8a6abc589b937e07ee72f252) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/context: ensure log directory existsChen Qi2018-12-081-0/+1
| | | | | | | | | | | Ensure log directory exists to avoid the following error. FileNotFoundError: [Errno 2] No such file or directory: '/.../build-selftest/tmp/log/oe-selftest-results-20181207043431.log' (From OE-Core rev: 6c41de48db76087fee596d9440d8f05346ab1094) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/esdk: Ensure parent directory existsRichard Purdie2018-12-081-0/+1
| | | | | | | | | | | | | | | | | | | | INFO - ====================================================================== INFO - ERROR: setUpClass (eSDK.oeSDKExtSelfTest) INFO - ---------------------------------------------------------------------- INFO - Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass clss.setUpClassMethod() File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/eSDK.py", line 76, in setUpClass cls.tmpdirobj = tempfile.TemporaryDirectory(prefix="selftest-esdk-", dir=bb_vars["WORKDIR"]) File "/usr/lib/python3.5/tempfile.py", line 929, in __init__ self.name = mkdtemp(suffix, prefix, dir) File "/usr/lib/python3.5/tempfile.py", line 507, in mkdtemp _os.mkdir(file, 0o700) FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-debian/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/selftest-esdk-q7ln84gc' (From OE-Core rev: 48719bc4d108df7e357e7f22f9f356cd72b3ebbd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/esdk: Fix typo causing test failureRichard Purdie2018-12-061-1/+1
| | | | | | | | | | | | | 2018-12-06 23:19:24,564 - oe-selftest - INFO - Traceback (most recent call last): File "/media/build1/poky-sumo/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass clss.setUpClassMethod() File "/media/build1/poky-sumo/meta/lib/oeqa/selftest/cases/eSDK.py", line 78, in setUpClass cls.tmpdir_eSDKQA = cls.tempdirobj.name AttributeError: type object 'oeSDKExtSelfTest' has no attribute 'tempdirobj' (From OE-Core rev: c6de27c2f6f598849dcb8036ee849f449ba7f327) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/esdk: run selftest inside workdir not /tmpRoss Burton2018-12-051-6/+7
| | | | | | | | | | We've seen issues with rootfs size calculations and we've seen systems like opensuse which have btrfs mounted on /tmp causing selftest failures. (From OE-Core rev: 61be3cd748d1b7321a1fc4cfe84efa9b26a6aee0) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: don't litter /tmp with temporary directoriesRoss Burton2018-12-052-3/+5
| | | | | | | | | | | | | | If we need to create a temporary directory in targetbuild or buildproject use tempfile.TemporaryDirectory so that when the test case is finished, the directory is deleted. Also synchronise the logic and don't possibly store the temporary directory in self.tmpdir as nothing uses that. (From OE-Core rev: db0e658097130d146752785d0d45f46a3e0bad71) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/qemurunner: Avoid tracebacks on closed filesRichard Purdie2018-12-051-2/+2
| | | | | | | | | | | | | | Reorder the shutdown/teardown to avoid: File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py", line 224, in launch op = self.getOutput(output) File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py", line 90, in getOutput fl = fcntl.fcntl(o, fcntl.F_GETFL) ValueError: I/O operation on closed file (From OE-Core rev: 8e7d756862d2a8d62f3c87497d6d65ddb3c1b962) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/runqemu: Improve testcase failure handlingRichard Purdie2018-12-051-12/+20
| | | | | | | | | | assertTrue doesn't give good debug information when things fail. Update several to use assertIn which gives information upon failure, for the others print the log information upon failure. (From OE-Core rev: c29cb75d5ce6b0873a934f4709b0c8824f7164d3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/utils: Improve multiprocess_lauch exception handlingRichard Purdie2018-12-051-0/+4
| | | | | | | | | | | | | | We've seen a cryptic: "ERROR: Fatal errors occurred in subprocesses, tracebacks printed above" message from oe-selftest with no other traceback information. Improve the traceback logging to try and give a better indication of any errors that is ocurring. (From OE-Core rev: 521dd3d00979a27b6932e58d5497de68abac26e1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/oelib/path: don't leak temporary directoriesRoss Burton2018-12-051-7/+3
| | | | | | | | | | | | | setUp() is used to populate a directory of temporary files, and deleted in __del__. However setUp() is called once *per test* so __del__ would only be able to remove the last directory created. Fix the code by using the natural counterpart to setUp, tearDown(), to clean up. (From OE-Core rev: 68b4723e6fb11d171869185bccf28f32f6284c18) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/sdk/python: add Python 2 and fix detectionRoss Burton2018-12-031-3/+17
| | | | | | | | | | | Add a Python 2 form to exercise that if present, and fix the setUp() so it actually looks for a package that exists (nativesdk-python3 is a virtual package, the interpretter is in nativesdk-python3-core). (From OE-Core rev: d286c2ad3eec24978557e16a8fa599476791109f) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: add some tests for recipeutils modulePaul Eggleton2018-12-031-0/+137
| | | | | | | | | | | | Add some tests for functions in meta/lib/oe/recipeutils.py, in particular for a few issues I've just fixed. I haven't added tests for all of the functions - some of them are already being tested via devtool in any case. (From OE-Core rev: 72d0cb3f8f1e69aeef93ea0bc90db3e8b8d6f94c) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/context: Improve log file handlingRichard Purdie2018-12-011-3/+8
| | | | | | | | | | | | | | | | The existing logfile is simply placed in the current directory. Since the test changes cwd to BUILDDIR, the symlink to the log can be placed in an invalid directory. We also see trackbacks if the symlink is invalid. Improve things by: * Placing logs in LOG_DIR (or BUILDDIR if unset). * Using a full path to the log meaning the log and link are placed in the same directory. * Using lexists instead of exists so invalid symlinks are handled correctly. (From OE-Core rev: 750ece11bed0e62a11e0003d1d16a81f7c219761) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/buildoptions: Ensure diskmon tests run consistentlyRichard Purdie2018-12-011-3/+3
| | | | | | | | | | | | | | | Heartbeat events default to once a second and we need to ensure we have enough time in the task to see them. Add a nostamp delay task 5s long so we can have a consistently timed task which doesn't need cleanup or have unneeded dependencies. This ensures we should deterministically see the disk moinitor events regardless of the state of the build. This is done in a way which doesn't corrupt build state or need cleanup and is efficient. (From OE-Core rev: ecc49ee8986929e2429d948000a0ca588fe63959) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/buildoptions: Improve ccache testRichard Purdie2018-12-011-0/+1
| | | | | | | | | | | | | This test occisionally fails as m4 doesn't recompile, meaning the logfile test then doesn't find mention of ccache. To ensure m4 does recompile, clean m4 before force compiling it. (Reading the test is confusing due to the test cleanup also involving a clean) (From OE-Core rev: 6e0b9214a0d57ed45a5df0ba5c9887a9045b89b1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/qemurunner: Remove resource python warningsRichard Purdie2018-12-011-2/+8
| | | | | | | | | If runqemu fails it would leak an unclosed socket and file. Ensure we close these in all cases to remove the resource warning. (From OE-Core rev: ed80e46ccbc8fe8e9148d80723152066fa00ba28) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/commands: Avoid log message duplicationRichard Purdie2018-12-011-1/+1
| | | | | | | | | | | | Each time a runqemu() fails, the log handler would be left behind meaning messages from any subsequent run would be duplicated (or worse/more). This ensures we remove the handler regardless and means we no longer have the duplication. (From OE-Core rev: 532984708436bdfa3a8cac2c684a425eb249bad0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/qemurunner: Fix python ResourceWarning for unclosed fileRichard Purdie2018-12-011-0/+2
| | | | | | | | | | | | Fixes: Stderr: /media/build1/poky/meta/lib/oeqa/utils/qemurunner.py:381: ResourceWarning: unclosed file <_io.BufferedWriter name=16> self.runqemu = None (From OE-Core rev: b9e0bf919e6fc1a58e02145a363ebe7066e5bf4f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/commands: Add extra qemu failure loggingRichard Purdie2018-12-011-1/+5
| | | | | | | | | | Rather than just referring the user to the logs containing the failure, print them on the console. This aids debugging with oe-selftest with parallelisation as the logs may otherwise be lost. (From OE-Core rev: 36a018e245a232f520ff946f152cc875927a6fb4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/buildoptions: Improve ccache test failure outputRichard Purdie2018-11-271-2/+3
| | | | | | | | | | The current failure mode doesn't show us what the logs actually looked like and later cleans can lose them. Show the whole log in case of failure to aid debugging intermittent problems on the autobuilder. (From OE-Core rev: 7c3a0dc5978cea898b1ca51decf4d6e7cf9d519f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/case: Use bb.utils.remove() instead of shutil.remove()Richard Purdie2018-11-271-1/+3
| | | | | | | | | | | | | | | | This avoids problems where shutil.remove will error with: File "/usr/lib/python3.5/shutil.py", line 436, in _rmtree_safe_fd os.unlink(name, dir_fd=topfd) FileNotFoundError: [Errno 2] No such file or directory: 'S.gpg-agent.extra' when there are races over file deletion (gpg agent may be slow to exit). We already worked around speed and race issues in bb.utils. (From OE-Core rev: 00a8fd5b93a5c19ce0b7498e2bc653ce8ad58aaf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/signing: Use do_populate_lic target instead of do_packageRichard Purdie2018-11-271-3/+3
| | | | | | | | | This should speed the test up signficiantly without any loss of functionality for the purposes of the test. (From OE-Core rev: 3dde0b749643575878bfbca2f8d2d9ec30bad166) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/signing: Allow tests not to need gpg on the hostRichard Purdie2018-11-271-19/+48
| | | | | | | | | | | | We ideally don't want to use gpg from the host. This is straightforward for package management but not for sstate. For sstate, create a second build directory to run the test in using gnupg-native from the original build directory. (From OE-Core rev: 10afa94c3f0d7eb7524a26deda86949073d55fde) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/signing: Skip tests if gpg isn't foundRichard Purdie2018-11-271-12/+11
| | | | | | | | | | | Raising an assertionError in the class setup isn't a particuarly good way to indicate gpg isn't installed. Instead skip the tests if the required binary isn't present. For the signing tests we do require it to be present and can't use a prebuilt one. (From OE-Core rev: 2d486af97e51b9daa9c40482c31d637c9ab4ae79) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/qemu: Add support for slirpYeoh Ee Peng2018-11-231-3/+19
| | | | | | | | | | | | Enable qemu for slirp. Initialize Qemurunner with slirp. Setup ip and port attribute to enable connection with qemu running with slirp. [YOCTO#10713] (From OE-Core rev: 815e05ac4abb42238d6fdeb15d7426759e059ce1) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* qemurunner: Add support for slirpYeoh Ee Peng2018-11-231-4/+13
| | | | | | | | | | | | Enable qemurunner for slirp. Retrieved the ip & port from host machine to connect to qemu from host machine. [YOCTO#10713] (From OE-Core rev: 1db6a6fc9cde28d0a29bcf6d24a8bfbe51d120b1) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/qemu & runtime: qemu do not need ip input from externalYeoh Ee Peng2018-11-232-4/+3
| | | | | | | | | | | Qemu do not use the ip input from external. It will retrieve ip from QemuRunner instance and assign ip value. (From OE-Core rev: 14d99dc6c39c963ba3e0d9a30274846bd5369210) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/testsdk: Split implementation into classesJoshua Watt2018-11-212-0/+246
| | | | | | | | | | | | | | | | | | | | | | | | | Splits the SDK test implementation into configurable Python classes. The classes used for the normal and extensible SDKs are ${TESTSDK_CLASS_NAME} and ${TESTSDKEXT_CLASS_NAME} respectively. This allows SDK machines to override the classes used to implement the tests. For the traditional SDK, a common "run()" function is provided by the class (oeqa.sdk.testsdk.TestSDK), with several hook member functions that can be overridden in child classes, making it easier to have consistent behavior. The extensible SDK class (oeqa.sdkext.testsdk.TestSDKEXT) also has a common "run()" function, but no hooks have yet been added as there is not currently a known use case for create derived classes. These changes should be purely organizational; no functional changes have been made to either the standard SDK or extensible SDK tests. [YOCTO #13020] (From OE-Core rev: a06d53928b22d5f88276023c4d57b206db2f27f9) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/httpserver: Rework to avoid hangs and improve loggingRichard Purdie2018-11-164-11/+32
| | | | | | | | | | | | | | | | | | testimage.bbclass installs a SIGTERM handler which conflicts with the use of multiprocessing here. This is paritcularly problematic if the http service is terminated before its started and hence before its had a chance to reset the default signal handler (as the code was written). Instead, temporarily remove testimage's handler whilst forking the http process which means the correct handler is installed and won't deadlock. Also take the opportunity to add in some log messages about the server start and shutdown so that future debugging is easier and its clearer what the code is doing. (From OE-Core rev: 0762b6021b87ceb1f37952f3a6d64a36e99ae6a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testimage/testsdk/selftest: Avoid platform.distro_identifier deprecation ↵Richard Purdie2018-11-161-1/+2
| | | | | | | | | | warnings Use our own lsb function instead as used elsewhere by the codebase. (From OE-Core rev: acac45a6fd604d28ef7c23d67482af3d7e8bcfe3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/concurrencytest: Avoid unclosed file warningsRichard Purdie2018-11-161-0/+3
| | | | | | | | | Avoid an unclosed file per thread warning when running selftests concurrently by closing the result stream. (From OE-Core rev: 33a4a076e8aa72a872807332501e7f5ae1cee0e2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/qemurunner.py: Fix python regex warningsRichard Purdie2018-11-161-5/+5
| | | | | | | | | | | | | | | | | | | Fix the warnings: meta/lib/oeqa/utils/qemurunner.py:250: DeprecationWarning: invalid escape sequence \. ips = re.findall("((?:[0-9]{1,3}\.){3}[0-9]{1,3})", cmdline.split("ip=")[1]) meta/lib/oeqa/utils/qemurunner.py:343: DeprecationWarning: invalid escape sequence \- if re.search("root@[a-zA-Z0-9\-]+:~#", output): poky/meta/lib/oeqa/utils/qemurunner.py:350: DeprecationWarning: invalid escape sequence \- if re.search("root@[a-zA-Z0-9\-]+:~#", output): meta/lib/oeqa/utils/qemurunner.py:448: DeprecationWarning: invalid escape sequence \- if re.search("[a-zA-Z0-9]+@[a-zA-Z0-9\-]+:~#", data): by correctly marking the regexs. (From OE-Core rev: 8e6987735002560fca714f77ea8ece9d4b28f7fa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/selftest/context: Replace deprecated imp module usageRichard Purdie2018-11-161-2/+2
| | | | | | | | | | | | | Avoid the warning: meta/lib/oeqa/selftest/context.py:8: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp In this case importlib is a direct replacement. (From OE-Core rev: db7a60c36a2d3eefc61ae6e1ede01680dc932035) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/commands: Avoid unclosed file warningsRichard Purdie2018-11-141-0/+3
| | | | | | | | | | | Avoid warnings such as: meta/lib/oeqa/utils/commands.py:213: ResourceWarning: unclosed file <_io.BufferedReader name=4> return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options) (From OE-Core rev: 6a68c42de08cffbadb59ebda63fa5e19f6e5acef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/core/runner: Correctly markup regexsRichard Purdie2018-11-141-2/+2
| | | | | | | | | Avoid the warning "DeprecationWarning: invalid escape sequence \(" by marking the regexs correctly. (From OE-Core rev: cb49980fa4a158d5529902df731dec61a8c9b3d4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/loader: Fix deprecation warningRichard Purdie2018-11-141-1/+1
| | | | | | | | | | Clean up the warning: meta/lib/oeqa/core/loader.py:27: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec() _failed_test_args = inspect.getargspec(unittest.loader._make_failed_test).args (From OE-Core rev: d2deb66830be2d44532fea3d5db763b57778252a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runner: Sort the test result output by result classRichard Purdie2018-11-141-1/+10
| | | | | | | | | | | We want to see failures/errors listed last since this is the most easily visible part of the log on consoles or autobuilder output and makes human processing easier rather than having to scroll up and scan for a single failure. (From OE-Core rev: 7954b19020c28a4120bc1671aa81b9e1e2b05fa2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runner: Always show a summary of success/fail/error/skip countsRichard Purdie2018-11-141-3/+1
| | | | | | | | | Its useful to have the counts of success/failure/error/skipped at the end of the results to allow for easier human reading of what happened. (From OE-Core rev: 080d8900d470a8e7f929b0c5c2765ad461744fbb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/cases: Improve dependencies of kernel/gcc/build testsRichard Purdie2018-11-146-30/+31
| | | | | | | | | Mark up these tests as needing a compiler, make and kernel source code as appropriate, the image feature requirements can then be retired. (From OE-Core rev: 9f64e6c25abdf494fb511e9cd401f8dcaa08be2a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/cases: Improve test dependency informationRichard Purdie2018-11-1410-25/+32
| | | | | | | | | | | | Add the OEHasPackage decorator to a variety of tests so they determine automatically if they should run against a given image. To ensure tests can do this we need to move target operations such as scp commands into the tests and out of the class startup/teardown. (From OE-Core rev: 60d6580b85714b8960a964e775d76a7f937f5e5a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* testsdk: fix skipped testcase output "UNKNOWN" status while multiprocess ↵Hongxu Jia2018-11-146-8/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | execution Usually skipped testcase output "SKIPPED" [snip serial execution] |RESULTS - buildgalculator.GalculatorTest.test_galculator - Testcase -1: SKIPPED (0.01s) |RESULTS - python.PythonTest.test_python3 - Testcase -1: SKIPPED (0.01s) [snip serial execution] But if enable multiprocess execution, skipped testcase output "UNKNOWN" status [snip enable multiprocess execution] |RESULTS - buildgalculator.GalculatorTest.test_galculator - Testcase -1: UNKNOWN |RESULTS - python.PythonTest.test_python3 - Testcase -1: UNKNOWN [snip enable multiprocess execution] Here is my investigation: There is a class pairs TestProtocolClient and TestProtocolServer provided by python3-subunit. The TestProtocolClient generates a subunit stream of TestResult from a test run, and TestProtocolServer parses the stream of subunit TestResult. The class ProtocolTestCase is a unittest.TestCase adapter and it uses TestProtocolServer to parse the stream of subunit TestResult. In Yocto testsdk, it forks multiple processes to execute testcases and use TestProtocolClient to generate TestResult stream; and then it creates multiple threads to use ProtocolTestCase to parse stream of subunit TestResult through pipe; finally it passes multiple ProtocolTestCase as TestCase instance to main process and output status result. The problem point is TestProtocolServer parses `skip:' directive after reading a `test:' directive. Without `test:' directive, `skip:' directive will be ignored. All above requires SkipTest should be raised inside a test method rather than setUpClass method. Throwing SkipTest inside setUp works correctly (From OE-Core rev: 4828a88556d59e4d06933164c2ebeb9361b7450e) Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime/utils/targetbuildproject: Use a subdir within ~/Richard Purdie2018-11-141-1/+3
| | | | | | | | | Without doing this, the code can and sometimes does try and do "rm ~/" which I think we'd all prefer it didn't. (From OE-Core rev: 9dc1de59330c366783ea043c68c1b59b1b49e707) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/utils/buildproject: Only clean files if we've done somethingRichard Purdie2018-11-141-0/+5
| | | | | | | | | We should only be wiping out things on target if the tests have actually run. (From OE-Core rev: d38c3eac0a5a1a9b0eb98385832e92f48145655e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/manual/build-appliance: Drop test which is already automatedRichard Purdie2018-11-141-26/+0
| | | | | | | | The nightly-build-appliance target already tests this particular build. (From OE-Core rev: 02c5f853f30568eb3f289c6b6fb5cacf712d15f7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/runtime: Add tests for gstreamer CLIYeoh Ee Peng2018-11-141-0/+14
| | | | | | | | | | | | | | | | | | | We should not be finding GStreamer issues right around release time but that's what we seem to do. Port gstreamer tests from Intel Ref Kit. https://github.com/intel/intel-iot-refkit/blob/master/meta-iotqa/ lib/oeqa/runtime/multimedia/gstreamer/gstreamer_cli.py Add tests for gstreamer commandline tools (gst-inspect-1.0 & gst-launch-1.0). [YOCTO #11418] (From OE-Core rev: 3e77f845b517e3f4b3e7d09d088a9a80bf171653) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-26 18:24:35 +0000