linux/poky.git/meta/lib/oeqa/core/runner.py, branch nanbield

oeqa/runner: Ensure class setup errors are shown to bitbake logging

2023-09-26T09:25:42+00:00

This took a bit of digging but failure messages from testimage are shown to bitbake's logging through stopTest. In the case of a setUpClass failure stopTest is never called and the bitbake logging never sees the error. It would still be in the task logfile. Add some code+comment to ensure logs not shown to the user mid stream are shown at the end. (From OE-Core rev: 7342c418bda2cc3c337a190089864ea731ff62e4) Signed-off-by: Richard Purdie

oeqa/core/runner: add helper to know about expected failures

2023-06-17T10:50:57+00:00

Testing framework currently uses the unittest.expectedFailure decorator for tests that can have intermittent failures (see PTEST_EXPECT_FAILURE = "1") in core-image-ptest.bb. While it allows upper layers to run tests without failing on "fragile" tests, it prevents those from knowing more about those failing tests since they are not accounting as failures (for example we could want to retrieve some logs about failed tests to improve them, and eventually to drop expectFailure decorator) Add a helper to allow upper layers to know about those failures which won't make global testing session (From OE-Core rev: 34595858b14f628a8282777b91c841add6ebe1ca) Signed-off-by: Alexis Lothoré Signed-off-by: Richard Purdie

oeqa: write @OETestTag content into json test reports for each case

2020-08-08T08:17:49+00:00

This allows using these tags for classification and filtering of test results according to various organization-specific criteria, such as teams responsible for the test, internal test ids, feature domains and so on. Test name itself meanwhile can stay short and human-readable. (From OE-Core rev: 3801b126eb52cd46efe417111afcd27f05d8f72b) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

testresults.json: add duration of the tests as well

2020-05-21T11:35:39+00:00

This is printed by testimage, but isn't actually saved. It's a useful metric for tracking execution times. (From OE-Core rev: 8fc19639f47b959a141dae231395bbababa644e1) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

oeqa: enable testresults.json for testexport

2020-03-12T22:49:28+00:00

Add the option --json-result-dir to oeqa core context to enable testresults.json creation for test runs via testexport. Eg. oe-test runtime --json-result-dir . (From OE-Core rev: 9d8edf33d1f5d89b310923b0aa3cc967317c7c49) Signed-off-by: Stefan Kral Signed-off-by: Richard Purdie

oeqa/core/runner.py: Fix OETestTag listing

2019-09-07T20:56:43+00:00

Use the __oeqa_testtags attribute added by OETestTag and display no, one or more tags separated by a comma. Also change the formatting of the output so that the list of tests is formatted as " ()" and remove the table header for "id" (it is no longer available). (From OE-Core rev: d62e577a4e1a5f6accbce9f7bff7317a1162d72d) Signed-off-by: Nathan Rossi Signed-off-by: Richard Purdie

oeqa/core: Implement proper extra result collection and serialization

2019-09-07T20:56:43+00:00

Implement handling of extra result (e.g. ptestresult) collection with the addition of a "extraresults" extraction function in OETestResult. In order to be able to serialize and deserialize the extraresults data, allow OETestResult add* calls to take a details kwarg. The subunit module can handle cross-process transfer of binary data for the details kwarg. With a TestResult proxy class to sit inbetween to encode and decode to and from json. (From OE-Core rev: b0831d43606415807af80e2aa1d0566d0b8c209c) Signed-off-by: Nathan Rossi Signed-off-by: Richard Purdie

oeqa/runner: Fix subunit setupClass/setupModule failure handling

2019-05-12T08:04:26+00:00

The string format for subunit setupClass/setupModule failures is slightly different, tweak the regex to correctly handle both cases. (From OE-Core rev: 425fbba9348420df72de5c6292f50acaf2023d0b) Signed-off-by: Richard Purdie

oeqa/core/runner: Handle unexpectedSucesses

2019-05-09T15:31:55+00:00

Instead of showing: RESULTS - ptest.PtestRunnerTest.test_ptestrunner - Testcase 1600: UNKNOWN (32.30s) map unexpectedSuccesses to PASSED and improve the way they're displayed. We expect/allow ptest runner to fail but if it passes we should handle it correctly. (From OE-Core rev: 7885939ca3dea9704185e93469fe515c17eb8017) Signed-off-by: Richard Purdie

meta/lib+scripts: Convert to SPDX license headers

2019-05-09T15:31:55+00:00

This adds SPDX license headers in place of the wide assortment of things currently in our script headers. We default to GPL-2.0-only except for the oeqa code where it was clearly submitted and marked as MIT on the most part or some scripts which had the "or later" GPL versioning. The patch also drops other obsolete bits of file headers where they were encoountered such as editor modelines, obsolete maintainer information or the phrase "All rights reserved" which is now obsolete and not required in copyright headers (in this case its actually confusing for licensing as all rights were not reserved). More work is needed for OE-Core but this takes care of the bulk of the scripts and meta/lib directories. The top level LICENSE files are tweaked to match the new structure and the SPDX naming. (From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e) Signed-off-by: Richard Purdie