linux/poky.git/meta/lib/oeqa/core/runner.py, branch hardknott

oeqa: write @OETestTag content into json test reports for each case

2020-08-08T08:17:49+00:00

This allows using these tags for classification and filtering of test results according to various organization-specific criteria, such as teams responsible for the test, internal test ids, feature domains and so on. Test name itself meanwhile can stay short and human-readable. (From OE-Core rev: 3801b126eb52cd46efe417111afcd27f05d8f72b) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

testresults.json: add duration of the tests as well

2020-05-21T11:35:39+00:00

This is printed by testimage, but isn't actually saved. It's a useful metric for tracking execution times. (From OE-Core rev: 8fc19639f47b959a141dae231395bbababa644e1) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

oeqa: enable testresults.json for testexport

2020-03-12T22:49:28+00:00

Add the option --json-result-dir to oeqa core context to enable testresults.json creation for test runs via testexport. Eg. oe-test runtime --json-result-dir . (From OE-Core rev: 9d8edf33d1f5d89b310923b0aa3cc967317c7c49) Signed-off-by: Stefan Kral Signed-off-by: Richard Purdie

oeqa/core/runner.py: Fix OETestTag listing

2019-09-07T20:56:43+00:00

Use the __oeqa_testtags attribute added by OETestTag and display no, one or more tags separated by a comma. Also change the formatting of the output so that the list of tests is formatted as " ()" and remove the table header for "id" (it is no longer available). (From OE-Core rev: d62e577a4e1a5f6accbce9f7bff7317a1162d72d) Signed-off-by: Nathan Rossi Signed-off-by: Richard Purdie

oeqa/core: Implement proper extra result collection and serialization

2019-09-07T20:56:43+00:00

Implement handling of extra result (e.g. ptestresult) collection with the addition of a "extraresults" extraction function in OETestResult. In order to be able to serialize and deserialize the extraresults data, allow OETestResult add* calls to take a details kwarg. The subunit module can handle cross-process transfer of binary data for the details kwarg. With a TestResult proxy class to sit inbetween to encode and decode to and from json. (From OE-Core rev: b0831d43606415807af80e2aa1d0566d0b8c209c) Signed-off-by: Nathan Rossi Signed-off-by: Richard Purdie

oeqa/runner: Fix subunit setupClass/setupModule failure handling

2019-05-12T08:04:26+00:00

The string format for subunit setupClass/setupModule failures is slightly different, tweak the regex to correctly handle both cases. (From OE-Core rev: 425fbba9348420df72de5c6292f50acaf2023d0b) Signed-off-by: Richard Purdie

oeqa/core/runner: Handle unexpectedSucesses

2019-05-09T15:31:55+00:00

Instead of showing: RESULTS - ptest.PtestRunnerTest.test_ptestrunner - Testcase 1600: UNKNOWN (32.30s) map unexpectedSuccesses to PASSED and improve the way they're displayed. We expect/allow ptest runner to fail but if it passes we should handle it correctly. (From OE-Core rev: 7885939ca3dea9704185e93469fe515c17eb8017) Signed-off-by: Richard Purdie

meta/lib+scripts: Convert to SPDX license headers

2019-05-09T15:31:55+00:00

This adds SPDX license headers in place of the wide assortment of things currently in our script headers. We default to GPL-2.0-only except for the oeqa code where it was clearly submitted and marked as MIT on the most part or some scripts which had the "or later" GPL versioning. The patch also drops other obsolete bits of file headers where they were encoountered such as editor modelines, obsolete maintainer information or the phrase "All rights reserved" which is now obsolete and not required in copyright headers (in this case its actually confusing for licensing as all rights were not reserved). More work is needed for OE-Core but this takes care of the bulk of the scripts and meta/lib directories. The top level LICENSE files are tweaked to match the new structure and the SPDX naming. (From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e) Signed-off-by: Richard Purdie

oeqa: Drop OETestID

2019-05-09T15:31:55+00:00

These IDs refer to testopia which we're no longer using. We would now use the test names to definitively reference tests and the IDs can be dropped, along with their supporting code. (From OE-Core rev: 8e2d0575e4e7036b5f60e632f377a8ab2b96ead8) Signed-off-by: Richard Purdie

oeqa/core/runner: dump stdout and stderr of each test case

2019-04-26T09:09:08+00:00

Some CI pipelines might perform further processing of the test output (for instance, to plot some metrics into a chart). However, Since `thud` we switched away from the XML-based jUnit reporting, and at the same time we lost the ability of collecting the stdout and stderr of the various tests. We now restore this functionality by adding `stdout` and `stderr` keys to the JSON reports. This behavior is off by default; in order to enable it, one must set the `TESTREPORT_FULLLOGS` variable in the bitbake configuration. (From OE-Core rev: fd0048630ece5b21efb3a79e97046be0ab2a1514) Signed-off-by: Alberto Mardegan Signed-off-by: Richard Purdie