| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
| |
This test keeps failing on the autobuilder and is proving extremely
annoying. It works much better in later releases but for zeus and
earlier, lets just stop running it as it doesn't really tell us
anything useful at this point, nobody has any plans to improve
the distro exclusions or otherwise fix it in the older releases.
(From OE-Core rev: 856674d6f75e2b99ae961d5ab869ff071ff5c362)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
Similarly to 04ee0e8b95cd8ed890374e0007f976684206b630, ensure only full
build paths are replaced in the environment to avoid breaking buildtools.
(From OE-Core rev: be07d93a4f59d4563f2d064be1997b39f05e9f0e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This avoids the seeing broken replacements like:
oe-selftest-centos/build/build-st-926tools/sysroots/x86_64-pokysdk-linux/etc/ssl/certs/ca-certificates.crt
which understandably break builds.
(From OE-Core rev: 04ee0e8b95cd8ed890374e0007f976684206b630)
(Cherry-picked from f930e2cadb9ee69759720b6c49aeeb6dd43a7edd but adjusted for thud)
(From OE-Core rev: b51a8c390714357713e91d828708c85cf5f68971)
(From OE-Core rev: cf51a1e4447010f927a9e3f4ff5a5c305be0c0c9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We want to test the SDK with PATH from the original host, not with our own
tools injected via HOSTTOOLS. It even uses some tools which aren't in
HOSTTOOLS.
This is necessary after changing the SDK to not reset PATH to the system
default which is bad for other reasons and brings the testing into sync
with that change.
(From OE-Core rev: 87c9602fd0dedc7bcf75b822aaf5f6ebfc17737c)
(From OE-Core rev: de3f405b6cf96311ced86bccf7202370e4d1481a)
(From OE-Core rev: aad4b41c49b1a63a767150a3451e10d15e8bcf31)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since this file is written during recipe parsing, having it in the
${BUILDDIR}/conf directory, which is covered by an inotify watcher,
will trigger a re-parse the next time bitbake is run and the resident
bitbake server is enabled. This causes the sanity_info file to be
updated again, which triggers a new parse the next time bitbake is run
ad infinitum. Moving it to ${BUILDDIR}/cache should avoid this.
(From OE-Core rev: a63d59f64a2d1f450a7639426cae8e0373a2d764)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f98103b548aa7dba6b1be6c8e02ef41858a8e85c)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We keep seeing "cannot allocate memory" errors from rpm when signing packages
on the autobuilder. The following were tried:
* checking locked memory use (isn't hitting limits)
* Restricting RPM_GPG_SIGN_CHUNK to 1
* Limiting to 10 parallel do_package_write_rpm tasks
* Allowing unlimied memory overcommit
* Disabling rpm parallel compression
and the test still failed. Further invetigation showed that the --auto-expand-secmem
wasn't being passed to gpg-agent which meant the secmem couldn't be expanded hence the
errors when there was pressure on the agent.
The reason this happens is that some of the early gpg commands can start the agent
without the option and it sticks around in memory so a version with the correct
option may or may not get started.
We therefore add the option to all the key gpg calls.
(From OE-Core rev: 028475f5b6ae9c9b181812c0c8a6aa382640f4bf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The output format for Python and GLib both can be of this form:
FAIL: foobar (Segmentation fault)
In this case the test is called foobar not foobar_segmentation_fault.
(From OE-Core rev: 95031da4f08295ad81efac1c082c48dd5c330fb0)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
TOOLCHAIN_OUTPUTNAME could be overridden. So use this variable directly
instead of its default value ${SDK_NAME}-toolchain-${SDK_VERSION}.
(From OE-Core rev: eb32dd2956da99813136842acdb010c8471f5e3c)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The current codes to skip test cases are logically correct, but they
do not work correctly in reality. It does skip the tests as the command
line argument specifies, but the related information is wrong.
e.g.
$ oe-selftest -R archiver bblayers runtime_test
2019-06-17 09:24:53,764 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it
2019-06-17 09:25:06,309 - oe-selftest - INFO - Adding layer libraries:
2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta/lib
2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-yocto-bsp/lib
2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-selftest/lib
2019-06-17 09:25:06,312 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-06-17 09:25:10,521 - oe-selftest - INFO - Adding: "include selftest.inc" in /buildarea5/chenqi/SWAT/poky/build-selftest/conf/local.conf
2019-06-17 09:25:10,521 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-06-17 09:25:10,522 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver)
2019-06-17 09:25:10,522 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "runtime_test"'
2019-06-17 09:25:10,522 - oe-selftest - INFO - Skip by the command line argument "runtime_test"
2019-06-17 09:25:10,523 - oe-selftest - INFO - test_archiver_filters_by_type (archiver.Archiver)
2019-06-17 09:25:10,523 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "runtime_test"'
2019-06-17 09:25:10,523 - oe-selftest - INFO - Skip by the command line argument "runtime_test"
The archiver.Archiver.xxx tests should be skipped by 'archiver' command line
argument, not 'runtime_test'.
Change to use a function generator to achieve the desired effect. After the change,
the effect is as follows.
$ oe-selftest -R archiver bblayers runtime_test
2019-06-17 09:19:06,223 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it
2019-06-17 09:19:19,598 - oe-selftest - INFO - Adding layer libraries:
2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta/lib
2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-yocto-bsp/lib
2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-selftest/lib
2019-06-17 09:19:19,602 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-06-17 09:19:24,368 - oe-selftest - INFO - Adding: "include selftest.inc" in /buildarea5/chenqi/SWAT/poky/build-selftest/conf/local.conf
2019-06-17 09:19:24,368 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-06-17 09:19:24,369 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver)
2019-06-17 09:19:24,369 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"'
2019-06-17 09:19:24,369 - oe-selftest - INFO - Skip by the command line argument "archiver"
2019-06-17 09:19:24,369 - oe-selftest - INFO - test_archiver_filters_by_type (archiver.Archiver)
2019-06-17 09:19:24,370 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"'
2019-06-17 09:19:24,370 - oe-selftest - INFO - Skip by the command line argument "archiver"
2019-06-17 09:19:24,370 - oe-selftest - INFO - test_archiver_filters_by_type_and_name (archiver.Archiver)
2019-06-17 09:19:24,370 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"'
2019-06-17 09:19:24,371 - oe-selftest - INFO - Skip by the command line argument "archiver"
2019-06-17 09:19:24,371 - oe-selftest - INFO - test_archiver_srpm_mode (archiver.Archiver)
2019-06-17 09:19:24,371 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"'
2019-06-17 09:19:24,372 - oe-selftest - INFO - Skip by the command line argument "archiver"
2019-06-17 09:19:24,372 - oe-selftest - INFO - test_bitbakelayers_add_remove (bblayers.BitbakeLayers)
2019-06-17 09:19:24,373 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "bblayers"'
2019-06-17 09:19:24,373 - oe-selftest - INFO - Skip by the command line argument "bblayers"
2019-06-17 09:19:24,373 - oe-selftest - INFO - test_bitbakelayers_createlayer (bblayers.BitbakeLayers)
2019-06-17 09:19:24,373 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "bblayers"'
2019-06-17 09:19:24,374 - oe-selftest - INFO - Skip by the command line argument "bblayers"
[snip]
(From OE-Core rev: efdde9ee80f17a302e2b435b6a1103b001c612dc)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
PyPi packages are now hosted at files.pythonhosted.org.
[YOCTO #13243]
(From OE-Core rev: d498a41d89adcead8995668f54c01a175c081bca)
Signed-off-by: Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
While hacking on this I got an Exception. It's better to define
variables also in python.
Signe:-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
(From OE-Core rev: 7c604a62356538de660835bd2d3ba26beccd810d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
(From OE-Core rev: 386795d14c23d2e4084563c6234779b90e4d400f)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
(From OE-Core rev: 8c0a03c24248bf69e9c8c0f2d949ff79efc9c71e)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
parameter
After the commit ad522ea6a64e ("runqemu: Let qemuparams override default
settings"), the order of the two "-serial" parameters when running the
qemu have been switched. The effect of this is that the logging thread
will use ttyS1 (of course can't capture the kernel boot message anymore),
and the test command will run on the ttyS0. So the output of the test
command may be mangled by the kernel message (such as call trace), and
let the test command produce a fake timeout error message. We can't fix
it by just adjusting the order of the threadport and serverport, since
it will break some machines such as qemuarm64 which use the virtio
serial. So using the tcpserial to setup both the threadport and
serverport.
[YOCTO Bug 13309]
(From OE-Core rev: 1ffea0c0fb3688f3cee0a5fa741e969581d01bd3)
Signed-off-by: Kevin Hao <kexin.hao@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently, the code will sum all of the different machine results into a
single report of the tests results. This can lead to confusion as to
which machine may be experiencing issues. Modify the code to store the
results in a per machine basis and report them accordingly.
(From OE-Core rev: 7ba5a0af31605eaf91eb75b5f673f4caf81361dc)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.
The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).
More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.
The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.
(From OE-Core rev: 3248a9e3c5a197321b1c4417509b9309cc3bae97)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
test runtimes in sec.
AIO: 14
MEM: 94
MSG: 89
SEM: 30
SIG: 194
THR: 399
TMR: 867
TPS: 23
(From OE-Core rev: 2ad81e503afdde2711ccc54942e83fce02f4cbe5)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds the framework for running ltp tests.
Here are some times:
math: 61
syscalls: 3957
dio: 18472
io: 29
mm: 551
ipc: 48
sched: 165
nptl: 46
pty: 37
containers: 52
controllers: 9625
filecaps: 27
cap_bounds: 27
fcntl-locktests: 29
connectors: 27
timers: 37
commands: 165
net.ipv6_lib: 30
input: 29
fs_perms_simple: 31
fs: 3476
fsx: 30
fs_bind: 28
fs_ext4: 28
cve: 675
(From OE-Core rev: 2a5767f584c159f4711afc5a01ee740d44c94d02)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As runqemu with launch_cmd means directly run the command, don't need set
rootfs or env vars.
Since commit [a847dd7202 runqemu: Let qemuparams override default settings]
applied in oe-core, if launch_cmd contains "qemuparams='***'", it does not
work, which is overridden by latter qemuparams="-serial tcp:127.0.0.1" in
QemuRunner.launch();
So we set qemuparams as a parameter in runqemu, the fix makes it work
(From OE-Core rev: 00c58ae552d7414116f13f43d4df2f6e57c0d1ba)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The commands only work with with bash. If /bin/sh is
dash like in Debian, the command execution fails with
errors like:
Standard Output: /bin/sh: 5: export: --sysroot: bad variable name
and all SDK tests fail.
(From OE-Core rev: 0423d5049492516a6df6caf3404ddaa1734ff1d9)
Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently setupClass errors were not being mapped back to the failing tests
and they were hence being marked as UNKNOWN and the test statistics were
inaccurate.
This is because whilst the errors were being encoded into the test results
stream, the decoder doesn't cope with an error outside a testStart event.
We patch in an addError handler to the outsideTest parser so that this
does get handled in a way similar to the non-concurrent case.
It would be nice if we didn't have to do this but there doesn't seem
to be any other way to fix this other than forking subunit.
We also make a minor change so another of our changes can cope with
tests without a start time.
(From OE-Core rev: 741cf82d0e7f9e797ba35157a1466fadbf74dc64)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
The string format for subunit setupClass/setupModule failures is slightly
different, tweak the regex to correctly handle both cases.
(From OE-Core rev: 3c645ecdb884ecafa7c492bdd8a952cce96d3906)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
We continued to see encoding problems with ssh commands run in oeqa. After much
research the conclusion was we should use ignore the errors since some occasional bad
locale encoding is better than the unicode decoding issues we were seeing which crashed
large parts of tests.
(From OE-Core rev: e4037c6911592700799d207ca8c3976afba18e25)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of showing:
RESULTS - ptest.PtestRunnerTest.test_ptestrunner - Testcase 1600: UNKNOWN (32.30s)
map unexpectedSuccesses to PASSED and improve the way they're displayed. We
expect/allow ptest runner to fail but if it passes we should handle it correctly.
(From OE-Core rev: 21079473e0cbd0e43682dc0d15a651e7e64db747)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
These IDs refer to testopia which we're no longer using. We would now use the test
names to definitively reference tests and the IDs can be dropped, along with their
supporting code.
(From OE-Core rev: 551153b0bd1ebbc05582f6014e3d88b9ce4a46d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[Backported to keep in-sync with future qa changes]
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This code really needs to be rewritten to not split potential
multibyte characters, for now work around it to avoid exceptions like:
File "/home/pokybuild/yocto-worker/qa-extras2/build/meta/lib/oeqa/core/target/ssh.py", line 211, in run
data = reader.read(1024, 4096)
File "/usr/lib64/python3.6/codecs.py", line 503, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 0: invalid start byte
(From OE-Core rev: 918e15e22c797d52d33c3399324e480aa1a2cc7f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some CI pipelines might perform further processing of the test output
(for instance, to plot some metrics into a chart). However, Since `thud`
we switched away from the XML-based jUnit reporting, and at the same
time we lost the ability of collecting the stdout and stderr of the
various tests.
We now restore this functionality by adding `stdout` and `stderr` keys
to the JSON reports. This behavior is off by default; in order to enable
it, one must set the `TESTREPORT_FULLLOGS` variable in the bitbake
configuration.
(From OE-Core rev: fd57b34d7c8a120273d65cd361be208fbdaeff50)
Signed-off-by: Alberto Mardegan <amardegan@luxoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
| |
(From OE-Core rev: e2b4b0580cd719cf9d48576aa7d0b88e4a286921)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
self.msg is a str(), bootlog is b'' so this code clearly doesn't work. Add
in a decode since its being used as a string.
(From OE-Core rev: e7546759d3102b576974bbab52594796f084721e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
All test id (eg. @alias) inside manual testcase file shall follow the same
test id naming convention from oeqa automated tests (eg. selftest,
runtime, sdk, etc), where the test id consists of
<test_module>.<test_suite>.<test_function>. Furthermore, there shall be
only 1 unique test_module per each manual testcases file, where
test_module match the file name itself.
This file was using test_module name that does not match the file name
itself. Fixed test_module name as well as the test_suite name.
(From OE-Core rev: f24178993e7b0830510900e80e879c7f9f8e28f0)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As part of the solution to replace Testopia, manual test cases
need to be migrated to OEQA. These manual test case json files
will serve two use cases.
Use case#1: as input to the future commandline-based manual test
runner script, where this script will display actions and expected
result information in user friendly text, then it will capture
user input for test result and log, finally it will write test
result and log into existing standardize test result json format
from OEQA framework for automated tests.
Use case#2: QA will open and read these json file manually for
planning manual test execution. Any reader interested in
understanding manual test cases will open and read these files.
(From OE-Core rev: fbeecb059d9c006f50a2a11bed5a42740722e3e8)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
All test id (eg. @alias) inside manual testcase file shall follow the same test id
naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where
the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore,
there shall be only 1 unique test_module per each manual testcases file.
This file was using more than 1 unique test_module for testcases. Furthermore,
some of the testcases were defined using different test_suite where it was not needed.
This patch fix the manual testcases file to have only 1 unique test_module
as well as test_suite to simplify test id naming.
(From OE-Core rev: be7cbe86f046bf29082c3da574377f9828aff5f4)
Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two changes made in oeqa/manual/compliance-test.json:
1. All test id (eg. @alias) inside manual testcase file shall follow the same test id
naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where
the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore,
there shall be only 1 unique test_module per each manual testcases file.
This file was using more than 1 unique test_module for testcases. Furthermore, some of
the testcases were defined using different test_suite where it was not needed. This patch
fix the manual testcases file to have only 1 unique test_module as well as test_suite to
simplify test id naming.
2. Added 2 missing test cases for Beaglebone Stress Test.
(From OE-Core rev: 5c5402bc6e6505da0203ca9245a3599bd4940dad)
Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two changes made in oeqa/manual/bsp-hw.json:
1. All test id (eg. @alias) inside manual testcase file shall follow the same test id naming
convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the
test id consists of <test_module>.<test_suite>.<test_function>. Furthermore,
there shall be only 1 unique test_module per each manual testcases file.
This file was using more than 1 unique test_module for testcases.
Furthermore, some of the testcases were defined using different test_suite where it was not
needed. This patch fix the manual testcases file to have only 1 unique
test_module as well as test_suite to simplify test id naming.
2. As per review by Intel and Windriver team, 7 testcases were found obsolete. Removed 7 testcases.
(From OE-Core rev: e1136bae81672894a277512d7084d27f4e3416b9)
Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If python3 git module is not installed, we fall back to some git
command to get info. A previous commit (b8d22ed6) accidently deleted
the return statement, causing errors like below.
Exception: UnboundLocalError: local variable 'InvalidGitRepositoryError' referenced before assignment
(From OE-Core rev: cd8aedc408afa4f8589f79e4d85befc7cac76ac6)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Now that we have a packagegroup that can be used to replace the overall
functionality of busybox as base-utils, add a test that we can continue
to build a fairly complex image without busybox being available. This
also doubles as documentation-by-test of how to do this.
(From OE-Core rev: 6319a59c1d30eeb8ad4871d43641e3469fb543ba)
Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Two identical classes exist for SystemdbootTarget,
introduced in commit f9a61d3400ad
("gummiboot: Remove/change gummiboot references with systemd-boot")
poky commit 2dce2648e3d8
Remove the duplicated copy.
(From OE-Core rev: a59562c7ddbda1a266f0ea22ab78fb86de4861a3)
Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
Note that the tests require that the host machine has a X display,
has mesa development files installed and is able to create OpenGL contexts.
(From OE-Core rev: 2868e8dfb9e62b49cd06f6c2d010405079d3a71c)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
Add a test for a multiconfig build which mixes tiny and musl builds
along with using the mcextend class to combine and package multiple images
into another image. This gives the multiconfig a decent test in a scenario
users may use.
(From OE-Core rev: 0c7fa15a7350808242754944243f01155bc6784c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
[YOCTO #13098]
(From OE-Core rev: 9c20f8c6e4bb5157550e347ffd6ef790eb6a1200)
Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
Currently if python3-git isn't installed we can get odd behaviours when the
commit_count is absent. Avoid this set of bugs by adding a fallback here.
(From OE-Core rev: b8d22ed681141ce360d742a96cec5f2925a20222)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
| |
(From OE-Core rev: ac4eba5415f39cd797a08071c0efae296ae61a70)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The parent code currently assumed that any parent branch is locally
checked out which may not be the case.
Use the local branch by default but fall back to the origin. This
also means removing the later saftey check as the branch may not exist
locally.
This fixes the autobuilder resulttool test pushing code.
(From OE-Core rev: 0ee636bc89d93282cb3d3cd747b3dd046f6e5a9c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a combined patch of the various tweaks and improvements I
made to resulttool:
* Avoid subprocess.run() as its a python 3.6 feature and we
have autobuilder workers with 3.5.
* Avoid python keywords as variable names
* Simplify dict accesses using .get()
* Rename resultsutils -> resultutils to match the resultstool ->
resulttool rename
* Formalised the handling of "file_name" to "TESTSERIES" which the code
will now add into the json configuration data if its not present, based
on the directory name.
* When we don't have failed test cases, print something saying so
instead of an empty table
* Tweak the table headers in the report to be more readable (reference
"Test Series" instead if file_id and ID instead of results_id)
* Improve/simplify the max string length handling
* Merge the counts and percentage data into one table in the report
since printing two reports of the same data confuses the user
* Removed the confusing header in the regression report
* Show matches, then regressions, then unmatched runs in the regression
report, also remove chatting unneeded output
* Try harder to "pair" up matching configurations to reduce noise in
the regressions report
* Abstracted the "mapping" table concept used to pairing in the
regression code to general code in resultutils
* Created multiple mappings for results analysis, results storage and
'flattening' results data in a merge
* Simplify the merge command to take a source and a destination,
letting the destination be a directory or a file, removing the need for
an output directory parameter
* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression
mappings
* Have the store command place the testresults files in a layout from
the mapping, making commits into the git repo for results storage more
useful for simple comparison purposes
* Set the oe-git-archive tag format appropriately for oeqa results
storage (and simplify the commit messages closer to their defaults)
* Fix oe-git-archive to use the commit/branch data from the results file
* Cleaned up the command option help to match other changes
* Follow the model of git branch/tag processing used by oe-build-perf-report
and use that to read the data using git show to avoid branch change
* Add ptest summary to the report command
* Update the tests to match the above changes
(From OE-Core rev: ff2c029b568f70aa9960dde04ddd207829812ea0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.
These scripts were developed as a test result tools to manage
these testresults.json file.
Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.
Using the "merge" operation, user can merge multiple testresults.json
files to a target file.
Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.
Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.
These resulttool operations expect the testresults.json file to use
the json format below.
{
"<testresult_1>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
...
"<testresult_n>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
}
To use these scripts, first source oe environment, then run the
entry point script to look for help.
$ resulttool
To store test result from oeqa automated tests, execute the below
$ resulttool store <source_dir> <git_branch>
To merge multiple testresults.json files, execute the below
$ resulttool merge <base_result_file> <target_result_file>
To report test report, execute the below
$ resulttool report <source_dir>
To perform regression file analysis, execute the below
$ resulttool regression-file <base_result_file> <target_result_file>
To perform regression dir analysis, execute the below
$ resulttool regression-dir <base_result_dir> <target_result_dir>
To perform regression git analysis, execute the below
$ resulttool regression-git <source_dir> <base_branch> <target_branch>
[YOCTO# 13012]
[YOCTO# 12654]
(From OE-Core rev: 78a322d7be402a5b9b5abf26ad35670a8535408a)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
These functions can be reused by the resulttool code so move to the common
function library for this purpose.
(From OE-Core rev: c66f848938c04e133259c5b6903dc592866ab385)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
During the test logrotate.LogrotateTest.test_1_logrotate_setup,
there is below logic:
# mkdir $HOME/logrotate_dir
# sed -i "s#wtmp {#wtmp {\n olddir $HOME/logrotate_dir#" /etc/logrotate.d/wtmp
After all logrotate.LogrotateTest finished, only cleanup
$HOME/logrotate_dir as below, but don't restore
the config file /etc/logrotate.d/wtmp.
[snip]
def tearDownClass(cls):
cls.tc.target.run('rm -rf $HOME/logrotate_dir')
[snip]
That's to say, there is one additional line added
to /etc/logrotate.d/wtmp and will make the logrotate
service start failed when run systemd.SystemdBasicTests.test_systemd_failed
Take an example as below when run test as root:
# cat /etc/logrotate.d/wtmp
# no packages own wtmp -- we'll rotate it here
/var/log/wtmp {
olddir /root/logrotate_dir
missingok
monthly
create 0664 root utmp
minsize 1M
rotate 1
}
# ls /root/logrotate_dir
ls: cannot access '/root/logrotate_dir': No such file or directory
# systemctl start logrotate
Job for logrotate.service failed because the control process exited with error code.
See "systemctl status logrotate.service" and "journalctl -xe" for details.
# systemctl status logrotate
logrotate.service - Rotate log files
Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset>
Active: failed (Result: exit-code) since Wed 2019-02-13 03:35:19 UTC; 7s ago
Docs: man:logrotate(8)
man:logrotate.conf(5)
Process: 540 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
Main PID: 540 (code=exited, status=1/FAILURE)
Feb 13 03:35:18 qemumips systemd[1]: Starting Rotate log files...
Feb 13 03:35:19 qemumips logrotate[540]: error: wtmp:9 error verifying olddir path /root/logrotate_dir: No such file or directory
Feb 13 03:35:19 qemumips logrotate[540]: error: found error in file wtmp, skipping
Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Failed with result 'exit-code'.
Feb 13 03:35:19 qemumips systemd[1]: Failed to start Rotate log files.
Add the logic to restore /etc/logrotate.d/wtmp to
make the cleanup complete to fix the above issue.
(From OE-Core rev: a2db9320d97d12d87524ff16a329f9c38a8da33f)
Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
This turns the core of the script into a library function. Ultimately this
will let us call that code with custom 'keywords' rather than relying
on the data parsed from bitbake metadata which can't be used when archiving
historical results.
(From OE-Core rev: 4820ca2b0850e29b04a4fd5659a6e9837d6714d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
QA team found that 10 manual bsp test cases from Testopia for
Beaglebone, EdgeRouter, and MPC need to be up streamed
[YOCTO #12650]
(From OE-Core rev: f47410899eb97cfd7db428f45b7ba354c609b4d5)
Signed-off-by: Mazliana <mazliana.mohamad@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
| |
Fix another load of regex escape sequence warnings for newer
python versions.
(From OE-Core rev: bd2c125bb9c362b6122e99dfdf4e1cfe12c26a90)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|