summaryrefslogtreecommitdiffstats
path: root/meta/recipes-devtools
diff options
context:
space:
mode:
authorTony Tascioglu <tony.tascioglu@windriver.com>2021-06-04 10:07:56 -0400
committerRichard Purdie <richard.purdie@linuxfoundation.org>2021-06-06 23:16:51 +0100
commitbe491f7343c7b384d791e4b4b1f6cb7bcbf9c2fb (patch)
tree047e066df3fc3151ff0328606b5b92bc70b8747f /meta/recipes-devtools
parent48ecf617b55e14d2e4d6f399144a5f4cbcef6777 (diff)
downloadpoky-be491f7343c7b384d791e4b4b1f6cb7bcbf9c2fb.tar.gz
valgrind: Improve non-deterministic ptest reliability
Several of the valgrind tests (particulaly helgrind) are unreliable and can fail with a different output. Particularly, there is a higher chance of failure on QEMU instances with SMP enabled and on systems with more interrupts such as laptops on powersave. The tests have been reported upstream as being unreliable dating back over 5 years, due in part to the ordering of threads during an "unwinding" process in helgrind. https://bugs.kde.org/show_bug.cgi?id=345121 https://bugs.kde.org/show_bug.cgi?id=430321 A workaround to improve the reliability of such tests is to force them to run on a single CPU core using taskset. This greatly reduces the chance of a failure. >From my testing, I have found it can help reduce the rate of failures on both a laptop and QEMU by over 5x. Stress-testing in QEMU for several hours did not result in a failure while running the test normally did. The flaky or undeterministic thread-based tests are defined in the taskset_nondeterministic_tests file. These test cases will be run with taskset 0x00000001 to run on a single CPU core rather then the regular test. The edited run-ptest executes the flaky tests first, then ignores them to not duplicate the results from the main tests. Everything modified is restored when testing is complete. The drawback is that this isn't a foolproof solution. It helps the tests fail much less frequently, and considering how this issue has been documented for a long time, a workaround such as this is needed. (From OE-Core rev: b318944dd72ca7b0408e955f3599381ab3ac3ba8) Signed-off-by: Tony Tascioglu <tony.tascioglu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'meta/recipes-devtools')
-rwxr-xr-xmeta/recipes-devtools/valgrind/valgrind/run-ptest12
-rw-r--r--meta/recipes-devtools/valgrind/valgrind/taskset_nondeterministic_tests4
2 files changed, 16 insertions, 0 deletions
diff --git a/meta/recipes-devtools/valgrind/valgrind/run-ptest b/meta/recipes-devtools/valgrind/valgrind/run-ptest
index 60d243276b..11050a8312 100755
--- a/meta/recipes-devtools/valgrind/valgrind/run-ptest
+++ b/meta/recipes-devtools/valgrind/valgrind/run-ptest
@@ -32,6 +32,13 @@ if [ "$arch" = "aarch64" ]; then
32 done 32 done
33fi 33fi
34 34
35echo "Run flaky tests using taskset to limit them to a single core."
36for i in `cat taskset_nondeterministic_tests`; do
37 taskset 0x00000001 perl tests/vg_regtest --valgrind=${VALGRIND_BIN} --valgrind-lib=${VALGRIND_LIBEXECDIR} --yocto-ptest $i 2>&1|tee ${LOG}
38 mv $i.vgtest $i.IGNORE
39done
40
41
35cd ${VALGRIND_LIB}/ptest && ./tests/vg_regtest \ 42cd ${VALGRIND_LIB}/ptest && ./tests/vg_regtest \
36 --valgrind=${VALGRIND_BIN} \ 43 --valgrind=${VALGRIND_BIN} \
37 --valgrind-lib=${VALGRIND_LIBEXECDIR} \ 44 --valgrind-lib=${VALGRIND_LIBEXECDIR} \
@@ -56,6 +63,11 @@ for i in `cat remove-for-all`; do
56 mv $i.IGNORE $i.vgtest; 63 mv $i.IGNORE $i.vgtest;
57done 64done
58 65
66echo "Restore flaky and other non-deterministic tests"
67for i in `cat taskset_nondeterministic_tests`; do
68 mv $i.IGNORE $i.vgtest;
69done
70
59echo "Failed test details..." 71echo "Failed test details..."
60failed_tests=`grep FAIL: ${LOG} | awk '{print $2}'` 72failed_tests=`grep FAIL: ${LOG} | awk '{print $2}'`
61for test in $failed_tests; do 73for test in $failed_tests; do
diff --git a/meta/recipes-devtools/valgrind/valgrind/taskset_nondeterministic_tests b/meta/recipes-devtools/valgrind/valgrind/taskset_nondeterministic_tests
new file mode 100644
index 0000000000..cf073fa927
--- /dev/null
+++ b/meta/recipes-devtools/valgrind/valgrind/taskset_nondeterministic_tests
@@ -0,0 +1,4 @@
1helgrind/tests/hg05_race2
2helgrind/tests/tc09_bad_unlock
3drd/tests/bar_bad
4drd/tests/bar_bad_xml