LTP in valgrind :)
Hi all,
in case you haven't read the news in LWN, LTP is now part of valgrind [1]:
The Linux Test Project (ltp) is integrated in the testsuite try 'make ltpchecks' (this will take a while and will point out various missing syscalls and valgrind crashes!)
FYI script they use [2]. They consider some test long running [3].
Kind regards,
Petr
[1] https://www.mail-archive.com/[email protected]/msg07691.html
[2] https://sourceware.org/git/?p=valgrind.git;a=blob;f=auxprogs/ltp-tester.sh;h=000cfaa7f336f8641f464d107a363af24f607614;hb=HEAD
[3] https://sourceware.org/git/?p=valgrind.git;a=blob;f=auxprogs/ltp-error-patterns.txt;h=ee5048bf42b652258fb469a632fb2d16afb72c38;hb=HEAD
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Petr,
what a nice surprise :-) I guess polishing and fixing LTP somehow helped.
Kind regards,
Andrea Cervesato
Quoted 17 lines · Petr Vorel
Hi all,
in case you haven't read the news in LWN, LTP is now part of valgrind [1]:
The Linux Test Project (ltp) is integrated in the testsuite try 'make ltpchecks' (this will take a while and will point out various missing syscalls and valgrind crashes!)
FYI script they use [2]. They consider some test long running [3].
Kind regards,
Petr[1] https://www.mail-archive.com/[email protected]/msg07691.html
[2] https://sourceware.org/git/?p=valgrind.git;a=blob;f=auxprogs/ltp-tester.sh;h=000cfaa7f336f8641f464d107a363af24f607614;hb=HEAD
[3] https://sourceware.org/git/?p=valgrind.git;a=blob;f=auxprogs/ltp-error-patterns.txt;h=ee5048bf42b652258fb469a632fb2d16afb72c38;hb=HEAD
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Awesome :-)
It also opens some interesting questions, i.e. how do we make comparing results from two different tests easier. Currently they grep the test results for a summary, but maybe we can do better.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 6 lines · Cyril Hrubis
Hi!
Awesome :-)It also opens some interesting questions, i.e. how do we make comparing results from two different tests easier. Currently they grep the test results for a summary, but maybe we can do better.
One option is to extract all TPASS/TFAIL/TWARN/TBROK/TCONF messages, discard any message contents past the file:line header and then compare whether the sanitized output is identical. That'll take care of random values in the output while ensuring that the test went through the same code paths as before. We could provide a sanitizer script for that.
Signature · 10 lines
Martin Doucha [email protected] SW Quality Engineer SUSE LINUX, s.r.o. CORSO IIa Krizikova 148/34 186 00 Prague 8 Czech Republic -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 9 lines
Quoted 3 lines
It also opens some interesting questions, i.e. how do we make comparing results from two different tests easier. Currently they grep the test results for a summary, but maybe we can do better.
One option is to extract all TPASS/TFAIL/TWARN/TBROK/TCONF messages, discard any message contents past the file:line header and then compare whether the sanitized output is identical. That'll take care of random values in the output while ensuring that the test went through the same code paths as before. We could provide a sanitizer script for that.
Maybe we can even add an option to the test library to supress the messages in output, that would be fairly simple.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 5 lines
Hi!
Quoted 3 lines
Quoted 3 lines
It also opens some interesting questions, i.e. how do we make comparing results from two different tests easier. Currently they grep the test results for a summary, but maybe we can do better.
Quoted 8 lines
Quoted 5 lines
One option is to extract all TPASS/TFAIL/TWARN/TBROK/TCONF messages, discard any message contents past the file:line header and then compare whether the sanitized output is identical. That'll take care of random values in the output while ensuring that the test went through the same code paths as before. We could provide a sanitizer script for that.
Maybe we can even add an option to the test library to supress the messages in output, that would be fairly simple.
+1
And Cc the authors :).
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi all,
[ Cc Martin and Mark from Valgrind project ]
Quoted 5 lines
Hi!
Quoted 3 lines
Quoted 3 lines
It also opens some interesting questions, i.e. how do we make comparing results from two different tests easier. Currently they grep the test results for a summary, but maybe we can do better.
Quoted 8 lines
Quoted 5 lines
One option is to extract all TPASS/TFAIL/TWARN/TBROK/TCONF messages, discard any message contents past the file:line header and then compare whether the sanitized output is identical. That'll take care of random values in the output while ensuring that the test went through the same code paths as before. We could provide a sanitizer script for that.
Maybe we can even add an option to the test library to supress the messages in output, that would be fairly simple.
@Martin @Mark: feel free to comment what we can do for you :). Whole thread:
https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#t
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hello guys,
Quoted 15 lines · Petr Vorel
Hi all,
[ Cc Martin and Mark from Valgrind project ]
Quoted 10 lines
Hi!
Quoted 9 lines
Quoted 3 lines
It also opens some interesting questions, i.e. how do we make comparing results from two different tests easier. Currently they grep the test results for a summary, but maybe we can do better.
One option is to extract all TPASS/TFAIL/TWARN/TBROK/TCONF messages, discard any message contents past the file:line header and then compare whether the sanitized output is identical. That'll take care of random values in the output while ensuring that the test went through the same code paths as before. We could provide a sanitizer script for that.
Quoted 6 lines
Quoted 2 lines
Maybe we can even add an option to the test library to supress the messages in output, that would be fairly simple.
@Martin @Mark: feel free to comment what we can do for you :). Whole thread:
https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#t
Nice to see you noticed! Primarily we're trying to identify valgrind gaps and bugs. For our use-case it's good to have reproducible test runs in a sense of identical test results for repeated test runs. We got pretty close to this by only taking into account the testcase summaries at the very end of the test logs. Any steps to get closer to reproducible test runs on the LTP side are interesting for us, and we'll try to reflect those!
Cheers,
Martin
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 17 lines
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
It also opens some interesting questions, i.e. how do we make comparing results from two different tests easier. Currently they grep the test results for a summary, but maybe we can do better.
One option is to extract all TPASS/TFAIL/TWARN/TBROK/TCONF messages, discard any message contents past the file:line header and then compare whether the sanitized output is identical. That'll take care of random values in the output while ensuring that the test went through the same code paths as before. We could provide a sanitizer script for that.
Quoted 6 lines
Quoted 2 lines
Maybe we can even add an option to the test library to supress the messages in output, that would be fairly simple.
@Martin @Mark: feel free to comment what we can do for you :). Whole thread:
https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#t
Quoted 7 lines
Nice to see you noticed! Primarily we're trying to identify valgrind gaps and bugs. For our use-case it's good to have reproducible test runs in a sense of identical test results for repeated test runs. We got pretty close to this by only taking into account the testcase summaries at the very end of the test logs. Any steps to get closer to reproducible test runs on the LTP side are interesting for us, and we'll try to reflect those!
I've proposed a simple solution that would cut the message from tests only to contain the line numbers and results in:
If that works for you we can get this in LTP before the next release.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Cyril, Hi ltp hackers,
Quoted 16 lines · Cyril Hrubis
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Maybe we can even add an option to the test library to supress the messages in output, that would be fairly simple.
@Martin @Mark: feel free to comment what we can do for you :). Whole thread:
https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#tQuoted 7 lines
Nice to see you noticed! Primarily we're trying to identify valgrind gaps and bugs. For our use-case it's good to have reproducible test runs in a sense of identical test results for repeated test runs. We got pretty close to this by only taking into account the testcase summaries at the very end of the test logs. Any steps to get closer to reproducible test runs on the LTP side are interesting for us, and we'll try to reflect those!
Quoted 7 lines
I've proposed a simple solution that would cut the message from tests only to contain the line numbers and results in:
If that works for you we can get this in LTP before the next release.
That looks pretty interesting, I just tried it and it turns:
$ ./testcases/kernel/syscalls/dup/dup01 tst_tmpdir.c:316: TINFO: Using /tmp/LTP_dupbmNNGn as tmpdir (tmpfs filesystem) tst_test.c:1907: TINFO: LTP version: VALGRIND_3_25_0-10-g0ce068434ec3 tst_test.c:1911: TINFO: Tested kernel: 6.13.11-200.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 10 19:02:09 UTC 2025 x86_64 tst_kconfig.c:88: TINFO: Parsing kernel config '/lib/modules/6.13.11-200.fc41.x86_64/build/.config' tst_test.c:1729: TINFO: Overall timeout per run is 0h 00m 30s dup01.c:24: TPASS: dup(fd) returned fd 4 dup01.c:27: TPASS: buf1.st_ino == buf2.st_ino (6921515)
Summary:
passed 2 failed 0 broken 0 skipped 0
warnings 0
into:
$ LTP_REPRODUCIBLE_OUTPUT=1 ./testcases/kernel/syscalls/dup/dup01 tst_tmpdir.c:316: tst_test.c:1907: tst_test.c:1911: tst_kconfig.c:88: tst_test.c:1729: dup01.c:24: dup01.c:27:
Summary:
passed 2 failed 0 broken 0 skipped 0
warnings 0
So that still shows the "code flow" and the test results Summary (all output goes to stderr). I think this would indeed be useful to compare if a testcase runs the same "native" and under valgrind.
Thanks,
Mark
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 34 lines
That looks pretty interesting, I just tried it and it turns:
$ ./testcases/kernel/syscalls/dup/dup01 tst_tmpdir.c:316: TINFO: Using /tmp/LTP_dupbmNNGn as tmpdir (tmpfs filesystem) tst_test.c:1907: TINFO: LTP version: VALGRIND_3_25_0-10-g0ce068434ec3 tst_test.c:1911: TINFO: Tested kernel: 6.13.11-200.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 10 19:02:09 UTC 2025 x86_64 tst_kconfig.c:88: TINFO: Parsing kernel config '/lib/modules/6.13.11-200.fc41.x86_64/build/.config' tst_test.c:1729: TINFO: Overall timeout per run is 0h 00m 30s dup01.c:24: TPASS: dup(fd) returned fd 4 dup01.c:27: TPASS: buf1.st_ino == buf2.st_ino (6921515)Summary:
passed 2 failed 0 broken 0 skipped 0warnings 0
into:
$ LTP_REPRODUCIBLE_OUTPUT=1 ./testcases/kernel/syscalls/dup/dup01 tst_tmpdir.c:316: tst_test.c:1907: tst_test.c:1911: tst_kconfig.c:88: tst_test.c:1729: dup01.c:24: dup01.c:27:Summary:
passed 2 failed 0 broken 0 skipped 0
Quoted 5 lines
warnings 0
So that still shows the "code flow" and the test results Summary (all output goes to stderr). I think this would indeed be useful to compare if a testcase runs the same "native" and under valgrind.
I will send a proper LTP patch.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 51 lines
-----Original Message-----
From: ltp <[email protected]> On Behalf Of Mark Wielaard Hi Cyril, Hi ltp hackers,Quoted 16 lines · Cyril Hrubis
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Maybe we can even add an option to the test library to supress the messages in output, that would be fairly simple.
@Martin @Mark: feel free to comment what we can do for you :). Whole thread:
https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#tQuoted 7 lines
Nice to see you noticed! Primarily we're trying to identify valgrind gaps and bugs. For our use-case it's good to have reproducible test runs in a sense of identical test results for repeated test runs. We got pretty close to this by only taking into account the testcase summaries at the very end of the test logs. Any steps to get closer to reproducible test runs on the LTP side are interesting for us, and we'll try to reflect those!
Quoted 7 lines
I've proposed a simple solution that would cut the message from tests only to contain the line numbers and results in:
If that works for you we can get this in LTP before the next release.
That looks pretty interesting, I just tried it and it turns:
$ ./testcases/kernel/syscalls/dup/dup01 tst_tmpdir.c:316: TINFO: Using /tmp/LTP_dupbmNNGn as tmpdir (tmpfs filesystem) tst_test.c:1907: TINFO: LTP version: VALGRIND_3_25_0-10-g0ce068434ec3 tst_test.c:1911: TINFO: Tested kernel: 6.13.11-200.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 10 19:02:09 UTC 2025 x86_64 tst_kconfig.c:88: TINFO: Parsing kernel config '/lib/modules/6.13.11-200.fc41.x86_64/build/.config' tst_test.c:1729: TINFO: Overall timeout per run is 0h 00m 30s dup01.c:24: TPASS: dup(fd) returned fd 4 dup01.c:27: TPASS: buf1.st_ino == buf2.st_ino (6921515)Summary:
passed 2 failed 0 broken 0 skipped 0warnings 0
into:
Quoted 8 lines
$ LTP_REPRODUCIBLE_OUTPUT=1 ./testcases/kernel/syscalls/dup/dup01 tst_tmpdir.c:316: tst_test.c:1907: tst_test.c:1911: tst_kconfig.c:88: tst_test.c:1729: dup01.c:24: dup01.c:27:
Don't you want to keep the status or result (TINFO and TPASS)?
This shouldn't change from run-to-run unless the test is flaky.
-- Tim
Quoted 17 lines
Summary:
passed 2 failed 0 broken 0 skipped 0warnings 0
So that still shows the "code flow" and the test results Summary (all output goes to stderr). I think this would indeed be useful to compare if a testcase runs the same "native" and under valgrind.
Thanks,
Mark
--
Mailing list info: https://lists.linux.it/listinfo/ltp
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 3 lines
Don't you want to keep the status or result (TINFO and TPASS)?
This shouldn't change from run-to-run unless the test is flaky.
Already addressed in v2 patch:
http://patchwork.ozlabs.org/project/ltp/patch/[email protected]/
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Petr, Hi ltp hackers,
Quoted 10 lines · Petr Vorel
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
It also opens some interesting questions, i.e. how do we make comparing results from two different tests easier. Currently they grep the test results for a summary, but maybe we can do better.
One option is to extract all TPASS/TFAIL/TWARN/TBROK/TCONF messages, discard any message contents past the file:line header and then compare whether the sanitized output is identical. That'll take care of random values in the output while ensuring that the test went through the same code paths as before. We could provide a sanitizer script for that.
Quoted 6 lines
Quoted 2 lines
Maybe we can even add an option to the test library to supress the messages in output, that would be fairly simple.
@Martin @Mark: feel free to comment what we can do for you :). Whole thread:
https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#t
Thanks. Something small, but if you could CC [email protected] on new ltp releases that would be great.
I saw a question at the top of that thread about "They consider some test long running". In general that is not about the actual ltp tests (we currently only run the testcases/kernel/syscalls ones), but because valgrind:
a) is basically a giant Just In Time compiler, which instruments all code, and might slow down your code 20x (yeah, we know).
b) serializes all threads, so only one thread runs at a time, all parallel code is executed as if it was serial code.
c) memcheck keeps track of all memory to see if values are "valid" (have a known value written to them). This cost about 25% more memory than the program itself uses (and might actually double the amount).
So for now we have the following exclude file:
https://sourceware.org/cgit/valgrind/tree/auxprogs/ltp-excludes.txt
bind06
epoll-ltp
fork14futex_cmp_requeue01futex_cmp_requeue02
inotify09
msgstress01pidfd_send_signal01pidfd_send_signal02pidfd_send_signal03
sendmsg03
setsockopt06
setsockopt07
signal05
signal06timerfd_settime02
We haven't yet analyzed them to see what exactly makes them so slow (under valgrind). It was more important to get the ltpchecks run on all our CI systems:
https://builder.sourceware.org/buildbot/#/builders?tags=valgrind
Some of those systems only have 8GB of memory or are fairly slow (specifically the arm32 and riscv builders).
The pidfd_send_signal tests are excluded because they seemed to kill the container they were running inside. Again, this could be a valgrind bug. valgrind doesn't implement a proper syscall wrapper for pidfd_send_signal so it might be possible we are just killing a random process...
Cheers,
Mark
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 6 lines
Hi!
Awesome :-)It also opens some interesting questions, i.e. how do we make comparing results from two different tests easier. Currently they grep the test results for a summary, but maybe we can do better.
Kirk and compare json?
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 5 lines
Quoted 3 lines
It also opens some interesting questions, i.e. how do we make comparing results from two different tests easier. Currently they grep the test results for a summary, but maybe we can do better.
Kirk and compare json?
That does sound like an overkill for something that is basically:
lib/tst_test.c +7 -0
@@ -64,6 +64,7 @@ static int mntpoint_mounted;static int ovl_mounted;static struct timespec tst_start_time;/* valid only for test pid */ static int tdebug; +static int reproducible_output;struct results {int passed;
@@ -299,6 +300,9 @@ static void print_result(const char *file, const int lineno, int ttype, str += ret; size -= ret; +if (reproducible_output) + goto print; +if (tst_color_enabled (STDERR_FILENO )) ret =snprintf (str, size,"%s%s: %s" ,tst_ttype2color (ttype), res,ANSI_COLOR_RESET );
@@ -324,6 +328,7 @@ static void print_result(const char *file, const int lineno, int ttype, "Next message is too long and truncated:"); } +print:snprintf (str, size,"\n" );/* we might be called from signal handler, so use write() */
@@ -1273,6 +1278,8 @@ static void do_setup(int argc, char *argv[])if (tst_test->tconf_msg)tst_brk (TCONF ,"%s" , tst_test->tconf_msg); + reproducible_output = !!getenv ("LTP_REPRODUCIBLE_OUTPUT" ); +assert_test_fn ();TCID = tid =get_tid (argv);
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Cyril,
Quoted 8 lines
Hi!
Quoted 5 lines
Quoted 3 lines
It also opens some interesting questions, i.e. how do we make comparing results from two different tests easier. Currently they grep the test results for a summary, but maybe we can do better.
Kirk and compare json?
That does sound like an overkill for something that is basically:
Agree. I sometimes think JSON output would be useful to be even in core LTP (not requiring kirk), but that's something different than reproducible output.
The patch above LGTM (if you send patch please remember to update doc/users/setup_tests.rst).
Kind regards,
Petr
Quoted 39 lines
lib/tst_test.c +7
-0@@ -64,6 +64,7 @@ static int mntpoint_mounted;static int ovl_mounted;static struct timespec tst_start_time;/* valid only for test pid */ static int tdebug; +static int reproducible_output;struct results {int passed;@@ -299,6 +300,9 @@ static void print_result(const char *file, const int lineno, int ttype, str += ret; size -= ret; +if (reproducible_output) + goto print; +if (tst_color_enabled (STDERR_FILENO )) ret =snprintf (str, size,"%s%s: %s" ,tst_ttype2color (ttype), res,ANSI_COLOR_RESET );@@ -324,6 +328,7 @@ static void print_result(const char *file, const int lineno, int ttype, "Next message is too long and truncated:"); } +print:snprintf (str, size,"\n" );/* we might be called from signal handler, so use write() */ @@ -1273,6 +1278,8 @@ static void do_setup(int argc, char *argv[])if (tst_test->tconf_msg)tst_brk (TCONF ,"%s" , tst_test->tconf_msg); + reproducible_output = !!getenv ("LTP_REPRODUCIBLE_OUTPUT" ); +assert_test_fn ();TCID = tid =get_tid (argv);
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp