Skip to contentSkip to conversation

selftests/mm: Log run_vmtests.sh results in TAP format

PATCH Applied Ryan Roberts 3 participants 12 messages Dec 14, 2023 – Dec 19, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 selftests/mm: Log run_vmtests.sh results in TAP format

When running tests on a CI system (e.g. LAVA) it is useful to output test results in TAP format so that the CI can parse the fine-grained results to show regressions. Many of the mm selftest binaries already output using the TAP format. And the kselftests runner (run_kselftest.sh) also uses the format. CI systems such as LAVA can already handle nested TAP reports. However, with the mm selftests we have 3 levels of nesting (run_kselftest.sh -> run_vmtests.sh -> individual test binaries) and the middle level did not previously support TAP, which breaks the parser.

Let's fix that by teaching run_vmtests.sh to output using the TAP format. Ideally this would be opt-in via a command line argument to avoid the possibility of breaking anyone's existing scripts that might scrape the output. However, it is not possible to pass arguments to tests invoked via run_kselftest.sh. So I've implemented an opt-out option (-n), which will revert to the existing output format.

Future changes to this file should be aware of 2 new conventions:

- output that is part of the TAP reporting is piped through tap_output
- general output is piped through tap_prefix

---
 tools/testing/selftests/mm/run_vmtests.sh | 51 +++++++++++++++++------
 1 file changed, 39 insertions(+), 12 deletions(-)
tools/testing/selftests/mm/run_vmtests.sh +39 -12
index 87f513f5cf91..246d53a5d7f2 100755
@@ -5,6 +5,7 @@
 # Kselftest framework requirement - SKIP code is 4.
 ksft_skip=4
 
+count_total=0
 count_pass=0
 count_fail=0
 count_skip=0
@@ -17,6 +18,7 @@ usage: ${BASH_SOURCE[0]:-$0} [ options ]
   -a: run all tests, including extra ones
   -t: specify specific categories to tests to run
   -h: display this message
+  -n: disable TAP output
 
 The default behavior is to run required tests only.  If -a is specified,
 will run all tests.
@@ -77,12 +79,14 @@ EOF
 }
 
 RUN_ALL=false
+TAP_PREFIX="# "
 
-while getopts "aht:" OPT; do
+while getopts "aht:n" OPT; do
 	case ${OPT} in
 		"a") RUN_ALL=true ;;
 		"h") usage ;;
 		"t") VM_SELFTEST_ITEMS=${OPTARG} ;;
+		"n") TAP_PREFIX= ;;
 	esac
 done
 shift $((OPTIND -1))
@@ -184,30 +188,52 @@ fi
 VADDR64=0
 echo "$ARCH64STR" | grep "$ARCH" &>/dev/null && VADDR64=1
 
+tap_prefix() {
+	sed -e "s/^/${TAP_PREFIX}/"
+}
+
+tap_output() {
+	if [[ ! -z "$TAP_PREFIX" ]]; then
+		read str
+		echo $str
+	fi
+}
+
+pretty_name() {
+	echo "$*" | sed -e 's/^\(bash \)\?\.\///'
+}
+
 # Usage: run_test [test binary] [arbitrary test arguments...]
 run_test() {
 	if test_selected ${CATEGORY}; then
+		local test=$(pretty_name "$*")
 		local title="running $*"
 		local sep=$(echo -n "$title" | tr "[:graph:][:space:]" -)
-		printf "%s\n%s\n%s\n" "$sep" "$title" "$sep"
+		printf "%s\n%s\n%s\n" "$sep" "$title" "$sep" | tap_prefix
 
-		"$@"
-		local ret=$?
+		("$@" 2>&1) | tap_prefix
+		local ret=${PIPESTATUS[0]}
+		count_total=$(( count_total + 1 ))
 		if [ $ret -eq 0 ]; then
 			count_pass=$(( count_pass + 1 ))
-			echo "[PASS]"
+			echo "[PASS]" | tap_prefix
+			echo "ok ${count_total} ${test}" | tap_output
 		elif [ $ret -eq $ksft_skip ]; then
 			count_skip=$(( count_skip + 1 ))
-			echo "[SKIP]"
+			echo "[SKIP]" | tap_prefix
+			echo "ok ${count_total} ${test} # SKIP" | tap_output
 			exitcode=$ksft_skip
 		else
 			count_fail=$(( count_fail + 1 ))
-			echo "[FAIL]"
+			echo "[FAIL]" | tap_prefix
+			echo "not ok ${count_total} ${test} # exit=$ret" | tap_output
 			exitcode=1
 		fi
 	fi # test_selected
 }
 
+echo "TAP version 13" | tap_output
+
 CATEGORY="hugetlb" run_test ./hugepage-mmap
 
 shmmax=$(cat /proc/sys/kernel/shmmax)
@@ -231,9 +257,9 @@ CATEGORY="hugetlb" run_test ./hugetlb_fault_after_madv
 echo "$nr_hugepages_tmp" > /proc/sys/vm/nr_hugepages
 
 if test_selected "hugetlb"; then
-	echo "NOTE: These hugetlb tests provide minimal coverage.  Use"
-	echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for"
-	echo "      hugetlb regression testing."
+	echo "NOTE: These hugetlb tests provide minimal coverage.  Use"	  | tap_prefix
+	echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for" | tap_prefix
+	echo "      hugetlb regression testing."			  | tap_prefix
 fi
 
 CATEGORY="mmap" run_test ./map_fixed_noreplace
@@ -312,7 +338,7 @@ CATEGORY="hmm" run_test bash ./test_hmm.sh smoke
 # MADV_POPULATE_READ and MADV_POPULATE_WRITE tests
 CATEGORY="madv_populate" run_test ./madv_populate
 
-echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
+(echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope 2>&1) | tap_prefix
 CATEGORY="memfd_secret" run_test ./memfd_secret
 
 # KSM KSM_MERGE_TIME_HUGE_PAGES test with size of 100
@@ -369,6 +395,7 @@ CATEGORY="mkdirty" run_test ./mkdirty
 
 CATEGORY="mdwe" run_test ./mdwe_test
 
-echo "SUMMARY: PASS=${count_pass} SKIP=${count_skip} FAIL=${count_fail}"
+echo "SUMMARY: PASS=${count_pass} SKIP=${count_skip} FAIL=${count_fail}" | tap_prefix
+echo "1..${count_total}" | tap_output
 
 exit $exitcode

--
2.25.1

Reviewed-by on v1 1/1
#1 lore raw
Quoted 9 lines · Ryan Roberts

When running tests on a CI system (e.g. LAVA) it is useful to output test results in TAP format so that the CI can parse the fine-grained results to show regressions. Many of the mm selftest binaries already output using the TAP format. And the kselftests runner (run_kselftest.sh) also uses the format. CI systems such as LAVA can already handle nested TAP reports. However, with the mm selftests we have 3 levels of nesting (run_kselftest.sh -> run_vmtests.sh -> individual test binaries) and the middle level did not previously support TAP, which breaks the parser.

Quoted 6 lines

Let's fix that by teaching run_vmtests.sh to output using the TAP format. Ideally this would be opt-in via a command line argument to avoid the possibility of breaking anyone's existing scripts that might scrape the output. However, it is not possible to pass arguments to tests invoked via run_kselftest.sh. So I've implemented an opt-out option (-n), which will revert to the existing output format.

What I did for ftrace which had a similar situation was make a wrapper script which invokes the test runner, make the test runner a TEST_PROGS_EXTENDED so it's not run by the kselftest infrastructure automatically and make the wrapper a normal TEST_PROGS. Neither option is especially lovely.

#2 lore raw
Quoted 11 lines · Mark Brown
Quoted 9 lines · Ryan Roberts

When running tests on a CI system (e.g. LAVA) it is useful to output test results in TAP format so that the CI can parse the fine-grained results to show regressions. Many of the mm selftest binaries already output using the TAP format. And the kselftests runner (run_kselftest.sh) also uses the format. CI systems such as LAVA can already handle nested TAP reports. However, with the mm selftests we have 3 levels of nesting (run_kselftest.sh -> run_vmtests.sh -> individual test binaries) and the middle level did not previously support TAP, which breaks the parser.

Quoted 1 line

Thanks!

Quoted 8 lines
Quoted 7 lines

Let's fix that by teaching run_vmtests.sh to output using the TAP format. Ideally this would be opt-in via a command line argument to avoid the possibility of breaking anyone's existing scripts that might scrape the output. However, it is not possible to pass arguments to tests invoked via run_kselftest.sh. So I've implemented an opt-out option (-n), which will revert to the existing output format.

Quoted 5 lines

What I did for ftrace which had a similar situation was make a wrapper script which invokes the test runner, make the test runner a TEST_PROGS_EXTENDED so it's not run by the kselftest infrastructure automatically and make the wrapper a normal TEST_PROGS. Neither option is especially lovely.

Yeah that's a good idea... I'll wait and see if anyone shouts that this has broken something. If nothing is broken, I think it is better to just make TAP the default rather than adding yet another wrapper.

#3 lore raw
Quoted 11 lines · Ryan Roberts
Quoted 5 lines · Mark Brown

What I did for ftrace which had a similar situation was make a wrapper script which invokes the test runner, make the test runner a TEST_PROGS_EXTENDED so it's not run by the kselftest infrastructure automatically and make the wrapper a normal TEST_PROGS. Neither option is especially lovely.

Yeah that's a good idea... I'll wait and see if anyone shouts that this has broken something. If nothing is broken, I think it is better to just make TAP the default rather than adding yet another wrapper.

I think it depends a bit how ergonomic the non-TAP output is for interactive use - TAP isn't amazing for humans so if there's something that's nicer it probably makes sense to keep that as the default. For these tests I'm not sure it's particularly an issue.

#4 lore raw
Quoted 9 lines · Mark Brown
Quoted 7 lines · Ryan Roberts
Quoted 5 lines · Mark Brown

What I did for ftrace which had a similar situation was make a wrapper script which invokes the test runner, make the test runner a TEST_PROGS_EXTENDED so it's not run by the kselftest infrastructure automatically and make the wrapper a normal TEST_PROGS. Neither option is especially lovely.

Quoted 8 lines
Quoted 3 lines

Yeah that's a good idea... I'll wait and see if anyone shouts that this has broken something. If nothing is broken, I think it is better to just make TAP the default rather than adding yet another wrapper.

I think it depends a bit how ergonomic the non-TAP output is for interactive use - TAP isn't amazing for humans so if there's something that's nicer it probably makes sense to keep that as the default. For these tests I'm not sure it's particularly an issue.

I've kept all the existing "pretty" output and results summary as is, it just gets a hash in front of it when TAP is enabled.

so this:

-----------------------
running ./hugepage-mmap
-----------------------
Returned address is 0xffff89e00000
First hex is 0
First hex is 3020100
[PASS]
SUMMARY: PASS=1 SKIP=0 FAIL=0

becomes this:

TAP version 13
# -----------------------

# running ./hugepage-mmap
# -----------------------
# Returned address is 0xffff89e00000
# First hex is 0
# First hex is 3020100
# [PASS]
ok 1 hugepage-mmap
# SUMMARY: PASS=1 SKIP=0 FAIL=0
1..1

If you think the latter is ofensive, then I can do the wrapping as you suggest.

#5 lore raw
Quoted 13 lines · Ryan Roberts

On 15/12/2023 14:08, Mark Brown wrote:

# -----------------------

# running ./hugepage-mmap
# -----------------------
# Returned address is 0xffff89e00000
# First hex is 0
# First hex is 3020100
# [PASS]
ok 1 hugepage-mmap
# SUMMARY: PASS=1 SKIP=0 FAIL=0
1..1
Quoted 1 line

If you think the latter is ofensive, then I can do the wrapping as you suggest.

I think it's fine here - it was more of an issue for ftrace.

Tested-by
#6 lore raw
In reply to Ryan Roberts

On 12/15/23 06:28, Ryan Roberts wrote:
...

Quoted 28 lines

I've kept all the existing "pretty" output and results summary as is, it just gets a hash in front of it when TAP is enabled.

so this:

-----------------------
running ./hugepage-mmap
-----------------------
Returned address is 0xffff89e00000
First hex is 0
First hex is 3020100
[PASS]
SUMMARY: PASS=1 SKIP=0 FAIL=0

becomes this:

TAP version 13
# -----------------------

# running ./hugepage-mmap
# -----------------------
# Returned address is 0xffff89e00000
# First hex is 0
# First hex is 3020100
# [PASS]
ok 1 hugepage-mmap
# SUMMARY: PASS=1 SKIP=0 FAIL=0
1..1
Quoted 1 line

If you think the latter is ofensive, then I can do the wrapping as you suggest.

I applied this and ran the tests, all while carefully reminding myself to "think like a human". :) And from that perspective, to me, the output is effectively the same: the leading '#' characters do not really change anything, from a readability point of view.

So IMHO you're on perfectly solid ground, if you just switch over directly to this format.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#7 lore raw
Quoted 37 lines · John Hubbard

On 12/15/23 06:28, Ryan Roberts wrote:
...

Quoted 28 lines

I've kept all the existing "pretty" output and results summary as is, it just gets a hash in front of it when TAP is enabled.

so this:

-----------------------
running ./hugepage-mmap
-----------------------
Returned address is 0xffff89e00000
First hex is 0
First hex is 3020100
[PASS]
SUMMARY: PASS=1 SKIP=0 FAIL=0

becomes this:

TAP version 13
# -----------------------

# running ./hugepage-mmap
# -----------------------
# Returned address is 0xffff89e00000
# First hex is 0
# First hex is 3020100
# [PASS]
ok 1 hugepage-mmap
# SUMMARY: PASS=1 SKIP=0 FAIL=0
1..1
Quoted 1 line

If you think the latter is ofensive, then I can do the wrapping as you suggest.

I applied this and ran the tests, all while carefully reminding myself to "think like a human". :) And from that perspective, to me, the output is effectively the same: the leading '#' characters do not really change anything, from a readability point of view.

Quoted 5 lines

So IMHO you're on perfectly solid ground, if you just switch over directly to this format.

I should also point out that some of the subtests already attempt a TAP output. So now we end up with TAP-within-TAP output for those programs.

For example:
# -----------------------
# running ./madv_populate
# -----------------------
# TAP version 13
# 1..21
# # [RUN] test_prot_read
# ok 1 MADV_POPULATE_READ with PROT_READ
# ok 2 MADV_POPULATE_WRITE with PROT_READ
# # [RUN] test_prot_write
# ok 3 MADV_POPULATE_READ with PROT_WRITE
...etc...

Note the double level of leading '#' characters.

Again, this is still readable enough for humans. But it should probably be removed in subsequent patches to the subtests.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#8 lore raw
Quoted 33 lines · John Hubbard
Quoted 32 lines · John Hubbard

On 12/15/23 06:28, Ryan Roberts wrote:
...

Quoted 29 lines

I've kept all the existing "pretty" output and results summary as is, it just gets a hash in front of it when TAP is enabled.

so this:

-----------------------
running ./hugepage-mmap
-----------------------
Returned address is 0xffff89e00000
First hex is 0
First hex is 3020100
[PASS]
SUMMARY: PASS=1 SKIP=0 FAIL=0

becomes this:

TAP version 13
# -----------------------

# running ./hugepage-mmap
# -----------------------
# Returned address is 0xffff89e00000
# First hex is 0
# First hex is 3020100
# [PASS]
ok 1 hugepage-mmap
# SUMMARY: PASS=1 SKIP=0 FAIL=0
1..1

If you think the latter is ofensive, then I can do the wrapping as you suggest.

Quoted 7 lines
Quoted 7 lines

I applied this and ran the tests, all while carefully reminding myself to "think like a human". :) And from that perspective, to me, the output is effectively the same: the leading '#' characters do not really change anything, from a readability point of view.

So IMHO you're on perfectly solid ground, if you just switch over directly to this format.

Great thanks for taking a look!

Quoted 6 lines
Quoted 3 lines

I should also point out that some of the subtests already attempt a TAP output. So now we end up with TAP-within-TAP output for those programs.

It's actually TAP-in-TAP-in-TAP if you're running from run_kselftest.sh :)

Quoted 14 lines

For example:
    # -----------------------

    # running ./madv_populate
    # -----------------------
    # TAP version 13
    # 1..21
    # # [RUN] test_prot_read
    # ok 1 MADV_POPULATE_READ with PROT_READ
    # ok 2 MADV_POPULATE_WRITE with PROT_READ
    # # [RUN] test_prot_write
    # ok 3 MADV_POPULATE_READ with PROT_WRITE
    ...etc...
Quoted 4 lines

Note the double level of leading '#' characters.

Again, this is still readable enough for humans. But it should probably be removed in subsequent patches to the subtests.

I personally don't agree with this. It would be difficult to flatten to a single TAP instance because the top level doesn't have a clue how many test cases the child is running. Trying to do this will make things more fragile and less modular. LAVA can certainly deal with nested test cases and correctly parses everything to test case names that contain the test name at each level of nesting. The thing I was trying to solve with this patch was that previously the top level (run_kselftest.sh) and the bottom level (individual mm test binaries) were using TAP, but the middle level (run_vmtests.sh) wasn't, and this was confusing the LAVA parser.

Quoted 3 lines

thanks,

#9 lore raw

On 12/18/23 03:32, Ryan Roberts wrote:
...

Quoted 19 lines
Quoted 2 lines

I should also point out that some of the subtests already attempt a TAP output. So now we end up with TAP-within-TAP output for those programs.

It's actually TAP-in-TAP-in-TAP if you're running from run_kselftest.sh :)

Quoted 14 lines

For example:
    # -----------------------

     # running ./madv_populate
     # -----------------------
     # TAP version 13
     # 1..21
     # # [RUN] test_prot_read
     # ok 1 MADV_POPULATE_READ with PROT_READ
     # ok 2 MADV_POPULATE_WRITE with PROT_READ
     # # [RUN] test_prot_write
     # ok 3 MADV_POPULATE_READ with PROT_WRITE
     ...etc...
Quoted 7 lines
Quoted 4 lines

Note the double level of leading '#' characters.

Again, this is still readable enough for humans. But it should probably be removed in subsequent patches to the subtests.

I personally don't agree with this. It would be difficult to flatten to a single TAP instance because the top level doesn't have a clue how many test cases the

That's not quite what I had in mind...

Quoted 8 lines

child is running. Trying to do this will make things more fragile and less modular. LAVA can certainly deal with nested test cases and correctly parses everything to test case names that contain the test name at each level of nesting. The thing I was trying to solve with this patch was that previously the top level (run_kselftest.sh) and the bottom level (individual mm test binaries) were using TAP, but the middle level (run_vmtests.sh) wasn't, and this was confusing the LAVA parser.

I was thinking more along these lines:

a) For the individual programs (binaries), there is actually neither need nor desire to create TAP output at that level, because frameworks like LAVA only care about running a lot of tests and parsing the output.

b) Therefore, just stop specifying TAP output at the leaf level, and let run_vmtests.sh and run_kselftest.sh do it.

Looking at madv_populate.c, I see that it scatters calls to ksft_*() around. And I was thinking that this is all just redundant, isn't it?

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#10 lore raw
Quoted 46 lines · John Hubbard

On 12/18/23 03:32, Ryan Roberts wrote:
...

Quoted 19 lines
Quoted 2 lines

I should also point out that some of the subtests already attempt a TAP output. So now we end up with TAP-within-TAP output for those programs.

It's actually TAP-in-TAP-in-TAP if you're running from run_kselftest.sh :)

Quoted 14 lines

For example:
     # -----------------------

     # running ./madv_populate
     # -----------------------
     # TAP version 13
     # 1..21
     # # [RUN] test_prot_read
     # ok 1 MADV_POPULATE_READ with PROT_READ
     # ok 2 MADV_POPULATE_WRITE with PROT_READ
     # # [RUN] test_prot_write
     # ok 3 MADV_POPULATE_READ with PROT_WRITE
     ...etc...
Quoted 7 lines
Quoted 4 lines

Note the double level of leading '#' characters.

Again, this is still readable enough for humans. But it should probably be removed in subsequent patches to the subtests.

I personally don't agree with this. It would be difficult to flatten to a single TAP instance because the top level doesn't have a clue how many test cases the

That's not quite what I had in mind...

Quoted 8 lines

child is running. Trying to do this will make things more fragile and less modular. LAVA can certainly deal with nested test cases and correctly parses everything to test case names that contain the test name at each level of nesting. The thing I was trying to solve with this patch was that previously the top level (run_kselftest.sh) and the bottom level (individual mm test binaries) were using TAP, but the middle level (run_vmtests.sh) wasn't, and this was confusing the LAVA parser.

I was thinking more along these lines:

a) For the individual programs (binaries), there is actually neither need nor desire to create TAP output at that level, because frameworks like LAVA only care about running a lot of tests and parsing the output.

Quoted 6 lines

b) Therefore, just stop specifying TAP output at the leaf level, and let run_vmtests.sh and run_kselftest.sh do it.

Looking at madv_populate.c, I see that it scatters calls to ksft_*() around. And I was thinking that this is all just redundant, isn't it?

Although I suppose that the counter argument is that the subtests in madv_populate.c really *do* want to be specifically printed in TAP format.

arggh, I guess this is just not worth fooling around with after all.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#11 lore raw
Quoted 54 lines · John Hubbard
Quoted 46 lines · John Hubbard

On 12/18/23 03:32, Ryan Roberts wrote:
...

Quoted 19 lines
Quoted 2 lines

I should also point out that some of the subtests already attempt a TAP output. So now we end up with TAP-within-TAP output for those programs.

It's actually TAP-in-TAP-in-TAP if you're running from run_kselftest.sh :)

Quoted 14 lines

For example:
     # -----------------------

     # running ./madv_populate
     # -----------------------
     # TAP version 13
     # 1..21
     # # [RUN] test_prot_read
     # ok 1 MADV_POPULATE_READ with PROT_READ
     # ok 2 MADV_POPULATE_WRITE with PROT_READ
     # # [RUN] test_prot_write
     # ok 3 MADV_POPULATE_READ with PROT_WRITE
     ...etc...
Quoted 7 lines
Quoted 4 lines

Note the double level of leading '#' characters.

Again, this is still readable enough for humans. But it should probably be removed in subsequent patches to the subtests.

I personally don't agree with this. It would be difficult to flatten to a single TAP instance because the top level doesn't have a clue how many test cases the

That's not quite what I had in mind...

Quoted 8 lines

child is running. Trying to do this will make things more fragile and less modular. LAVA can certainly deal with nested test cases and correctly parses everything to test case names that contain the test name at each level of nesting. The thing I was trying to solve with this patch was that previously the top level (run_kselftest.sh) and the bottom level (individual mm test binaries) were using TAP, but the middle level (run_vmtests.sh) wasn't, and this was confusing the LAVA parser.

I was thinking more along these lines:

a) For the individual programs (binaries), there is actually neither need nor desire to create TAP output at that level, because frameworks like LAVA only care about running a lot of tests and parsing the output.

Quoted 6 lines

b) Therefore, just stop specifying TAP output at the leaf level, and let run_vmtests.sh and run_kselftest.sh do it.

Looking at madv_populate.c, I see that it scatters calls to ksft_*() around. And I was thinking that this is all just redundant, isn't it?

Quoted 5 lines

Although I suppose that the counter argument is that the subtests in madv_populate.c really *do* want to be specifically printed in TAP format.

arggh, I guess this is just not worth fooling around with after all.

Yes; I wouldn't want to lose the fine granularity we have currently. For example cow.c has ~900 test cases now that I've multiplied everything up for mTHP. 16 of those are known to fail (hugetlb issue) and 1 is skipped. I wouldn't want to reduce that down to a single cow test case that always fails; that's not helpful to understand if I've regressed something.

But sounds like we are both on the same page now.

Quoted 3 lines

thanks,