Skip to contentSkip to conversation

lsmod01.sh: retry test couple times to lower false positives

PATCH New Jan Stancek 3 participants 11 messages Oct 18, 2019 – Oct 24, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives

Test sporadically fails with:
lsmod01 1 TFAIL: lsmod output different from /proc/modules. 36c36
< loop 42057 2
---

Quoted 1 line

loop 42057 1

commands runtest file runs mkswap01 before this test. That test is using loop device, and udev is presumably still holding a reference by the time lsmod01 test starts.

Repeat the test couple times to avoid racing with rest of the system.

---
 testcases/commands/lsmod/lsmod01.sh | 28 +++++++++++++++++++---------
 1 file changed, 19 insertions(+), 9 deletions(-)
testcases/commands/lsmod/lsmod01.sh +19 -9
index ad170dcd41b8..38ba8e0c94ad 100755
@@ -10,31 +10,41 @@ TST_NEEDS_TMPDIR=1
 TST_NEEDS_CMDS="lsmod"
 . tst_test.sh
 
-lsmod_test()
+lsmod_matches_proc_modules()
 {
 	lsmod_output=$(lsmod | awk '!/Module/{print $1, $2, $3}' | sort)
 	if [ -z "$lsmod_output" ]; then
-		tst_res TFAIL "Failed to parse the output from lsmod"
-		return
+		tst_brk TBROK "Failed to parse the output from lsmod"
 	fi
 
-	modules_output=$(awk '{print $1, $2, $3}' /proc/modules | sort)
+	modules_output=$(awk '{print $1, $2, $3} 1' /proc/modules | sort)
 	if [ -z "$modules_output" ]; then
-		tst_res TFAIL "Failed to parse /proc/modules"
-		return
+		tst_brk TBROK "Failed to parse /proc/modules"
 	fi
 
 	if [ "$lsmod_output" != "$modules_output" ]; then
-		tst_res TFAIL "lsmod output different from /proc/modules."
+		tst_res TINFO "lsmod output different from /proc/modules."
 
 		echo "$lsmod_output" > temp1
 		echo "$modules_output" > temp2
 		diff temp1 temp2
 
-		return
+		return 1
 	fi
+	return 0
+}
 
-	tst_res TPASS "'lsmod' passed."
+lsmod_test()
+{
+	for i in $(seq 1 5); do
+		if lsmod_matches_proc_modules; then
+			tst_res TPASS "'lsmod' passed."
+			return
+		fi
+		tst_res TINFO "Trying again"
+		sleep 1
+	done
+	tst_res TFAIL "'lsmod' doesn't match /proc/modules output"
 }
 
 tst_run
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives

Hi Jan,

Quoted 7 lines

Test sporadically fails with:
lsmod01 1 TFAIL: lsmod output different from /proc/modules. 36c36
< loop 42057 2
---

Quoted 1 line

loop 42057 1

Quoted 7 lines

commands runtest file runs mkswap01 before this test. That test is using loop device, and udev is presumably still holding a reference by the time lsmod01 test starts.

Repeat the test couple times to avoid racing with rest of the system.

...

Quoted 7 lines

+++ b/testcases/commands/lsmod/lsmod01.sh

testcases/commands/lsmod/lsmod01.sh +1 -1
@@ -10,31 +10,41 @@ TST_NEEDS_TMPDIR=1
 TST_NEEDS_CMDS="lsmod"
 . tst_test.sh
 
-lsmod_test()
+lsmod_matches_proc_modules()

testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 20

Quoted 10 lines
testcases/commands/lsmod/lsmod01.sh +2 -3
 {
 	lsmod_output=$(lsmod | awk '!/Module/{print $1, $2, $3}' | sort)
 	if [ -z "$lsmod_output" ]; then
-		tst_res TFAIL "Failed to parse the output from lsmod"
-		return
+		tst_brk TBROK "Failed to parse the output from lsmod"
 	fi
 
-	modules_output=$(awk '{print $1, $2, $3}' /proc/modules | sort)
+	modules_output=$(awk '{print $1, $2, $3} 1' /proc/modules | sort)

This is a regression. Please keep the old version (without 1).

- modules_output=$(awk '{print $1, $2, $3} 1' /proc/modules | sort)
+ modules_output=$(awk '{print $1, $2, $3}' /proc/modules | sort)

testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 26

Quoted 9 lines
testcases/commands/lsmod/lsmod01.sh +2 -3
 	if [ -z "$modules_output" ]; then
-		tst_res TFAIL "Failed to parse /proc/modules"
-		return
+		tst_brk TBROK "Failed to parse /proc/modules"
 	fi
 
 	if [ "$lsmod_output" != "$modules_output" ]; then
-		tst_res TFAIL "lsmod output different from /proc/modules."
+		tst_res TINFO "lsmod output different from /proc/modules."

nit: please remove dots at the end of messages, when you're touching that file.

The rest looks good to me.

Kind regards,
Petr

#2 lore raw

PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives

----- Original Message -----

Quoted 34 lines

Hi Jan,

Quoted 7 lines

Test sporadically fails with:
lsmod01 1 TFAIL: lsmod output different from /proc/modules. 36c36
< loop 42057 2
---

Quoted 1 line

loop 42057 1

Quoted 7 lines

commands runtest file runs mkswap01 before this test. That test is using loop device, and udev is presumably still holding a reference by the time lsmod01 test starts.

Repeat the test couple times to avoid racing with rest of the system.

...

Quoted 14 lines

+++ b/testcases/commands/lsmod/lsmod01.sh

testcases/commands/lsmod/lsmod01.sh +2 -3
@@ -10,31 +10,41 @@ TST_NEEDS_TMPDIR=1
 TST_NEEDS_CMDS="lsmod"
 . tst_test.sh
 
-lsmod_test()
+lsmod_matches_proc_modules()
 {
 	lsmod_output=$(lsmod | awk '!/Module/{print $1, $2, $3}' | sort)
 	if [ -z "$lsmod_output" ]; then
-		tst_res TFAIL "Failed to parse the output from lsmod"
-		return
+		tst_brk TBROK "Failed to parse the output from lsmod"
 	fi
Quoted 3 lines

testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 20

Quoted 2 lines
testcases/commands/lsmod/lsmod01.sh +1 -1
-	modules_output=$(awk '{print $1, $2, $3}' /proc/modules | sort)
+	modules_output=$(awk '{print $1, $2, $3} 1' /proc/modules | sort)

This is a regression. Please keep the old version (without 1).

My bad, I left it in by accident after testing.

Quoted 11 lines
(quoted diff) +1 -1
-	modules_output=$(awk '{print $1, $2, $3} 1' /proc/modules | sort)
+	modules_output=$(awk '{print $1, $2, $3}' /proc/modules | sort)
Quoted 6 lines
testcases/commands/lsmod/lsmod01.sh +1 -2
 	if [ -z "$modules_output" ]; then
-		tst_res TFAIL "Failed to parse /proc/modules"
-		return
+		tst_brk TBROK "Failed to parse /proc/modules"
 	fi
Quoted 5 lines

testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 26

Quoted 3 lines
testcases/commands/lsmod/lsmod01.sh +1 -1
 	if [ "$lsmod_output" != "$modules_output" ]; then
-		tst_res TFAIL "lsmod output different from /proc/modules."
+		tst_res TINFO "lsmod output different from /proc/modules."

nit: please remove dots at the end of messages, when you're touching that file.

will do

Quoted 6 lines

The rest looks good to me.

Kind regards,
Petr

#3 lore raw

PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives

In reply to Jan Stancek

Hi Jan,

...

testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 30

Quoted 7 lines
testcases/commands/lsmod/lsmod01.sh +1 -1
 	if [ "$lsmod_output" != "$modules_output" ]; then
-		tst_res TFAIL "lsmod output different from /proc/modules."
+		tst_res TINFO "lsmod output different from /proc/modules."
 
 		echo "$lsmod_output" > temp1
 		echo "$modules_output" > temp2
 		diff temp1 temp2

Also this code could be wrapped with
if tst_cmd_available diff; then

(I wouldn't add it to TST_NEEDS_CMDS)

Kind regards,
Petr

#4 lore raw

PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives

----- Original Message -----

Quoted 7 lines

Hi Jan,

...

Quoted 3 lines
testcases/commands/lsmod/lsmod01.sh +1 -1
 	if [ "$lsmod_output" != "$modules_output" ]; then
-		tst_res TFAIL "lsmod output different from /proc/modules."
+		tst_res TINFO "lsmod output different from /proc/modules."
Quoted 5 lines

testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 30

Quoted 3 lines
testcases/commands/lsmod/lsmod01.sh +0 -0
 		echo "$lsmod_output" > temp1
 		echo "$modules_output" > temp2
 		diff temp1 temp2

Also this code could be wrapped with
if tst_cmd_available diff; then

Added this as well and pushed.

#5 lore raw

PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives

In reply to Jan Stancek

Hi Jan,

...

Quoted 33 lines
testcases/commands/lsmod/lsmod01.sh +8 -9
-lsmod_test()
+lsmod_matches_proc_modules()
 {
 	lsmod_output=$(lsmod | awk '!/Module/{print $1, $2, $3}' | sort)
 	if [ -z "$lsmod_output" ]; then
-		tst_res TFAIL "Failed to parse the output from lsmod"
-		return
+		tst_brk TBROK "Failed to parse the output from lsmod"
 	fi
 
-	modules_output=$(awk '{print $1, $2, $3}' /proc/modules | sort)
+	modules_output=$(awk '{print $1, $2, $3} 1' /proc/modules | sort)
 	if [ -z "$modules_output" ]; then
-		tst_res TFAIL "Failed to parse /proc/modules"
-		return
+		tst_brk TBROK "Failed to parse /proc/modules"
 	fi
 
 	if [ "$lsmod_output" != "$modules_output" ]; then
-		tst_res TFAIL "lsmod output different from /proc/modules."
+		tst_res TINFO "lsmod output different from /proc/modules."
 
 		echo "$lsmod_output" > temp1
 		echo "$modules_output" > temp2
 		diff temp1 temp2
 
-		return
+		return 1
 	fi
+	return 0
+}
 
-	tst_res TPASS "'lsmod' passed."

testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 46

Quoted 10 lines
testcases/commands/lsmod/lsmod01.sh +10 -0
+lsmod_test()
+{
+	for i in $(seq 1 5); do
+		if lsmod_matches_proc_modules; then
+			tst_res TPASS "'lsmod' passed."
+			return
+		fi
+		tst_res TINFO "Trying again"
+		sleep 1
+	done

This is similar pattern to TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() (for both shell and C). I wonder if we also have use for TPASS/TFAIL instead of just TBROK and specifying number of tries instead of time to be setup.
C and shell usage is a bit different, so maybe TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() doesn't make much sense for shell (actually nothing uses them in shell) and I don't see much usage for my proposal in C.

testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 47

Quoted 1 line
testcases/commands/lsmod/lsmod01.sh +1 -0
+	tst_res TFAIL "'lsmod' doesn't match /proc/modules output"

+ we forget to use local (for lsmod_outputa and i), but that's not that important.

Kind regards,
Petr

#6 lore raw

PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives

----- Original Message -----

Quoted 16 lines

testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 46

Quoted 12 lines
testcases/commands/lsmod/lsmod01.sh +10 -1
-	tst_res TPASS "'lsmod' passed."
+lsmod_test()
+{
+	for i in $(seq 1 5); do
+		if lsmod_matches_proc_modules; then
+			tst_res TPASS "'lsmod' passed."
+			return
+		fi
+		tst_res TINFO "Trying again"
+		sleep 1
+	done

This is similar pattern to TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() (for both shell and C). I wonder if we also have use for TPASS/TFAIL instead of just TBROK and specifying number of tries instead of time to be setup.

I think TFAIL fits more here, it's outcome of what we are testing. TBROK in my mind is failure unrelated to subject of test.

But functionally TST_RETRY_FUNC should work too.

Quoted 2 lines

C and shell usage is a bit different, so maybe TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() doesn't make much sense for shell

I see it used in mkswap01.sh and numa01.sh.

I wonder if we need to TBROK in TST_RETRY_FUNC(). We could just return what the FUNC returns and let the test decide. TST_RETRY_FUNC_BRK() could be a wrapper that TBROKs on timeout.

#7 lore raw

PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives

Hi Jan,

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
testcases/commands/lsmod/lsmod01.sh +6 -1
-	tst_res TPASS "'lsmod' passed."
+lsmod_test()
+{
+	for i in $(seq 1 5); do
+		if lsmod_matches_proc_modules; then
+			tst_res TPASS "'lsmod' passed."
+			return
Quoted 8 lines

testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 46

Quoted 4 lines
testcases/commands/lsmod/lsmod01.sh +4 -0
+		fi
+		tst_res TINFO "Trying again"
+		sleep 1
+	done

This is similar pattern to TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() (for both shell and C). I wonder if we also have use for TPASS/TFAIL instead of just TBROK and specifying number of tries instead of time to be setup.

Quoted 2 lines

I think TFAIL fits more here, it's outcome of what we are testing. TBROK in my mind is failure unrelated to subject of test.

I express myself wrong. Sure, I meant to have TPASS/TFAIL,
just to use some helper function instead of writing the wrapper in the test.

Quoted 6 lines

But functionally TST_RETRY_FUNC should work too.

Quoted 2 lines

C and shell usage is a bit different, so maybe TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() doesn't make much sense for shell

I see it used in mkswap01.sh and numa01.sh.

Sorry, I searched just TST_RETRY_FN_EXP_BACKOFF.
Correct, TST_RETRY_FUNC is used there.

Quoted 3 lines

I wonder if we need to TBROK in TST_RETRY_FUNC(). We could just return what the FUNC returns and let the test decide. TST_RETRY_FUNC_BRK() could be a wrapper that TBROKs on timeout.

That could work (apart from the fact it diverges the functionality from C).
+ there could be the third one, which TPASS/TFAIL (instead of nothing/TBROK).

But this should be based on TST_RETRY_FN_EXP_BACKOFF (TST_RETRY_FUNC is reusing TST_RETRY_FN_EXP_BACKOFF) + add also TST_RETRY_FUNC wrappers.

Do we need similar functionality in C?

Kind regards,
Petr

#8 lore raw

PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives

Quoted 42 lines · Petr Vorel

Hi Jan,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

- tst_res TPASS "'lsmod' passed."

testcases/commands/lsmod/lsmod01.sh +5 -0
+lsmod_test()
+{
+ for i in $(seq 1 5); do
+         if lsmod_matches_proc_modules; then
+                 tst_res TPASS "'lsmod' passed."
Quoted 8 lines
Quoted 8 lines

testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 46

Quoted 5 lines
testcases/commands/lsmod/lsmod01.sh +5 -0
+                 return
+         fi
+         tst_res TINFO "Trying again"
+         sleep 1
+ done

This is similar pattern to TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() (for both shell and C). I wonder if we also have use for TPASS/TFAIL instead of just TBROK and specifying number of tries instead of time

to be

Quoted 4 lines
Quoted 1 line

setup.

I think TFAIL fits more here, it's outcome of what we are testing. TBROK in my mind is failure unrelated to subject of test.

I express myself wrong. Sure, I meant to have TPASS/TFAIL, just to use some helper function instead of writing the wrapper in the test.

Quoted 4 lines

But functionally TST_RETRY_FUNC should work too.

Quoted 2 lines

C and shell usage is a bit different, so maybe TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() doesn't make much sense

for shell

Quoted 1 line

I see it used in mkswap01.sh and numa01.sh.

Sorry, I searched just TST_RETRY_FN_EXP_BACKOFF.
Correct, TST_RETRY_FUNC is used there.

Quoted 3 lines

I wonder if we need to TBROK in TST_RETRY_FUNC(). We could just return what the FUNC returns and let the test decide. TST_RETRY_FUNC_BRK() could be a wrapper that TBROKs on timeout.

That could work (apart from the fact it diverges the functionality from C).
+ there could be the third one, which TPASS/TFAIL (instead of nothing/TBROK).

Quoted 6 lines

But this should be based on TST_RETRY_FN_EXP_BACKOFF (TST_RETRY_FUNC is reusing
TST_RETRY_FN_EXP_BACKOFF) + add also TST_RETRY_FUNC wrappers.

Do we need similar functionality in C?

Not sure, but we could collect the requirement for the EXP_BACKOFF series macro. I'm also thinking about to extend the functionality for more situations.

e.g http://lists.linux.it/pipermail/ltp/2019-October/013896.html

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20191024/e640bf64/attachment.htm>
#9 lore raw

PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives

Hi Li, Jan,

...

Quoted 11 lines
Quoted 10 lines
Quoted 3 lines

I wonder if we need to TBROK in TST_RETRY_FUNC(). We could just return what the FUNC returns and let the test decide. TST_RETRY_FUNC_BRK() could be a wrapper that TBROKs on timeout.

That could work (apart from the fact it diverges the functionality from C).
+ there could be the third one, which TPASS/TFAIL (instead of nothing/TBROK).

But this should be based on TST_RETRY_FN_EXP_BACKOFF (TST_RETRY_FUNC is reusing
TST_RETRY_FN_EXP_BACKOFF) + add also TST_RETRY_FUNC wrappers.

Quoted 8 lines
Quoted 1 line

Do we need similar functionality in C?

Not sure, but we could collect the requirement for the EXP_BACKOFF series macro. I'm also thinking about to extend the functionality for more situations.

e.g http://lists.linux.it/pipermail/ltp/2019-October/013896.html

+1 for @INFI - 1: retry infinitely, 0: retry in limit times

Combining with TPASS/TFAIL vs. nothing/TBROK it might make sense to either use enum flags for C implementation, which would save one parameter and allow further extension. Shell could have getopts instead of more parameters. And it's a question whether cover all variants with wrappers like TST_RETRY_FUNC* or not.

Kind regards,
Petr

#10 lore raw

PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives

In reply to Petr Vorel

----- Original Message -----

Quoted 37 lines

Hi Jan,

Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
testcases/commands/lsmod/lsmod01.sh +7 -1
-	tst_res TPASS "'lsmod' passed."
+lsmod_test()
+{
+	for i in $(seq 1 5); do
+		if lsmod_matches_proc_modules; then
+			tst_res TPASS "'lsmod' passed."
+			return
+		fi
Quoted 8 lines

testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 46

Quoted 3 lines
testcases/commands/lsmod/lsmod01.sh +3 -0
+		tst_res TINFO "Trying again"
+		sleep 1
+	done

This is similar pattern to TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() (for both shell and C). I wonder if we also have use for TPASS/TFAIL instead of just TBROK and specifying number of tries instead of time to be
setup.

Quoted 2 lines

I think TFAIL fits more here, it's outcome of what we are testing. TBROK in my mind is failure unrelated to subject of test.

I express myself wrong. Sure, I meant to have TPASS/TFAIL,
just to use some helper function instead of writing the wrapper in the test.

Quoted 7 lines

But functionally TST_RETRY_FUNC should work too.

Quoted 3 lines

C and shell usage is a bit different, so maybe TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() doesn't make much sense for shell

I see it used in mkswap01.sh and numa01.sh.

Sorry, I searched just TST_RETRY_FN_EXP_BACKOFF.
Correct, TST_RETRY_FUNC is used there.

Quoted 3 lines

I wonder if we need to TBROK in TST_RETRY_FUNC(). We could just return what the FUNC returns and let the test decide. TST_RETRY_FUNC_BRK() could be a wrapper that TBROKs on timeout.

Quoted 8 lines

That could work (apart from the fact it diverges the functionality from C).
+ there could be the third one, which TPASS/TFAIL (instead of nothing/TBROK).

But this should be based on TST_RETRY_FN_EXP_BACKOFF (TST_RETRY_FUNC is reusing
TST_RETRY_FN_EXP_BACKOFF) + add also TST_RETRY_FUNC wrappers.

Do we need similar functionality in C?

If we make modifications we should keep it consistent with C.