lsmod01.sh: retry test couple times to lower false positives
PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives
Test sporadically fails with:
lsmod01 1 TFAIL: lsmod output different from /proc/modules. 36c36
< loop 42057 2
---
Quoted 1 line
loop 42057 1
commands runtest file runs mkswap01 before this test. That test is using loop device, and udev is presumably still holding a reference by the time lsmod01 test starts.
Repeat the test couple times to avoid racing with rest of the system.
--- testcases/commands/lsmod/lsmod01.sh | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-)
testcases/commands/lsmod/lsmod01.sh +19 -9
@@ -10,31 +10,41 @@ TST_NEEDS_TMPDIR=1 TST_NEEDS_CMDS="lsmod" . tst_test.sh -lsmod_test() +lsmod_matches_proc_modules() { lsmod_output=$(lsmod | awk '!/Module/{print $1, $2, $3}' | sort) if [ -z "$lsmod_output" ]; then - tst_res TFAIL "Failed to parse the output from lsmod" - return + tst_brk TBROK "Failed to parse the output from lsmod" fi - modules_output=$(awk '{print $1, $2, $3}' /proc/modules | sort) + modules_output=$(awk '{print $1, $2, $3} 1' /proc/modules | sort) if [ -z "$modules_output" ]; then - tst_res TFAIL "Failed to parse /proc/modules" - return + tst_brk TBROK "Failed to parse /proc/modules" fi if [ "$lsmod_output" != "$modules_output" ]; then - tst_res TFAIL "lsmod output different from /proc/modules." + tst_res TINFO "lsmod output different from /proc/modules." echo "$lsmod_output" > temp1 echo "$modules_output" > temp2 diff temp1 temp2 - return + return 1 fi + return 0 +} - tst_res TPASS "'lsmod' passed." +lsmod_test() +{ + for i in $(seq 1 5); do + if lsmod_matches_proc_modules; then + tst_res TPASS "'lsmod' passed." + return + fi + tst_res TINFO "Trying again" + sleep 1 + done + tst_res TFAIL "'lsmod' doesn't match /proc/modules output" } tst_run
Signature · 1 line
1.8.3.1
PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives
Hi Jan,
Quoted 7 lines
Test sporadically fails with:
lsmod01 1 TFAIL: lsmod output different from/proc/modules. 36c36
< loop 42057 2
---Quoted 1 line
loop 42057 1
Quoted 7 lines
commands runtest file runs mkswap01 before this test. That test is using loop device, and udev is presumably still holding a reference by the time lsmod01 test starts.
Repeat the test couple times to avoid racing with rest of the system.
Signed-off-by Jan Stancek <[email protected]>
...
Quoted 7 lines
+++
b/testcases/commands/lsmod/lsmod01.shtestcases/commands/lsmod/lsmod01.sh +1
-1@@ -10,31 +10,41 @@ TST_NEEDS_TMPDIR=1 TST_NEEDS_CMDS="lsmod" . tst_test.sh -lsmod_test() +lsmod_matches_proc_modules()
testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 20
Quoted 10 lines
testcases/commands/lsmod/lsmod01.sh +2
-3{ lsmod_output=$(lsmod | awk '!/Module/{print $1, $2, $3}' | sort) if [ -z "$lsmod_output" ]; then - tst_res TFAIL "Failed to parse the output from lsmod" - return + tst_brk TBROK "Failed to parse the output from lsmod" fi - modules_output=$(awk '{print $1, $2, $3}' /proc/modules | sort) + modules_output=$(awk '{print $1, $2, $3} 1' /proc/modules | sort)
This is a regression. Please keep the old version (without 1).
- modules_output=$(awk '{print $1, $2, $3} 1' /proc/modules | sort)
+ modules_output=$(awk '{print $1, $2, $3}' /proc/modules | sort)
testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 26
Quoted 9 lines
testcases/commands/lsmod/lsmod01.sh +2
-3if [ -z "$modules_output" ]; then - tst_res TFAIL "Failed to parse /proc/modules" - return + tst_brk TBROK "Failed to parse /proc/modules" fi if [ "$lsmod_output" != "$modules_output" ]; then - tst_res TFAIL "lsmod output different from /proc/modules." + tst_res TINFO "lsmod output different from /proc/modules."
nit: please remove dots at the end of messages, when you're touching that file.
The rest looks good to me.
Kind regards,
Petr
PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives
----- Original Message -----
Quoted 34 lines
Hi Jan,
Quoted 7 lines
Test sporadically fails with:
lsmod01 1 TFAIL: lsmod output different from/proc/modules. 36c36
< loop 42057 2
---Quoted 1 line
loop 42057 1
Quoted 7 lines
commands runtest file runs mkswap01 before this test. That test is using loop device, and udev is presumably still holding a reference by the time lsmod01 test starts.
Repeat the test couple times to avoid racing with rest of the system.
Signed-off-by Jan Stancek <[email protected]>Reviewed-by Petr Vorel <[email protected]>...
Quoted 14 lines
+++
b/testcases/commands/lsmod/lsmod01.shtestcases/commands/lsmod/lsmod01.sh +2
-3@@ -10,31 +10,41 @@ TST_NEEDS_TMPDIR=1 TST_NEEDS_CMDS="lsmod" . tst_test.sh -lsmod_test() +lsmod_matches_proc_modules() { lsmod_output=$(lsmod | awk '!/Module/{print $1, $2, $3}' | sort) if [ -z "$lsmod_output" ]; then - tst_res TFAIL "Failed to parse the output from lsmod" - return + tst_brk TBROK "Failed to parse the output from lsmod" fi
Quoted 3 lines
testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 20
Quoted 2 lines
testcases/commands/lsmod/lsmod01.sh +1
-1- modules_output=$(awk '{print $1, $2, $3}' /proc/modules | sort) + modules_output=$(awk '{print $1, $2, $3} 1' /proc/modules | sort)This is a regression. Please keep the old version (without 1).
My bad, I left it in by accident after testing.
Quoted 11 lines
(quoted diff) +1
-1- modules_output=$(awk '{print $1, $2, $3} 1' /proc/modules | sort) + modules_output=$(awk '{print $1, $2, $3}' /proc/modules | sort)Quoted 6 lines
testcases/commands/lsmod/lsmod01.sh +1
-2if [ -z "$modules_output" ]; then - tst_res TFAIL "Failed to parse /proc/modules" - return + tst_brk TBROK "Failed to parse /proc/modules" fi
Quoted 5 lines
testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 26
Quoted 3 lines
testcases/commands/lsmod/lsmod01.sh +1
-1if [ "$lsmod_output" != "$modules_output" ]; then - tst_res TFAIL "lsmod output different from /proc/modules." + tst_res TINFO "lsmod output different from /proc/modules."nit: please remove dots at the end of messages, when you're touching that file.
will do
Quoted 6 lines
The rest looks good to me.
Kind regards,
Petr
PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives
In reply to Jan StancekHi Jan,
...
testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 30
Quoted 7 lines
testcases/commands/lsmod/lsmod01.sh +1
-1if [ "$lsmod_output" != "$modules_output" ]; then - tst_res TFAIL "lsmod output different from /proc/modules." + tst_res TINFO "lsmod output different from /proc/modules." echo "$lsmod_output" > temp1 echo "$modules_output" > temp2 diff temp1 temp2
Also this code could be wrapped with
if tst_cmd_available diff; then
(I wouldn't add it to TST_NEEDS_CMDS)
Kind regards,
Petr
PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives
----- Original Message -----
Quoted 7 lines
Hi Jan,
...
Quoted 3 lines
testcases/commands/lsmod/lsmod01.sh +1
-1if [ "$lsmod_output" != "$modules_output" ]; then - tst_res TFAIL "lsmod output different from /proc/modules." + tst_res TINFO "lsmod output different from /proc/modules."
Quoted 5 lines
testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 30
Quoted 3 lines
testcases/commands/lsmod/lsmod01.sh +0
-0echo "$lsmod_output" > temp1 echo "$modules_output" > temp2 diff temp1 temp2Also this code could be wrapped with
iftst_cmd_availablediff; then
Added this as well and pushed.
PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives
In reply to Jan StancekHi Jan,
...
Quoted 33 lines
testcases/commands/lsmod/lsmod01.sh +8
-9-lsmod_test() +lsmod_matches_proc_modules() { lsmod_output=$(lsmod | awk '!/Module/{print $1, $2, $3}' | sort) if [ -z "$lsmod_output" ]; then - tst_res TFAIL "Failed to parse the output from lsmod" - return + tst_brk TBROK "Failed to parse the output from lsmod" fi - modules_output=$(awk '{print $1, $2, $3}' /proc/modules | sort) + modules_output=$(awk '{print $1, $2, $3} 1' /proc/modules | sort) if [ -z "$modules_output" ]; then - tst_res TFAIL "Failed to parse /proc/modules" - return + tst_brk TBROK "Failed to parse /proc/modules" fi if [ "$lsmod_output" != "$modules_output" ]; then - tst_res TFAIL "lsmod output different from /proc/modules." + tst_res TINFO "lsmod output different from /proc/modules." echo "$lsmod_output" > temp1 echo "$modules_output" > temp2 diff temp1 temp2 - return + return 1 fi + return 0 +} - tst_res TPASS "'lsmod' passed."
testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 46
Quoted 10 lines
testcases/commands/lsmod/lsmod01.sh +10
-0+lsmod_test() +{ + for i in $(seq 1 5); do + if lsmod_matches_proc_modules; then + tst_res TPASS "'lsmod' passed." + return + fi + tst_res TINFO "Trying again" + sleep 1 + done
This is similar pattern to TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() (for both shell and C). I wonder if we also have use for TPASS/TFAIL instead of just TBROK and specifying number of tries instead of time to be setup.
C and shell usage is a bit different, so maybe TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() doesn't make much sense for shell (actually nothing uses them in shell) and I don't see much usage for my proposal in C.
testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 47
Quoted 1 line
testcases/commands/lsmod/lsmod01.sh +1
-0+ tst_res TFAIL "'lsmod' doesn't match /proc/modules output"
+ we forget to use local (for lsmod_outputa and i), but that's not that important.
Kind regards,
Petr
PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives
----- Original Message -----
Quoted 16 lines
testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 46
Quoted 12 lines
testcases/commands/lsmod/lsmod01.sh +10
-1- tst_res TPASS "'lsmod' passed." +lsmod_test() +{ + for i in $(seq 1 5); do + if lsmod_matches_proc_modules; then + tst_res TPASS "'lsmod' passed." + return + fi + tst_res TINFO "Trying again" + sleep 1 + doneThis is similar pattern to TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() (for both shell and C). I wonder if we also have use for TPASS/TFAIL instead of just TBROK and specifying number of tries instead of time to be setup.
I think TFAIL fits more here, it's outcome of what we are testing. TBROK in my mind is failure unrelated to subject of test.
But functionally TST_RETRY_FUNC should work too.
Quoted 2 lines
C and shell usage is a bit different, so maybe TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() doesn't make much sense for shell
I see it used in mkswap01.sh and numa01.sh.
I wonder if we need to TBROK in TST_RETRY_FUNC(). We could just return what the FUNC returns and let the test decide. TST_RETRY_FUNC_BRK() could be a wrapper that TBROKs on timeout.
PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives
Hi Jan,
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
testcases/commands/lsmod/lsmod01.sh +6
-1- tst_res TPASS "'lsmod' passed." +lsmod_test() +{ + for i in $(seq 1 5); do + if lsmod_matches_proc_modules; then + tst_res TPASS "'lsmod' passed." + returnQuoted 8 lines
testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 46
Quoted 4 lines
testcases/commands/lsmod/lsmod01.sh +4
-0+ fi + tst_res TINFO "Trying again" + sleep 1 + doneThis is similar pattern to TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() (for both shell and C). I wonder if we also have use for TPASS/TFAIL instead of just TBROK and specifying number of tries instead of time to be setup.
Quoted 2 lines
I think TFAIL fits more here, it's outcome of what we are testing. TBROK in my mind is failure unrelated to subject of test.
I express myself wrong. Sure, I meant to have TPASS/TFAIL,
just to use some helper function instead of writing the wrapper in the test.
Quoted 6 lines
But functionally
TST_RETRY_FUNCshould work too.Quoted 2 lines
C and shell usage is a bit different, so maybe TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() doesn't make much sense for shell
I see it used in
mkswap01.shandnuma01.sh.
Sorry, I searched just TST_RETRY_FN_EXP_BACKOFF.
Correct, TST_RETRY_FUNC is used there.
Quoted 3 lines
I wonder if we need to TBROK in
TST_RETRY_FUNC(). We could just return what the FUNC returns and let the test decide.TST_RETRY_FUNC_BRK()could be a wrapper that TBROKs on timeout.
That could work (apart from the fact it diverges the functionality from C).
+ there could be the third one, which TPASS/TFAIL (instead of nothing/TBROK).
But this should be based on TST_RETRY_FN_EXP_BACKOFF (TST_RETRY_FUNC is reusing TST_RETRY_FN_EXP_BACKOFF) + add also TST_RETRY_FUNC wrappers.
Do we need similar functionality in C?
Kind regards,
Petr
PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives
Quoted 42 lines · Petr Vorel
Hi Jan,
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
-
tst_resTPASS "'lsmod' passed."testcases/commands/lsmod/lsmod01.sh +5
-0+lsmod_test() +{ + for i in $(seq 1 5); do + if lsmod_matches_proc_modules; then + tst_res TPASS "'lsmod' passed."Quoted 8 lines
Quoted 8 lines
testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 46
Quoted 5 lines
testcases/commands/lsmod/lsmod01.sh +5
-0+ return + fi + tst_res TINFO "Trying again" + sleep 1 + doneThis is similar pattern to TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() (for both shell and C). I wonder if we also have use for TPASS/TFAIL instead of just TBROK and specifying number of tries instead of time
to be
Quoted 4 lines
Quoted 1 line
setup.
I think TFAIL fits more here, it's outcome of what we are testing. TBROK in my mind is failure unrelated to subject of test.
I express myself wrong. Sure, I meant to have TPASS/TFAIL, just to use some helper function instead of writing the wrapper in the test.
Quoted 4 lines
But functionally
TST_RETRY_FUNCshould work too.Quoted 2 lines
C and shell usage is a bit different, so maybe TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() doesn't make much sense
for shell
Quoted 1 line
I see it used in
mkswap01.shandnuma01.sh.Sorry, I searched just
TST_RETRY_FN_EXP_BACKOFF.
Correct,TST_RETRY_FUNCis used there.Quoted 3 lines
I wonder if we need to TBROK in
TST_RETRY_FUNC(). We could just return what the FUNC returns and let the test decide.TST_RETRY_FUNC_BRK()could be a wrapper that TBROKs on timeout.That could work (apart from the fact it diverges the functionality from C).
+ there could be the third one, which TPASS/TFAIL (instead of nothing/TBROK).
Quoted 6 lines
But this should be based on
TST_RETRY_FN_EXP_BACKOFF(TST_RETRY_FUNCis reusingTST_RETRY_FN_EXP_BACKOFF) + add alsoTST_RETRY_FUNCwrappers.Do we need similar functionality in C?
Not sure, but we could collect the requirement for the EXP_BACKOFF series macro. I'm also thinking about to extend the functionality for more situations.
e.g http://lists.linux.it/pipermail/ltp/2019-October/013896.html
Signature · 5 lines
Regards, Li Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linux.it/pipermail/ltp/attachments/20191024/e640bf64/attachment.htm>
PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives
Hi Li, Jan,
...
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
I wonder if we need to TBROK in
TST_RETRY_FUNC(). We could just return what the FUNC returns and let the test decide.TST_RETRY_FUNC_BRK()could be a wrapper that TBROKs on timeout.That could work (apart from the fact it diverges the functionality from C).
+ there could be the third one, which TPASS/TFAIL (instead of nothing/TBROK).But this should be based on
TST_RETRY_FN_EXP_BACKOFF(TST_RETRY_FUNCis reusingTST_RETRY_FN_EXP_BACKOFF) + add alsoTST_RETRY_FUNCwrappers.
Quoted 8 lines
Quoted 1 line
Do we need similar functionality in C?
Not sure, but we could collect the requirement for the
EXP_BACKOFFseries macro. I'm also thinking about to extend the functionality for more situations.e.g http://lists.linux.it/pipermail/ltp/2019-October/013896.html
+1 for @INFI - 1: retry infinitely, 0: retry in limit times
Combining with TPASS/TFAIL vs. nothing/TBROK it might make sense to either use enum flags for C implementation, which would save one parameter and allow further extension. Shell could have getopts instead of more parameters. And it's a question whether cover all variants with wrappers like TST_RETRY_FUNC* or not.
Kind regards,
Petr
PATCH 1/1 lsmod01.sh: retry test couple times to lower false positives
In reply to Petr Vorel----- Original Message -----
Quoted 37 lines
Hi Jan,
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
testcases/commands/lsmod/lsmod01.sh +7
-1- tst_res TPASS "'lsmod' passed." +lsmod_test() +{ + for i in $(seq 1 5); do + if lsmod_matches_proc_modules; then + tst_res TPASS "'lsmod' passed." + return + fiQuoted 8 lines
testcases/commands/lsmod/lsmod01.sh TST_NEEDS_TMPDIR=1 · line 46
Quoted 3 lines
testcases/commands/lsmod/lsmod01.sh +3
-0+ tst_res TINFO "Trying again" + sleep 1 + doneThis is similar pattern to TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() (for both shell and C). I wonder if we also have use for TPASS/TFAIL instead of just TBROK and specifying number of tries instead of time to be
setup.Quoted 2 lines
I think TFAIL fits more here, it's outcome of what we are testing. TBROK in my mind is failure unrelated to subject of test.
I express myself wrong. Sure, I meant to have TPASS/TFAIL,
just to use some helper function instead of writing the wrapper in the test.Quoted 7 lines
But functionally
TST_RETRY_FUNCshould work too.Quoted 3 lines
C and shell usage is a bit different, so maybe TST_RETRY_FUNC()/TST_RETRY_FN_EXP_BACKOFF() doesn't make much sense for shell
I see it used in
mkswap01.shandnuma01.sh.Sorry, I searched just
TST_RETRY_FN_EXP_BACKOFF.
Correct,TST_RETRY_FUNCis used there.Quoted 3 lines
I wonder if we need to TBROK in
TST_RETRY_FUNC(). We could just return what the FUNC returns and let the test decide.TST_RETRY_FUNC_BRK()could be a wrapper that TBROKs on timeout.
Quoted 8 lines
That could work (apart from the fact it diverges the functionality from C).
+ there could be the third one, which TPASS/TFAIL (instead of nothing/TBROK).But this should be based on
TST_RETRY_FN_EXP_BACKOFF(TST_RETRY_FUNCis reusingTST_RETRY_FN_EXP_BACKOFF) + add alsoTST_RETRY_FUNCwrappers.Do we need similar functionality in C?
If we make modifications we should keep it consistent with C.