Skip to contentSkip to conversation

check: add CLI option to repeat tests

PATCH In review Dave Chinner 2 participants 3 messages Jul 8, 2020 – Jul 12, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 check: add CLI option to repeat tests

From: Dave Chinner <[email protected]>

Frequently when trying to reproduce a problem I want to run a set of specific tests in a loop, over and over again. I run fstests from a set of run scripts that have non-trivial overhead (e.g. patterning block devices before the runs start), so if all I want to do is run the same test 100x, using a shell loop over the entire run scripts reduces the iteration rate substantially.

Hence add an option to check to allow fstests to loop a number of times over the configured test set without stopping.

---
 check | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
check +17 -4
index 2e148e57..2676b619 100755
@@ -26,6 +26,7 @@ subdir_xfile=""
 brief_test_summary=false
 do_report=false
 DUMP_OUTPUT=false
+iterations=1
 
 # This is a global variable used to pass test failure text to reporting gunk
 _err_msg=""
@@ -66,6 +67,7 @@ check options
     -n			show me, do not run tests
     -T			output timestamps
     -r			randomize test order
+    -i <n>		iterate the test list <n> times
     -d			dump test output to stdout
     -b			brief test summary
     -R fmt[,fmt]	generate report in formats specified. Supported format: [xunit]
@@ -297,7 +299,7 @@ while [ $# -gt 0 ]; do
 
 	-n)	showme=true ;;
         -r)	randomize=true ;;
-
+	-i)	iterations=$2; shift ;;
 	-T)	timestamp=true ;;
 	-d)	DUMP_OUTPUT=true ;;
 	-b)	brief_test_summary=true;;
@@ -525,7 +527,10 @@ else
 	trap "_wrapup; exit \$status" 0 1 2 3 15
 fi
 
-for section in $HOST_OPTIONS_SECTIONS; do
+function run_section()
+{
+	local section=$1
+
 	OLD_FSTYP=$FSTYP
 	OLD_TEST_FS_MOUNT_OPTS=$TEST_FS_MOUNT_OPTS
 	get_next_config $section
@@ -540,7 +545,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 			fi
 		done
 		if $skip; then
-			continue
+			return
 		fi
 	fi
 
@@ -554,7 +559,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 			fi
 		done
 		if $skip; then
-			continue
+			return
 		fi
 	fi
 
@@ -700,6 +705,8 @@ for section in $HOST_OPTIONS_SECTIONS; do
 		seqres="$REPORT_DIR/$seqnum"
 
 		mkdir -p $RESULT_DIR
+		rm -f ${RESULT_DIR}/require_scratch*
+		rm -f ${RESULT_DIR}/require_test*
 		echo -n "$seqnum"
 
 		if $showme; then
@@ -858,6 +865,12 @@ for section in $HOST_OPTIONS_SECTIONS; do
 
 	_test_unmount 2> /dev/null
 	_scratch_unmount 2> /dev/null
+}
+
+for ((iters = 0; iters < $iterations; iters++)) do
+	for section in $HOST_OPTIONS_SECTIONS; do
+		run_section $section
+	done
 done
 
 interrupt=false
Signature · 1 line
2.26.2.761.g0e0b3e54be
#1 lore raw
Quoted 11 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Frequently when trying to reproduce a problem I want to run a set of specific tests in a loop, over and over again. I run fstests from a set of run scripts that have non-trivial overhead (e.g. patterning block devices before the runs start), so if all I want to do is run the same test 100x, using a shell loop over the entire run scripts reduces the iteration rate substantially.

Hence add an option to check to allow fstests to loop a number of times over the configured test set without stopping.

Hi Dave,

this is useful as I am often doing the same. However what I am really missing is the ability to end the loop when a test fails and the ability to run tests in infinite loop (or until the test fails).

It should be easy modification to make, what do you think ?

-Lukas

Quoted 90 lines
---
 check | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
check +17 -4
index 2e148e57..2676b619 100755
@@ -26,6 +26,7 @@ subdir_xfile=""
 brief_test_summary=false
 do_report=false
 DUMP_OUTPUT=false
+iterations=1
 
 # This is a global variable used to pass test failure text to reporting gunk
 _err_msg=""
@@ -66,6 +67,7 @@ check options
     -n			show me, do not run tests
     -T			output timestamps
     -r			randomize test order
+    -i <n>		iterate the test list <n> times
     -d			dump test output to stdout
     -b			brief test summary
     -R fmt[,fmt]	generate report in formats specified. Supported format: [xunit]
@@ -297,7 +299,7 @@ while [ $# -gt 0 ]; do
 
 	-n)	showme=true ;;
         -r)	randomize=true ;;
-
+	-i)	iterations=$2; shift ;;
 	-T)	timestamp=true ;;
 	-d)	DUMP_OUTPUT=true ;;
 	-b)	brief_test_summary=true;;
@@ -525,7 +527,10 @@ else
 	trap "_wrapup; exit \$status" 0 1 2 3 15
 fi
 
-for section in $HOST_OPTIONS_SECTIONS; do
+function run_section()
+{
+	local section=$1
+
 	OLD_FSTYP=$FSTYP
 	OLD_TEST_FS_MOUNT_OPTS=$TEST_FS_MOUNT_OPTS
 	get_next_config $section
@@ -540,7 +545,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 			fi
 		done
 		if $skip; then
-			continue
+			return
 		fi
 	fi
 
@@ -554,7 +559,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 			fi
 		done
 		if $skip; then
-			continue
+			return
 		fi
 	fi
 
@@ -700,6 +705,8 @@ for section in $HOST_OPTIONS_SECTIONS; do
 		seqres="$REPORT_DIR/$seqnum"
 
 		mkdir -p $RESULT_DIR
+		rm -f ${RESULT_DIR}/require_scratch*
+		rm -f ${RESULT_DIR}/require_test*
 		echo -n "$seqnum"
 
 		if $showme; then
@@ -858,6 +865,12 @@ for section in $HOST_OPTIONS_SECTIONS; do
 
 	_test_unmount 2> /dev/null
 	_scratch_unmount 2> /dev/null
+}
+
+for ((iters = 0; iters < $iterations; iters++)) do
+	for section in $HOST_OPTIONS_SECTIONS; do
+		run_section $section
+	done
 done
 
 interrupt=false
Signature · 1 line
2.26.2.761.g0e0b3e54be
#2 lore raw
Quoted 10 lines · Lukas Czerner
Quoted 9 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Frequently when trying to reproduce a problem I want to run a set of specific tests in a loop, over and over again. I run fstests from a set of run scripts that have non-trivial overhead (e.g. patterning block devices before the runs start), so if all I want to do is run the same test 100x, using a shell loop over the entire run scripts reduces the iteration rate substantially.

Quoted 8 lines
Quoted 2 lines

Hence add an option to check to allow fstests to loop a number of times over the configured test set without stopping.

Hi Dave,

this is useful as I am often doing the same. However what I am really missing is the ability to end the loop when a test fails and the ability to run tests in infinite loop (or until the test fails).

To run until a test set fails, I have just been using "-i 10000" because if it passes 10000 loops, it isn't going to fail.. :)

Hence I didn't see a need to complicate things with a specific "loop forever" options.

Quoted 1 line

It should be easy modification to make, what do you think ?

Feel free to add send a patch on top of this one to support that - all of the failures I've been looking at with this have been "test does not complete" failures, so it wasn't important to me.

As it is, right now I don't have time to feature creep simple modifications like this into more complex functionality - I really only sent this out because Willy mentioned on #xfs that he wished there was a way to loop a test and I said "I've got a patch on my local tree for that"....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]