Skip to contentSkip to conversation

fstests: direct specification of looping test duration

PATCH Superseded Darrick J. Wong 3 participants 16 messages Apr 11, 2023 – Apr 25, 2023

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover fstests: direct specification of looping test duration

Hi all,

One of the things that I do as a maintainer is to designate a handful of VMs to run fstests for unusually long periods of time. This practice I call long term soak testing. There are actually three separate fleets for this -- one runs alongside the nightly builds, one runs alongside weekly rebases, and the last one runs stable releases.

My interactions with all three fleets is pretty much the same -- load current builds of software, and try to run the exerciser tests for a duration of time -- 12 hours, 6.5 days, 30 days, etc. TIME_FACTOR does not work well for this usage model, because it is difficult to guess the correct time factor given that the VMs are hetergeneous and the IO completion rate is not perfectly predictable.

Worse yet, if you want to run (say) all the recoveryloop tests on one VM (because recoveryloop is prone to crashing), it's impossible to set a TIME_FACTOR so that each loop test gets equal runtime. That can be hacked around with config sections, but that doesn't solve the first problem.

This series introduces a new configuration variable, SOAK_DURATION, that allows test runners to control directly various long soak and looping recovery tests. This is intended to be an alternative to TIME_FACTOR, since that variable usually adjusts operation counts, which are proportional to runtime but otherwise not a direct measure of time.

With this override in place, I can configure the long soak fleet to run for exactly as long as I want them to, and they actually hit the time budget targets. The recoveryloop fleet now divides looping-test time equally among the four that are in that group so that they all get ~3 hours of coverage every night.

There are more tests that could use this than I actually modified here, but I've done enough to show this off as a proof of concept.

If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=soak-duration

---
 check                 |   14 +++++++++
 common/config         |    7 ++++
 common/fuzzy          |    7 ++++
 common/rc             |   34 +++++++++++++++++++++
 common/report         |    1 +
 ltp/fsstress.c        |   78 +++++++++++++++++++++++++++++++++++++++++++++++--
 ltp/fsx.c             |   50 +++++++++++++++++++++++++++++++
 src/soak_duration.awk |   23 ++++++++++++++
 tests/generic/019     |    1 +
 tests/generic/388     |    2 +
 tests/generic/475     |    2 +
 tests/generic/476     |    7 +++-
 tests/generic/482     |    5 +++
 tests/generic/521     |    1 +
 tests/generic/522     |    1 +
 tests/generic/642     |    1 +
 tests/generic/648     |    8 +++--
 17 files changed, 229 insertions(+), 13 deletions(-)
 create mode 100644 src/soak_duration.awk
#1 lore raw

PATCH 1/3 generic/476: reclassify this test as a long running soak stress test

From: Darrick J. Wong <[email protected]>

This test is a long(ish) running stress test, so add it to those groups.

---
 tests/generic/476 |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/476 +1 -1
index 212373d17c..edb0be7b50 100755
@@ -8,7 +8,7 @@
 # bugs in the write path.
 #
 . ./common/preamble
-_begin_fstest auto rw
+_begin_fstest auto rw soak long_rw stress
 
 # Override the default cleanup function.
 _cleanup()
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] generic/476: reclassify this test as a long running soak stress test

Quoted 15 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

This test is a long(ish) running stress test, so add it to those groups.

---
 tests/generic/476 |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/476 +0 -0
index 212373d17c..edb0be7b50 100755

tests/generic/476 line 11

Quoted 5 lines
tests/generic/476 +1 -1
@@ -8,7 +8,7 @@
 # bugs in the write path.
 #
 . ./common/preamble
-_begin_fstest auto rw
+_begin_fstest auto rw soak long_rw stress

Sorry for late reviewing. I thought a bit more about this change. I think the "soak", "long_rw" and "stress" tags are a bit overlap. If the "stress" group means "fsstress", then I think the fsstress test can be in soak group too, and currently the test cases in "soak" group are same with the "long_rw" group [1].

So I think we can give the "soak" tag to more test cases with random I/Os (fsstress or fsx or others). And rename "long_rw" to "long_soak" for those soak group cases which need long soaking time. Then we have two group tags for random loading/stress test cases, the testers can (decide to) run these random load test cases seperately with more time or loop count.

Anyway, above things can be done in another patchset, I just speak out to get more talking:) For this patch:

Thanks,
Zorro

[1]

# ./check -n -g soak
SECTION       -- simpledev
FSTYP         -- xfs (non-debug)
PLATFORM      -- Linux/x86_64
MKFS_OPTIONS  -- -f -m rmapbt=1 /dev/sda3
MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sda3 /mnt/scratch

generic/521
generic/522
generic/642

# ./check -n -g long_rw
SECTION       -- simpledev
FSTYP         -- xfs (non-debug)
PLATFORM      -- Linux/x86_64
MKFS_OPTIONS  -- -f -m rmapbt=1 /dev/sda3
MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sda3 /mnt/scratch

generic/521
generic/522
generic/642

Quoted 4 lines
tests/generic/476 +0 -0
 # Override the default cleanup function.
 _cleanup()
 
#3 lore raw

Re: [PATCH 1/3] generic/476: reclassify this test as a long running soak stress test

Quoted 11 lines · Zorro Lang
Quoted 10 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

This test is a long(ish) running stress test, so add it to those groups.

---
 tests/generic/476 |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 16 lines

tests/generic/476 line 11

Quoted 10 lines
tests/generic/476 +1 -1
index 212373d17c..edb0be7b50 100755
@@ -8,7 +8,7 @@
 # bugs in the write path.
 #
 . ./common/preamble
-_begin_fstest auto rw
+_begin_fstest auto rw soak long_rw stress

Sorry for late reviewing. I thought a bit more about this change. I think the "soak", "long_rw" and "stress" tags are a bit overlap. If the "stress" group means "fsstress", then I think the fsstress test can be in soak group too, and currently the test cases in "soak" group are same with the "long_rw" group [1].

Hm. Given the current definitions of each group:

long_rw                 long-soak read write IO path exercisers
rw                      read/write IO tests
soak                    long running soak tests of any kind
stress                  fsstress filesystem exerciser

I think these all can apply to generic/476 -- it's definitely a read-write IO test; it's definitely one that does RW for a long time; and it uses fsstress.

Quoted 5 lines

So I think we can give the "soak" tag to more test cases with random I/Os (fsstress or fsx or others). And rename "long_rw" to "long_soak" for those soak group cases which need long soaking time. Then we have two group tags for random loading/stress test cases, the testers can (decide to) run these random load test cases seperately with more time or loop count.

I have a counterproposal -- what do you think about redefining 'soak' to mean "all tests where SOAK_DURATION can be used to control the test runtime directly"? This shouldn't break anyone's scripts, since the only members of 'soak' are the ones that get modified by this patchset.

--D

Quoted 39 lines

Anyway, above things can be done in another patchset, I just speak out to get more talking:) For this patch:

Thanks,
Zorro

[1]

# ./check -n -g soak
SECTION       -- simpledev
FSTYP         -- xfs (non-debug)
PLATFORM      -- Linux/x86_64
MKFS_OPTIONS  -- -f -m rmapbt=1 /dev/sda3
MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sda3 /mnt/scratch

generic/521
generic/522
generic/642

# ./check -n -g long_rw
SECTION       -- simpledev
FSTYP         -- xfs (non-debug)
PLATFORM      -- Linux/x86_64
MKFS_OPTIONS  -- -f -m rmapbt=1 /dev/sda3
MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sda3 /mnt/scratch

generic/521
generic/522
generic/642

Quoted 4 lines
tests/generic/476 +0 -0
 # Override the default cleanup function.
 _cleanup()
 
#4 lore raw

Re: [PATCH 1/3] generic/476: reclassify this test as a long running soak stress test

Quoted 46 lines · Darrick J. Wong
Quoted 11 lines · Zorro Lang
Quoted 10 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

This test is a long(ish) running stress test, so add it to those groups.

---
 tests/generic/476 |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 16 lines

tests/generic/476 line 11

Quoted 10 lines
tests/generic/476 +1 -1
index 212373d17c..edb0be7b50 100755
@@ -8,7 +8,7 @@
 # bugs in the write path.
 #
 . ./common/preamble
-_begin_fstest auto rw
+_begin_fstest auto rw soak long_rw stress

Sorry for late reviewing. I thought a bit more about this change. I think the "soak", "long_rw" and "stress" tags are a bit overlap. If the "stress" group means "fsstress", then I think the fsstress test can be in soak group too, and currently the test cases in "soak" group are same with the "long_rw" group [1].

Hm. Given the current definitions of each group:

long_rw                 long-soak read write IO path exercisers
rw                      read/write IO tests
soak                    long running soak tests of any kind
stress                  fsstress filesystem exerciser

I think these all can apply to generic/476 -- it's definitely a read-write IO test; it's definitely one that does RW for a long time; and it uses fsstress.

Quoted 5 lines

So I think we can give the "soak" tag to more test cases with random I/Os (fsstress or fsx or others). And rename "long_rw" to "long_soak" for those soak group cases which need long soaking time. Then we have two group tags for random loading/stress test cases, the testers can (decide to) run these random load test cases seperately with more time or loop count.

Quoted 4 lines

I have a counterproposal -- what do you think about redefining 'soak' to mean "all tests where SOAK_DURATION can be used to control the test runtime directly"? This shouldn't break anyone's scripts, since the only members of 'soak' are the ones that get modified by this patchset.

Sure, we can check if more cases can use the SOAK_DURATION later, then we add them to soak group.

Thanks,
Zorro

Quoted 43 lines

--D

Quoted 39 lines

Anyway, above things can be done in another patchset, I just speak out to get more talking:) For this patch:

Thanks,
Zorro

[1]

# ./check -n -g soak
SECTION       -- simpledev
FSTYP         -- xfs (non-debug)
PLATFORM      -- Linux/x86_64
MKFS_OPTIONS  -- -f -m rmapbt=1 /dev/sda3
MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sda3 /mnt/scratch

generic/521
generic/522
generic/642

# ./check -n -g long_rw
SECTION       -- simpledev
FSTYP         -- xfs (non-debug)
PLATFORM      -- Linux/x86_64
MKFS_OPTIONS  -- -f -m rmapbt=1 /dev/sda3
MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sda3 /mnt/scratch

generic/521
generic/522
generic/642

Quoted 4 lines
tests/generic/476 +0 -0
 # Override the default cleanup function.
 _cleanup()
 
#5 lore raw

PATCH 2/3 misc: add duration for long soak tests

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Make it so that test runners can schedule long soak stress test programs for an exact number of seconds by setting the STRESS_DURATION config variable.

---
 check                 |   14 +++++++++
 common/config         |    7 ++++
 common/fuzzy          |    7 ++++
 common/report         |    1 +
 ltp/fsstress.c        |   78 +++++++++++++++++++++++++++++++++++++++++++++++--
 ltp/fsx.c             |   50 +++++++++++++++++++++++++++++++
 src/soak_duration.awk |   23 ++++++++++++++
 tests/generic/476     |    5 +++
 tests/generic/521     |    1 +
 tests/generic/522     |    1 +
 tests/generic/642     |    1 +
 11 files changed, 182 insertions(+), 6 deletions(-)
 create mode 100644 src/soak_duration.awk
check +14 -0
index 1a58a2b269..b065277287 100755
@@ -366,6 +366,20 @@ if ! . ./common/rc; then
 	exit 1
 fi
 
+# If the test config specified a soak test duration, see if there are any
+# unit suffixes that need converting to an integer seconds count.
+if [ -n "$SOAK_DURATION" ]; then
+	SOAK_DURATION="$(echo "$SOAK_DURATION" | \
+		sed -e 's/^\([.0-9]*\)\([a-z]\)*/\1 \2/g' | \
+		$AWK_PROG -f $here/src/soak_duration.awk)"
+	if [ $? -ne 0 ]; then
+		echo "$SOAK_DURATION"
+		status=1
+		exit 1
+	fi
+	export SOAK_DURATION
+fi
+
 if [ -n "$subdir_xfile" ]; then
 	for d in $SRC_GROUPS $FSTYP; do
 		[ -f $SRC_DIR/$d/$subdir_xfile ] || continue
common/config +7 -0
index 6c8cb3a5ba..fdd0aadbeb 100644
@@ -57,6 +57,13 @@ export SOAK_PROC=3             # -p option to fsstress
 export SOAK_STRESS=10000       # -n option to fsstress
 export SOAK_PASSES=-1          # count of repetitions of fsstress (while soaking)
 export EMAIL=root@localhost    # where auto-qa will send its status messages
+
+# For certain tests that run in tight loops, setting this variable allows the
+# test runner to specify exactly how long the test should continue looping.
+# This is independent of TIME_FACTOR.  Floating point numbers are allowed, and
+# the unit suffixes m(inutes), h(ours), d(ays), and w(eeks) are supported.
+export SOAK_DURATION=${SOAK_DURATION:=}
+
 export HOST_OPTIONS=${HOST_OPTIONS:=local.config}
 export CHECK_OPTIONS=${CHECK_OPTIONS:="-g auto"}
 export BENCH_PASSES=${BENCH_PASSES:=5}
common/fuzzy +6 -1
index 961bedc717..4365e7343e 100644
@@ -1351,7 +1351,12 @@ _scratch_xfs_stress_scrub() {
 	fi
 
 	local start="$(date +%s)"
-	local end="$((start + (30 * TIME_FACTOR) ))"
+	local end
+	if [ -n "$SOAK_DURATION" ]; then
+		end="$((start + SOAK_DURATION))"
+	else
+		end="$((start + (30 * TIME_FACTOR) ))"
+	fi
 	local scrub_startat="$((start + scrub_delay))"
 	test "$scrub_startat" -gt "$((end - 10))" &&
 		scrub_startat="$((end - 10))"
common/report +1 -0
index be930e0b06..9bfa09ecce 100644
@@ -67,6 +67,7 @@ __generate_report_vars() {
 	REPORT_VARS["CPUS"]="$(getconf _NPROCESSORS_ONLN 2>/dev/null)"
 	REPORT_VARS["MEM_KB"]="$(grep MemTotal: /proc/meminfo | awk '{print $2}')"
 	REPORT_VARS["SWAP_KB"]="$(grep SwapTotal: /proc/meminfo | awk '{print $2}')"
+	test -n "$SOAK_DURATION" && REPORT_VARS["SOAK_DURATION"]="$SOAK_DURATION"
 
 	test -e /sys/devices/system/node/possible && \
 		REPORT_VARS["NUMA_NODES"]="$(cat /sys/devices/system/node/possible 2>/dev/null)"
ltp/fsstress.c +75 -3

132 lines not on this page · raw message

ltp/fsx.c +49 -1

89 lines not on this page · raw message

src/soak_duration.awk new file +23 -0
new file mode 100644
index 0000000000..6c38d09b39
@@ -0,0 +1,23 @@
+#!/usr/bin/awk
+#
+# Convert time interval specifications with suffixes to an integer number of
+# seconds.
+{
+	nr = $1;
+	if ($2 == "" || $2 ~ /s/)	# seconds
+		;
+	else if ($2 ~ /m/)		# minutes
+		nr *= 60;
+	else if ($2 ~ /h/)		# hours
+		nr *= 3600;
+	else if ($2 ~ /d/)		# days
+		nr *= 86400;
+	else if ($2 ~ /w/)		# weeks
+		nr *= 604800;
+	else {
+		printf("%s: unknown suffix\n", $2);
+		exit 1;
+	}
+
+	printf("%d\n", nr);
+}
tests/generic/476 +4 -1
index edb0be7b50..a162cda6b1 100755
@@ -33,7 +33,10 @@ _scratch_mount >> $seqres.full 2>&1
 
 nr_cpus=$((LOAD_FACTOR * 4))
 nr_ops=$((25000 * nr_cpus * TIME_FACTOR))
-$FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
+fsstress_args=(-w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus)
+test -n "$SOAK_DURATION" && fsstress_args+=(--duration="$SOAK_DURATION")
+
+$FSSTRESS_PROG $FSSTRESS_AVOID "${fsstress_args[@]}" >> $seqres.full
 
 # success, all done
 status=0
tests/generic/521 +1 -0
index cde9d44775..22dd31a8ec 100755
@@ -35,6 +35,7 @@ fsx_args+=(-r $min_dio_sz)
 fsx_args+=(-t $min_dio_sz)
 fsx_args+=(-w $min_dio_sz)
 fsx_args+=(-Z)
+test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION")
 
 run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'
 
tests/generic/522 +1 -0
index ae84fe04bb..f0cbcb245c 100755
@@ -29,6 +29,7 @@ fsx_args+=(-N $nr_ops)
 fsx_args+=(-p $((nr_ops / 100)))
 fsx_args+=(-o $op_sz)
 fsx_args+=(-l $file_sz)
+test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION")
 
 run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'
 
tests/generic/642 +1 -0
index c0e274d843..eba90903a3 100755
@@ -49,6 +49,7 @@ for verb in attr_remove removefattr; do
 done
 args+=('-f' "setfattr=20")
 args+=('-f' "attr_set=60")	# sets larger xattrs
+test -n "$DURATION" && args+=(--duration="$DURATION")
 
 $FSSTRESS_PROG "${args[@]}" $FSSTRESS_AVOID -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
 
#6 lore raw

PATCH v2 2/3 misc: add duration for long soak tests

From: Darrick J. Wong <[email protected]>

Make it so that test runners can schedule long soak stress test programs for an exact number of seconds by setting the SOAK_DURATION config variable.

---
v2: fix commit message

---
 check                 |   14 +++++++++
 common/config         |    7 ++++
 common/fuzzy          |    7 ++++
 common/report         |    1 +
 ltp/fsstress.c        |   78 +++++++++++++++++++++++++++++++++++++++++++++++--
 ltp/fsx.c             |   50 +++++++++++++++++++++++++++++++
 src/soak_duration.awk |   23 ++++++++++++++
 tests/generic/476     |    5 +++
 tests/generic/521     |    1 +
 tests/generic/522     |    1 +
 tests/generic/642     |    1 +
 11 files changed, 182 insertions(+), 6 deletions(-)
 create mode 100644 src/soak_duration.awk
check +14 -0
index e32b70d301..1d78cf27f4 100755
@@ -366,6 +366,20 @@ if ! . ./common/rc; then
 	exit 1
 fi
 
+# If the test config specified a soak test duration, see if there are any
+# unit suffixes that need converting to an integer seconds count.
+if [ -n "$SOAK_DURATION" ]; then
+	SOAK_DURATION="$(echo "$SOAK_DURATION" | \
+		sed -e 's/^\([.0-9]*\)\([a-z]\)*/\1 \2/g' | \
+		$AWK_PROG -f $here/src/soak_duration.awk)"
+	if [ $? -ne 0 ]; then
+		echo "$SOAK_DURATION"
+		status=1
+		exit 1
+	fi
+	export SOAK_DURATION
+fi
+
 if [ -n "$subdir_xfile" ]; then
 	for d in $SRC_GROUPS $FSTYP; do
 		[ -f $SRC_DIR/$d/$subdir_xfile ] || continue
common/config +7 -0
index 6c8cb3a5ba..fdd0aadbeb 100644
@@ -57,6 +57,13 @@ export SOAK_PROC=3             # -p option to fsstress
 export SOAK_STRESS=10000       # -n option to fsstress
 export SOAK_PASSES=-1          # count of repetitions of fsstress (while soaking)
 export EMAIL=root@localhost    # where auto-qa will send its status messages
+
+# For certain tests that run in tight loops, setting this variable allows the
+# test runner to specify exactly how long the test should continue looping.
+# This is independent of TIME_FACTOR.  Floating point numbers are allowed, and
+# the unit suffixes m(inutes), h(ours), d(ays), and w(eeks) are supported.
+export SOAK_DURATION=${SOAK_DURATION:=}
+
 export HOST_OPTIONS=${HOST_OPTIONS:=local.config}
 export CHECK_OPTIONS=${CHECK_OPTIONS:="-g auto"}
 export BENCH_PASSES=${BENCH_PASSES:=5}
common/fuzzy +6 -1
index 744d9ed65d..9c04bb5318 100644
@@ -1360,7 +1360,12 @@ _scratch_xfs_stress_scrub() {
 	fi
 
 	local start="$(date +%s)"
-	local end="$((start + (30 * TIME_FACTOR) ))"
+	local end
+	if [ -n "$SOAK_DURATION" ]; then
+		end="$((start + SOAK_DURATION))"
+	else
+		end="$((start + (30 * TIME_FACTOR) ))"
+	fi
 	local scrub_startat="$((start + scrub_delay))"
 	test "$scrub_startat" -gt "$((end - 10))" &&
 		scrub_startat="$((end - 10))"
common/report +1 -0
index be930e0b06..9bfa09ecce 100644
@@ -67,6 +67,7 @@ __generate_report_vars() {
 	REPORT_VARS["CPUS"]="$(getconf _NPROCESSORS_ONLN 2>/dev/null)"
 	REPORT_VARS["MEM_KB"]="$(grep MemTotal: /proc/meminfo | awk '{print $2}')"
 	REPORT_VARS["SWAP_KB"]="$(grep SwapTotal: /proc/meminfo | awk '{print $2}')"
+	test -n "$SOAK_DURATION" && REPORT_VARS["SOAK_DURATION"]="$SOAK_DURATION"
 
 	test -e /sys/devices/system/node/possible && \
 		REPORT_VARS["NUMA_NODES"]="$(cat /sys/devices/system/node/possible 2>/dev/null)"
ltp/fsstress.c +75 -3

132 lines not on this page · raw message

ltp/fsx.c +49 -1

89 lines not on this page · raw message

src/soak_duration.awk new file +23 -0
new file mode 100644
index 0000000000..6c38d09b39
@@ -0,0 +1,23 @@
+#!/usr/bin/awk
+#
+# Convert time interval specifications with suffixes to an integer number of
+# seconds.
+{
+	nr = $1;
+	if ($2 == "" || $2 ~ /s/)	# seconds
+		;
+	else if ($2 ~ /m/)		# minutes
+		nr *= 60;
+	else if ($2 ~ /h/)		# hours
+		nr *= 3600;
+	else if ($2 ~ /d/)		# days
+		nr *= 86400;
+	else if ($2 ~ /w/)		# weeks
+		nr *= 604800;
+	else {
+		printf("%s: unknown suffix\n", $2);
+		exit 1;
+	}
+
+	printf("%d\n", nr);
+}
tests/generic/476 +4 -1
index edb0be7b50..a162cda6b1 100755
@@ -33,7 +33,10 @@ _scratch_mount >> $seqres.full 2>&1
 
 nr_cpus=$((LOAD_FACTOR * 4))
 nr_ops=$((25000 * nr_cpus * TIME_FACTOR))
-$FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
+fsstress_args=(-w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus)
+test -n "$SOAK_DURATION" && fsstress_args+=(--duration="$SOAK_DURATION")
+
+$FSSTRESS_PROG $FSSTRESS_AVOID "${fsstress_args[@]}" >> $seqres.full
 
 # success, all done
 status=0
tests/generic/521 +1 -0
index cde9d44775..22dd31a8ec 100755
@@ -35,6 +35,7 @@ fsx_args+=(-r $min_dio_sz)
 fsx_args+=(-t $min_dio_sz)
 fsx_args+=(-w $min_dio_sz)
 fsx_args+=(-Z)
+test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION")
 
 run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'
 
tests/generic/522 +1 -0
index ae84fe04bb..f0cbcb245c 100755
@@ -29,6 +29,7 @@ fsx_args+=(-N $nr_ops)
 fsx_args+=(-p $((nr_ops / 100)))
 fsx_args+=(-o $op_sz)
 fsx_args+=(-l $file_sz)
+test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION")
 
 run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'
 
tests/generic/642 +1 -0
index c0e274d843..eba90903a3 100755
@@ -49,6 +49,7 @@ for verb in attr_remove removefattr; do
 done
 args+=('-f' "setfattr=20")
 args+=('-f' "attr_set=60")	# sets larger xattrs
+test -n "$DURATION" && args+=(--duration="$DURATION")
 
 $FSSTRESS_PROG "${args[@]}" $FSSTRESS_AVOID -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
 
#7 lore raw

Re: [PATCH v2 2/3] misc: add duration for long soak tests

Quoted 31 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Make it so that test runners can schedule long soak stress test programs for an exact number of seconds by setting the SOAK_DURATION config variable.

---
v2: fix commit message

---
 check                 |   14 +++++++++
 common/config         |    7 ++++
 common/fuzzy          |    7 ++++
 common/report         |    1 +
 ltp/fsstress.c        |   78 +++++++++++++++++++++++++++++++++++++++++++++++--
 ltp/fsx.c             |   50 +++++++++++++++++++++++++++++++
 src/soak_duration.awk |   23 ++++++++++++++
 tests/generic/476     |    5 +++
 tests/generic/521     |    1 +
 tests/generic/522     |    1 +
 tests/generic/642     |    1 +
 11 files changed, 182 insertions(+), 6 deletions(-)
 create mode 100644 src/soak_duration.awk
check +0 -0
index e32b70d301..1d78cf27f4 100755
@@ -366,6 +366,20 @@ if ! . ./common/rc; then
 	exit 1

check if ! . ./common/rc; then · line 376

Quoted 10 lines
check +8 -0
 fi
 
+# If the test config specified a soak test duration, see if there are any
+# unit suffixes that need converting to an integer seconds count.
+if [ -n "$SOAK_DURATION" ]; then
+	SOAK_DURATION="$(echo "$SOAK_DURATION" | \
+		sed -e 's/^\([.0-9]*\)\([a-z]\)*/\1 \2/g' | \
+		$AWK_PROG -f $here/src/soak_duration.awk)"
+	if [ $? -ne 0 ]; then
+		echo "$SOAK_DURATION"

echo SOAK_DURATION=$SOAK_DURATION ?

check if ! . ./common/rc; then · line 380

Quoted 4 lines
check +4 -0
+		status=1
+		exit 1
+	fi
+	export SOAK_DURATION

Is this necessary? As you've exported SOAK_DURATION in common/config:

export SOAK_DURATION=${SOAK_DURATION:=}

Quoted 10 lines
check +2 -0
+fi
+
 if [ -n "$subdir_xfile" ]; then
 	for d in $SRC_GROUPS $FSTYP; do
 		[ -f $SRC_DIR/$d/$subdir_xfile ] || continue
common/config +0 -0
index 6c8cb3a5ba..fdd0aadbeb 100644

common/config export SOAK_PROC=3 # -p option to fsstress · line 65

Quoted 9 lines
common/config +6 -0
@@ -57,6 +57,13 @@ export SOAK_PROC=3             # -p option to fsstress
 export SOAK_STRESS=10000       # -n option to fsstress
 export SOAK_PASSES=-1          # count of repetitions of fsstress (while soaking)
 export EMAIL=root@localhost    # where auto-qa will send its status messages
+
+# For certain tests that run in tight loops, setting this variable allows the
+# test runner to specify exactly how long the test should continue looping.
+# This is independent of TIME_FACTOR.  Floating point numbers are allowed, and
+# the unit suffixes m(inutes), h(ours), d(ays), and w(eeks) are supported.
+export SOAK_DURATION=${SOAK_DURATION:=}

Better to describe more about the SOAK_DURATION (and TIME/LOAD_FACTOR) in documentation (README?). To clarify the relationship of SOAK_DURATION with other related parameters, and how the SOAK_DURATION works/be used.

Quoted 66 lines
common/config +1 -0
+
 export HOST_OPTIONS=${HOST_OPTIONS:=local.config}
 export CHECK_OPTIONS=${CHECK_OPTIONS:="-g auto"}
 export BENCH_PASSES=${BENCH_PASSES:=5}
common/fuzzy +6 -1
index 744d9ed65d..9c04bb5318 100644
@@ -1360,7 +1360,12 @@ _scratch_xfs_stress_scrub() {
 	fi
 
 	local start="$(date +%s)"
-	local end="$((start + (30 * TIME_FACTOR) ))"
+	local end
+	if [ -n "$SOAK_DURATION" ]; then
+		end="$((start + SOAK_DURATION))"
+	else
+		end="$((start + (30 * TIME_FACTOR) ))"
+	fi
 	local scrub_startat="$((start + scrub_delay))"
 	test "$scrub_startat" -gt "$((end - 10))" &&
 		scrub_startat="$((end - 10))"
common/report +1 -0
index be930e0b06..9bfa09ecce 100644
@@ -67,6 +67,7 @@ __generate_report_vars() {
 	REPORT_VARS["CPUS"]="$(getconf _NPROCESSORS_ONLN 2>/dev/null)"
 	REPORT_VARS["MEM_KB"]="$(grep MemTotal: /proc/meminfo | awk '{print $2}')"
 	REPORT_VARS["SWAP_KB"]="$(grep SwapTotal: /proc/meminfo | awk '{print $2}')"
+	test -n "$SOAK_DURATION" && REPORT_VARS["SOAK_DURATION"]="$SOAK_DURATION"
 
 	test -e /sys/devices/system/node/possible && \
 		REPORT_VARS["NUMA_NODES"]="$(cat /sys/devices/system/node/possible 2>/dev/null)"
ltp/fsstress.c +17 -0
index e60f2da929..0dc6545448 100644
@@ -386,6 +386,8 @@ char		*execute_cmd = NULL;
 int		execute_freq = 1;
 struct print_string	flag_str = {0};
 
+struct timespec deadline = { 0 };
+
 void	add_to_flist(int, int, int, int);
 void	append_pathname(pathname_t *, char *);
 int	attr_list_path(pathname_t *, char *, const int);
@@ -459,6 +461,34 @@ void sg_handler(int signum)
 	}
 }
 
+bool
+keep_looping(int i, int loops)
+{
+	int ret;
+
+	if (deadline.tv_nsec) {
+		struct timespec now;
+
+		ret = clock_gettime(CLOCK_MONOTONIC, &now);
+		if (ret) {
+			perror("CLOCK_MONOTONIC");
+			return false;
+		}
+
+		return now.tv_sec <= deadline.tv_sec;

ltp/fsstress.c void sg_handler(int signum) · line 488

Quoted 10 lines
ltp/fsstress.c +10 -0
+	}
+
+	if (!loops)
+		return true;
+
+	return i < loops;
+}
+
+static struct option longopts[] = {
+	{"duration", optional_argument, 0, 256},

OK, we'd better to check those test cases use fsstress "-n" later, think about if some of them should be replaced by "--duration".

Thanks,
Zorro

Quoted 283 lines
ltp/fsstress.c +48 -3
+	{ }
+};
+
 int main(int argc, char **argv)
 {
 	char		buf[10];
@@ -478,13 +508,14 @@ int main(int argc, char **argv)
 	struct sigaction action;
 	int		loops = 1;
 	const char	*allopts = "cd:e:f:i:l:m:M:n:o:p:rRs:S:vVwx:X:zH";
+	long long	duration;
 
 	errrange = errtag = 0;
 	umask(0);
 	nops = sizeof(ops) / sizeof(ops[0]);
 	ops_end = &ops[nops];
 	myprog = argv[0];
-	while ((c = getopt(argc, argv, allopts)) != -1) {
+	while ((c = getopt_long(argc, argv, allopts, longopts, NULL)) != -1) {
 		switch (c) {
 		case 'c':
 			cleanup = 1;
@@ -579,6 +610,26 @@ int main(int argc, char **argv)
 		case 'X':
 			execute_freq = strtoul(optarg, NULL, 0);
 			break;
+		case 256:  /* --duration */
+			if (!optarg) {
+				fprintf(stderr, "Specify time with --duration=\n");
+				exit(87);
+			}
+			duration = strtoll(optarg, NULL, 0);
+			if (duration < 1) {
+				fprintf(stderr, "%lld: invalid duration\n", duration);
+				exit(88);
+			}
+
+			i = clock_gettime(CLOCK_MONOTONIC, &deadline);
+			if (i) {
+				perror("CLOCK_MONOTONIC");
+				exit(89);
+			}
+
+			deadline.tv_sec += duration;
+			deadline.tv_nsec = 1;
+			break;
 		case '?':
 			fprintf(stderr, "%s - invalid parameters\n",
 				myprog);
@@ -721,7 +772,7 @@ int main(int argc, char **argv)
 				}
 			}
 #endif
-			for (i = 0; !loops || (i < loops); i++)
+			for (i = 0; keep_looping(i, loops); i++)
 				doproc();
 #ifdef AIO
 			if(io_destroy(io_ctx) != 0) {
@@ -1121,6 +1172,26 @@ dirid_to_fent(int dirid)
 	return NULL;
 }
 
+bool
+keep_running(opnum_t opno, opnum_t operations)
+{
+	int ret;
+
+	if (deadline.tv_nsec) {
+		struct timespec now;
+
+		ret = clock_gettime(CLOCK_MONOTONIC, &now);
+		if (ret) {
+			perror("CLOCK_MONOTONIC");
+			return false;
+		}
+
+		return now.tv_sec <= deadline.tv_sec;
+	}
+
+	return opno < operations;
+}
+
 void
 doproc(void)
 {
@@ -1149,7 +1220,7 @@ doproc(void)
 	srandom(seed);
 	if (namerand)
 		namerand = random();
-	for (opno = 0; opno < operations; opno++) {
+	for (opno = 0; keep_running(opno, operations); opno++) {
 		if (execute_cmd && opno && opno % dividend == 0) {
 			if (verbose)
 				printf("%lld: execute command %s\n", opno,
@@ -1935,6 +2006,7 @@ usage(void)
 	printf("   -V               specifies verifiable logging mode (omitting inode numbers)\n");
 	printf("   -X ncmd          number of calls to the -x command (default 1)\n");
 	printf("   -H               prints usage and exits\n");
+	printf("   --duration=s     run for this many seconds\n");
 }
 
 void
ltp/fsx.c +49 -1
index ee4b8fe45d..761d5e467f 100644
@@ -193,6 +193,8 @@ int fsx_rw(int rw, int fd, char *buf, unsigned len, unsigned offset);
 #define fsxread(a,b,c,d)	fsx_rw(READ, a,b,c,d)
 #define fsxwrite(a,b,c,d)	fsx_rw(WRITE, a,b,c,d)
 
+struct timespec deadline;
+
 const char *replayops = NULL;
 const char *recordops = NULL;
 FILE *	fsxlogf = NULL;
@@ -2457,6 +2459,7 @@ usage(void)
         -Z: O_DIRECT (use -R, -W, -r and -w too)\n\
 	--replay-ops opsfile: replay ops from recorded .fsxops file\n\
 	--record-ops[=opsfile]: dump ops file also on success. optionally specify ops file name\n\
+	--duration=seconds: run for this many seconds\n\
 	fname: this filename is REQUIRED (no default)\n");
 	exit(90);
 }
@@ -2739,9 +2742,33 @@ __test_fallocate(int mode, const char *mode_str)
 #endif
 }
 
+bool
+keep_running(void)
+{
+	int ret;
+
+	if (deadline.tv_nsec) {
+		struct timespec now;
+
+		ret = clock_gettime(CLOCK_MONOTONIC, &now);
+		if (ret) {
+			perror("CLOCK_MONOTONIC");
+			return false;
+		}
+
+		return now.tv_sec <= deadline.tv_sec;
+	}
+
+	if (numops == -1)
+		return true;
+
+	return numops-- != 0;
+}
+
 static struct option longopts[] = {
 	{"replay-ops", required_argument, 0, 256},
 	{"record-ops", optional_argument, 0, 255},
+	{"duration", optional_argument, 0, 254},
 	{ }
 };
 
@@ -2753,6 +2780,7 @@ main(int argc, char **argv)
 	char logfile[PATH_MAX];
 	struct stat statbuf;
 	int o_flags = O_RDWR|O_CREAT|O_TRUNC;
+	long long duration;
 
 	logfile[0] = 0;
 	dname[0] = 0;
@@ -2950,6 +2978,26 @@ main(int argc, char **argv)
 			o_direct = O_DIRECT;
 			o_flags |= O_DIRECT;
 			break;
+		case 254:  /* --duration */
+			if (!optarg) {
+				fprintf(stderr, "Specify time with --duration=\n");
+				exit(87);
+			}
+			duration = strtoll(optarg, NULL, 0);
+			if (duration < 1) {
+				fprintf(stderr, "%lld: invalid duration\n", duration);
+				exit(88);
+			}
+
+			i = clock_gettime(CLOCK_MONOTONIC, &deadline);
+			if (i) {
+				perror("CLOCK_MONOTONIC");
+				exit(89);
+			}
+
+			deadline.tv_sec += duration;
+			deadline.tv_nsec = 1;
+			break;
 		case 255:  /* --record-ops */
 			if (optarg)
 				snprintf(opsfile, sizeof(opsfile), "%s", optarg);
@@ -3145,7 +3193,7 @@ main(int argc, char **argv)
 	if (xchg_range_calls)
 		xchg_range_calls = test_xchg_range();
 
-	while (numops == -1 || numops--)
+	while (keep_running())
 		if (!test())
 			break;
 
src/soak_duration.awk new file +23 -0
new file mode 100644
index 0000000000..6c38d09b39
@@ -0,0 +1,23 @@
+#!/usr/bin/awk
+#
+# Convert time interval specifications with suffixes to an integer number of
+# seconds.
+{
+	nr = $1;
+	if ($2 == "" || $2 ~ /s/)	# seconds
+		;
+	else if ($2 ~ /m/)		# minutes
+		nr *= 60;
+	else if ($2 ~ /h/)		# hours
+		nr *= 3600;
+	else if ($2 ~ /d/)		# days
+		nr *= 86400;
+	else if ($2 ~ /w/)		# weeks
+		nr *= 604800;
+	else {
+		printf("%s: unknown suffix\n", $2);
+		exit 1;
+	}
+
+	printf("%d\n", nr);
+}
tests/generic/476 +4 -1
index edb0be7b50..a162cda6b1 100755
@@ -33,7 +33,10 @@ _scratch_mount >> $seqres.full 2>&1
 
 nr_cpus=$((LOAD_FACTOR * 4))
 nr_ops=$((25000 * nr_cpus * TIME_FACTOR))
-$FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
+fsstress_args=(-w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus)
+test -n "$SOAK_DURATION" && fsstress_args+=(--duration="$SOAK_DURATION")
+
+$FSSTRESS_PROG $FSSTRESS_AVOID "${fsstress_args[@]}" >> $seqres.full
 
 # success, all done
 status=0
tests/generic/521 +1 -0
index cde9d44775..22dd31a8ec 100755
@@ -35,6 +35,7 @@ fsx_args+=(-r $min_dio_sz)
 fsx_args+=(-t $min_dio_sz)
 fsx_args+=(-w $min_dio_sz)
 fsx_args+=(-Z)
+test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION")
 
 run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'
 
tests/generic/522 +1 -0
index ae84fe04bb..f0cbcb245c 100755
@@ -29,6 +29,7 @@ fsx_args+=(-N $nr_ops)
 fsx_args+=(-p $((nr_ops / 100)))
 fsx_args+=(-o $op_sz)
 fsx_args+=(-l $file_sz)
+test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION")
 
 run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'
 
tests/generic/642 +1 -0
index c0e274d843..eba90903a3 100755
@@ -49,6 +49,7 @@ for verb in attr_remove removefattr; do
 done
 args+=('-f' "setfattr=20")
 args+=('-f' "attr_set=60")	# sets larger xattrs
+test -n "$DURATION" && args+=(--duration="$DURATION")
 
 $FSSTRESS_PROG "${args[@]}" $FSSTRESS_AVOID -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
 
#8 lore raw

Re: [PATCH v2 2/3] misc: add duration for long soak tests

Quoted 34 lines · Zorro Lang
Quoted 33 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Make it so that test runners can schedule long soak stress test programs for an exact number of seconds by setting the SOAK_DURATION config variable.

---
v2: fix commit message

---
 check                 |   14 +++++++++
 common/config         |    7 ++++
 common/fuzzy          |    7 ++++
 common/report         |    1 +
 ltp/fsstress.c        |   78 +++++++++++++++++++++++++++++++++++++++++++++++--
 ltp/fsx.c             |   50 +++++++++++++++++++++++++++++++
 src/soak_duration.awk |   23 ++++++++++++++
 tests/generic/476     |    5 +++
 tests/generic/521     |    1 +
 tests/generic/522     |    1 +
 tests/generic/642     |    1 +
 11 files changed, 182 insertions(+), 6 deletions(-)
 create mode 100644 src/soak_duration.awk
check +0 -0
index e32b70d301..1d78cf27f4 100755
@@ -366,6 +366,20 @@ if ! . ./common/rc; then
 	exit 1
 fi
 
Quoted 10 lines

check if ! . ./common/rc; then · line 376

Quoted 8 lines
check +8 -0
+# If the test config specified a soak test duration, see if there are any
+# unit suffixes that need converting to an integer seconds count.
+if [ -n "$SOAK_DURATION" ]; then
+	SOAK_DURATION="$(echo "$SOAK_DURATION" | \
+		sed -e 's/^\([.0-9]*\)\([a-z]\)*/\1 \2/g' | \
+		$AWK_PROG -f $here/src/soak_duration.awk)"
+	if [ $? -ne 0 ]; then
+		echo "$SOAK_DURATION"

echo SOAK_DURATION=$SOAK_DURATION ?

Hmm. On nonzero return, the awk script already specified the reason that the input was rejected, so I think this line can go away.

Quoted 8 lines

check if ! . ./common/rc; then · line 380

Quoted 4 lines
check +4 -0
+		status=1
+		exit 1
+	fi
+	export SOAK_DURATION

Is this necessary? As you've exported SOAK_DURATION in common/config:

export SOAK_DURATION=${SOAK_DURATION:=}

You're correct, it does not need to be re-exported.

Quoted 15 lines
Quoted 15 lines
check +2 -0
+fi
+
 if [ -n "$subdir_xfile" ]; then
 	for d in $SRC_GROUPS $FSTYP; do
 		[ -f $SRC_DIR/$d/$subdir_xfile ] || continue
common/config +2 -0
index 6c8cb3a5ba..fdd0aadbeb 100644
@@ -57,6 +57,13 @@ export SOAK_PROC=3             # -p option to fsstress
 export SOAK_STRESS=10000       # -n option to fsstress
 export SOAK_PASSES=-1          # count of repetitions of fsstress (while soaking)
 export EMAIL=root@localhost    # where auto-qa will send its status messages
+
+# For certain tests that run in tight loops, setting this variable allows the
Quoted 8 lines

common/config export SOAK_PROC=3 # -p option to fsstress · line 65

Quoted 4 lines
common/config +4 -0
+# test runner to specify exactly how long the test should continue looping.
+# This is independent of TIME_FACTOR.  Floating point numbers are allowed, and
+# the unit suffixes m(inutes), h(ours), d(ays), and w(eeks) are supported.
+export SOAK_DURATION=${SOAK_DURATION:=}

Better to describe more about the SOAK_DURATION (and TIME/LOAD_FACTOR) in documentation (README?). To clarify the relationship of SOAK_DURATION with other related parameters, and how the SOAK_DURATION works/be used.

TIME_FACTOR is not documented in the README. I'll add another patch to do that...

Quoted 71 lines
Quoted 71 lines
common/config +1 -0
+
 export HOST_OPTIONS=${HOST_OPTIONS:=local.config}
 export CHECK_OPTIONS=${CHECK_OPTIONS:="-g auto"}
 export BENCH_PASSES=${BENCH_PASSES:=5}
common/fuzzy +6 -1
index 744d9ed65d..9c04bb5318 100644
@@ -1360,7 +1360,12 @@ _scratch_xfs_stress_scrub() {
 	fi
 
 	local start="$(date +%s)"
-	local end="$((start + (30 * TIME_FACTOR) ))"
+	local end
+	if [ -n "$SOAK_DURATION" ]; then
+		end="$((start + SOAK_DURATION))"
+	else
+		end="$((start + (30 * TIME_FACTOR) ))"
+	fi
 	local scrub_startat="$((start + scrub_delay))"
 	test "$scrub_startat" -gt "$((end - 10))" &&
 		scrub_startat="$((end - 10))"
common/report +1 -0
index be930e0b06..9bfa09ecce 100644
@@ -67,6 +67,7 @@ __generate_report_vars() {
 	REPORT_VARS["CPUS"]="$(getconf _NPROCESSORS_ONLN 2>/dev/null)"
 	REPORT_VARS["MEM_KB"]="$(grep MemTotal: /proc/meminfo | awk '{print $2}')"
 	REPORT_VARS["SWAP_KB"]="$(grep SwapTotal: /proc/meminfo | awk '{print $2}')"
+	test -n "$SOAK_DURATION" && REPORT_VARS["SOAK_DURATION"]="$SOAK_DURATION"
 
 	test -e /sys/devices/system/node/possible && \
 		REPORT_VARS["NUMA_NODES"]="$(cat /sys/devices/system/node/possible 2>/dev/null)"
ltp/fsstress.c +22 -0
index e60f2da929..0dc6545448 100644
@@ -386,6 +386,8 @@ char		*execute_cmd = NULL;
 int		execute_freq = 1;
 struct print_string	flag_str = {0};
 
+struct timespec deadline = { 0 };
+
 void	add_to_flist(int, int, int, int);
 void	append_pathname(pathname_t *, char *);
 int	attr_list_path(pathname_t *, char *, const int);
@@ -459,6 +461,34 @@ void sg_handler(int signum)
 	}
 }
 
+bool
+keep_looping(int i, int loops)
+{
+	int ret;
+
+	if (deadline.tv_nsec) {
+		struct timespec now;
+
+		ret = clock_gettime(CLOCK_MONOTONIC, &now);
+		if (ret) {
+			perror("CLOCK_MONOTONIC");
+			return false;
+		}
+
+		return now.tv_sec <= deadline.tv_sec;
+	}
+
+	if (!loops)
+		return true;
+
Quoted 8 lines

ltp/fsstress.c void sg_handler(int signum) · line 488

Quoted 5 lines
ltp/fsstress.c +5 -0
+	return i < loops;
+}
+
+static struct option longopts[] = {
+	{"duration", optional_argument, 0, 256},

OK, we'd better to check those test cases use fsstress "-n" later, think about if some of them should be replaced by "--duration".

<nod> I'll update the fsstress/fsx --help screens to state that --duration overrides -n/-N.

--D

Quoted 287 lines

Thanks,
Zorro

Quoted 283 lines
ltp/fsstress.c +48 -3
+	{ }
+};
+
 int main(int argc, char **argv)
 {
 	char		buf[10];
@@ -478,13 +508,14 @@ int main(int argc, char **argv)
 	struct sigaction action;
 	int		loops = 1;
 	const char	*allopts = "cd:e:f:i:l:m:M:n:o:p:rRs:S:vVwx:X:zH";
+	long long	duration;
 
 	errrange = errtag = 0;
 	umask(0);
 	nops = sizeof(ops) / sizeof(ops[0]);
 	ops_end = &ops[nops];
 	myprog = argv[0];
-	while ((c = getopt(argc, argv, allopts)) != -1) {
+	while ((c = getopt_long(argc, argv, allopts, longopts, NULL)) != -1) {
 		switch (c) {
 		case 'c':
 			cleanup = 1;
@@ -579,6 +610,26 @@ int main(int argc, char **argv)
 		case 'X':
 			execute_freq = strtoul(optarg, NULL, 0);
 			break;
+		case 256:  /* --duration */
+			if (!optarg) {
+				fprintf(stderr, "Specify time with --duration=\n");
+				exit(87);
+			}
+			duration = strtoll(optarg, NULL, 0);
+			if (duration < 1) {
+				fprintf(stderr, "%lld: invalid duration\n", duration);
+				exit(88);
+			}
+
+			i = clock_gettime(CLOCK_MONOTONIC, &deadline);
+			if (i) {
+				perror("CLOCK_MONOTONIC");
+				exit(89);
+			}
+
+			deadline.tv_sec += duration;
+			deadline.tv_nsec = 1;
+			break;
 		case '?':
 			fprintf(stderr, "%s - invalid parameters\n",
 				myprog);
@@ -721,7 +772,7 @@ int main(int argc, char **argv)
 				}
 			}
 #endif
-			for (i = 0; !loops || (i < loops); i++)
+			for (i = 0; keep_looping(i, loops); i++)
 				doproc();
 #ifdef AIO
 			if(io_destroy(io_ctx) != 0) {
@@ -1121,6 +1172,26 @@ dirid_to_fent(int dirid)
 	return NULL;
 }
 
+bool
+keep_running(opnum_t opno, opnum_t operations)
+{
+	int ret;
+
+	if (deadline.tv_nsec) {
+		struct timespec now;
+
+		ret = clock_gettime(CLOCK_MONOTONIC, &now);
+		if (ret) {
+			perror("CLOCK_MONOTONIC");
+			return false;
+		}
+
+		return now.tv_sec <= deadline.tv_sec;
+	}
+
+	return opno < operations;
+}
+
 void
 doproc(void)
 {
@@ -1149,7 +1220,7 @@ doproc(void)
 	srandom(seed);
 	if (namerand)
 		namerand = random();
-	for (opno = 0; opno < operations; opno++) {
+	for (opno = 0; keep_running(opno, operations); opno++) {
 		if (execute_cmd && opno && opno % dividend == 0) {
 			if (verbose)
 				printf("%lld: execute command %s\n", opno,
@@ -1935,6 +2006,7 @@ usage(void)
 	printf("   -V               specifies verifiable logging mode (omitting inode numbers)\n");
 	printf("   -X ncmd          number of calls to the -x command (default 1)\n");
 	printf("   -H               prints usage and exits\n");
+	printf("   --duration=s     run for this many seconds\n");
 }
 
 void
ltp/fsx.c +49 -1
index ee4b8fe45d..761d5e467f 100644
@@ -193,6 +193,8 @@ int fsx_rw(int rw, int fd, char *buf, unsigned len, unsigned offset);
 #define fsxread(a,b,c,d)	fsx_rw(READ, a,b,c,d)
 #define fsxwrite(a,b,c,d)	fsx_rw(WRITE, a,b,c,d)
 
+struct timespec deadline;
+
 const char *replayops = NULL;
 const char *recordops = NULL;
 FILE *	fsxlogf = NULL;
@@ -2457,6 +2459,7 @@ usage(void)
         -Z: O_DIRECT (use -R, -W, -r and -w too)\n\
 	--replay-ops opsfile: replay ops from recorded .fsxops file\n\
 	--record-ops[=opsfile]: dump ops file also on success. optionally specify ops file name\n\
+	--duration=seconds: run for this many seconds\n\
 	fname: this filename is REQUIRED (no default)\n");
 	exit(90);
 }
@@ -2739,9 +2742,33 @@ __test_fallocate(int mode, const char *mode_str)
 #endif
 }
 
+bool
+keep_running(void)
+{
+	int ret;
+
+	if (deadline.tv_nsec) {
+		struct timespec now;
+
+		ret = clock_gettime(CLOCK_MONOTONIC, &now);
+		if (ret) {
+			perror("CLOCK_MONOTONIC");
+			return false;
+		}
+
+		return now.tv_sec <= deadline.tv_sec;
+	}
+
+	if (numops == -1)
+		return true;
+
+	return numops-- != 0;
+}
+
 static struct option longopts[] = {
 	{"replay-ops", required_argument, 0, 256},
 	{"record-ops", optional_argument, 0, 255},
+	{"duration", optional_argument, 0, 254},
 	{ }
 };
 
@@ -2753,6 +2780,7 @@ main(int argc, char **argv)
 	char logfile[PATH_MAX];
 	struct stat statbuf;
 	int o_flags = O_RDWR|O_CREAT|O_TRUNC;
+	long long duration;
 
 	logfile[0] = 0;
 	dname[0] = 0;
@@ -2950,6 +2978,26 @@ main(int argc, char **argv)
 			o_direct = O_DIRECT;
 			o_flags |= O_DIRECT;
 			break;
+		case 254:  /* --duration */
+			if (!optarg) {
+				fprintf(stderr, "Specify time with --duration=\n");
+				exit(87);
+			}
+			duration = strtoll(optarg, NULL, 0);
+			if (duration < 1) {
+				fprintf(stderr, "%lld: invalid duration\n", duration);
+				exit(88);
+			}
+
+			i = clock_gettime(CLOCK_MONOTONIC, &deadline);
+			if (i) {
+				perror("CLOCK_MONOTONIC");
+				exit(89);
+			}
+
+			deadline.tv_sec += duration;
+			deadline.tv_nsec = 1;
+			break;
 		case 255:  /* --record-ops */
 			if (optarg)
 				snprintf(opsfile, sizeof(opsfile), "%s", optarg);
@@ -3145,7 +3193,7 @@ main(int argc, char **argv)
 	if (xchg_range_calls)
 		xchg_range_calls = test_xchg_range();
 
-	while (numops == -1 || numops--)
+	while (keep_running())
 		if (!test())
 			break;
 
src/soak_duration.awk new file +23 -0
new file mode 100644
index 0000000000..6c38d09b39
@@ -0,0 +1,23 @@
+#!/usr/bin/awk
+#
+# Convert time interval specifications with suffixes to an integer number of
+# seconds.
+{
+	nr = $1;
+	if ($2 == "" || $2 ~ /s/)	# seconds
+		;
+	else if ($2 ~ /m/)		# minutes
+		nr *= 60;
+	else if ($2 ~ /h/)		# hours
+		nr *= 3600;
+	else if ($2 ~ /d/)		# days
+		nr *= 86400;
+	else if ($2 ~ /w/)		# weeks
+		nr *= 604800;
+	else {
+		printf("%s: unknown suffix\n", $2);
+		exit 1;
+	}
+
+	printf("%d\n", nr);
+}
tests/generic/476 +4 -1
index edb0be7b50..a162cda6b1 100755
@@ -33,7 +33,10 @@ _scratch_mount >> $seqres.full 2>&1
 
 nr_cpus=$((LOAD_FACTOR * 4))
 nr_ops=$((25000 * nr_cpus * TIME_FACTOR))
-$FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
+fsstress_args=(-w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus)
+test -n "$SOAK_DURATION" && fsstress_args+=(--duration="$SOAK_DURATION")
+
+$FSSTRESS_PROG $FSSTRESS_AVOID "${fsstress_args[@]}" >> $seqres.full
 
 # success, all done
 status=0
tests/generic/521 +1 -0
index cde9d44775..22dd31a8ec 100755
@@ -35,6 +35,7 @@ fsx_args+=(-r $min_dio_sz)
 fsx_args+=(-t $min_dio_sz)
 fsx_args+=(-w $min_dio_sz)
 fsx_args+=(-Z)
+test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION")
 
 run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'
 
tests/generic/522 +1 -0
index ae84fe04bb..f0cbcb245c 100755
@@ -29,6 +29,7 @@ fsx_args+=(-N $nr_ops)
 fsx_args+=(-p $((nr_ops / 100)))
 fsx_args+=(-o $op_sz)
 fsx_args+=(-l $file_sz)
+test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION")
 
 run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'
 
tests/generic/642 +1 -0
index c0e274d843..eba90903a3 100755
@@ -49,6 +49,7 @@ for verb in attr_remove removefattr; do
 done
 args+=('-f' "setfattr=20")
 args+=('-f' "attr_set=60")	# sets larger xattrs
+test -n "$DURATION" && args+=(--duration="$DURATION")
 
 $FSSTRESS_PROG "${args[@]}" $FSSTRESS_AVOID -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
 
#9 lore raw

PATCH 3/3 misc: add duration for recovery loop tests

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Make it so that we can run recovery loop tests for an exact number of seconds.

---
 common/rc         |   34 ++++++++++++++++++++++++++++++++++
 tests/generic/019 |    1 +
 tests/generic/388 |    2 +-
 tests/generic/475 |    2 +-
 tests/generic/482 |    5 +++++
 tests/generic/648 |    8 ++++----
 6 files changed, 46 insertions(+), 6 deletions(-)
common/rc +34 -0
index e89b0a3794..090f3d4938 100644
@@ -5078,6 +5078,40 @@ _save_coredump()
 	$COREDUMP_COMPRESSOR -f "$out_file"
 }
 
+# Decide if a soak test should continue looping.  The sole parameter is the
+# number of soak loops that the test wants to run by default.  The actual
+# loop iteration number is stored in SOAK_LOOPIDX until the loop completes.
+#
+# If the test runner set a SOAK_DURATION value, this predicate will keep
+# looping until it has run for at least that long.
+_soak_loop_running() {
+	local max_soak_loops="$1"
+
+	test -z "$SOAK_LOOPIDX" && SOAK_LOOPIDX=1
+
+	if [ -n "$SOAK_DURATION" ]; then
+		if [ -z "$SOAK_DEADLINE" ]; then
+			SOAK_DEADLINE="$(( $(date +%s) + SOAK_DURATION))"
+		fi
+
+		local now="$(date +%s)"
+		if [ "$now" -gt "$SOAK_DEADLINE" ]; then
+			unset SOAK_DEADLINE
+			unset SOAK_LOOPIDX
+			return 1
+		fi
+		SOAK_LOOPIDX=$((SOAK_LOOPIDX + 1))
+		return 0
+	fi
+
+	if [ "$SOAK_LOOPIDX" -gt "$max_soak_loops" ]; then
+		unset SOAK_LOOPIDX
+		return 1
+	fi
+	SOAK_LOOPIDX=$((SOAK_LOOPIDX + 1))
+	return 0
+}
+
 init_rc
 
 ################################################################################
tests/generic/019 +1 -0
index b68dd90c0d..b81c1d17ba 100755
@@ -30,6 +30,7 @@ _cleanup()
 }
 
 RUN_TIME=$((20+10*$TIME_FACTOR))
+test -n "$SOAK_DURATION" && RUN_TIME="$SOAK_DURATION"
 NUM_JOBS=$((4*LOAD_FACTOR))
 BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
 FILE_SIZE=$((BLK_DEV_SIZE * 512))
tests/generic/388 +1 -1
index 9cd737e8eb..4a5be6698c 100755
@@ -42,7 +42,7 @@ _scratch_mkfs >> $seqres.full 2>&1
 _require_metadata_journaling $SCRATCH_DEV
 _scratch_mount
 
-for i in $(seq 1 $((50 * TIME_FACTOR)) ); do
+while _soak_loop_running $((50 * TIME_FACTOR)); do
 	($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p 4 >> $seqres.full &) \
 		> /dev/null 2>&1
 
tests/generic/475 +1 -1
index c426402ede..0cbf5131c2 100755
@@ -41,7 +41,7 @@ _require_metadata_journaling $SCRATCH_DEV
 _dmerror_init
 _dmerror_mount
 
-for i in $(seq 1 $((50 * TIME_FACTOR)) ); do
+while _soak_loop_running $((50 * TIME_FACTOR)); do
 	($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4)) >> $seqres.full &) \
 		> /dev/null 2>&1
 
tests/generic/482 +5 -0
index 28c83a232e..b980826b14 100755
@@ -62,8 +62,13 @@ nr_cpus=$("$here/src/feature" -o)
 if [ $nr_cpus -gt 8 ]; then
 	nr_cpus=8
 fi
+
 fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 512 -p $nr_cpus \
 		$FSSTRESS_AVOID)
+
+# XXX dm-logwrites pins kernel memory for every write!
+# test -n "$SOAK_DURATION" && fsstress_args="$fsstress_args --duration=$SOAK_DURATION"
+
 devsize=$((1024*1024*200 / 512))	# 200m phys/virt size
 csize=$((1024*64 / 512))		# 64k cluster size
 lowspace=$((1024*1024 / 512))		# 1m low space threshold
tests/generic/648 +4 -4
index d7bf5697e1..3b3544ff49 100755
@@ -74,14 +74,14 @@ snap_loop_fs() {
 
 fsstress=($FSSTRESS_PROG $FSSTRESS_AVOID -d "$loopmnt" -n 999999 -p "$((LOAD_FACTOR * 4))")
 
-for i in $(seq 1 $((25 * TIME_FACTOR)) ); do
+while _soak_loop_running $((25 * TIME_FACTOR)); do
 	touch $scratch_aliveflag
 	snap_loop_fs >> $seqres.full 2>&1 &
 
 	if ! _mount $loopimg $loopmnt -o loop; then
 		rm -f $scratch_aliveflag
 		_metadump_dev $loopimg $seqres.loop.$i.md
-		_fail "iteration $i loopimg mount failed"
+		_fail "iteration $SOAK_LOOPIDX loopimg mount failed"
 		break
 	fi
 
@@ -126,12 +126,12 @@ for i in $(seq 1 $((25 * TIME_FACTOR)) ); do
 	done
 	if [ $is_unmounted -ne 0 ];then
 		cat $tmp.unmount.err
-		_fail "iteration $i scratch unmount failed"
+		_fail "iteration $SOAK_LOOPIDX scratch unmount failed"
 	fi
 	_dmerror_load_working_table
 	if ! _dmerror_mount; then
 		_metadump_dev $DMERROR_DEV $seqres.scratch.$i.md
-		_fail "iteration $i scratch mount failed"
+		_fail "iteration $SOAK_LOOPIDX scratch mount failed"
 	fi
 done
 
#10 lore raw

Re: [PATCH 3/3] misc: add duration for recovery loop tests

Quoted 104 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Make it so that we can run recovery loop tests for an exact number of seconds.

---
 common/rc         |   34 ++++++++++++++++++++++++++++++++++
 tests/generic/019 |    1 +
 tests/generic/388 |    2 +-
 tests/generic/475 |    2 +-
 tests/generic/482 |    5 +++++
 tests/generic/648 |    8 ++++----
 6 files changed, 46 insertions(+), 6 deletions(-)
common/rc +34 -0
index e89b0a3794..090f3d4938 100644
@@ -5078,6 +5078,40 @@ _save_coredump()
 	$COREDUMP_COMPRESSOR -f "$out_file"
 }
 
+# Decide if a soak test should continue looping.  The sole parameter is the
+# number of soak loops that the test wants to run by default.  The actual
+# loop iteration number is stored in SOAK_LOOPIDX until the loop completes.
+#
+# If the test runner set a SOAK_DURATION value, this predicate will keep
+# looping until it has run for at least that long.
+_soak_loop_running() {
+	local max_soak_loops="$1"
+
+	test -z "$SOAK_LOOPIDX" && SOAK_LOOPIDX=1
+
+	if [ -n "$SOAK_DURATION" ]; then
+		if [ -z "$SOAK_DEADLINE" ]; then
+			SOAK_DEADLINE="$(( $(date +%s) + SOAK_DURATION))"
+		fi
+
+		local now="$(date +%s)"
+		if [ "$now" -gt "$SOAK_DEADLINE" ]; then
+			unset SOAK_DEADLINE
+			unset SOAK_LOOPIDX
+			return 1
+		fi
+		SOAK_LOOPIDX=$((SOAK_LOOPIDX + 1))
+		return 0
+	fi
+
+	if [ "$SOAK_LOOPIDX" -gt "$max_soak_loops" ]; then
+		unset SOAK_LOOPIDX
+		return 1
+	fi
+	SOAK_LOOPIDX=$((SOAK_LOOPIDX + 1))
+	return 0
+}
+
 init_rc
 
 ################################################################################
tests/generic/019 +1 -0
index b68dd90c0d..b81c1d17ba 100755
@@ -30,6 +30,7 @@ _cleanup()
 }
 
 RUN_TIME=$((20+10*$TIME_FACTOR))
+test -n "$SOAK_DURATION" && RUN_TIME="$SOAK_DURATION"
 NUM_JOBS=$((4*LOAD_FACTOR))
 BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
 FILE_SIZE=$((BLK_DEV_SIZE * 512))
tests/generic/388 +1 -1
index 9cd737e8eb..4a5be6698c 100755
@@ -42,7 +42,7 @@ _scratch_mkfs >> $seqres.full 2>&1
 _require_metadata_journaling $SCRATCH_DEV
 _scratch_mount
 
-for i in $(seq 1 $((50 * TIME_FACTOR)) ); do
+while _soak_loop_running $((50 * TIME_FACTOR)); do
 	($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p 4 >> $seqres.full &) \
 		> /dev/null 2>&1
 
tests/generic/475 +1 -1
index c426402ede..0cbf5131c2 100755
@@ -41,7 +41,7 @@ _require_metadata_journaling $SCRATCH_DEV
 _dmerror_init
 _dmerror_mount
 
-for i in $(seq 1 $((50 * TIME_FACTOR)) ); do
+while _soak_loop_running $((50 * TIME_FACTOR)); do
 	($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4)) >> $seqres.full &) \
 		> /dev/null 2>&1
 
tests/generic/482 +0 -0
index 28c83a232e..b980826b14 100755

tests/generic/482 nr_cpus=$("$here/src/feature" -o) · line 70

Quoted 9 lines
tests/generic/482 +4 -0
@@ -62,8 +62,13 @@ nr_cpus=$("$here/src/feature" -o)
 if [ $nr_cpus -gt 8 ]; then
 	nr_cpus=8
 fi
+
 fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 512 -p $nr_cpus \
 		$FSSTRESS_AVOID)
+
+# XXX dm-logwrites pins kernel memory for every write!
+# test -n "$SOAK_DURATION" && fsstress_args="$fsstress_args --duration=$SOAK_DURATION"

Do you expect the second comment is a comment?

Others looks good to me. I'll test V2 and merge it if no regression from it.

Thanks,
Zorro

#11 lore raw

Re: [PATCH 3/3] misc: add duration for recovery loop tests

Quoted 100 lines · Zorro Lang
Quoted 99 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Make it so that we can run recovery loop tests for an exact number of seconds.

---
 common/rc         |   34 ++++++++++++++++++++++++++++++++++
 tests/generic/019 |    1 +
 tests/generic/388 |    2 +-
 tests/generic/475 |    2 +-
 tests/generic/482 |    5 +++++
 tests/generic/648 |    8 ++++----
 6 files changed, 46 insertions(+), 6 deletions(-)
common/rc +34 -0
index e89b0a3794..090f3d4938 100644
@@ -5078,6 +5078,40 @@ _save_coredump()
 	$COREDUMP_COMPRESSOR -f "$out_file"
 }
 
+# Decide if a soak test should continue looping.  The sole parameter is the
+# number of soak loops that the test wants to run by default.  The actual
+# loop iteration number is stored in SOAK_LOOPIDX until the loop completes.
+#
+# If the test runner set a SOAK_DURATION value, this predicate will keep
+# looping until it has run for at least that long.
+_soak_loop_running() {
+	local max_soak_loops="$1"
+
+	test -z "$SOAK_LOOPIDX" && SOAK_LOOPIDX=1
+
+	if [ -n "$SOAK_DURATION" ]; then
+		if [ -z "$SOAK_DEADLINE" ]; then
+			SOAK_DEADLINE="$(( $(date +%s) + SOAK_DURATION))"
+		fi
+
+		local now="$(date +%s)"
+		if [ "$now" -gt "$SOAK_DEADLINE" ]; then
+			unset SOAK_DEADLINE
+			unset SOAK_LOOPIDX
+			return 1
+		fi
+		SOAK_LOOPIDX=$((SOAK_LOOPIDX + 1))
+		return 0
+	fi
+
+	if [ "$SOAK_LOOPIDX" -gt "$max_soak_loops" ]; then
+		unset SOAK_LOOPIDX
+		return 1
+	fi
+	SOAK_LOOPIDX=$((SOAK_LOOPIDX + 1))
+	return 0
+}
+
 init_rc
 
 ################################################################################
tests/generic/019 +1 -0
index b68dd90c0d..b81c1d17ba 100755
@@ -30,6 +30,7 @@ _cleanup()
 }
 
 RUN_TIME=$((20+10*$TIME_FACTOR))
+test -n "$SOAK_DURATION" && RUN_TIME="$SOAK_DURATION"
 NUM_JOBS=$((4*LOAD_FACTOR))
 BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
 FILE_SIZE=$((BLK_DEV_SIZE * 512))
tests/generic/388 +1 -1
index 9cd737e8eb..4a5be6698c 100755
@@ -42,7 +42,7 @@ _scratch_mkfs >> $seqres.full 2>&1
 _require_metadata_journaling $SCRATCH_DEV
 _scratch_mount
 
-for i in $(seq 1 $((50 * TIME_FACTOR)) ); do
+while _soak_loop_running $((50 * TIME_FACTOR)); do
 	($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p 4 >> $seqres.full &) \
 		> /dev/null 2>&1
 
tests/generic/475 +1 -1
index c426402ede..0cbf5131c2 100755
@@ -41,7 +41,7 @@ _require_metadata_journaling $SCRATCH_DEV
 _dmerror_init
 _dmerror_mount
 
-for i in $(seq 1 $((50 * TIME_FACTOR)) ); do
+while _soak_loop_running $((50 * TIME_FACTOR)); do
 	($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4)) >> $seqres.full &) \
 		> /dev/null 2>&1
 
Quoted 16 lines
Quoted 5 lines
tests/generic/482 +0 -0
index 28c83a232e..b980826b14 100755

tests/generic/482 nr_cpus=$("$here/src/feature" -o) · line 70

Quoted 9 lines
tests/generic/482 +4 -0
@@ -62,8 +62,13 @@ nr_cpus=$("$here/src/feature" -o)
 if [ $nr_cpus -gt 8 ]; then
 	nr_cpus=8
 fi
+
 fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 512 -p $nr_cpus \
 		$FSSTRESS_AVOID)
+
+# XXX dm-logwrites pins kernel memory for every write!
+# test -n "$SOAK_DURATION" && fsstress_args="$fsstress_args --duration=$SOAK_DURATION"

Do you expect the second comment is a comment?

Actually... I'd like to withdraw this chunk so that people aren't tempted to uncomment it. I had previously wanted generic/482 to be a SOAK_DURATION controllable test, but all that does is increase kernel memory usage until the VM OOMs, and it OOMs pretty quickly if you don't provision it with the same amount of RAM as the fs is writing to disk.

I'll repost this without the changes to g/482.

--D

Quoted 5 lines

Others looks good to me. I'll test V2 and merge it if no regression from it.

Thanks,
Zorro

Reviewed-by on the v1 series
#12 lore raw
In reply to Darrick J. Wong
Quoted 58 lines · Darrick J. Wong

Hi all,

One of the things that I do as a maintainer is to designate a handful of VMs to run fstests for unusually long periods of time. This practice I call long term soak testing. There are actually three separate fleets for this -- one runs alongside the nightly builds, one runs alongside weekly rebases, and the last one runs stable releases.

My interactions with all three fleets is pretty much the same -- load current builds of software, and try to run the exerciser tests for a duration of time -- 12 hours, 6.5 days, 30 days, etc. TIME_FACTOR does not work well for this usage model, because it is difficult to guess the correct time factor given that the VMs are hetergeneous and the IO completion rate is not perfectly predictable.

Worse yet, if you want to run (say) all the recoveryloop tests on one VM (because recoveryloop is prone to crashing), it's impossible to set a TIME_FACTOR so that each loop test gets equal runtime. That can be hacked around with config sections, but that doesn't solve the first problem.

This series introduces a new configuration variable, SOAK_DURATION, that allows test runners to control directly various long soak and looping recovery tests. This is intended to be an alternative to TIME_FACTOR, since that variable usually adjusts operation counts, which are proportional to runtime but otherwise not a direct measure of time.

With this override in place, I can configure the long soak fleet to run for exactly as long as I want them to, and they actually hit the time budget targets. The recoveryloop fleet now divides looping-test time equally among the four that are in that group so that they all get ~3 hours of coverage every night.

There are more tests that could use this than I actually modified here, but I've done enough to show this off as a proof of concept.

If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=soak-duration

---
 check                 |   14 +++++++++
 common/config         |    7 ++++
 common/fuzzy          |    7 ++++
 common/rc             |   34 +++++++++++++++++++++
 common/report         |    1 +
 ltp/fsstress.c        |   78 +++++++++++++++++++++++++++++++++++++++++++++++--
 ltp/fsx.c             |   50 +++++++++++++++++++++++++++++++
 src/soak_duration.awk |   23 ++++++++++++++
 tests/generic/019     |    1 +
 tests/generic/388     |    2 +
 tests/generic/475     |    2 +
Quoted 9 lines
 tests/generic/476     |    7 +++-
 tests/generic/482     |    5 +++
 tests/generic/521     |    1 +
 tests/generic/522     |    1 +
 tests/generic/642     |    1 +
 tests/generic/648     |    8 +++--
 17 files changed, 229 insertions(+), 13 deletions(-)
 create mode 100644 src/soak_duration.awk

The set looks good to me (the second commit has different var name, but fine by me)

Signature · 1 line
- Andrey
#13 lore raw
Quoted 69 lines · Andrey Albershteyn
Quoted 58 lines · Darrick J. Wong

Hi all,

One of the things that I do as a maintainer is to designate a handful of VMs to run fstests for unusually long periods of time. This practice I call long term soak testing. There are actually three separate fleets for this -- one runs alongside the nightly builds, one runs alongside weekly rebases, and the last one runs stable releases.

My interactions with all three fleets is pretty much the same -- load current builds of software, and try to run the exerciser tests for a duration of time -- 12 hours, 6.5 days, 30 days, etc. TIME_FACTOR does not work well for this usage model, because it is difficult to guess the correct time factor given that the VMs are hetergeneous and the IO completion rate is not perfectly predictable.

Worse yet, if you want to run (say) all the recoveryloop tests on one VM (because recoveryloop is prone to crashing), it's impossible to set a TIME_FACTOR so that each loop test gets equal runtime. That can be hacked around with config sections, but that doesn't solve the first problem.

This series introduces a new configuration variable, SOAK_DURATION, that allows test runners to control directly various long soak and looping recovery tests. This is intended to be an alternative to TIME_FACTOR, since that variable usually adjusts operation counts, which are proportional to runtime but otherwise not a direct measure of time.

With this override in place, I can configure the long soak fleet to run for exactly as long as I want them to, and they actually hit the time budget targets. The recoveryloop fleet now divides looping-test time equally among the four that are in that group so that they all get ~3 hours of coverage every night.

There are more tests that could use this than I actually modified here, but I've done enough to show this off as a proof of concept.

If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=soak-duration

---
 check                 |   14 +++++++++
 common/config         |    7 ++++
 common/fuzzy          |    7 ++++
 common/rc             |   34 +++++++++++++++++++++
 common/report         |    1 +
 ltp/fsstress.c        |   78 +++++++++++++++++++++++++++++++++++++++++++++++--
 ltp/fsx.c             |   50 +++++++++++++++++++++++++++++++
 src/soak_duration.awk |   23 ++++++++++++++
 tests/generic/019     |    1 +
 tests/generic/388     |    2 +
 tests/generic/475     |    2 +
Quoted 9 lines
 tests/generic/476     |    7 +++-
 tests/generic/482     |    5 +++
 tests/generic/521     |    1 +
 tests/generic/522     |    1 +
 tests/generic/642     |    1 +
 tests/generic/648     |    8 +++--
 17 files changed, 229 insertions(+), 13 deletions(-)
 create mode 100644 src/soak_duration.awk
Quoted 2 lines

The set looks good to me (the second commit has different var name, but fine by me)

Which variable name, specifically?

--D

Quoted 5 lines
Signature · 1 line
- Andrey
#14 lore raw
Quoted 10 lines · Darrick J. Wong
Quoted 7 lines · Andrey Albershteyn
Quoted 4 lines
 tests/generic/648     |    8 +++--
 17 files changed, 229 insertions(+), 13 deletions(-)
 create mode 100644 src/soak_duration.awk

The set looks good to me (the second commit has different var name, but fine by me)

Which variable name, specifically?

STRESS_DURATION in the commit message

Signature · 1 line
- Andrey
#15 lore raw
Quoted 7 lines · Andrey Albershteyn
Quoted 6 lines · Darrick J. Wong
Quoted 5 lines · Andrey Albershteyn
Quoted 4 lines
 tests/generic/648     |    8 +++--
 17 files changed, 229 insertions(+), 13 deletions(-)
 create mode 100644 src/soak_duration.awk
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

The set looks good to me (the second commit has different var name, but fine by me)

Which variable name, specifically?

STRESS_DURATION in the commit message

Ah, will fix and resend. Thanks for noticing!

--D

Quoted 3 lines
Signature · 1 line
- Andrey