fstests: direct specification of looping test duration
cover fstests: direct specification of looping test duration
Hi all,
One of the things that I do as a maintainer is to designate a handful of VMs to run fstests for unusually long periods of time. This practice I call long term soak testing. There are actually three separate fleets for this -- one runs alongside the nightly builds, one runs alongside weekly rebases, and the last one runs stable releases.
My interactions with all three fleets is pretty much the same -- load current builds of software, and try to run the exerciser tests for a duration of time -- 12 hours, 6.5 days, 30 days, etc. TIME_FACTOR does not work well for this usage model, because it is difficult to guess the correct time factor given that the VMs are hetergeneous and the IO completion rate is not perfectly predictable.
Worse yet, if you want to run (say) all the recoveryloop tests on one VM (because recoveryloop is prone to crashing), it's impossible to set a TIME_FACTOR so that each loop test gets equal runtime. That can be hacked around with config sections, but that doesn't solve the first problem.
This series introduces a new configuration variable, SOAK_DURATION, that allows test runners to control directly various long soak and looping recovery tests. This is intended to be an alternative to TIME_FACTOR, since that variable usually adjusts operation counts, which are proportional to runtime but otherwise not a direct measure of time.
With this override in place, I can configure the long soak fleet to run for exactly as long as I want them to, and they actually hit the time budget targets. The recoveryloop fleet now divides looping-test time equally among the four that are in that group so that they all get ~3 hours of coverage every night.
There are more tests that could use this than I actually modified here, but I've done enough to show this off as a proof of concept.
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome.
--D
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=soak-duration
--- check | 14 +++++++++ common/config | 7 ++++ common/fuzzy | 7 ++++ common/rc | 34 +++++++++++++++++++++ common/report | 1 + ltp/fsstress.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++-- ltp/fsx.c | 50 +++++++++++++++++++++++++++++++ src/soak_duration.awk | 23 ++++++++++++++ tests/generic/019 | 1 + tests/generic/388 | 2 + tests/generic/475 | 2 + tests/generic/476 | 7 +++- tests/generic/482 | 5 +++ tests/generic/521 | 1 + tests/generic/522 | 1 + tests/generic/642 | 1 + tests/generic/648 | 8 +++-- 17 files changed, 229 insertions(+), 13 deletions(-) create mode 100644 src/soak_duration.awk
PATCH 1/3 generic/476: reclassify this test as a long running soak stress test
From: Darrick J. Wong <[email protected]>
This test is a long(ish) running stress test, so add it to those groups.
--- tests/generic/476 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/476 +1 -1
@@ -8,7 +8,7 @@ # bugs in the write path. # . ./common/preamble -_begin_fstest auto rw +_begin_fstest auto rw soak long_rw stress # Override the default cleanup function. _cleanup()
Re: [PATCH 1/3] generic/476: reclassify this test as a long running soak stress test
Quoted 15 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
This test is a long(ish) running stress test, so add it to those groups.
Signed-off-by Darrick J. Wong <[email protected]>--- tests/generic/476 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/476 +0
-0
tests/generic/476 line 11
Quoted 5 lines
tests/generic/476 +1
-1@@ -8,7 +8,7 @@ # bugs in the write path. # . ./common/preamble -_begin_fstest auto rw +_begin_fstest auto rw soak long_rw stress
Sorry for late reviewing. I thought a bit more about this change. I think the "soak", "long_rw" and "stress" tags are a bit overlap. If the "stress" group means "fsstress", then I think the fsstress test can be in soak group too, and currently the test cases in "soak" group are same with the "long_rw" group [1].
So I think we can give the "soak" tag to more test cases with random I/Os (fsstress or fsx or others). And rename "long_rw" to "long_soak" for those soak group cases which need long soaking time. Then we have two group tags for random loading/stress test cases, the testers can (decide to) run these random load test cases seperately with more time or loop count.
Anyway, above things can be done in another patchset, I just speak out to get more talking:) For this patch:
Thanks,
Zorro
[1]
# ./check -n -g soak SECTION -- simpledev FSTYP -- xfs (non-debug) PLATFORM -- Linux/x86_64 MKFS_OPTIONS -- -f -m rmapbt=1 /dev/sda3 MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sda3 /mnt/scratch
generic/521
generic/522
generic/642
# ./check -n -g long_rw SECTION -- simpledev FSTYP -- xfs (non-debug) PLATFORM -- Linux/x86_64 MKFS_OPTIONS -- -f -m rmapbt=1 /dev/sda3 MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sda3 /mnt/scratch
generic/521
generic/522
generic/642
Quoted 4 lines
tests/generic/476 +0
-0# Override the default cleanup function. _cleanup()
Re: [PATCH 1/3] generic/476: reclassify this test as a long running soak stress test
Quoted 11 lines · Zorro Lang
Quoted 10 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
This test is a long(ish) running stress test, so add it to those groups.
Signed-off-by Darrick J. Wong <[email protected]>--- tests/generic/476 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 16 lines
tests/generic/476 line 11
Quoted 10 lines
tests/generic/476 +1
-1@@ -8,7 +8,7 @@ # bugs in the write path. # . ./common/preamble -_begin_fstest auto rw +_begin_fstest auto rw soak long_rw stressSorry for late reviewing. I thought a bit more about this change. I think the "soak", "
long_rw" and "stress" tags are a bit overlap. If the "stress" group means "fsstress", then I think the fsstress test can be in soak group too, and currently the test cases in "soak" group are same with the "long_rw" group [1].
Hm. Given the current definitions of each group:
long_rw long-soak read write IO path exercisers rw read/write IO tests soak long running soak tests of any kind stress fsstress filesystem exerciser
I think these all can apply to generic/476 -- it's definitely a read-write IO test; it's definitely one that does RW for a long time; and it uses fsstress.
Quoted 5 lines
So I think we can give the "soak" tag to more test cases with random I/Os (fsstress or fsx or others). And rename "
long_rw" to "long_soak" for those soak group cases which need long soaking time. Then we have two group tags for random loading/stress test cases, the testers can (decide to) run these random load test cases seperately with more time or loop count.
I have a counterproposal -- what do you think about redefining 'soak' to mean "all tests where SOAK_DURATION can be used to control the test runtime directly"? This shouldn't break anyone's scripts, since the only members of 'soak' are the ones that get modified by this patchset.
--D
Quoted 39 lines
Anyway, above things can be done in another patchset, I just speak out to get more talking:) For this patch:
Reviewed-by Zorro Lang <[email protected]>Thanks,
Zorro[1]
# ./check -n -g soak SECTION -- simpledev FSTYP -- xfs (non-debug) PLATFORM -- Linux/x86_64 MKFS_OPTIONS -- -f -m rmapbt=1 /dev/sda3 MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sda3 /mnt/scratchgeneric/521
generic/522
generic/642# ./check -n -g long_rw SECTION -- simpledev FSTYP -- xfs (non-debug) PLATFORM -- Linux/x86_64 MKFS_OPTIONS -- -f -m rmapbt=1 /dev/sda3 MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sda3 /mnt/scratchgeneric/521
generic/522
generic/642Quoted 4 lines
tests/generic/476 +0
-0# Override the default cleanup function. _cleanup()
Re: [PATCH 1/3] generic/476: reclassify this test as a long running soak stress test
Quoted 46 lines · Darrick J. Wong
Quoted 11 lines · Zorro Lang
Quoted 10 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
This test is a long(ish) running stress test, so add it to those groups.
Signed-off-by Darrick J. Wong <[email protected]>--- tests/generic/476 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 16 lines
tests/generic/476 line 11
Quoted 10 lines
tests/generic/476 +1
-1@@ -8,7 +8,7 @@ # bugs in the write path. # . ./common/preamble -_begin_fstest auto rw +_begin_fstest auto rw soak long_rw stressSorry for late reviewing. I thought a bit more about this change. I think the "soak", "
long_rw" and "stress" tags are a bit overlap. If the "stress" group means "fsstress", then I think the fsstress test can be in soak group too, and currently the test cases in "soak" group are same with the "long_rw" group [1].Hm. Given the current definitions of each group:
long_rw long-soak read write IO path exercisers rw read/write IO tests soak long running soak tests of any kind stress fsstress filesystem exerciserI think these all can apply to generic/476 -- it's definitely a read-write IO test; it's definitely one that does RW for a long time; and it uses fsstress.
Quoted 5 lines
So I think we can give the "soak" tag to more test cases with random I/Os (fsstress or fsx or others). And rename "
long_rw" to "long_soak" for those soak group cases which need long soaking time. Then we have two group tags for random loading/stress test cases, the testers can (decide to) run these random load test cases seperately with more time or loop count.
Quoted 4 lines
I have a counterproposal -- what do you think about redefining 'soak' to mean "all tests where
SOAK_DURATIONcan be used to control the test runtime directly"? This shouldn't break anyone's scripts, since the only members of 'soak' are the ones that get modified by this patchset.
Sure, we can check if more cases can use the SOAK_DURATION later, then we add them to soak group.
Thanks,
Zorro
Quoted 43 lines
--D
Quoted 39 lines
Anyway, above things can be done in another patchset, I just speak out to get more talking:) For this patch:
Reviewed-by Zorro Lang <[email protected]>Thanks,
Zorro[1]
# ./check -n -g soak SECTION -- simpledev FSTYP -- xfs (non-debug) PLATFORM -- Linux/x86_64 MKFS_OPTIONS -- -f -m rmapbt=1 /dev/sda3 MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sda3 /mnt/scratchgeneric/521
generic/522
generic/642# ./check -n -g long_rw SECTION -- simpledev FSTYP -- xfs (non-debug) PLATFORM -- Linux/x86_64 MKFS_OPTIONS -- -f -m rmapbt=1 /dev/sda3 MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sda3 /mnt/scratchgeneric/521
generic/522
generic/642Quoted 4 lines
tests/generic/476 +0
-0# Override the default cleanup function. _cleanup()
PATCH 2/3 misc: add duration for long soak tests
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Make it so that test runners can schedule long soak stress test programs for an exact number of seconds by setting the STRESS_DURATION config variable.
--- check | 14 +++++++++ common/config | 7 ++++ common/fuzzy | 7 ++++ common/report | 1 + ltp/fsstress.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++-- ltp/fsx.c | 50 +++++++++++++++++++++++++++++++ src/soak_duration.awk | 23 ++++++++++++++ tests/generic/476 | 5 +++ tests/generic/521 | 1 + tests/generic/522 | 1 + tests/generic/642 | 1 + 11 files changed, 182 insertions(+), 6 deletions(-) create mode 100644 src/soak_duration.awk
check +14 -0
@@ -366,6 +366,20 @@ if ! . ./common/rc; then exit 1 fi +# If the test config specified a soak test duration, see if there are any +# unit suffixes that need converting to an integer seconds count. +if [ -n "$SOAK_DURATION" ]; then + SOAK_DURATION="$(echo "$SOAK_DURATION" | \ + sed -e 's/^\([.0-9]*\)\([a-z]\)*/\1 \2/g' | \ + $AWK_PROG -f $here/src/soak_duration.awk)" + if [ $? -ne 0 ]; then + echo "$SOAK_DURATION" + status=1 + exit 1 + fi + export SOAK_DURATION +fi + if [ -n "$subdir_xfile" ]; then for d in $SRC_GROUPS $FSTYP; do [ -f $SRC_DIR/$d/$subdir_xfile ] || continue
common/config +7 -0
@@ -57,6 +57,13 @@ export SOAK_PROC=3 # -p option to fsstress export SOAK_STRESS=10000 # -n option to fsstress export SOAK_PASSES=-1 # count of repetitions of fsstress (while soaking) export EMAIL=root@localhost # where auto-qa will send its status messages + +# For certain tests that run in tight loops, setting this variable allows the +# test runner to specify exactly how long the test should continue looping. +# This is independent of TIME_FACTOR. Floating point numbers are allowed, and +# the unit suffixes m(inutes), h(ours), d(ays), and w(eeks) are supported. +export SOAK_DURATION=${SOAK_DURATION:=} + export HOST_OPTIONS=${HOST_OPTIONS:=local.config} export CHECK_OPTIONS=${CHECK_OPTIONS:="-g auto"} export BENCH_PASSES=${BENCH_PASSES:=5}
common/fuzzy +6 -1
@@ -1351,7 +1351,12 @@ _scratch_xfs_stress_scrub() { fi local start="$(date +%s)" - local end="$((start + (30 * TIME_FACTOR) ))" + local end + if [ -n "$SOAK_DURATION" ]; then + end="$((start + SOAK_DURATION))" + else + end="$((start + (30 * TIME_FACTOR) ))" + fi local scrub_startat="$((start + scrub_delay))" test "$scrub_startat" -gt "$((end - 10))" && scrub_startat="$((end - 10))"
common/report +1 -0
@@ -67,6 +67,7 @@ __generate_report_vars() { REPORT_VARS["CPUS"]="$(getconf _NPROCESSORS_ONLN 2>/dev/null)" REPORT_VARS["MEM_KB"]="$(grep MemTotal: /proc/meminfo | awk '{print $2}')" REPORT_VARS["SWAP_KB"]="$(grep SwapTotal: /proc/meminfo | awk '{print $2}')" + test -n "$SOAK_DURATION" && REPORT_VARS["SOAK_DURATION"]="$SOAK_DURATION" test -e /sys/devices/system/node/possible && \ REPORT_VARS["NUMA_NODES"]="$(cat /sys/devices/system/node/possible 2>/dev/null)"
ltp/fsstress.c +75 -3
132 lines not on this page · raw message
ltp/fsx.c +49 -1
89 lines not on this page · raw message
src/soak_duration.awk new file +23 -0
@@ -0,0 +1,23 @@ +#!/usr/bin/awk +# +# Convert time interval specifications with suffixes to an integer number of +# seconds. +{ + nr = $1; + if ($2 == "" || $2 ~ /s/) # seconds + ; + else if ($2 ~ /m/) # minutes + nr *= 60; + else if ($2 ~ /h/) # hours + nr *= 3600; + else if ($2 ~ /d/) # days + nr *= 86400; + else if ($2 ~ /w/) # weeks + nr *= 604800; + else { + printf("%s: unknown suffix\n", $2); + exit 1; + } + + printf("%d\n", nr); +}
tests/generic/476 +4 -1
@@ -33,7 +33,10 @@ _scratch_mount >> $seqres.full 2>&1 nr_cpus=$((LOAD_FACTOR * 4)) nr_ops=$((25000 * nr_cpus * TIME_FACTOR)) -$FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full +fsstress_args=(-w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus) +test -n "$SOAK_DURATION" && fsstress_args+=(--duration="$SOAK_DURATION") + +$FSSTRESS_PROG $FSSTRESS_AVOID "${fsstress_args[@]}" >> $seqres.full # success, all done status=0
tests/generic/521 +1 -0
@@ -35,6 +35,7 @@ fsx_args+=(-r $min_dio_sz) fsx_args+=(-t $min_dio_sz) fsx_args+=(-w $min_dio_sz) fsx_args+=(-Z) +test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION") run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'
tests/generic/522 +1 -0
@@ -29,6 +29,7 @@ fsx_args+=(-N $nr_ops) fsx_args+=(-p $((nr_ops / 100))) fsx_args+=(-o $op_sz) fsx_args+=(-l $file_sz) +test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION") run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'
tests/generic/642 +1 -0
@@ -49,6 +49,7 @@ for verb in attr_remove removefattr; do done args+=('-f' "setfattr=20") args+=('-f' "attr_set=60") # sets larger xattrs +test -n "$DURATION" && args+=(--duration="$DURATION") $FSSTRESS_PROG "${args[@]}" $FSSTRESS_AVOID -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
PATCH v2 2/3 misc: add duration for long soak tests
From: Darrick J. Wong <[email protected]>
Make it so that test runners can schedule long soak stress test programs for an exact number of seconds by setting the SOAK_DURATION config variable.
---
v2: fix commit message
--- check | 14 +++++++++ common/config | 7 ++++ common/fuzzy | 7 ++++ common/report | 1 + ltp/fsstress.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++-- ltp/fsx.c | 50 +++++++++++++++++++++++++++++++ src/soak_duration.awk | 23 ++++++++++++++ tests/generic/476 | 5 +++ tests/generic/521 | 1 + tests/generic/522 | 1 + tests/generic/642 | 1 + 11 files changed, 182 insertions(+), 6 deletions(-) create mode 100644 src/soak_duration.awk
check +14 -0
@@ -366,6 +366,20 @@ if ! . ./common/rc; then exit 1 fi +# If the test config specified a soak test duration, see if there are any +# unit suffixes that need converting to an integer seconds count. +if [ -n "$SOAK_DURATION" ]; then + SOAK_DURATION="$(echo "$SOAK_DURATION" | \ + sed -e 's/^\([.0-9]*\)\([a-z]\)*/\1 \2/g' | \ + $AWK_PROG -f $here/src/soak_duration.awk)" + if [ $? -ne 0 ]; then + echo "$SOAK_DURATION" + status=1 + exit 1 + fi + export SOAK_DURATION +fi + if [ -n "$subdir_xfile" ]; then for d in $SRC_GROUPS $FSTYP; do [ -f $SRC_DIR/$d/$subdir_xfile ] || continue
common/config +7 -0
@@ -57,6 +57,13 @@ export SOAK_PROC=3 # -p option to fsstress export SOAK_STRESS=10000 # -n option to fsstress export SOAK_PASSES=-1 # count of repetitions of fsstress (while soaking) export EMAIL=root@localhost # where auto-qa will send its status messages + +# For certain tests that run in tight loops, setting this variable allows the +# test runner to specify exactly how long the test should continue looping. +# This is independent of TIME_FACTOR. Floating point numbers are allowed, and +# the unit suffixes m(inutes), h(ours), d(ays), and w(eeks) are supported. +export SOAK_DURATION=${SOAK_DURATION:=} + export HOST_OPTIONS=${HOST_OPTIONS:=local.config} export CHECK_OPTIONS=${CHECK_OPTIONS:="-g auto"} export BENCH_PASSES=${BENCH_PASSES:=5}
common/fuzzy +6 -1
@@ -1360,7 +1360,12 @@ _scratch_xfs_stress_scrub() { fi local start="$(date +%s)" - local end="$((start + (30 * TIME_FACTOR) ))" + local end + if [ -n "$SOAK_DURATION" ]; then + end="$((start + SOAK_DURATION))" + else + end="$((start + (30 * TIME_FACTOR) ))" + fi local scrub_startat="$((start + scrub_delay))" test "$scrub_startat" -gt "$((end - 10))" && scrub_startat="$((end - 10))"
common/report +1 -0
@@ -67,6 +67,7 @@ __generate_report_vars() { REPORT_VARS["CPUS"]="$(getconf _NPROCESSORS_ONLN 2>/dev/null)" REPORT_VARS["MEM_KB"]="$(grep MemTotal: /proc/meminfo | awk '{print $2}')" REPORT_VARS["SWAP_KB"]="$(grep SwapTotal: /proc/meminfo | awk '{print $2}')" + test -n "$SOAK_DURATION" && REPORT_VARS["SOAK_DURATION"]="$SOAK_DURATION" test -e /sys/devices/system/node/possible && \ REPORT_VARS["NUMA_NODES"]="$(cat /sys/devices/system/node/possible 2>/dev/null)"
ltp/fsstress.c +75 -3
132 lines not on this page · raw message
ltp/fsx.c +49 -1
89 lines not on this page · raw message
src/soak_duration.awk new file +23 -0
@@ -0,0 +1,23 @@ +#!/usr/bin/awk +# +# Convert time interval specifications with suffixes to an integer number of +# seconds. +{ + nr = $1; + if ($2 == "" || $2 ~ /s/) # seconds + ; + else if ($2 ~ /m/) # minutes + nr *= 60; + else if ($2 ~ /h/) # hours + nr *= 3600; + else if ($2 ~ /d/) # days + nr *= 86400; + else if ($2 ~ /w/) # weeks + nr *= 604800; + else { + printf("%s: unknown suffix\n", $2); + exit 1; + } + + printf("%d\n", nr); +}
tests/generic/476 +4 -1
@@ -33,7 +33,10 @@ _scratch_mount >> $seqres.full 2>&1 nr_cpus=$((LOAD_FACTOR * 4)) nr_ops=$((25000 * nr_cpus * TIME_FACTOR)) -$FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full +fsstress_args=(-w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus) +test -n "$SOAK_DURATION" && fsstress_args+=(--duration="$SOAK_DURATION") + +$FSSTRESS_PROG $FSSTRESS_AVOID "${fsstress_args[@]}" >> $seqres.full # success, all done status=0
tests/generic/521 +1 -0
@@ -35,6 +35,7 @@ fsx_args+=(-r $min_dio_sz) fsx_args+=(-t $min_dio_sz) fsx_args+=(-w $min_dio_sz) fsx_args+=(-Z) +test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION") run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'
tests/generic/522 +1 -0
@@ -29,6 +29,7 @@ fsx_args+=(-N $nr_ops) fsx_args+=(-p $((nr_ops / 100))) fsx_args+=(-o $op_sz) fsx_args+=(-l $file_sz) +test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION") run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'
tests/generic/642 +1 -0
@@ -49,6 +49,7 @@ for verb in attr_remove removefattr; do done args+=('-f' "setfattr=20") args+=('-f' "attr_set=60") # sets larger xattrs +test -n "$DURATION" && args+=(--duration="$DURATION") $FSSTRESS_PROG "${args[@]}" $FSSTRESS_AVOID -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
Re: [PATCH v2 2/3] misc: add duration for long soak tests
Quoted 31 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Make it so that test runners can schedule long soak stress test programs for an exact number of seconds by setting the
SOAK_DURATIONconfig variable.Signed-off-by Darrick J. Wong <[email protected]>Reviewed-by Andrey Albershteyn <[email protected]>---
v2: fix commit message--- check | 14 +++++++++ common/config | 7 ++++ common/fuzzy | 7 ++++ common/report | 1 + ltp/fsstress.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++-- ltp/fsx.c | 50 +++++++++++++++++++++++++++++++ src/soak_duration.awk | 23 ++++++++++++++ tests/generic/476 | 5 +++ tests/generic/521 | 1 + tests/generic/522 | 1 + tests/generic/642 | 1 + 11 files changed, 182 insertions(+), 6 deletions(-) create mode 100644 src/soak_duration.awkcheck +0
-0@@ -366,6 +366,20 @@ if ! . ./common/rc; then exit 1
check if ! . ./common/rc; then · line 376
Quoted 10 lines
check +8
-0fi +# If the test config specified a soak test duration, see if there are any +# unit suffixes that need converting to an integer seconds count. +if [ -n "$SOAK_DURATION" ]; then + SOAK_DURATION="$(echo "$SOAK_DURATION" | \ + sed -e 's/^\([.0-9]*\)\([a-z]\)*/\1 \2/g' | \ + $AWK_PROG -f $here/src/soak_duration.awk)" + if [ $? -ne 0 ]; then + echo "$SOAK_DURATION"
echo SOAK_DURATION=$SOAK_DURATION ?
check if ! . ./common/rc; then · line 380
Quoted 4 lines
check +4
-0+ status=1 + exit 1 + fi + export SOAK_DURATION
Is this necessary? As you've exported SOAK_DURATION in common/config:
export SOAK_DURATION=${SOAK_DURATION:=}
Quoted 10 lines
check +2
-0+fi + if [ -n "$subdir_xfile" ]; then for d in $SRC_GROUPS $FSTYP; do [ -f $SRC_DIR/$d/$subdir_xfile ] || continuecommon/config +0
-0
common/config export SOAK_PROC=3 # -p option to fsstress · line 65
Quoted 9 lines
common/config +6
-0@@ -57,6 +57,13 @@ export SOAK_PROC=3 # -p option to fsstress export SOAK_STRESS=10000 # -n option to fsstress export SOAK_PASSES=-1 # count of repetitions of fsstress (while soaking) export EMAIL=root@localhost # where auto-qa will send its status messages + +# For certain tests that run in tight loops, setting this variable allows the +# test runner to specify exactly how long the test should continue looping. +# This is independent of TIME_FACTOR. Floating point numbers are allowed, and +# the unit suffixes m(inutes), h(ours), d(ays), and w(eeks) are supported. +export SOAK_DURATION=${SOAK_DURATION:=}
Better to describe more about the SOAK_DURATION (and TIME/LOAD_FACTOR) in documentation (README?). To clarify the relationship of SOAK_DURATION with other related parameters, and how the SOAK_DURATION works/be used.
Quoted 66 lines
common/config +1
-0+ export HOST_OPTIONS=${HOST_OPTIONS:=local.config} export CHECK_OPTIONS=${CHECK_OPTIONS:="-g auto"} export BENCH_PASSES=${BENCH_PASSES:=5}common/fuzzy +6
-1@@ -1360,7 +1360,12 @@ _scratch_xfs_stress_scrub() { fi local start="$(date +%s)" - local end="$((start + (30 * TIME_FACTOR) ))" + local end + if [ -n "$SOAK_DURATION" ]; then + end="$((start + SOAK_DURATION))" + else + end="$((start + (30 * TIME_FACTOR) ))" + fi local scrub_startat="$((start + scrub_delay))" test "$scrub_startat" -gt "$((end - 10))" && scrub_startat="$((end - 10))"common/report +1
-0@@ -67,6 +67,7 @@ __generate_report_vars() { REPORT_VARS["CPUS"]="$(getconf _NPROCESSORS_ONLN 2>/dev/null)" REPORT_VARS["MEM_KB"]="$(grep MemTotal: /proc/meminfo | awk '{print $2}')" REPORT_VARS["SWAP_KB"]="$(grep SwapTotal: /proc/meminfo | awk '{print $2}')" + test -n "$SOAK_DURATION" && REPORT_VARS["SOAK_DURATION"]="$SOAK_DURATION" test -e /sys/devices/system/node/possible && \ REPORT_VARS["NUMA_NODES"]="$(cat /sys/devices/system/node/possible 2>/dev/null)"ltp/fsstress.c +17
-0@@ -386,6 +386,8 @@ char *execute_cmd = NULL;int execute_freq =1 ;struct print_string flag_str = {0 }; +struct timespec deadline = {0 }; +void add_to_flist (int ,int ,int ,int );void append_pathname (pathname_t *,char *);int attr_list_path (pathname_t *,char *,const int );@@ -459,6 +461,34 @@ void sg_handler(int signum) } } +bool +keep_looping (int i,int loops) +{ +int ret; + +if (deadline.tv_nsec) { +struct timespec now; + + ret =clock_gettime (CLOCK_MONOTONIC , &now); +if (ret) { +perror ("CLOCK_MONOTONIC" ); +return false; + } + +return now.tv_sec <= deadline.tv_sec;
ltp/fsstress.c void sg_handler(int signum) · line 488
Quoted 10 lines
ltp/fsstress.c +10
-0+ } + +if (!loops) +return true; + +return i < loops; +} + +static struct option longopts[] = { + {"duration" , optional_argument,0 ,256 },
OK, we'd better to check those test cases use fsstress "-n" later, think about if some of them should be replaced by "--duration".
Thanks,
Zorro
Quoted 283 lines
ltp/fsstress.c +48
-3+ { } +}; +int main (int argc,char **argv) {char buf[10 ];@@ -478,13 +508,14 @@ int main(int argc, char **argv)struct sigaction action;int loops =1 ;const char *allopts ="cd:e:f:i:l:m:M:n:o:p:rRs:S:vVwx:X:zH" ; +long long duration; errrange = errtag =0 ;umask (0 ); nops =sizeof (ops) /sizeof (ops[0 ]); ops_end = &ops[nops]; myprog = argv[0 ]; -while ((c =getopt (argc, argv, allopts)) !=-1 ) { +while ((c =getopt_long (argc, argv, allopts, longopts,NULL )) !=-1 ) {switch (c) {case 'c' : cleanup =1 ;@@ -579,6 +610,26 @@ int main(int argc, char **argv)case 'X' : execute_freq =strtoul (optarg,NULL ,0 );break ; +case 256 :/* --duration */ +if (!optarg) { +fprintf (stderr,"Specify time with --duration=\n" ); +exit (87 ); + } + duration =strtoll (optarg,NULL ,0 ); +if (duration <1 ) { +fprintf (stderr,"%lld: invalid duration\n" , duration); +exit (88 ); + } + + i =clock_gettime (CLOCK_MONOTONIC , &deadline); +if (i) { +perror ("CLOCK_MONOTONIC" ); +exit (89 ); + } + + deadline.tv_sec += duration; + deadline.tv_nsec =1 ; +break ;case '?' :fprintf (stderr,"%s - invalid parameters\n" , myprog);@@ -721,7 +772,7 @@ int main(int argc, char **argv) } }#endif -for (i =0 ; !loops || (i < loops); i++) +for (i =0 ;keep_looping (i, loops); i++)doproc ();#ifdef AIO if (io_destroy (io_ctx) !=0 ) {@@ -1121,6 +1172,26 @@ dirid_to_fent(int dirid)return NULL ; } +bool +keep_running (opnum_t opno,opnum_t operations) +{ +int ret; + +if (deadline.tv_nsec) { +struct timespec now; + + ret =clock_gettime (CLOCK_MONOTONIC , &now); +if (ret) { +perror ("CLOCK_MONOTONIC" ); +return false; + } + +return now.tv_sec <= deadline.tv_sec; + } + +return opno < operations; +} +void doproc (void ) {@@ -1149,7 +1220,7 @@ doproc(void)srandom (seed);if (namerand) namerand =random (); -for (opno =0 ; opno < operations; opno++) { +for (opno =0 ;keep_running (opno, operations); opno++) {if (execute_cmd && opno && opno % dividend ==0 ) {if (verbose) printf("%lld: execute command %s\n", opno,@@ -1935,6 +2006,7 @@ usage(void)printf (" -V specifies verifiable logging mode (omitting inode numbers)\n" );printf (" -X ncmd number of calls to the -x command (default 1)\n" );printf (" -H prints usage and exits\n" ); +printf (" --duration=s run for this many seconds\n" ); }void ltp/fsx.c +49
-1@@ -193,6 +193,8 @@ int fsx_rw(int rw, int fd, char *buf, unsigned len, unsigned offset);#define fsxread (a,b,c,d) fsx_rw(READ, a,b,c,d)#define fsxwrite (a,b,c,d) fsx_rw(WRITE, a,b,c,d) +struct timespec deadline; +const char *replayops =NULL ;const char *recordops =NULL ;FILE * fsxlogf =NULL ;@@ -2457,6 +2459,7 @@ usage(void) -Z: O_DIRECT (use -R, -W, -r and -w too)\n\ --replay-ops opsfile: replay ops from recorded .fsxops file\n\ --record-ops[=opsfile]: dump ops file also on success. optionally specify ops file name\n\ + --duration=seconds: run for this many seconds\n\ fname: this filename is REQUIRED (no default)\n"); exit(90); }@@ -2739,9 +2742,33 @@ __test_fallocate(int mode, const char *mode_str)#endif } +bool +keep_running (void ) +{ +int ret; + +if (deadline.tv_nsec) { +struct timespec now; + + ret =clock_gettime (CLOCK_MONOTONIC , &now); +if (ret) { +perror ("CLOCK_MONOTONIC" ); +return false; + } + +return now.tv_sec <= deadline.tv_sec; + } + +if (numops ==-1 ) +return true; + +return numops-- !=0 ; +} +static struct option longopts[] = { {"replay-ops" , required_argument,0 ,256 }, {"record-ops" , optional_argument,0 ,255 }, + {"duration" , optional_argument,0 ,254 }, { } };@@ -2753,6 +2780,7 @@ main(int argc, char **argv)char logfile[PATH_MAX ];struct stat statbuf;int o_flags =O_RDWR |O_CREAT |O_TRUNC ; +long long duration; logfile[0 ] =0 ; dname[0 ] =0 ;@@ -2950,6 +2978,26 @@ main(int argc, char **argv) o_direct =O_DIRECT ; o_flags |=O_DIRECT ;break ; +case 254 :/* --duration */ +if (!optarg) { +fprintf (stderr,"Specify time with --duration=\n" ); +exit (87 ); + } + duration =strtoll (optarg,NULL ,0 ); +if (duration <1 ) { +fprintf (stderr,"%lld: invalid duration\n" , duration); +exit (88 ); + } + + i =clock_gettime (CLOCK_MONOTONIC , &deadline); +if (i) { +perror ("CLOCK_MONOTONIC" ); +exit (89 ); + } + + deadline.tv_sec += duration; + deadline.tv_nsec =1 ; +break ;case 255 :/* --record-ops */ if (optarg)snprintf (opsfile,sizeof (opsfile),"%s" , optarg);@@ -3145,7 +3193,7 @@ main(int argc, char **argv)if (xchg_range_calls) xchg_range_calls =test_xchg_range (); -while (numops ==-1 || numops--) +while (keep_running ())if (!test ())break ;src/soak_duration.awk new file +23
-0@@ -0,0 +1,23 @@ +#!/usr/bin/awk +# +# Convert time interval specifications with suffixes to an integer number of +# seconds. +{ + nr = $1; + if ($2 == "" || $2 ~ /s/) # seconds + ; + else if ($2 ~ /m/) # minutes + nr *= 60; + else if ($2 ~ /h/) # hours + nr *= 3600; + else if ($2 ~ /d/) # days + nr *= 86400; + else if ($2 ~ /w/) # weeks + nr *= 604800; + else { + printf("%s: unknown suffix\n", $2); + exit 1; + } + + printf("%d\n", nr); +}tests/generic/476 +4
-1@@ -33,7 +33,10 @@ _scratch_mount >> $seqres.full 2>&1 nr_cpus=$((LOAD_FACTOR * 4)) nr_ops=$((25000 * nr_cpus * TIME_FACTOR)) -$FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full +fsstress_args=(-w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus) +test -n "$SOAK_DURATION" && fsstress_args+=(--duration="$SOAK_DURATION") + +$FSSTRESS_PROG $FSSTRESS_AVOID "${fsstress_args[@]}" >> $seqres.full # success, all done status=0tests/generic/521 +1
-0@@ -35,6 +35,7 @@ fsx_args+=(-r $min_dio_sz) fsx_args+=(-t $min_dio_sz) fsx_args+=(-w $min_dio_sz) fsx_args+=(-Z) +test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION") run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'tests/generic/522 +1
-0@@ -29,6 +29,7 @@ fsx_args+=(-N $nr_ops) fsx_args+=(-p $((nr_ops / 100))) fsx_args+=(-o $op_sz) fsx_args+=(-l $file_sz) +test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION") run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'tests/generic/642 +1
-0@@ -49,6 +49,7 @@ for verb in attr_remove removefattr; do done args+=('-f' "setfattr=20") args+=('-f' "attr_set=60") # sets larger xattrs +test -n "$DURATION" && args+=(--duration="$DURATION") $FSSTRESS_PROG "${args[@]}" $FSSTRESS_AVOID -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
Re: [PATCH v2 2/3] misc: add duration for long soak tests
Quoted 34 lines · Zorro Lang
Quoted 33 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Make it so that test runners can schedule long soak stress test programs for an exact number of seconds by setting the
SOAK_DURATIONconfig variable.Signed-off-by Darrick J. Wong <[email protected]>Reviewed-by Andrey Albershteyn <[email protected]>---
v2: fix commit message--- check | 14 +++++++++ common/config | 7 ++++ common/fuzzy | 7 ++++ common/report | 1 + ltp/fsstress.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++-- ltp/fsx.c | 50 +++++++++++++++++++++++++++++++ src/soak_duration.awk | 23 ++++++++++++++ tests/generic/476 | 5 +++ tests/generic/521 | 1 + tests/generic/522 | 1 + tests/generic/642 | 1 + 11 files changed, 182 insertions(+), 6 deletions(-) create mode 100644 src/soak_duration.awkcheck +0
-0@@ -366,6 +366,20 @@ if ! . ./common/rc; then exit 1 fi
Quoted 10 lines
check if ! . ./common/rc; then · line 376
Quoted 8 lines
check +8
-0+# If the test config specified a soak test duration, see if there are any +# unit suffixes that need converting to an integer seconds count. +if [ -n "$SOAK_DURATION" ]; then + SOAK_DURATION="$(echo "$SOAK_DURATION" | \ + sed -e 's/^\([.0-9]*\)\([a-z]\)*/\1 \2/g' | \ + $AWK_PROG -f $here/src/soak_duration.awk)" + if [ $? -ne 0 ]; then + echo "$SOAK_DURATION"echo SOAK_DURATION=$SOAK_DURATION ?
Hmm. On nonzero return, the awk script already specified the reason that the input was rejected, so I think this line can go away.
Quoted 8 lines
check if ! . ./common/rc; then · line 380
Quoted 4 lines
check +4
-0+ status=1 + exit 1 + fi + export SOAK_DURATIONIs this necessary? As you've exported
SOAK_DURATIONin common/config:export SOAK_DURATION=${SOAK_DURATION:=}
You're correct, it does not need to be re-exported.
Quoted 15 lines
Quoted 15 lines
check +2
-0+fi + if [ -n "$subdir_xfile" ]; then for d in $SRC_GROUPS $FSTYP; do [ -f $SRC_DIR/$d/$subdir_xfile ] || continuecommon/config +2
-0@@ -57,6 +57,13 @@ export SOAK_PROC=3 # -p option to fsstress export SOAK_STRESS=10000 # -n option to fsstress export SOAK_PASSES=-1 # count of repetitions of fsstress (while soaking) export EMAIL=root@localhost # where auto-qa will send its status messages + +# For certain tests that run in tight loops, setting this variable allows the
Quoted 8 lines
common/config export SOAK_PROC=3 # -p option to fsstress · line 65
Quoted 4 lines
common/config +4
-0+# test runner to specify exactly how long the test should continue looping. +# This is independent of TIME_FACTOR. Floating point numbers are allowed, and +# the unit suffixes m(inutes), h(ours), d(ays), and w(eeks) are supported. +export SOAK_DURATION=${SOAK_DURATION:=}Better to describe more about the
SOAK_DURATION(and TIME/LOAD_FACTOR) in documentation (README?). To clarify the relationship ofSOAK_DURATIONwith other related parameters, and how theSOAK_DURATIONworks/be used.
TIME_FACTOR is not documented in the README. I'll add another patch to do that...
Quoted 71 lines
Quoted 71 lines
common/config +1
-0+ export HOST_OPTIONS=${HOST_OPTIONS:=local.config} export CHECK_OPTIONS=${CHECK_OPTIONS:="-g auto"} export BENCH_PASSES=${BENCH_PASSES:=5}common/fuzzy +6
-1@@ -1360,7 +1360,12 @@ _scratch_xfs_stress_scrub() { fi local start="$(date +%s)" - local end="$((start + (30 * TIME_FACTOR) ))" + local end + if [ -n "$SOAK_DURATION" ]; then + end="$((start + SOAK_DURATION))" + else + end="$((start + (30 * TIME_FACTOR) ))" + fi local scrub_startat="$((start + scrub_delay))" test "$scrub_startat" -gt "$((end - 10))" && scrub_startat="$((end - 10))"common/report +1
-0@@ -67,6 +67,7 @@ __generate_report_vars() { REPORT_VARS["CPUS"]="$(getconf _NPROCESSORS_ONLN 2>/dev/null)" REPORT_VARS["MEM_KB"]="$(grep MemTotal: /proc/meminfo | awk '{print $2}')" REPORT_VARS["SWAP_KB"]="$(grep SwapTotal: /proc/meminfo | awk '{print $2}')" + test -n "$SOAK_DURATION" && REPORT_VARS["SOAK_DURATION"]="$SOAK_DURATION" test -e /sys/devices/system/node/possible && \ REPORT_VARS["NUMA_NODES"]="$(cat /sys/devices/system/node/possible 2>/dev/null)"ltp/fsstress.c +22
-0@@ -386,6 +386,8 @@ char *execute_cmd = NULL;int execute_freq =1 ;struct print_string flag_str = {0 }; +struct timespec deadline = {0 }; +void add_to_flist (int ,int ,int ,int );void append_pathname (pathname_t *,char *);int attr_list_path (pathname_t *,char *,const int );@@ -459,6 +461,34 @@ void sg_handler(int signum) } } +bool +keep_looping (int i,int loops) +{ +int ret; + +if (deadline.tv_nsec) { +struct timespec now; + + ret =clock_gettime (CLOCK_MONOTONIC , &now); +if (ret) { +perror ("CLOCK_MONOTONIC" ); +return false; + } + +return now.tv_sec <= deadline.tv_sec; + } + +if (!loops) +return true; +
Quoted 8 lines
ltp/fsstress.c void sg_handler(int signum) · line 488
Quoted 5 lines
ltp/fsstress.c +5
-0+return i < loops; +} + +static struct option longopts[] = { + {"duration" , optional_argument,0 ,256 },OK, we'd better to check those test cases use fsstress "-n" later, think about if some of them should be replaced by "--duration".
<nod> I'll update the fsstress/fsx --help screens to state that --duration overrides -n/-N.
--D
Quoted 287 lines
Thanks,
ZorroQuoted 283 lines
ltp/fsstress.c +48
-3+ { } +}; +int main (int argc,char **argv) {char buf[10 ];@@ -478,13 +508,14 @@ int main(int argc, char **argv)struct sigaction action;int loops =1 ;const char *allopts ="cd:e:f:i:l:m:M:n:o:p:rRs:S:vVwx:X:zH" ; +long long duration; errrange = errtag =0 ;umask (0 ); nops =sizeof (ops) /sizeof (ops[0 ]); ops_end = &ops[nops]; myprog = argv[0 ]; -while ((c =getopt (argc, argv, allopts)) !=-1 ) { +while ((c =getopt_long (argc, argv, allopts, longopts,NULL )) !=-1 ) {switch (c) {case 'c' : cleanup =1 ;@@ -579,6 +610,26 @@ int main(int argc, char **argv)case 'X' : execute_freq =strtoul (optarg,NULL ,0 );break ; +case 256 :/* --duration */ +if (!optarg) { +fprintf (stderr,"Specify time with --duration=\n" ); +exit (87 ); + } + duration =strtoll (optarg,NULL ,0 ); +if (duration <1 ) { +fprintf (stderr,"%lld: invalid duration\n" , duration); +exit (88 ); + } + + i =clock_gettime (CLOCK_MONOTONIC , &deadline); +if (i) { +perror ("CLOCK_MONOTONIC" ); +exit (89 ); + } + + deadline.tv_sec += duration; + deadline.tv_nsec =1 ; +break ;case '?' :fprintf (stderr,"%s - invalid parameters\n" , myprog);@@ -721,7 +772,7 @@ int main(int argc, char **argv) } }#endif -for (i =0 ; !loops || (i < loops); i++) +for (i =0 ;keep_looping (i, loops); i++)doproc ();#ifdef AIO if (io_destroy (io_ctx) !=0 ) {@@ -1121,6 +1172,26 @@ dirid_to_fent(int dirid)return NULL ; } +bool +keep_running (opnum_t opno,opnum_t operations) +{ +int ret; + +if (deadline.tv_nsec) { +struct timespec now; + + ret =clock_gettime (CLOCK_MONOTONIC , &now); +if (ret) { +perror ("CLOCK_MONOTONIC" ); +return false; + } + +return now.tv_sec <= deadline.tv_sec; + } + +return opno < operations; +} +void doproc (void ) {@@ -1149,7 +1220,7 @@ doproc(void)srandom (seed);if (namerand) namerand =random (); -for (opno =0 ; opno < operations; opno++) { +for (opno =0 ;keep_running (opno, operations); opno++) {if (execute_cmd && opno && opno % dividend ==0 ) {if (verbose) printf("%lld: execute command %s\n", opno,@@ -1935,6 +2006,7 @@ usage(void)printf (" -V specifies verifiable logging mode (omitting inode numbers)\n" );printf (" -X ncmd number of calls to the -x command (default 1)\n" );printf (" -H prints usage and exits\n" ); +printf (" --duration=s run for this many seconds\n" ); }void ltp/fsx.c +49
-1@@ -193,6 +193,8 @@ int fsx_rw(int rw, int fd, char *buf, unsigned len, unsigned offset);#define fsxread (a,b,c,d) fsx_rw(READ, a,b,c,d)#define fsxwrite (a,b,c,d) fsx_rw(WRITE, a,b,c,d) +struct timespec deadline; +const char *replayops =NULL ;const char *recordops =NULL ;FILE * fsxlogf =NULL ;@@ -2457,6 +2459,7 @@ usage(void) -Z: O_DIRECT (use -R, -W, -r and -w too)\n\ --replay-ops opsfile: replay ops from recorded .fsxops file\n\ --record-ops[=opsfile]: dump ops file also on success. optionally specify ops file name\n\ + --duration=seconds: run for this many seconds\n\ fname: this filename is REQUIRED (no default)\n"); exit(90); }@@ -2739,9 +2742,33 @@ __test_fallocate(int mode, const char *mode_str)#endif } +bool +keep_running (void ) +{ +int ret; + +if (deadline.tv_nsec) { +struct timespec now; + + ret =clock_gettime (CLOCK_MONOTONIC , &now); +if (ret) { +perror ("CLOCK_MONOTONIC" ); +return false; + } + +return now.tv_sec <= deadline.tv_sec; + } + +if (numops ==-1 ) +return true; + +return numops-- !=0 ; +} +static struct option longopts[] = { {"replay-ops" , required_argument,0 ,256 }, {"record-ops" , optional_argument,0 ,255 }, + {"duration" , optional_argument,0 ,254 }, { } };@@ -2753,6 +2780,7 @@ main(int argc, char **argv)char logfile[PATH_MAX ];struct stat statbuf;int o_flags =O_RDWR |O_CREAT |O_TRUNC ; +long long duration; logfile[0 ] =0 ; dname[0 ] =0 ;@@ -2950,6 +2978,26 @@ main(int argc, char **argv) o_direct =O_DIRECT ; o_flags |=O_DIRECT ;break ; +case 254 :/* --duration */ +if (!optarg) { +fprintf (stderr,"Specify time with --duration=\n" ); +exit (87 ); + } + duration =strtoll (optarg,NULL ,0 ); +if (duration <1 ) { +fprintf (stderr,"%lld: invalid duration\n" , duration); +exit (88 ); + } + + i =clock_gettime (CLOCK_MONOTONIC , &deadline); +if (i) { +perror ("CLOCK_MONOTONIC" ); +exit (89 ); + } + + deadline.tv_sec += duration; + deadline.tv_nsec =1 ; +break ;case 255 :/* --record-ops */ if (optarg)snprintf (opsfile,sizeof (opsfile),"%s" , optarg);@@ -3145,7 +3193,7 @@ main(int argc, char **argv)if (xchg_range_calls) xchg_range_calls =test_xchg_range (); -while (numops ==-1 || numops--) +while (keep_running ())if (!test ())break ;src/soak_duration.awk new file +23
-0@@ -0,0 +1,23 @@ +#!/usr/bin/awk +# +# Convert time interval specifications with suffixes to an integer number of +# seconds. +{ + nr = $1; + if ($2 == "" || $2 ~ /s/) # seconds + ; + else if ($2 ~ /m/) # minutes + nr *= 60; + else if ($2 ~ /h/) # hours + nr *= 3600; + else if ($2 ~ /d/) # days + nr *= 86400; + else if ($2 ~ /w/) # weeks + nr *= 604800; + else { + printf("%s: unknown suffix\n", $2); + exit 1; + } + + printf("%d\n", nr); +}tests/generic/476 +4
-1@@ -33,7 +33,10 @@ _scratch_mount >> $seqres.full 2>&1 nr_cpus=$((LOAD_FACTOR * 4)) nr_ops=$((25000 * nr_cpus * TIME_FACTOR)) -$FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full +fsstress_args=(-w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus) +test -n "$SOAK_DURATION" && fsstress_args+=(--duration="$SOAK_DURATION") + +$FSSTRESS_PROG $FSSTRESS_AVOID "${fsstress_args[@]}" >> $seqres.full # success, all done status=0tests/generic/521 +1
-0@@ -35,6 +35,7 @@ fsx_args+=(-r $min_dio_sz) fsx_args+=(-t $min_dio_sz) fsx_args+=(-w $min_dio_sz) fsx_args+=(-Z) +test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION") run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'tests/generic/522 +1
-0@@ -29,6 +29,7 @@ fsx_args+=(-N $nr_ops) fsx_args+=(-p $((nr_ops / 100))) fsx_args+=(-o $op_sz) fsx_args+=(-l $file_sz) +test -n "$SOAK_DURATION" && fsx_args+=(--duration="$SOAK_DURATION") run_fsx "${fsx_args[@]}" | sed -e '/^fsx.*/d'tests/generic/642 +1
-0@@ -49,6 +49,7 @@ for verb in attr_remove removefattr; do done args+=('-f' "setfattr=20") args+=('-f' "attr_set=60") # sets larger xattrs +test -n "$DURATION" && args+=(--duration="$DURATION") $FSSTRESS_PROG "${args[@]}" $FSSTRESS_AVOID -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
PATCH 3/3 misc: add duration for recovery loop tests
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Make it so that we can run recovery loop tests for an exact number of seconds.
--- common/rc | 34 ++++++++++++++++++++++++++++++++++ tests/generic/019 | 1 + tests/generic/388 | 2 +- tests/generic/475 | 2 +- tests/generic/482 | 5 +++++ tests/generic/648 | 8 ++++---- 6 files changed, 46 insertions(+), 6 deletions(-)
common/rc +34 -0
@@ -5078,6 +5078,40 @@ _save_coredump() $COREDUMP_COMPRESSOR -f "$out_file" } +# Decide if a soak test should continue looping. The sole parameter is the +# number of soak loops that the test wants to run by default. The actual +# loop iteration number is stored in SOAK_LOOPIDX until the loop completes. +# +# If the test runner set a SOAK_DURATION value, this predicate will keep +# looping until it has run for at least that long. +_soak_loop_running() { + local max_soak_loops="$1" + + test -z "$SOAK_LOOPIDX" && SOAK_LOOPIDX=1 + + if [ -n "$SOAK_DURATION" ]; then + if [ -z "$SOAK_DEADLINE" ]; then + SOAK_DEADLINE="$(( $(date +%s) + SOAK_DURATION))" + fi + + local now="$(date +%s)" + if [ "$now" -gt "$SOAK_DEADLINE" ]; then + unset SOAK_DEADLINE + unset SOAK_LOOPIDX + return 1 + fi + SOAK_LOOPIDX=$((SOAK_LOOPIDX + 1)) + return 0 + fi + + if [ "$SOAK_LOOPIDX" -gt "$max_soak_loops" ]; then + unset SOAK_LOOPIDX + return 1 + fi + SOAK_LOOPIDX=$((SOAK_LOOPIDX + 1)) + return 0 +} + init_rc ################################################################################
tests/generic/019 +1 -0
@@ -30,6 +30,7 @@ _cleanup() } RUN_TIME=$((20+10*$TIME_FACTOR)) +test -n "$SOAK_DURATION" && RUN_TIME="$SOAK_DURATION" NUM_JOBS=$((4*LOAD_FACTOR)) BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV` FILE_SIZE=$((BLK_DEV_SIZE * 512))
tests/generic/388 +1 -1
@@ -42,7 +42,7 @@ _scratch_mkfs >> $seqres.full 2>&1 _require_metadata_journaling $SCRATCH_DEV _scratch_mount -for i in $(seq 1 $((50 * TIME_FACTOR)) ); do +while _soak_loop_running $((50 * TIME_FACTOR)); do ($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p 4 >> $seqres.full &) \ > /dev/null 2>&1
tests/generic/475 +1 -1
@@ -41,7 +41,7 @@ _require_metadata_journaling $SCRATCH_DEV _dmerror_init _dmerror_mount -for i in $(seq 1 $((50 * TIME_FACTOR)) ); do +while _soak_loop_running $((50 * TIME_FACTOR)); do ($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4)) >> $seqres.full &) \ > /dev/null 2>&1
tests/generic/482 +5 -0
@@ -62,8 +62,13 @@ nr_cpus=$("$here/src/feature" -o) if [ $nr_cpus -gt 8 ]; then nr_cpus=8 fi + fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 512 -p $nr_cpus \ $FSSTRESS_AVOID) + +# XXX dm-logwrites pins kernel memory for every write! +# test -n "$SOAK_DURATION" && fsstress_args="$fsstress_args --duration=$SOAK_DURATION" + devsize=$((1024*1024*200 / 512)) # 200m phys/virt size csize=$((1024*64 / 512)) # 64k cluster size lowspace=$((1024*1024 / 512)) # 1m low space threshold
tests/generic/648 +4 -4
@@ -74,14 +74,14 @@ snap_loop_fs() { fsstress=($FSSTRESS_PROG $FSSTRESS_AVOID -d "$loopmnt" -n 999999 -p "$((LOAD_FACTOR * 4))") -for i in $(seq 1 $((25 * TIME_FACTOR)) ); do +while _soak_loop_running $((25 * TIME_FACTOR)); do touch $scratch_aliveflag snap_loop_fs >> $seqres.full 2>&1 & if ! _mount $loopimg $loopmnt -o loop; then rm -f $scratch_aliveflag _metadump_dev $loopimg $seqres.loop.$i.md - _fail "iteration $i loopimg mount failed" + _fail "iteration $SOAK_LOOPIDX loopimg mount failed" break fi
@@ -126,12 +126,12 @@ for i in $(seq 1 $((25 * TIME_FACTOR)) ); do done if [ $is_unmounted -ne 0 ];then cat $tmp.unmount.err - _fail "iteration $i scratch unmount failed" + _fail "iteration $SOAK_LOOPIDX scratch unmount failed" fi _dmerror_load_working_table if ! _dmerror_mount; then _metadump_dev $DMERROR_DEV $seqres.scratch.$i.md - _fail "iteration $i scratch mount failed" + _fail "iteration $SOAK_LOOPIDX scratch mount failed" fi done
Re: [PATCH 3/3] misc: add duration for recovery loop tests
Quoted 104 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Make it so that we can run recovery loop tests for an exact number of seconds.
Signed-off-by Darrick J. Wong <[email protected]>--- common/rc | 34 ++++++++++++++++++++++++++++++++++ tests/generic/019 | 1 + tests/generic/388 | 2 +- tests/generic/475 | 2 +- tests/generic/482 | 5 +++++ tests/generic/648 | 8 ++++---- 6 files changed, 46 insertions(+), 6 deletions(-)common/rc +34
-0@@ -5078,6 +5078,40 @@ _save_coredump() $COREDUMP_COMPRESSOR -f "$out_file" } +# Decide if a soak test should continue looping. The sole parameter is the +# number of soak loops that the test wants to run by default. The actual +# loop iteration number is stored in SOAK_LOOPIDX until the loop completes. +# +# If the test runner set a SOAK_DURATION value, this predicate will keep +# looping until it has run for at least that long. +_soak_loop_running() { + local max_soak_loops="$1" + + test -z "$SOAK_LOOPIDX" && SOAK_LOOPIDX=1 + + if [ -n "$SOAK_DURATION" ]; then + if [ -z "$SOAK_DEADLINE" ]; then + SOAK_DEADLINE="$(( $(date +%s) + SOAK_DURATION))" + fi + + local now="$(date +%s)" + if [ "$now" -gt "$SOAK_DEADLINE" ]; then + unset SOAK_DEADLINE + unset SOAK_LOOPIDX + return 1 + fi + SOAK_LOOPIDX=$((SOAK_LOOPIDX + 1)) + return 0 + fi + + if [ "$SOAK_LOOPIDX" -gt "$max_soak_loops" ]; then + unset SOAK_LOOPIDX + return 1 + fi + SOAK_LOOPIDX=$((SOAK_LOOPIDX + 1)) + return 0 +} + init_rc ################################################################################tests/generic/019 +1
-0@@ -30,6 +30,7 @@ _cleanup() } RUN_TIME=$((20+10*$TIME_FACTOR)) +test -n "$SOAK_DURATION" && RUN_TIME="$SOAK_DURATION" NUM_JOBS=$((4*LOAD_FACTOR)) BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV` FILE_SIZE=$((BLK_DEV_SIZE * 512))tests/generic/388 +1
-1@@ -42,7 +42,7 @@ _scratch_mkfs >> $seqres.full 2>&1 _require_metadata_journaling $SCRATCH_DEV _scratch_mount -for i in $(seq 1 $((50 * TIME_FACTOR)) ); do +while _soak_loop_running $((50 * TIME_FACTOR)); do ($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p 4 >> $seqres.full &) \ > /dev/null 2>&1tests/generic/475 +1
-1@@ -41,7 +41,7 @@ _require_metadata_journaling $SCRATCH_DEV _dmerror_init _dmerror_mount -for i in $(seq 1 $((50 * TIME_FACTOR)) ); do +while _soak_loop_running $((50 * TIME_FACTOR)); do ($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4)) >> $seqres.full &) \ > /dev/null 2>&1tests/generic/482 +0
-0
tests/generic/482 nr_cpus=$("$here/src/feature" -o) · line 70
Quoted 9 lines
tests/generic/482 +4
-0@@ -62,8 +62,13 @@ nr_cpus=$("$here/src/feature" -o) if [ $nr_cpus -gt 8 ]; then nr_cpus=8 fi + fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 512 -p $nr_cpus \ $FSSTRESS_AVOID) + +# XXX dm-logwrites pins kernel memory for every write! +# test -n "$SOAK_DURATION" && fsstress_args="$fsstress_args --duration=$SOAK_DURATION"
Do you expect the second comment is a comment?
Others looks good to me. I'll test V2 and merge it if no regression from it.
Thanks,
Zorro
Re: [PATCH 3/3] misc: add duration for recovery loop tests
Quoted 100 lines · Zorro Lang
Quoted 99 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Make it so that we can run recovery loop tests for an exact number of seconds.
Signed-off-by Darrick J. Wong <[email protected]>--- common/rc | 34 ++++++++++++++++++++++++++++++++++ tests/generic/019 | 1 + tests/generic/388 | 2 +- tests/generic/475 | 2 +- tests/generic/482 | 5 +++++ tests/generic/648 | 8 ++++---- 6 files changed, 46 insertions(+), 6 deletions(-)common/rc +34
-0@@ -5078,6 +5078,40 @@ _save_coredump() $COREDUMP_COMPRESSOR -f "$out_file" } +# Decide if a soak test should continue looping. The sole parameter is the +# number of soak loops that the test wants to run by default. The actual +# loop iteration number is stored in SOAK_LOOPIDX until the loop completes. +# +# If the test runner set a SOAK_DURATION value, this predicate will keep +# looping until it has run for at least that long. +_soak_loop_running() { + local max_soak_loops="$1" + + test -z "$SOAK_LOOPIDX" && SOAK_LOOPIDX=1 + + if [ -n "$SOAK_DURATION" ]; then + if [ -z "$SOAK_DEADLINE" ]; then + SOAK_DEADLINE="$(( $(date +%s) + SOAK_DURATION))" + fi + + local now="$(date +%s)" + if [ "$now" -gt "$SOAK_DEADLINE" ]; then + unset SOAK_DEADLINE + unset SOAK_LOOPIDX + return 1 + fi + SOAK_LOOPIDX=$((SOAK_LOOPIDX + 1)) + return 0 + fi + + if [ "$SOAK_LOOPIDX" -gt "$max_soak_loops" ]; then + unset SOAK_LOOPIDX + return 1 + fi + SOAK_LOOPIDX=$((SOAK_LOOPIDX + 1)) + return 0 +} + init_rc ################################################################################tests/generic/019 +1
-0@@ -30,6 +30,7 @@ _cleanup() } RUN_TIME=$((20+10*$TIME_FACTOR)) +test -n "$SOAK_DURATION" && RUN_TIME="$SOAK_DURATION" NUM_JOBS=$((4*LOAD_FACTOR)) BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV` FILE_SIZE=$((BLK_DEV_SIZE * 512))tests/generic/388 +1
-1@@ -42,7 +42,7 @@ _scratch_mkfs >> $seqres.full 2>&1 _require_metadata_journaling $SCRATCH_DEV _scratch_mount -for i in $(seq 1 $((50 * TIME_FACTOR)) ); do +while _soak_loop_running $((50 * TIME_FACTOR)); do ($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p 4 >> $seqres.full &) \ > /dev/null 2>&1tests/generic/475 +1
-1@@ -41,7 +41,7 @@ _require_metadata_journaling $SCRATCH_DEV _dmerror_init _dmerror_mount -for i in $(seq 1 $((50 * TIME_FACTOR)) ); do +while _soak_loop_running $((50 * TIME_FACTOR)); do ($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4)) >> $seqres.full &) \ > /dev/null 2>&1
Quoted 16 lines
Quoted 5 lines
tests/generic/482 +0
-0tests/generic/482 nr_cpus=$("$here/src/feature" -o) · line 70
Quoted 9 lines
tests/generic/482 +4
-0@@ -62,8 +62,13 @@ nr_cpus=$("$here/src/feature" -o) if [ $nr_cpus -gt 8 ]; then nr_cpus=8 fi + fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 512 -p $nr_cpus \ $FSSTRESS_AVOID) + +# XXX dm-logwrites pins kernel memory for every write! +# test -n "$SOAK_DURATION" && fsstress_args="$fsstress_args --duration=$SOAK_DURATION"Do you expect the second comment is a comment?
Actually... I'd like to withdraw this chunk so that people aren't tempted to uncomment it. I had previously wanted generic/482 to be a SOAK_DURATION controllable test, but all that does is increase kernel memory usage until the VM OOMs, and it OOMs pretty quickly if you don't provision it with the same amount of RAM as the fs is writing to disk.
I'll repost this without the changes to g/482.
--D
Quoted 5 lines
Others looks good to me. I'll test V2 and merge it if no regression from it.
Thanks,
Zorro
Quoted 58 lines · Darrick J. Wong
Hi all,
One of the things that I do as a maintainer is to designate a handful of VMs to run fstests for unusually long periods of time. This practice I call long term soak testing. There are actually three separate fleets for this -- one runs alongside the nightly builds, one runs alongside weekly rebases, and the last one runs stable releases.
My interactions with all three fleets is pretty much the same -- load current builds of software, and try to run the exerciser tests for a duration of time -- 12 hours, 6.5 days, 30 days, etc.
TIME_FACTORdoes not work well for this usage model, because it is difficult to guess the correct time factor given that the VMs are hetergeneous and the IO completion rate is not perfectly predictable.Worse yet, if you want to run (say) all the recoveryloop tests on one VM (because recoveryloop is prone to crashing), it's impossible to set a
TIME_FACTORso that each loop test gets equal runtime. That can be hacked around with config sections, but that doesn't solve the first problem.This series introduces a new configuration variable,
SOAK_DURATION, that allows test runners to control directly various long soak and looping recovery tests. This is intended to be an alternative toTIME_FACTOR, since that variable usually adjusts operation counts, which are proportional to runtime but otherwise not a direct measure of time.With this override in place, I can configure the long soak fleet to run for exactly as long as I want them to, and they actually hit the time budget targets. The recoveryloop fleet now divides looping-test time equally among the four that are in that group so that they all get ~3 hours of coverage every night.
There are more tests that could use this than I actually modified here, but I've done enough to show this off as a proof of concept.
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome.
--D
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=soak-duration--- check | 14 +++++++++ common/config | 7 ++++ common/fuzzy | 7 ++++ common/rc | 34 +++++++++++++++++++++ common/report | 1 + ltp/fsstress.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++-- ltp/fsx.c | 50 +++++++++++++++++++++++++++++++ src/soak_duration.awk | 23 ++++++++++++++ tests/generic/019 | 1 + tests/generic/388 | 2 + tests/generic/475 | 2 +
Quoted 9 lines
tests/generic/476 | 7 +++- tests/generic/482 | 5 +++ tests/generic/521 | 1 + tests/generic/522 | 1 + tests/generic/642 | 1 + tests/generic/648 | 8 +++-- 17 files changed, 229 insertions(+), 13 deletions(-) create mode 100644 src/soak_duration.awk
The set looks good to me (the second commit has different var name, but fine by me)
Signature · 1 line
- Andrey
Quoted 69 lines · Andrey Albershteyn
Quoted 58 lines · Darrick J. Wong
Hi all,
One of the things that I do as a maintainer is to designate a handful of VMs to run fstests for unusually long periods of time. This practice I call long term soak testing. There are actually three separate fleets for this -- one runs alongside the nightly builds, one runs alongside weekly rebases, and the last one runs stable releases.
My interactions with all three fleets is pretty much the same -- load current builds of software, and try to run the exerciser tests for a duration of time -- 12 hours, 6.5 days, 30 days, etc.
TIME_FACTORdoes not work well for this usage model, because it is difficult to guess the correct time factor given that the VMs are hetergeneous and the IO completion rate is not perfectly predictable.Worse yet, if you want to run (say) all the recoveryloop tests on one VM (because recoveryloop is prone to crashing), it's impossible to set a
TIME_FACTORso that each loop test gets equal runtime. That can be hacked around with config sections, but that doesn't solve the first problem.This series introduces a new configuration variable,
SOAK_DURATION, that allows test runners to control directly various long soak and looping recovery tests. This is intended to be an alternative toTIME_FACTOR, since that variable usually adjusts operation counts, which are proportional to runtime but otherwise not a direct measure of time.With this override in place, I can configure the long soak fleet to run for exactly as long as I want them to, and they actually hit the time budget targets. The recoveryloop fleet now divides looping-test time equally among the four that are in that group so that they all get ~3 hours of coverage every night.
There are more tests that could use this than I actually modified here, but I've done enough to show this off as a proof of concept.
If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome.
--D
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=soak-duration--- check | 14 +++++++++ common/config | 7 ++++ common/fuzzy | 7 ++++ common/rc | 34 +++++++++++++++++++++ common/report | 1 + ltp/fsstress.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++-- ltp/fsx.c | 50 +++++++++++++++++++++++++++++++ src/soak_duration.awk | 23 ++++++++++++++ tests/generic/019 | 1 + tests/generic/388 | 2 + tests/generic/475 | 2 +Quoted 9 lines
tests/generic/476 | 7 +++- tests/generic/482 | 5 +++ tests/generic/521 | 1 + tests/generic/522 | 1 + tests/generic/642 | 1 + tests/generic/648 | 8 +++-- 17 files changed, 229 insertions(+), 13 deletions(-) create mode 100644 src/soak_duration.awk
Quoted 2 lines
The set looks good to me (the second commit has different var name, but fine by me)
Which variable name, specifically?
--D
Quoted 5 lines
Reviewed-by Andrey Albershteyn <[email protected]>Signature · 1 line
- Andrey
Quoted 10 lines · Darrick J. Wong
Quoted 7 lines · Andrey Albershteyn
Quoted 4 lines
tests/generic/648 | 8 +++-- 17 files changed, 229 insertions(+), 13 deletions(-) create mode 100644 src/soak_duration.awkThe set looks good to me (the second commit has different var name, but fine by me)
Which variable name, specifically?
STRESS_DURATION in the commit message
Signature · 1 line
- Andrey
Quoted 7 lines · Andrey Albershteyn
Quoted 6 lines · Darrick J. Wong
Quoted 5 lines · Andrey Albershteyn
Quoted 4 lines
tests/generic/648 | 8 +++-- 17 files changed, 229 insertions(+), 13 deletions(-) create mode 100644 src/soak_duration.awk
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
The set looks good to me (the second commit has different var name, but fine by me)
Which variable name, specifically?
STRESS_DURATIONin the commit message
Ah, will fix and resend. Thanks for noticing!
--D
Quoted 3 lines
Signature · 1 line
- Andrey