xfs: testcases for shrinking free space in the last AG
v2 cover xfs: testcases for shrinking free space in the last AG
Hi,
This version matches
kernel v8: https://lore.kernel.org/r/[email protected]
xfsprogs RFC: https://lore.kernel.org/r/[email protected]
and mainly addresses comments for the previous version.
Thanks,
Gao Xinag
changes since v1:
- add a _require_xfs_shrink helper (Zorro, Darrick);
- introduce a basic functionality test (Darrick);
- drop prefix "_" for all functions in tests (Darrick);
- introduce TIME_FACTOR and LOAD_FACTOR to adjust the load (Darrick);
- move $decsize out of the loop (Darrick);
- use $XFS_GROWFS_PROG instead of xfs_growfs (Darrick);
- query the running size instead of calculating (Darrick).
common/xfs | 10 ++++ tests/xfs/990 | 59 ++++++++++++++++++++++ tests/xfs/990.out | 12 +++++ tests/xfs/991 | 121 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/991.out | 8 +++ tests/xfs/group | 2 + 6 files changed, 212 insertions(+) create mode 100755 tests/xfs/990 create mode 100644 tests/xfs/990.out create mode 100755 tests/xfs/991 create mode 100644 tests/xfs/991.out
Signature · 1 line
2.27.0
RFC PATCH v2 1/3 common/xfs: add a _require_xfs_shrink helper
In order to detect whether the current kernel supports XFS shrinking.
---
use -D1 rather than -D0 since xfs_growfs would report unchanged size instead.
common/xfs | 10 ++++++++++ 1 file changed, 10 insertions(+)
common/xfs +10 -0
@@ -432,6 +432,16 @@ _supports_xfs_scrub() return 0 } +_require_xfs_shrink() +{ + _scratch_mkfs_xfs >/dev/null 2>&1 + + _scratch_mount + $XFS_GROWFS_PROG -D1 "$SCRATCH_MNT" 2>&1 | grep -q 'Invalid argument' || { \ + _scratch_unmount; _notrun "kernel does not support shrinking"; } + _scratch_unmount +} + # run xfs_check and friends on a FS. _check_xfs_filesystem() {
Signature · 1 line
2.27.0
Re: [RFC PATCH v2 1/3] common/xfs: add a _require_xfs_shrink helper
Quoted 15 lines · Gao Xiang
In order to detect whether the current kernel supports XFS shrinking.
Signed-off-by Gao Xiang <[email protected]>---
use -D1 rather than -D0 sincexfs_growfswould report unchanged size instead.common/xfs | 10 ++++++++++ 1 file changed, 10 insertions(+)common/xfs +0
-0
common/xfs _supports_xfs_scrub() · line 441
Quoted 10 lines
common/xfs +7
-0@@ -432,6 +432,16 @@ _supports_xfs_scrub() return 0 } +_require_xfs_shrink() +{ + _scratch_mkfs_xfs >/dev/null 2>&1 + + _scratch_mount + $XFS_GROWFS_PROG -D1 "$SCRATCH_MNT" 2>&1 | grep -q 'Invalid argument' || { \ + _scratch_unmount; _notrun "kernel does not support shrinking"; }
^^^^
I think this unmount isn't necessary, due to after "_notrun" the $SCRATCH_DEV will be umounted "automatically".
Quoted 9 lines
common/xfs +3
-0+ _scratch_unmount +} + # run xfs_check and friends on a FS. _check_xfs_filesystem() {Signature · 1 line
2.27.0
Re: [RFC PATCH v2 1/3] common/xfs: add a _require_xfs_shrink helper
Hi Zorro,
Quoted 19 lines · Zorro Lang
Quoted 18 lines · Gao Xiang
In order to detect whether the current kernel supports XFS shrinking.
Signed-off-by Gao Xiang <[email protected]>---
use -D1 rather than -D0 sincexfs_growfswould report unchanged size instead.common/xfs | 10 ++++++++++ 1 file changed, 10 insertions(+)common/xfs +0
-0@@ -432,6 +432,16 @@ _supports_xfs_scrub() return 0 }
Quoted 10 lines
common/xfs _supports_xfs_scrub() · line 441
Quoted 7 lines
common/xfs +7
-0+_require_xfs_shrink() +{ + _scratch_mkfs_xfs >/dev/null 2>&1 + + _scratch_mount + $XFS_GROWFS_PROG -D1 "$SCRATCH_MNT" 2>&1 | grep -q 'Invalid argument' || { \ + _scratch_unmount; _notrun "kernel does not support shrinking"; }^^^^
I think this unmount isn't necessary, due to after "_notrun" the $SCRATCH_DEV will be umounted "automatically".
I didn't dig into it more. Just follow _require_xfs_scratch_rmapbt().
If it's not necessary (assumed ./check does that), will update it in the next version. Thanks for your suggestion.
Thanks,
Gao Xiang
RFC PATCH v2 2/3 xfs: basic functionality test for shrinking free space in the last AG
In reply to Gao XiangAdd basic test to make sure the functionality works as expected.
--- tests/xfs/990 | 59 +++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/990.out | 12 ++++++++++ tests/xfs/group | 1 + 3 files changed, 72 insertions(+) create mode 100755 tests/xfs/990 create mode 100644 tests/xfs/990.out
tests/xfs/990 new file +59 -0
@@ -0,0 +1,59 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2021 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test 990 +# +# XFS shrinkfs basic functionality test +# +# This test attempts to shrink with a small size (512K), half AG size and +# an out-of-bound size (agsize + 1) to observe if it works as expected. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs xfs +_require_xfs_shrink + +rm -f $seqres.full +echo "Format and mount" +size="$((512 * 1024 * 1024))" +_scratch_mkfs -dsize=$size -dagcount=3 2>&1 | \ + tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs +. $tmp.mkfs +_scratch_mount >> $seqres.full 2>&1 + +echo "Shrink fs (small size)" +$XFS_GROWFS_PROG -D $((dblocks-512*1024/dbsize)) $SCRATCH_MNT \ + >> $seqres.full 2>&1 || echo failure +_scratch_cycle_mount + +echo "Shrink fs (half AG)" +$XFS_GROWFS_PROG -D $((dblocks-agsize/2)) $SCRATCH_MNT \ + >> $seqres.full 2>&1 || echo failure +_scratch_cycle_mount + +echo "Shrink fs (out-of-bound)" +$XFS_GROWFS_PROG -D $((dblocks-agsize-1)) $SCRATCH_MNT \ + >> $seqres.full 2>&1 && echo failure +_scratch_cycle_mount + +$XFS_INFO_PROG $SCRATCH_MNT >> $seqres.full + +_scratch_unmount +echo "*** done" + +# success, all done +status=0 +exit
tests/xfs/990.out new file +12 -0
@@ -0,0 +1,12 @@ +QA output created by 990 +Format and mount +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks +data = bsize=XXX blocks=XXX, imaxpct=PCT + = sunit=XXX swidth=XXX, unwritten=X +naming =VERN bsize=XXX +log =LDEV bsize=XXX blocks=XXX +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX +Shrink fs (small size) +Shrink fs (half AG) +Shrink fs (out-of-bound) +*** done
tests/xfs/group +1 -0
@@ -525,3 +525,4 @@ 525 auto quick mkfs 526 auto quick mkfs 527 auto quick quota +990 auto quick growfs
Signature · 1 line
2.27.0
Re: [RFC PATCH v2 2/3] xfs: basic functionality test for shrinking free space in the last AG
Quoted 33 lines · Gao Xiang
Add basic test to make sure the functionality works as expected.
Signed-off-by Gao Xiang <[email protected]>--- tests/xfs/990 | 59 +++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/990.out | 12 ++++++++++ tests/xfs/group | 1 + 3 files changed, 72 insertions(+) create mode 100755 tests/xfs/990 create mode 100644 tests/xfs/990.outtests/xfs/990 new file +16
-0@@ -0,0 +1,59 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2021 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test 990 +# +# XFS shrinkfs basic functionality test +# +# This test attempts to shrink with a small size (512K), half AG size and +# an out-of-bound size (agsize + 1) to observe if it works as expected. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd`
tests/xfs/990 line 26
Quoted 10 lines
tests/xfs/990 new file +10
-0+tmp=/tmp/$$ +status=1 # failure is the default! +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs xfs
_require_scratch
tests/xfs/990 line 31
Quoted 5 lines
tests/xfs/990 new file +5
-0+_require_xfs_shrink + +rm -f $seqres.full +echo "Format and mount" +size="$((512 * 1024 * 1024))"
Is the fixed size necessary? Is that better to let testers run this test with their different device/XFS geometry.
tests/xfs/990 line 40
Quoted 9 lines
tests/xfs/990 new file +9
-0+_scratch_mkfs -dsize=$size -dagcount=3 2>&1 | \ + tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs +. $tmp.mkfs +_scratch_mount >> $seqres.full 2>&1 + +echo "Shrink fs (small size)" +$XFS_GROWFS_PROG -D $((dblocks-512*1024/dbsize)) $SCRATCH_MNT \ + >> $seqres.full 2>&1 || echo failure +_scratch_cycle_mount
I don't understand the XFS Shrink new feature that much, is the "cycle_mount" necessary? If it's not, can we get more chances to find bugs without "cycle_mount", or with a fsck?
Another question is, should we verify the new size after shrink?
Quoted 4 lines
tests/xfs/990 new file +4
-0+ +echo "Shrink fs (half AG)" +$XFS_GROWFS_PROG -D $((dblocks-agsize/2)) $SCRATCH_MNT \ + >> $seqres.full 2>&1 || echo failure
tests/xfs/990 line 54
Quoted 10 lines
tests/xfs/990 new file +10
-0+_scratch_cycle_mount + +echo "Shrink fs (out-of-bound)" +$XFS_GROWFS_PROG -D $((dblocks-agsize-1)) $SCRATCH_MNT \ + >> $seqres.full 2>&1 && echo failure +_scratch_cycle_mount + +$XFS_INFO_PROG $SCRATCH_MNT >> $seqres.full + +_scratch_unmount
^^^
It's not necessary.
Quoted 35 lines
tests/xfs/990 new file +5
-0+echo "*** done" + +# success, all done +status=0 +exittests/xfs/990.out new file +12
-0@@ -0,0 +1,12 @@ +QA output created by 990 +Format and mount +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks +data = bsize=XXX blocks=XXX, imaxpct=PCT + = sunit=XXX swidth=XXX, unwritten=X +naming =VERN bsize=XXX +log =LDEV bsize=XXX blocks=XXX +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX +Shrink fs (small size) +Shrink fs (half AG) +Shrink fs (out-of-bound) +*** donetests/xfs/group +1
-0@@ -525,3 +525,4 @@ 525 auto quick mkfs 526 auto quick mkfs 527 auto quick quota +990 auto quick growfsSignature · 1 line
2.27.0
Re: [RFC PATCH v2 2/3] xfs: basic functionality test for shrinking free space in the last AG
Quoted 38 lines · Zorro Lang
Quoted 37 lines · Gao Xiang
Add basic test to make sure the functionality works as expected.
Signed-off-by Gao Xiang <[email protected]>--- tests/xfs/990 | 59 +++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/990.out | 12 ++++++++++ tests/xfs/group | 1 + 3 files changed, 72 insertions(+) create mode 100755 tests/xfs/990 create mode 100644 tests/xfs/990.outtests/xfs/990 new file +20
-0@@ -0,0 +1,59 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2021 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test 990 +# +# XFS shrinkfs basic functionality test +# +# This test attempts to shrink with a small size (512K), half AG size and +# an out-of-bound size (agsize + 1) to observe if it works as expected. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15 +
Quoted 8 lines
tests/xfs/990 line 26
Quoted 6 lines
tests/xfs/990 new file +6
-0+# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# real QA test starts here +_supported_fs xfs
_require_scratch
Will fix.
Quoted 9 lines
tests/xfs/990 line 31
Quoted 6 lines
tests/xfs/990 new file +5
-0+_require_xfs_shrink + +rm -f $seqres.full +echo "Format and mount" +size="$((512 * 1024 * 1024))"Is the fixed size necessary? Is that better to let testers run this test with their different device/XFS geometry.
I'm fine with either way since it's a simple functionality test, yet for most common cases, stratch devices are somewhat large. I tend to use a relative controllable small value.
Actually, this case was from xfs/127 with some modification.
Quoted 14 lines
tests/xfs/990 line 40
Quoted 10 lines
tests/xfs/990 new file +9
-0+_scratch_mkfs -dsize=$size -dagcount=3 2>&1 | \ + tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs +. $tmp.mkfs +_scratch_mount >> $seqres.full 2>&1 + +echo "Shrink fs (small size)" +$XFS_GROWFS_PROG -D $((dblocks-512*1024/dbsize)) $SCRATCH_MNT \ + >> $seqres.full 2>&1 || echo failure +_scratch_cycle_mountI don't understand the XFS Shrink new feature that much, is the "
cycle_mount" necessary? If it's not, can we get more chances to find bugs without "cycle_mount", or with a fsck?
maybe it's useful to test unmount here. Yeah, I think it's better to try fsck here. Good idea.
Quoted 2 lines
Another question is, should we verify the new size after shrink?
Yeah, will add xfs_info.
Quoted 17 lines
Quoted 5 lines
tests/xfs/990 new file +4
-0+ +echo "Shrink fs (half AG)" +$XFS_GROWFS_PROG -D $((dblocks-agsize/2)) $SCRATCH_MNT \ + >> $seqres.full 2>&1 || echo failuretests/xfs/990 line 54
Quoted 10 lines
tests/xfs/990 new file +10
-0+_scratch_cycle_mount + +echo "Shrink fs (out-of-bound)" +$XFS_GROWFS_PROG -D $((dblocks-agsize-1)) $SCRATCH_MNT \ + >> $seqres.full 2>&1 && echo failure +_scratch_cycle_mount + +$XFS_INFO_PROG $SCRATCH_MNT >> $seqres.full + +_scratch_unmount^^^
It's not necessary.
ok. It seems that ./check will fsck scratch device as well. Will update it.
Thanks,
Gao Xiang
RFC PATCH v2 3/3 xfs: stress test for shrinking free space in the last AG
In reply to Gao XiangThis adds a stress testcase to shrink free space as much as possible in the last AG with background fsstress workload.
The expectation is that no crash happens with expected output.
---
Note that I don't use _fill_fs instead, since fill_scratch here mainly to eat 125M to make fsstress more effectively, rather than fill data as much as possible.
tests/xfs/991 | 121 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/991.out | 8 +++ tests/xfs/group | 1 + 3 files changed, 130 insertions(+) create mode 100755 tests/xfs/991 create mode 100644 tests/xfs/991.out
tests/xfs/991 new file +121 -0
@@ -0,0 +1,121 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2020-2021 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test 991 +# +# XFS online shrinkfs stress test +# +# This test attempts to shrink unused space as much as possible with +# background fsstress workload. It will decrease the shrink size if +# larger size fails. And totally repeat 2 * TIME_FACTOR times. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +create_scratch() +{ + _scratch_mkfs_xfs $@ | tee -a $seqres.full | \ + _filter_mkfs 2>$tmp.mkfs >/dev/null + . $tmp.mkfs + + if ! _try_scratch_mount 2>/dev/null; then + echo "failed to mount $SCRATCH_DEV" + exit 1 + fi + + # fix the reserve block pool to a known size so that the enospc + # calculations work out correctly. + _scratch_resvblks 1024 > /dev/null 2>&1 +} + +fill_scratch() +{ + $XFS_IO_PROG -f -c "resvsp 0 ${1}" $SCRATCH_MNT/resvfile +} + +stress_scratch() +{ + procs=3 + nops=$((1000 * LOAD_FACTOR)) + # -w ensures that the only ops are ones which cause write I/O + FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \ + -n $nops $FSSTRESS_AVOID` + $FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 & +} + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_shrink +_require_xfs_io_command "falloc" + +rm -f $seqres.full +_scratch_mkfs_xfs | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs +. $tmp.mkfs # extract blocksize and data size for scratch device + +decsize=`expr 42 \* 1048576` # shrink in chunks of this size at most +endsize=`expr 125 \* 1048576` # stop after shrinking this big +[ `expr $endsize / $dbsize` -lt $dblocks ] || _notrun "Scratch device too small" + +nags=2 +totalcount=$((2 * TIME_FACTOR)) + +while [ $totalcount -gt 0 ]; do + size=`expr 1010 \* 1048576` # 1010 megabytes initially + logblks=$(_scratch_find_xfs_min_logblocks -dsize=${size} -dagcount=${nags}) + + create_scratch -lsize=${logblks}b -dsize=${size} -dagcount=${nags} + + for i in `seq 125 -1 90`; do + fillsize=`expr $i \* 1048576` + out="$(fill_scratch $fillsize 2>&1)" + echo "$out" | grep -q 'No space left on device' && continue + test -n "${out}" && echo "$out" + break + done + + while [ $size -gt $endsize ]; do + stress_scratch + sleep 1 + + decb=`expr $decsize / $dbsize` # in data blocks + while [ $decb -gt 0 ]; do + sizeb=`expr $size / $dbsize - $decb` + + $XFS_GROWFS_PROG -D ${sizeb} $SCRATCH_MNT \ + >> $seqres.full 2>&1 && break + + [ $decb -gt 100 ] && decb=`expr $decb + $RANDOM % 10` + decb=`expr $decb / 2` + done + + wait + [ $decb -eq 0 ] && break + + # get latest dblocks + $XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null + . $tmp.growfs + + size=`expr $dblocks \* $dbsize` + _scratch_cycle_mount + done + + _scratch_unmount + _repair_scratch_fs >> $seqres.full + totalcount=`expr $totalcount - 1` +done + +echo "*** done" +status=0 +exit
tests/xfs/991.out new file +8 -0
@@ -0,0 +1,8 @@ +QA output created by 991 +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks +data = bsize=XXX blocks=XXX, imaxpct=PCT + = sunit=XXX swidth=XXX, unwritten=X +naming =VERN bsize=XXX +log =LDEV bsize=XXX blocks=XXX +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX +*** done
tests/xfs/group +1 -0
@@ -526,3 +526,4 @@ 526 auto quick mkfs 527 auto quick quota 990 auto quick growfs +991 auto quick growfs
Signature · 1 line
2.27.0
Re: [RFC PATCH v2 3/3] xfs: stress test for shrinking free space in the last AG
Quoted 10 lines · Gao Xiang
This adds a stress testcase to shrink free space as much as possible in the last AG with background fsstress workload.
The expectation is that no crash happens with expected output.
Signed-off-by Gao Xiang <[email protected]>---
Note that I don't use_fill_fsinstead, sincefill_scratchhere mainly to eat 125M to make fsstress more effectively, rather than fill data as much as possible.
As Darrick had given lots of review points to this case, I just have 2 picky questions as below:)
Quoted 65 lines
tests/xfs/991 | 121 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/991.out | 8 +++ tests/xfs/group | 1 + 3 files changed, 130 insertions(+) create mode 100755 tests/xfs/991 create mode 100644 tests/xfs/991.outtests/xfs/991 new file +51
-0@@ -0,0 +1,121 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2020-2021 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test 991 +# +# XFS online shrinkfs stress test +# +# This test attempts to shrink unused space as much as possible with +# background fsstress workload. It will decrease the shrink size if +# larger size fails. And totally repeat 2 * TIME_FACTOR times. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +create_scratch() +{ + _scratch_mkfs_xfs $@ | tee -a $seqres.full | \ + _filter_mkfs 2>$tmp.mkfs >/dev/null + . $tmp.mkfs + + if ! _try_scratch_mount 2>/dev/null; then + echo "failed to mount $SCRATCH_DEV" + exit 1 + fi + + # fix the reserve block pool to a known size so that the enospc + # calculations work out correctly. + _scratch_resvblks 1024 > /dev/null 2>&1 +} + +fill_scratch() +{ + $XFS_IO_PROG -f -c "resvsp 0 ${1}" $SCRATCH_MNT/resvfile +} + +stress_scratch() +{ + procs=3 + nops=$((1000 * LOAD_FACTOR)) + # -w ensures that the only ops are ones which cause write I/O
tests/xfs/991 line 61
Quoted 10 lines
tests/xfs/991 new file +10
-0+ FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \ + -n $nops $FSSTRESS_AVOID` + $FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 & +} + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_shrink +_require_xfs_io_command "falloc"
Do I miss something? I only found you use xfs_io "resvsp", why you need "falloc" cmd?
Quoted 79 lines
tests/xfs/991 new file +60
-0+ +rm -f $seqres.full +_scratch_mkfs_xfs | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs +. $tmp.mkfs # extract blocksize and data size for scratch device + +decsize=`expr 42 \* 1048576` # shrink in chunks of this size at most +endsize=`expr 125 \* 1048576` # stop after shrinking this big +[ `expr $endsize / $dbsize` -lt $dblocks ] || _notrun "Scratch device too small" + +nags=2 +totalcount=$((2 * TIME_FACTOR)) + +while [ $totalcount -gt 0 ]; do + size=`expr 1010 \* 1048576` # 1010 megabytes initially + logblks=$(_scratch_find_xfs_min_logblocks -dsize=${size} -dagcount=${nags}) + + create_scratch -lsize=${logblks}b -dsize=${size} -dagcount=${nags} + + for i in `seq 125 -1 90`; do + fillsize=`expr $i \* 1048576` + out="$(fill_scratch $fillsize 2>&1)" + echo "$out" | grep -q 'No space left on device' && continue + test -n "${out}" && echo "$out" + break + done + + while [ $size -gt $endsize ]; do + stress_scratch + sleep 1 + + decb=`expr $decsize / $dbsize` # in data blocks + while [ $decb -gt 0 ]; do + sizeb=`expr $size / $dbsize - $decb` + + $XFS_GROWFS_PROG -D ${sizeb} $SCRATCH_MNT \ + >> $seqres.full 2>&1 && break + + [ $decb -gt 100 ] && decb=`expr $decb + $RANDOM % 10` + decb=`expr $decb / 2` + done + + wait + [ $decb -eq 0 ] && break + + # get latest dblocks + $XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null + . $tmp.growfs + + size=`expr $dblocks \* $dbsize` + _scratch_cycle_mount + done + + _scratch_unmount + _repair_scratch_fs >> $seqres.full + totalcount=`expr $totalcount - 1` +done + +echo "*** done" +status=0 +exittests/xfs/991.out new file +8
-0@@ -0,0 +1,8 @@ +QA output created by 991 +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks +data = bsize=XXX blocks=XXX, imaxpct=PCT + = sunit=XXX swidth=XXX, unwritten=X +naming =VERN bsize=XXX +log =LDEV bsize=XXX blocks=XXX +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX +*** donetests/xfs/group +0
-0
tests/xfs/group line 529
Quoted 4 lines
tests/xfs/group +1
-0@@ -526,3 +526,4 @@ 526 auto quick mkfs 527 auto quick quota 990 auto quick growfs +991 auto quick growfs
Is this "stress" test case really "quick" :)
Quoted 3 lines
Signature · 1 line
2.27.0
Re: [RFC PATCH v2 3/3] xfs: stress test for shrinking free space in the last AG
Quoted 84 lines · Zorro Lang
Quoted 10 lines · Gao Xiang
This adds a stress testcase to shrink free space as much as possible in the last AG with background fsstress workload.
The expectation is that no crash happens with expected output.
Signed-off-by Gao Xiang <[email protected]>---
Note that I don't use_fill_fsinstead, sincefill_scratchhere mainly to eat 125M to make fsstress more effectively, rather than fill data as much as possible.As Darrick had given lots of review points to this case, I just have 2 picky questions as below:)
Quoted 69 lines
tests/xfs/991 | 121 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/991.out | 8 +++ tests/xfs/group | 1 + 3 files changed, 130 insertions(+) create mode 100755 tests/xfs/991 create mode 100644 tests/xfs/991.outtests/xfs/991 new file +55
-0@@ -0,0 +1,121 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2020-2021 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test 991 +# +# XFS online shrinkfs stress test +# +# This test attempts to shrink unused space as much as possible with +# background fsstress workload. It will decrease the shrink size if +# larger size fails. And totally repeat 2 * TIME_FACTOR times. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +create_scratch() +{ + _scratch_mkfs_xfs $@ | tee -a $seqres.full | \ + _filter_mkfs 2>$tmp.mkfs >/dev/null + . $tmp.mkfs + + if ! _try_scratch_mount 2>/dev/null; then + echo "failed to mount $SCRATCH_DEV" + exit 1 + fi + + # fix the reserve block pool to a known size so that the enospc + # calculations work out correctly. + _scratch_resvblks 1024 > /dev/null 2>&1 +} + +fill_scratch() +{ + $XFS_IO_PROG -f -c "resvsp 0 ${1}" $SCRATCH_MNT/resvfile +} + +stress_scratch() +{ + procs=3 + nops=$((1000 * LOAD_FACTOR)) + # -w ensures that the only ops are ones which cause write I/O + FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \ + -n $nops $FSSTRESS_AVOID` + $FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 & +}
Quoted 8 lines
tests/xfs/991 line 61
Quoted 6 lines
tests/xfs/991 new file +6
-0+ +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_shrink +_require_xfs_io_command "falloc"Do I miss something? I only found you use
xfs_io"resvsp", why you need "falloc" cmd?
As I mentioned before, the testcase was derived from xfs/104 with some modification.
At a quick glance, this line was added by commit 09e94f84d929 ("xfs: don't assume preallocation is always supported on XFS"). I have no more background yet.
Quoted 78 lines
Quoted 78 lines
tests/xfs/991 new file +60
-0+ +rm -f $seqres.full +_scratch_mkfs_xfs | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs +. $tmp.mkfs # extract blocksize and data size for scratch device + +decsize=`expr 42 \* 1048576` # shrink in chunks of this size at most +endsize=`expr 125 \* 1048576` # stop after shrinking this big +[ `expr $endsize / $dbsize` -lt $dblocks ] || _notrun "Scratch device too small" + +nags=2 +totalcount=$((2 * TIME_FACTOR)) + +while [ $totalcount -gt 0 ]; do + size=`expr 1010 \* 1048576` # 1010 megabytes initially + logblks=$(_scratch_find_xfs_min_logblocks -dsize=${size} -dagcount=${nags}) + + create_scratch -lsize=${logblks}b -dsize=${size} -dagcount=${nags} + + for i in `seq 125 -1 90`; do + fillsize=`expr $i \* 1048576` + out="$(fill_scratch $fillsize 2>&1)" + echo "$out" | grep -q 'No space left on device' && continue + test -n "${out}" && echo "$out" + break + done + + while [ $size -gt $endsize ]; do + stress_scratch + sleep 1 + + decb=`expr $decsize / $dbsize` # in data blocks + while [ $decb -gt 0 ]; do + sizeb=`expr $size / $dbsize - $decb` + + $XFS_GROWFS_PROG -D ${sizeb} $SCRATCH_MNT \ + >> $seqres.full 2>&1 && break + + [ $decb -gt 100 ] && decb=`expr $decb + $RANDOM % 10` + decb=`expr $decb / 2` + done + + wait + [ $decb -eq 0 ] && break + + # get latest dblocks + $XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null + . $tmp.growfs + + size=`expr $dblocks \* $dbsize` + _scratch_cycle_mount + done + + _scratch_unmount + _repair_scratch_fs >> $seqres.full + totalcount=`expr $totalcount - 1` +done + +echo "*** done" +status=0 +exittests/xfs/991.out new file +8
-0@@ -0,0 +1,8 @@ +QA output created by 991 +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks +data = bsize=XXX blocks=XXX, imaxpct=PCT + = sunit=XXX swidth=XXX, unwritten=X +naming =VERN bsize=XXX +log =LDEV bsize=XXX blocks=XXX +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX +*** donetests/xfs/group +0
-1--- a/tests/xfs/group
Quoted 8 lines
tests/xfs/group line 529
Quoted 6 lines
+++ b/tests/xfs/group
tests/xfs/group +1
-0@@ -526,3 +526,4 @@ 526 auto quick mkfs 527 auto quick quota 990 auto quick growfs +991 auto quick growfsIs this "stress" test case really "quick" :)
ok, will update.
Thanks,
Gao Xiang
Quoted 5 lines
Quoted 4 lines
Signature · 2 lines
-- 2.27.0
Re: [RFC PATCH v2 3/3] xfs: stress test for shrinking free space in the last AG
Quoted 94 lines · Gao Xiang
Quoted 84 lines · Zorro Lang
Quoted 10 lines · Gao Xiang
This adds a stress testcase to shrink free space as much as possible in the last AG with background fsstress workload.
The expectation is that no crash happens with expected output.
Signed-off-by Gao Xiang <[email protected]>---
Note that I don't use_fill_fsinstead, sincefill_scratchhere mainly to eat 125M to make fsstress more effectively, rather than fill data as much as possible.As Darrick had given lots of review points to this case, I just have 2 picky questions as below:)
Quoted 69 lines
tests/xfs/991 | 121 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/991.out | 8 +++ tests/xfs/group | 1 + 3 files changed, 130 insertions(+) create mode 100755 tests/xfs/991 create mode 100644 tests/xfs/991.outtests/xfs/991 new file +55
-0@@ -0,0 +1,121 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2020-2021 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test 991 +# +# XFS online shrinkfs stress test +# +# This test attempts to shrink unused space as much as possible with +# background fsstress workload. It will decrease the shrink size if +# larger size fails. And totally repeat 2 * TIME_FACTOR times. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +create_scratch() +{ + _scratch_mkfs_xfs $@ | tee -a $seqres.full | \ + _filter_mkfs 2>$tmp.mkfs >/dev/null + . $tmp.mkfs + + if ! _try_scratch_mount 2>/dev/null; then + echo "failed to mount $SCRATCH_DEV" + exit 1 + fi + + # fix the reserve block pool to a known size so that the enospc + # calculations work out correctly. + _scratch_resvblks 1024 > /dev/null 2>&1 +} + +fill_scratch() +{ + $XFS_IO_PROG -f -c "resvsp 0 ${1}" $SCRATCH_MNT/resvfile +} + +stress_scratch() +{ + procs=3 + nops=$((1000 * LOAD_FACTOR)) + # -w ensures that the only ops are ones which cause write I/O + FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \ + -n $nops $FSSTRESS_AVOID` + $FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 & +}Quoted 8 lines
tests/xfs/991 line 61
Quoted 6 lines
tests/xfs/991 new file +6
-0+ +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_shrink +_require_xfs_io_command "falloc"Do I miss something? I only found you use
xfs_io"resvsp", why you need "falloc" cmd?
Quoted 6 lines
As I mentioned before, the testcase was derived from xfs/104 with some modification.
At a quick glance, this line was added by commit
09e94f84d929("xfs: don't assume preallocation is always supported on XFS"). I have no more background yet.
Why not use xfs_io falloc in the test? fallocate is the successor to resvsp.
--D
Quoted 98 lines
Quoted 78 lines
Quoted 78 lines
tests/xfs/991 new file +60
-0+ +rm -f $seqres.full +_scratch_mkfs_xfs | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs +. $tmp.mkfs # extract blocksize and data size for scratch device + +decsize=`expr 42 \* 1048576` # shrink in chunks of this size at most +endsize=`expr 125 \* 1048576` # stop after shrinking this big +[ `expr $endsize / $dbsize` -lt $dblocks ] || _notrun "Scratch device too small" + +nags=2 +totalcount=$((2 * TIME_FACTOR)) + +while [ $totalcount -gt 0 ]; do + size=`expr 1010 \* 1048576` # 1010 megabytes initially + logblks=$(_scratch_find_xfs_min_logblocks -dsize=${size} -dagcount=${nags}) + + create_scratch -lsize=${logblks}b -dsize=${size} -dagcount=${nags} + + for i in `seq 125 -1 90`; do + fillsize=`expr $i \* 1048576` + out="$(fill_scratch $fillsize 2>&1)" + echo "$out" | grep -q 'No space left on device' && continue + test -n "${out}" && echo "$out" + break + done + + while [ $size -gt $endsize ]; do + stress_scratch + sleep 1 + + decb=`expr $decsize / $dbsize` # in data blocks + while [ $decb -gt 0 ]; do + sizeb=`expr $size / $dbsize - $decb` + + $XFS_GROWFS_PROG -D ${sizeb} $SCRATCH_MNT \ + >> $seqres.full 2>&1 && break + + [ $decb -gt 100 ] && decb=`expr $decb + $RANDOM % 10` + decb=`expr $decb / 2` + done + + wait + [ $decb -eq 0 ] && break + + # get latest dblocks + $XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null + . $tmp.growfs + + size=`expr $dblocks \* $dbsize` + _scratch_cycle_mount + done + + _scratch_unmount + _repair_scratch_fs >> $seqres.full + totalcount=`expr $totalcount - 1` +done + +echo "*** done" +status=0 +exittests/xfs/991.out new file +8
-0@@ -0,0 +1,8 @@ +QA output created by 991 +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks +data = bsize=XXX blocks=XXX, imaxpct=PCT + = sunit=XXX swidth=XXX, unwritten=X +naming =VERN bsize=XXX +log =LDEV bsize=XXX blocks=XXX +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX +*** donetests/xfs/group +0
-1--- a/tests/xfs/groupQuoted 8 lines
tests/xfs/group line 529
Quoted 6 lines
+++ b/tests/xfs/group
tests/xfs/group +1
-0@@ -526,3 +526,4 @@ 526 auto quick mkfs 527 auto quick quota 990 auto quick growfs +991 auto quick growfsIs this "stress" test case really "quick" :)
ok, will update.
Thanks,
Gao XiangQuoted 5 lines
Quoted 4 lines
Signature · 2 lines
-- 2.27.0
Re: [RFC PATCH v2 3/3] xfs: stress test for shrinking free space in the last AG
Quoted 98 lines · Darrick J. Wong
Quoted 97 lines · Gao Xiang
Quoted 84 lines · Zorro Lang
Quoted 10 lines · Gao Xiang
This adds a stress testcase to shrink free space as much as possible in the last AG with background fsstress workload.
The expectation is that no crash happens with expected output.
Signed-off-by Gao Xiang <[email protected]>---
Note that I don't use_fill_fsinstead, sincefill_scratchhere mainly to eat 125M to make fsstress more effectively, rather than fill data as much as possible.As Darrick had given lots of review points to this case, I just have 2 picky questions as below:)
Quoted 69 lines
tests/xfs/991 | 121 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/991.out | 8 +++ tests/xfs/group | 1 + 3 files changed, 130 insertions(+) create mode 100755 tests/xfs/991 create mode 100644 tests/xfs/991.outtests/xfs/991 new file +55
-0@@ -0,0 +1,121 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2020-2021 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test 991 +# +# XFS online shrinkfs stress test +# +# This test attempts to shrink unused space as much as possible with +# background fsstress workload. It will decrease the shrink size if +# larger size fails. And totally repeat 2 * TIME_FACTOR times. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +create_scratch() +{ + _scratch_mkfs_xfs $@ | tee -a $seqres.full | \ + _filter_mkfs 2>$tmp.mkfs >/dev/null + . $tmp.mkfs + + if ! _try_scratch_mount 2>/dev/null; then + echo "failed to mount $SCRATCH_DEV" + exit 1 + fi + + # fix the reserve block pool to a known size so that the enospc + # calculations work out correctly. + _scratch_resvblks 1024 > /dev/null 2>&1 +} + +fill_scratch() +{ + $XFS_IO_PROG -f -c "resvsp 0 ${1}" $SCRATCH_MNT/resvfile +} + +stress_scratch() +{ + procs=3 + nops=$((1000 * LOAD_FACTOR)) + # -w ensures that the only ops are ones which cause write I/O + FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \ + -n $nops $FSSTRESS_AVOID` + $FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 & +}Quoted 8 lines
tests/xfs/991 line 61
Quoted 6 lines
tests/xfs/991 new file +6
-0+ +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_shrink +_require_xfs_io_command "falloc"Do I miss something? I only found you use
xfs_io"resvsp", why you need "falloc" cmd?As I mentioned before, the testcase was derived from xfs/104 with some modification.
Quoted 6 lines
Quoted 3 lines
At a quick glance, this line was added by commit
09e94f84d929("xfs: don't assume preallocation is always supported on XFS"). I have no more background yet.Why not use
xfs_iofalloc in the test? fallocate is the successor to resvsp.
Yeah, general falloc seems better, and it seems _require_xfs_io_command here is used for always_cow inode feature. Will update it. Thanks!
Thanks,
Gao Xiang
Quoted 2 lines
--D