Skip to contentSkip to conversation

xfs: testcases for shrinking free space in the last AG

RFC PATCH Superseded Gao Xiang 3 participants 12 messages Mar 12, 2021

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Gao Xiang

author
#0 lore raw

v2 cover xfs: testcases for shrinking free space in the last AG

Hi,

This version matches
kernel v8: https://lore.kernel.org/r/[email protected]
xfsprogs RFC: https://lore.kernel.org/r/[email protected]

and mainly addresses comments for the previous version.

Thanks,
Gao Xinag

changes since v1:
- add a _require_xfs_shrink helper (Zorro, Darrick);
- introduce a basic functionality test (Darrick);
- drop prefix "_" for all functions in tests (Darrick);
- introduce TIME_FACTOR and LOAD_FACTOR to adjust the load (Darrick);
- move $decsize out of the loop (Darrick);
- use $XFS_GROWFS_PROG instead of xfs_growfs (Darrick);
- query the running size instead of calculating (Darrick).

 common/xfs        |  10 ++++
 tests/xfs/990     |  59 ++++++++++++++++++++++
 tests/xfs/990.out |  12 +++++
 tests/xfs/991     | 121 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/991.out |   8 +++
 tests/xfs/group   |   2 +
 6 files changed, 212 insertions(+)
 create mode 100755 tests/xfs/990
 create mode 100644 tests/xfs/990.out
 create mode 100755 tests/xfs/991
 create mode 100644 tests/xfs/991.out
Signature · 1 line
2.27.0
author
#1 lore raw

RFC PATCH v2 1/3 common/xfs: add a _require_xfs_shrink helper

In order to detect whether the current kernel supports XFS shrinking.

---
use -D1 rather than -D0 since xfs_growfs would report unchanged size instead.

 common/xfs | 10 ++++++++++
 1 file changed, 10 insertions(+)
common/xfs +10 -0
index 2156749d..326edacc 100644
@@ -432,6 +432,16 @@ _supports_xfs_scrub()
 	return 0
 }
 
+_require_xfs_shrink()
+{
+	_scratch_mkfs_xfs >/dev/null 2>&1
+
+	_scratch_mount
+	$XFS_GROWFS_PROG -D1 "$SCRATCH_MNT" 2>&1 | grep -q 'Invalid argument' || { \
+		_scratch_unmount; _notrun "kernel does not support shrinking"; }
+	_scratch_unmount
+}
+
 # run xfs_check and friends on a FS.
 _check_xfs_filesystem()
 {
Signature · 1 line
2.27.0
#2 lore raw

Re: [RFC PATCH v2 1/3] common/xfs: add a _require_xfs_shrink helper

Quoted 15 lines · Gao Xiang

In order to detect whether the current kernel supports XFS shrinking.

---
use -D1 rather than -D0 since xfs_growfs would report unchanged size instead.

 common/xfs | 10 ++++++++++
 1 file changed, 10 insertions(+)
common/xfs +0 -0
index 2156749d..326edacc 100644

common/xfs _supports_xfs_scrub() · line 441

Quoted 10 lines
common/xfs +7 -0
@@ -432,6 +432,16 @@ _supports_xfs_scrub()
 	return 0
 }
 
+_require_xfs_shrink()
+{
+	_scratch_mkfs_xfs >/dev/null 2>&1
+
+	_scratch_mount
+	$XFS_GROWFS_PROG -D1 "$SCRATCH_MNT" 2>&1 | grep -q 'Invalid argument' || { \
+		_scratch_unmount; _notrun "kernel does not support shrinking"; }

^^^^
I think this unmount isn't necessary, due to after "_notrun" the $SCRATCH_DEV will be umounted "automatically".

Quoted 9 lines
common/xfs +3 -0
+	_scratch_unmount
+}
+
 # run xfs_check and friends on a FS.
 _check_xfs_filesystem()
 {
Signature · 1 line
2.27.0
author
#3 lore raw

Re: [RFC PATCH v2 1/3] common/xfs: add a _require_xfs_shrink helper

Hi Zorro,

Quoted 19 lines · Zorro Lang
Quoted 18 lines · Gao Xiang

In order to detect whether the current kernel supports XFS shrinking.

---
use -D1 rather than -D0 since xfs_growfs would report unchanged size instead.

 common/xfs | 10 ++++++++++
 1 file changed, 10 insertions(+)
common/xfs +0 -0
index 2156749d..326edacc 100644
@@ -432,6 +432,16 @@ _supports_xfs_scrub()
 	return 0
 }
 
Quoted 10 lines

common/xfs _supports_xfs_scrub() · line 441

Quoted 7 lines
common/xfs +7 -0
+_require_xfs_shrink()
+{
+	_scratch_mkfs_xfs >/dev/null 2>&1
+
+	_scratch_mount
+	$XFS_GROWFS_PROG -D1 "$SCRATCH_MNT" 2>&1 | grep -q 'Invalid argument' || { \
+		_scratch_unmount; _notrun "kernel does not support shrinking"; }

^^^^
I think this unmount isn't necessary, due to after "_notrun" the $SCRATCH_DEV will be umounted "automatically".

I didn't dig into it more. Just follow _require_xfs_scratch_rmapbt().

If it's not necessary (assumed ./check does that), will update it in the next version. Thanks for your suggestion.

Thanks,
Gao Xiang

author
#4 lore raw

RFC PATCH v2 2/3 xfs: basic functionality test for shrinking free space in the last AG

In reply to Gao Xiang

Add basic test to make sure the functionality works as expected.

---
 tests/xfs/990     | 59 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/990.out | 12 ++++++++++
 tests/xfs/group   |  1 +
 3 files changed, 72 insertions(+)
 create mode 100755 tests/xfs/990
 create mode 100644 tests/xfs/990.out
tests/xfs/990 new file +59 -0
new file mode 100755
index 00000000..551c4784
@@ -0,0 +1,59 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test 990
+#
+# XFS shrinkfs basic functionality test
+#
+# This test attempts to shrink with a small size (512K), half AG size and
+# an out-of-bound size (agsize + 1) to observe if it works as expected.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs xfs
+_require_xfs_shrink
+
+rm -f $seqres.full
+echo "Format and mount"
+size="$((512 * 1024 * 1024))"
+_scratch_mkfs -dsize=$size -dagcount=3 2>&1 | \
+	tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
+. $tmp.mkfs
+_scratch_mount >> $seqres.full 2>&1
+
+echo "Shrink fs (small size)"
+$XFS_GROWFS_PROG -D $((dblocks-512*1024/dbsize)) $SCRATCH_MNT \
+	>> $seqres.full 2>&1 || echo failure
+_scratch_cycle_mount
+
+echo "Shrink fs (half AG)"
+$XFS_GROWFS_PROG -D $((dblocks-agsize/2)) $SCRATCH_MNT \
+	>> $seqres.full 2>&1 || echo failure
+_scratch_cycle_mount
+
+echo "Shrink fs (out-of-bound)"
+$XFS_GROWFS_PROG -D $((dblocks-agsize-1)) $SCRATCH_MNT \
+	>> $seqres.full 2>&1 && echo failure
+_scratch_cycle_mount
+
+$XFS_INFO_PROG $SCRATCH_MNT >> $seqres.full
+
+_scratch_unmount
+echo "*** done"
+
+# success, all done
+status=0
+exit
tests/xfs/990.out new file +12 -0
new file mode 100644
index 00000000..812f89ef
@@ -0,0 +1,12 @@
+QA output created by 990
+Format and mount
+meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks
+data     = bsize=XXX blocks=XXX, imaxpct=PCT
+         = sunit=XXX swidth=XXX, unwritten=X
+naming   =VERN bsize=XXX
+log      =LDEV bsize=XXX blocks=XXX
+realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
+Shrink fs (small size)
+Shrink fs (half AG)
+Shrink fs (out-of-bound)
+*** done
tests/xfs/group +1 -0
index e861cec9..a7981b67 100644
@@ -525,3 +525,4 @@
 525 auto quick mkfs
 526 auto quick mkfs
 527 auto quick quota
+990 auto quick growfs
Signature · 1 line
2.27.0
#5 lore raw

Re: [RFC PATCH v2 2/3] xfs: basic functionality test for shrinking free space in the last AG

Quoted 33 lines · Gao Xiang

Add basic test to make sure the functionality works as expected.

---
 tests/xfs/990     | 59 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/990.out | 12 ++++++++++
 tests/xfs/group   |  1 +
 3 files changed, 72 insertions(+)
 create mode 100755 tests/xfs/990
 create mode 100644 tests/xfs/990.out
tests/xfs/990 new file +16 -0
new file mode 100755
index 00000000..551c4784
@@ -0,0 +1,59 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test 990
+#
+# XFS shrinkfs basic functionality test
+#
+# This test attempts to shrink with a small size (512K), half AG size and
+# an out-of-bound size (agsize + 1) to observe if it works as expected.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`

tests/xfs/990 line 26

Quoted 10 lines
tests/xfs/990 new file +10 -0
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs xfs

_require_scratch

tests/xfs/990 line 31

Quoted 5 lines
tests/xfs/990 new file +5 -0
+_require_xfs_shrink
+
+rm -f $seqres.full
+echo "Format and mount"
+size="$((512 * 1024 * 1024))"

Is the fixed size necessary? Is that better to let testers run this test with their different device/XFS geometry.

tests/xfs/990 line 40

Quoted 9 lines
tests/xfs/990 new file +9 -0
+_scratch_mkfs -dsize=$size -dagcount=3 2>&1 | \
+	tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
+. $tmp.mkfs
+_scratch_mount >> $seqres.full 2>&1
+
+echo "Shrink fs (small size)"
+$XFS_GROWFS_PROG -D $((dblocks-512*1024/dbsize)) $SCRATCH_MNT \
+	>> $seqres.full 2>&1 || echo failure
+_scratch_cycle_mount

I don't understand the XFS Shrink new feature that much, is the "cycle_mount" necessary? If it's not, can we get more chances to find bugs without "cycle_mount", or with a fsck?

Another question is, should we verify the new size after shrink?

Quoted 4 lines
tests/xfs/990 new file +4 -0
+
+echo "Shrink fs (half AG)"
+$XFS_GROWFS_PROG -D $((dblocks-agsize/2)) $SCRATCH_MNT \
+	>> $seqres.full 2>&1 || echo failure

tests/xfs/990 line 54

Quoted 10 lines
tests/xfs/990 new file +10 -0
+_scratch_cycle_mount
+
+echo "Shrink fs (out-of-bound)"
+$XFS_GROWFS_PROG -D $((dblocks-agsize-1)) $SCRATCH_MNT \
+	>> $seqres.full 2>&1 && echo failure
+_scratch_cycle_mount
+
+$XFS_INFO_PROG $SCRATCH_MNT >> $seqres.full
+
+_scratch_unmount

^^^
It's not necessary.

Quoted 35 lines
tests/xfs/990 new file +5 -0
+echo "*** done"
+
+# success, all done
+status=0
+exit
tests/xfs/990.out new file +12 -0
new file mode 100644
index 00000000..812f89ef
@@ -0,0 +1,12 @@
+QA output created by 990
+Format and mount
+meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks
+data     = bsize=XXX blocks=XXX, imaxpct=PCT
+         = sunit=XXX swidth=XXX, unwritten=X
+naming   =VERN bsize=XXX
+log      =LDEV bsize=XXX blocks=XXX
+realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
+Shrink fs (small size)
+Shrink fs (half AG)
+Shrink fs (out-of-bound)
+*** done
tests/xfs/group +1 -0
index e861cec9..a7981b67 100644
@@ -525,3 +525,4 @@
 525 auto quick mkfs
 526 auto quick mkfs
 527 auto quick quota
+990 auto quick growfs
Signature · 1 line
2.27.0
author
#6 lore raw

Re: [RFC PATCH v2 2/3] xfs: basic functionality test for shrinking free space in the last AG

Quoted 38 lines · Zorro Lang
Quoted 37 lines · Gao Xiang

Add basic test to make sure the functionality works as expected.

---
 tests/xfs/990     | 59 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/990.out | 12 ++++++++++
 tests/xfs/group   |  1 +
 3 files changed, 72 insertions(+)
 create mode 100755 tests/xfs/990
 create mode 100644 tests/xfs/990.out
tests/xfs/990 new file +20 -0
new file mode 100755
index 00000000..551c4784
@@ -0,0 +1,59 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test 990
+#
+# XFS shrinkfs basic functionality test
+#
+# This test attempts to shrink with a small size (512K), half AG size and
+# an out-of-bound size (agsize + 1) to observe if it works as expected.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
Quoted 8 lines

tests/xfs/990 line 26

Quoted 6 lines
tests/xfs/990 new file +6 -0
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs xfs

_require_scratch

Will fix.

Quoted 9 lines

tests/xfs/990 line 31

Quoted 6 lines
tests/xfs/990 new file +5 -0
+_require_xfs_shrink
+
+rm -f $seqres.full
+echo "Format and mount"
+size="$((512 * 1024 * 1024))"

Is the fixed size necessary? Is that better to let testers run this test with their different device/XFS geometry.

I'm fine with either way since it's a simple functionality test, yet for most common cases, stratch devices are somewhat large. I tend to use a relative controllable small value.

Actually, this case was from xfs/127 with some modification.

Quoted 14 lines

tests/xfs/990 line 40

Quoted 10 lines
tests/xfs/990 new file +9 -0
+_scratch_mkfs -dsize=$size -dagcount=3 2>&1 | \
+	tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
+. $tmp.mkfs
+_scratch_mount >> $seqres.full 2>&1
+
+echo "Shrink fs (small size)"
+$XFS_GROWFS_PROG -D $((dblocks-512*1024/dbsize)) $SCRATCH_MNT \
+	>> $seqres.full 2>&1 || echo failure
+_scratch_cycle_mount

I don't understand the XFS Shrink new feature that much, is the "cycle_mount" necessary? If it's not, can we get more chances to find bugs without "cycle_mount", or with a fsck?

maybe it's useful to test unmount here. Yeah, I think it's better to try fsck here. Good idea.

Quoted 2 lines

Another question is, should we verify the new size after shrink?

Yeah, will add xfs_info.

Quoted 17 lines
Quoted 5 lines
tests/xfs/990 new file +4 -0
+
+echo "Shrink fs (half AG)"
+$XFS_GROWFS_PROG -D $((dblocks-agsize/2)) $SCRATCH_MNT \
+	>> $seqres.full 2>&1 || echo failure

tests/xfs/990 line 54

Quoted 10 lines
tests/xfs/990 new file +10 -0
+_scratch_cycle_mount
+
+echo "Shrink fs (out-of-bound)"
+$XFS_GROWFS_PROG -D $((dblocks-agsize-1)) $SCRATCH_MNT \
+	>> $seqres.full 2>&1 && echo failure
+_scratch_cycle_mount
+
+$XFS_INFO_PROG $SCRATCH_MNT >> $seqres.full
+
+_scratch_unmount

^^^
It's not necessary.

ok. It seems that ./check will fsck scratch device as well. Will update it.

Thanks,
Gao Xiang

author
#7 lore raw

RFC PATCH v2 3/3 xfs: stress test for shrinking free space in the last AG

In reply to Gao Xiang

This adds a stress testcase to shrink free space as much as possible in the last AG with background fsstress workload.

The expectation is that no crash happens with expected output.

---
Note that I don't use _fill_fs instead, since fill_scratch here mainly to eat 125M to make fsstress more effectively, rather than fill data as much as possible.

 tests/xfs/991     | 121 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/991.out |   8 +++
 tests/xfs/group   |   1 +
 3 files changed, 130 insertions(+)
 create mode 100755 tests/xfs/991
 create mode 100644 tests/xfs/991.out
tests/xfs/991 new file +121 -0
new file mode 100755
index 00000000..22a5ac81
@@ -0,0 +1,121 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2020-2021 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test 991
+#
+# XFS online shrinkfs stress test
+#
+# This test attempts to shrink unused space as much as possible with
+# background fsstress workload. It will decrease the shrink size if
+# larger size fails. And totally repeat 2 * TIME_FACTOR times.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+create_scratch()
+{
+	_scratch_mkfs_xfs $@ | tee -a $seqres.full | \
+		_filter_mkfs 2>$tmp.mkfs >/dev/null
+	. $tmp.mkfs
+
+	if ! _try_scratch_mount 2>/dev/null; then
+		echo "failed to mount $SCRATCH_DEV"
+		exit 1
+	fi
+
+	# fix the reserve block pool to a known size so that the enospc
+	# calculations work out correctly.
+	_scratch_resvblks 1024 > /dev/null 2>&1
+}
+
+fill_scratch()
+{
+	$XFS_IO_PROG -f -c "resvsp 0 ${1}" $SCRATCH_MNT/resvfile
+}
+
+stress_scratch()
+{
+	procs=3
+	nops=$((1000 * LOAD_FACTOR))
+	# -w ensures that the only ops are ones which cause write I/O
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
+	    -n $nops $FSSTRESS_AVOID`
+	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+}
+
+# real QA test starts here
+_supported_fs xfs
+_require_scratch
+_require_xfs_shrink
+_require_xfs_io_command "falloc"
+
+rm -f $seqres.full
+_scratch_mkfs_xfs | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
+. $tmp.mkfs	# extract blocksize and data size for scratch device
+
+decsize=`expr  42 \* 1048576`	# shrink in chunks of this size at most
+endsize=`expr 125 \* 1048576`	# stop after shrinking this big
+[ `expr $endsize / $dbsize` -lt $dblocks ] || _notrun "Scratch device too small"
+
+nags=2
+totalcount=$((2 * TIME_FACTOR))
+
+while [ $totalcount -gt 0 ]; do
+	size=`expr 1010 \* 1048576`	# 1010 megabytes initially
+	logblks=$(_scratch_find_xfs_min_logblocks -dsize=${size} -dagcount=${nags})
+
+	create_scratch -lsize=${logblks}b -dsize=${size} -dagcount=${nags}
+
+	for i in `seq 125 -1 90`; do
+		fillsize=`expr $i \* 1048576`
+		out="$(fill_scratch $fillsize 2>&1)"
+		echo "$out" | grep -q 'No space left on device' && continue
+		test -n "${out}" && echo "$out"
+		break
+	done
+
+	while [ $size -gt $endsize ]; do
+		stress_scratch
+		sleep 1
+
+		decb=`expr $decsize / $dbsize`    # in data blocks
+		while [ $decb -gt 0 ]; do
+			sizeb=`expr $size / $dbsize - $decb`
+
+			$XFS_GROWFS_PROG -D ${sizeb} $SCRATCH_MNT \
+				>> $seqres.full 2>&1 && break
+
+			[ $decb -gt 100 ] && decb=`expr $decb + $RANDOM % 10`
+			decb=`expr $decb / 2`
+		done
+
+		wait
+		[ $decb -eq 0 ] && break
+
+		# get latest dblocks
+		$XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null
+		. $tmp.growfs
+
+		size=`expr $dblocks \* $dbsize`
+		_scratch_cycle_mount
+	done
+
+	_scratch_unmount
+	_repair_scratch_fs >> $seqres.full
+	totalcount=`expr $totalcount - 1`
+done
+
+echo "*** done"
+status=0
+exit
tests/xfs/991.out new file +8 -0
new file mode 100644
index 00000000..e8209672
@@ -0,0 +1,8 @@
+QA output created by 991
+meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks
+data     = bsize=XXX blocks=XXX, imaxpct=PCT
+         = sunit=XXX swidth=XXX, unwritten=X
+naming   =VERN bsize=XXX
+log      =LDEV bsize=XXX blocks=XXX
+realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
+*** done
tests/xfs/group +1 -0
index a7981b67..b479ed3a 100644
@@ -526,3 +526,4 @@
 526 auto quick mkfs
 527 auto quick quota
 990 auto quick growfs
+991 auto quick growfs
Signature · 1 line
2.27.0
#8 lore raw

Re: [RFC PATCH v2 3/3] xfs: stress test for shrinking free space in the last AG

Quoted 10 lines · Gao Xiang

This adds a stress testcase to shrink free space as much as possible in the last AG with background fsstress workload.

The expectation is that no crash happens with expected output.

---
Note that I don't use _fill_fs instead, since fill_scratch here mainly to eat 125M to make fsstress more effectively, rather than fill data as much as possible.

As Darrick had given lots of review points to this case, I just have 2 picky questions as below:)

Quoted 65 lines
 tests/xfs/991     | 121 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/991.out |   8 +++
 tests/xfs/group   |   1 +
 3 files changed, 130 insertions(+)
 create mode 100755 tests/xfs/991
 create mode 100644 tests/xfs/991.out
tests/xfs/991 new file +51 -0
new file mode 100755
index 00000000..22a5ac81
@@ -0,0 +1,121 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2020-2021 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test 991
+#
+# XFS online shrinkfs stress test
+#
+# This test attempts to shrink unused space as much as possible with
+# background fsstress workload. It will decrease the shrink size if
+# larger size fails. And totally repeat 2 * TIME_FACTOR times.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+create_scratch()
+{
+	_scratch_mkfs_xfs $@ | tee -a $seqres.full | \
+		_filter_mkfs 2>$tmp.mkfs >/dev/null
+	. $tmp.mkfs
+
+	if ! _try_scratch_mount 2>/dev/null; then
+		echo "failed to mount $SCRATCH_DEV"
+		exit 1
+	fi
+
+	# fix the reserve block pool to a known size so that the enospc
+	# calculations work out correctly.
+	_scratch_resvblks 1024 > /dev/null 2>&1
+}
+
+fill_scratch()
+{
+	$XFS_IO_PROG -f -c "resvsp 0 ${1}" $SCRATCH_MNT/resvfile
+}
+
+stress_scratch()
+{
+	procs=3
+	nops=$((1000 * LOAD_FACTOR))
+	# -w ensures that the only ops are ones which cause write I/O

tests/xfs/991 line 61

Quoted 10 lines
tests/xfs/991 new file +10 -0
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
+	    -n $nops $FSSTRESS_AVOID`
+	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+}
+
+# real QA test starts here
+_supported_fs xfs
+_require_scratch
+_require_xfs_shrink
+_require_xfs_io_command "falloc"

Do I miss something? I only found you use xfs_io "resvsp", why you need "falloc" cmd?

Quoted 79 lines
tests/xfs/991 new file +60 -0
+
+rm -f $seqres.full
+_scratch_mkfs_xfs | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
+. $tmp.mkfs	# extract blocksize and data size for scratch device
+
+decsize=`expr  42 \* 1048576`	# shrink in chunks of this size at most
+endsize=`expr 125 \* 1048576`	# stop after shrinking this big
+[ `expr $endsize / $dbsize` -lt $dblocks ] || _notrun "Scratch device too small"
+
+nags=2
+totalcount=$((2 * TIME_FACTOR))
+
+while [ $totalcount -gt 0 ]; do
+	size=`expr 1010 \* 1048576`	# 1010 megabytes initially
+	logblks=$(_scratch_find_xfs_min_logblocks -dsize=${size} -dagcount=${nags})
+
+	create_scratch -lsize=${logblks}b -dsize=${size} -dagcount=${nags}
+
+	for i in `seq 125 -1 90`; do
+		fillsize=`expr $i \* 1048576`
+		out="$(fill_scratch $fillsize 2>&1)"
+		echo "$out" | grep -q 'No space left on device' && continue
+		test -n "${out}" && echo "$out"
+		break
+	done
+
+	while [ $size -gt $endsize ]; do
+		stress_scratch
+		sleep 1
+
+		decb=`expr $decsize / $dbsize`    # in data blocks
+		while [ $decb -gt 0 ]; do
+			sizeb=`expr $size / $dbsize - $decb`
+
+			$XFS_GROWFS_PROG -D ${sizeb} $SCRATCH_MNT \
+				>> $seqres.full 2>&1 && break
+
+			[ $decb -gt 100 ] && decb=`expr $decb + $RANDOM % 10`
+			decb=`expr $decb / 2`
+		done
+
+		wait
+		[ $decb -eq 0 ] && break
+
+		# get latest dblocks
+		$XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null
+		. $tmp.growfs
+
+		size=`expr $dblocks \* $dbsize`
+		_scratch_cycle_mount
+	done
+
+	_scratch_unmount
+	_repair_scratch_fs >> $seqres.full
+	totalcount=`expr $totalcount - 1`
+done
+
+echo "*** done"
+status=0
+exit
tests/xfs/991.out new file +8 -0
new file mode 100644
index 00000000..e8209672
@@ -0,0 +1,8 @@
+QA output created by 991
+meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks
+data     = bsize=XXX blocks=XXX, imaxpct=PCT
+         = sunit=XXX swidth=XXX, unwritten=X
+naming   =VERN bsize=XXX
+log      =LDEV bsize=XXX blocks=XXX
+realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
+*** done
tests/xfs/group +0 -0
index a7981b67..b479ed3a 100644

tests/xfs/group line 529

Quoted 4 lines
tests/xfs/group +1 -0
@@ -526,3 +526,4 @@
 526 auto quick mkfs
 527 auto quick quota
 990 auto quick growfs
+991 auto quick growfs

Is this "stress" test case really "quick" :)

Quoted 3 lines
Signature · 1 line
2.27.0
author
#9 lore raw

Re: [RFC PATCH v2 3/3] xfs: stress test for shrinking free space in the last AG

Quoted 84 lines · Zorro Lang
Quoted 10 lines · Gao Xiang

This adds a stress testcase to shrink free space as much as possible in the last AG with background fsstress workload.

The expectation is that no crash happens with expected output.

---
Note that I don't use _fill_fs instead, since fill_scratch here mainly to eat 125M to make fsstress more effectively, rather than fill data as much as possible.

As Darrick had given lots of review points to this case, I just have 2 picky questions as below:)

Quoted 69 lines
 tests/xfs/991     | 121 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/991.out |   8 +++
 tests/xfs/group   |   1 +
 3 files changed, 130 insertions(+)
 create mode 100755 tests/xfs/991
 create mode 100644 tests/xfs/991.out
tests/xfs/991 new file +55 -0
new file mode 100755
index 00000000..22a5ac81
@@ -0,0 +1,121 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2020-2021 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test 991
+#
+# XFS online shrinkfs stress test
+#
+# This test attempts to shrink unused space as much as possible with
+# background fsstress workload. It will decrease the shrink size if
+# larger size fails. And totally repeat 2 * TIME_FACTOR times.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+create_scratch()
+{
+	_scratch_mkfs_xfs $@ | tee -a $seqres.full | \
+		_filter_mkfs 2>$tmp.mkfs >/dev/null
+	. $tmp.mkfs
+
+	if ! _try_scratch_mount 2>/dev/null; then
+		echo "failed to mount $SCRATCH_DEV"
+		exit 1
+	fi
+
+	# fix the reserve block pool to a known size so that the enospc
+	# calculations work out correctly.
+	_scratch_resvblks 1024 > /dev/null 2>&1
+}
+
+fill_scratch()
+{
+	$XFS_IO_PROG -f -c "resvsp 0 ${1}" $SCRATCH_MNT/resvfile
+}
+
+stress_scratch()
+{
+	procs=3
+	nops=$((1000 * LOAD_FACTOR))
+	# -w ensures that the only ops are ones which cause write I/O
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
+	    -n $nops $FSSTRESS_AVOID`
+	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+}
Quoted 8 lines

tests/xfs/991 line 61

Quoted 6 lines
tests/xfs/991 new file +6 -0
+
+# real QA test starts here
+_supported_fs xfs
+_require_scratch
+_require_xfs_shrink
+_require_xfs_io_command "falloc"

Do I miss something? I only found you use xfs_io "resvsp", why you need "falloc" cmd?

As I mentioned before, the testcase was derived from xfs/104 with some modification.

At a quick glance, this line was added by commit 09e94f84d929 ("xfs: don't assume preallocation is always supported on XFS"). I have no more background yet.

Quoted 78 lines
Quoted 78 lines
tests/xfs/991 new file +60 -0
+
+rm -f $seqres.full
+_scratch_mkfs_xfs | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
+. $tmp.mkfs	# extract blocksize and data size for scratch device
+
+decsize=`expr  42 \* 1048576`	# shrink in chunks of this size at most
+endsize=`expr 125 \* 1048576`	# stop after shrinking this big
+[ `expr $endsize / $dbsize` -lt $dblocks ] || _notrun "Scratch device too small"
+
+nags=2
+totalcount=$((2 * TIME_FACTOR))
+
+while [ $totalcount -gt 0 ]; do
+	size=`expr 1010 \* 1048576`	# 1010 megabytes initially
+	logblks=$(_scratch_find_xfs_min_logblocks -dsize=${size} -dagcount=${nags})
+
+	create_scratch -lsize=${logblks}b -dsize=${size} -dagcount=${nags}
+
+	for i in `seq 125 -1 90`; do
+		fillsize=`expr $i \* 1048576`
+		out="$(fill_scratch $fillsize 2>&1)"
+		echo "$out" | grep -q 'No space left on device' && continue
+		test -n "${out}" && echo "$out"
+		break
+	done
+
+	while [ $size -gt $endsize ]; do
+		stress_scratch
+		sleep 1
+
+		decb=`expr $decsize / $dbsize`    # in data blocks
+		while [ $decb -gt 0 ]; do
+			sizeb=`expr $size / $dbsize - $decb`
+
+			$XFS_GROWFS_PROG -D ${sizeb} $SCRATCH_MNT \
+				>> $seqres.full 2>&1 && break
+
+			[ $decb -gt 100 ] && decb=`expr $decb + $RANDOM % 10`
+			decb=`expr $decb / 2`
+		done
+
+		wait
+		[ $decb -eq 0 ] && break
+
+		# get latest dblocks
+		$XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null
+		. $tmp.growfs
+
+		size=`expr $dblocks \* $dbsize`
+		_scratch_cycle_mount
+	done
+
+	_scratch_unmount
+	_repair_scratch_fs >> $seqres.full
+	totalcount=`expr $totalcount - 1`
+done
+
+echo "*** done"
+status=0
+exit
tests/xfs/991.out new file +8 -0
new file mode 100644
index 00000000..e8209672
@@ -0,0 +1,8 @@
+QA output created by 991
+meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks
+data     = bsize=XXX blocks=XXX, imaxpct=PCT
+         = sunit=XXX swidth=XXX, unwritten=X
+naming   =VERN bsize=XXX
+log      =LDEV bsize=XXX blocks=XXX
+realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
+*** done
tests/xfs/group +0 -1
index a7981b67..b479ed3a 100644
--- a/tests/xfs/group
Quoted 8 lines

tests/xfs/group line 529

Quoted 6 lines

+++ b/tests/xfs/group

tests/xfs/group +1 -0
@@ -526,3 +526,4 @@
 526 auto quick mkfs
 527 auto quick quota
 990 auto quick growfs
+991 auto quick growfs

Is this "stress" test case really "quick" :)

ok, will update.

Thanks,
Gao Xiang

Quoted 5 lines
Quoted 4 lines
Signature · 2 lines
-- 
2.27.0
#10 lore raw

Re: [RFC PATCH v2 3/3] xfs: stress test for shrinking free space in the last AG

Quoted 94 lines · Gao Xiang
Quoted 84 lines · Zorro Lang
Quoted 10 lines · Gao Xiang

This adds a stress testcase to shrink free space as much as possible in the last AG with background fsstress workload.

The expectation is that no crash happens with expected output.

---
Note that I don't use _fill_fs instead, since fill_scratch here mainly to eat 125M to make fsstress more effectively, rather than fill data as much as possible.

As Darrick had given lots of review points to this case, I just have 2 picky questions as below:)

Quoted 69 lines
 tests/xfs/991     | 121 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/991.out |   8 +++
 tests/xfs/group   |   1 +
 3 files changed, 130 insertions(+)
 create mode 100755 tests/xfs/991
 create mode 100644 tests/xfs/991.out
tests/xfs/991 new file +55 -0
new file mode 100755
index 00000000..22a5ac81
@@ -0,0 +1,121 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2020-2021 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test 991
+#
+# XFS online shrinkfs stress test
+#
+# This test attempts to shrink unused space as much as possible with
+# background fsstress workload. It will decrease the shrink size if
+# larger size fails. And totally repeat 2 * TIME_FACTOR times.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+create_scratch()
+{
+	_scratch_mkfs_xfs $@ | tee -a $seqres.full | \
+		_filter_mkfs 2>$tmp.mkfs >/dev/null
+	. $tmp.mkfs
+
+	if ! _try_scratch_mount 2>/dev/null; then
+		echo "failed to mount $SCRATCH_DEV"
+		exit 1
+	fi
+
+	# fix the reserve block pool to a known size so that the enospc
+	# calculations work out correctly.
+	_scratch_resvblks 1024 > /dev/null 2>&1
+}
+
+fill_scratch()
+{
+	$XFS_IO_PROG -f -c "resvsp 0 ${1}" $SCRATCH_MNT/resvfile
+}
+
+stress_scratch()
+{
+	procs=3
+	nops=$((1000 * LOAD_FACTOR))
+	# -w ensures that the only ops are ones which cause write I/O
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
+	    -n $nops $FSSTRESS_AVOID`
+	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+}
Quoted 8 lines

tests/xfs/991 line 61

Quoted 6 lines
tests/xfs/991 new file +6 -0
+
+# real QA test starts here
+_supported_fs xfs
+_require_scratch
+_require_xfs_shrink
+_require_xfs_io_command "falloc"

Do I miss something? I only found you use xfs_io "resvsp", why you need "falloc" cmd?

Quoted 6 lines

As I mentioned before, the testcase was derived from xfs/104 with some modification.

At a quick glance, this line was added by commit 09e94f84d929 ("xfs: don't assume preallocation is always supported on XFS"). I have no more background yet.

Why not use xfs_io falloc in the test? fallocate is the successor to resvsp.

--D

Quoted 98 lines
Quoted 78 lines
Quoted 78 lines
tests/xfs/991 new file +60 -0
+
+rm -f $seqres.full
+_scratch_mkfs_xfs | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
+. $tmp.mkfs	# extract blocksize and data size for scratch device
+
+decsize=`expr  42 \* 1048576`	# shrink in chunks of this size at most
+endsize=`expr 125 \* 1048576`	# stop after shrinking this big
+[ `expr $endsize / $dbsize` -lt $dblocks ] || _notrun "Scratch device too small"
+
+nags=2
+totalcount=$((2 * TIME_FACTOR))
+
+while [ $totalcount -gt 0 ]; do
+	size=`expr 1010 \* 1048576`	# 1010 megabytes initially
+	logblks=$(_scratch_find_xfs_min_logblocks -dsize=${size} -dagcount=${nags})
+
+	create_scratch -lsize=${logblks}b -dsize=${size} -dagcount=${nags}
+
+	for i in `seq 125 -1 90`; do
+		fillsize=`expr $i \* 1048576`
+		out="$(fill_scratch $fillsize 2>&1)"
+		echo "$out" | grep -q 'No space left on device' && continue
+		test -n "${out}" && echo "$out"
+		break
+	done
+
+	while [ $size -gt $endsize ]; do
+		stress_scratch
+		sleep 1
+
+		decb=`expr $decsize / $dbsize`    # in data blocks
+		while [ $decb -gt 0 ]; do
+			sizeb=`expr $size / $dbsize - $decb`
+
+			$XFS_GROWFS_PROG -D ${sizeb} $SCRATCH_MNT \
+				>> $seqres.full 2>&1 && break
+
+			[ $decb -gt 100 ] && decb=`expr $decb + $RANDOM % 10`
+			decb=`expr $decb / 2`
+		done
+
+		wait
+		[ $decb -eq 0 ] && break
+
+		# get latest dblocks
+		$XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null
+		. $tmp.growfs
+
+		size=`expr $dblocks \* $dbsize`
+		_scratch_cycle_mount
+	done
+
+	_scratch_unmount
+	_repair_scratch_fs >> $seqres.full
+	totalcount=`expr $totalcount - 1`
+done
+
+echo "*** done"
+status=0
+exit
tests/xfs/991.out new file +8 -0
new file mode 100644
index 00000000..e8209672
@@ -0,0 +1,8 @@
+QA output created by 991
+meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks
+data     = bsize=XXX blocks=XXX, imaxpct=PCT
+         = sunit=XXX swidth=XXX, unwritten=X
+naming   =VERN bsize=XXX
+log      =LDEV bsize=XXX blocks=XXX
+realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
+*** done
tests/xfs/group +0 -1
index a7981b67..b479ed3a 100644
--- a/tests/xfs/group
Quoted 8 lines

tests/xfs/group line 529

Quoted 6 lines

+++ b/tests/xfs/group

tests/xfs/group +1 -0
@@ -526,3 +526,4 @@
 526 auto quick mkfs
 527 auto quick quota
 990 auto quick growfs
+991 auto quick growfs

Is this "stress" test case really "quick" :)

ok, will update.

Thanks,
Gao Xiang

Quoted 5 lines
Quoted 4 lines
Signature · 2 lines
-- 
2.27.0
author
#11 lore raw

Re: [RFC PATCH v2 3/3] xfs: stress test for shrinking free space in the last AG

Quoted 98 lines · Darrick J. Wong
Quoted 97 lines · Gao Xiang
Quoted 84 lines · Zorro Lang
Quoted 10 lines · Gao Xiang

This adds a stress testcase to shrink free space as much as possible in the last AG with background fsstress workload.

The expectation is that no crash happens with expected output.

---
Note that I don't use _fill_fs instead, since fill_scratch here mainly to eat 125M to make fsstress more effectively, rather than fill data as much as possible.

As Darrick had given lots of review points to this case, I just have 2 picky questions as below:)

Quoted 69 lines
 tests/xfs/991     | 121 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/991.out |   8 +++
 tests/xfs/group   |   1 +
 3 files changed, 130 insertions(+)
 create mode 100755 tests/xfs/991
 create mode 100644 tests/xfs/991.out
tests/xfs/991 new file +55 -0
new file mode 100755
index 00000000..22a5ac81
@@ -0,0 +1,121 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2020-2021 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test 991
+#
+# XFS online shrinkfs stress test
+#
+# This test attempts to shrink unused space as much as possible with
+# background fsstress workload. It will decrease the shrink size if
+# larger size fails. And totally repeat 2 * TIME_FACTOR times.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+create_scratch()
+{
+	_scratch_mkfs_xfs $@ | tee -a $seqres.full | \
+		_filter_mkfs 2>$tmp.mkfs >/dev/null
+	. $tmp.mkfs
+
+	if ! _try_scratch_mount 2>/dev/null; then
+		echo "failed to mount $SCRATCH_DEV"
+		exit 1
+	fi
+
+	# fix the reserve block pool to a known size so that the enospc
+	# calculations work out correctly.
+	_scratch_resvblks 1024 > /dev/null 2>&1
+}
+
+fill_scratch()
+{
+	$XFS_IO_PROG -f -c "resvsp 0 ${1}" $SCRATCH_MNT/resvfile
+}
+
+stress_scratch()
+{
+	procs=3
+	nops=$((1000 * LOAD_FACTOR))
+	# -w ensures that the only ops are ones which cause write I/O
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
+	    -n $nops $FSSTRESS_AVOID`
+	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+}
Quoted 8 lines

tests/xfs/991 line 61

Quoted 6 lines
tests/xfs/991 new file +6 -0
+
+# real QA test starts here
+_supported_fs xfs
+_require_scratch
+_require_xfs_shrink
+_require_xfs_io_command "falloc"

Do I miss something? I only found you use xfs_io "resvsp", why you need "falloc" cmd?

As I mentioned before, the testcase was derived from xfs/104 with some modification.

Quoted 6 lines
Quoted 3 lines

At a quick glance, this line was added by commit 09e94f84d929 ("xfs: don't assume preallocation is always supported on XFS"). I have no more background yet.

Why not use xfs_io falloc in the test? fallocate is the successor to resvsp.

Yeah, general falloc seems better, and it seems _require_xfs_io_command here is used for always_cow inode feature. Will update it. Thanks!

Thanks,
Gao Xiang

Quoted 2 lines

--D