fstests/xfs: a couple growfs log recovery tests
cover fstests/xfs: a couple growfs log recovery tests
Hi all,
This is first pass of a growfs crash and log recovery test I cooked up for XFS. A bit more background and context on this is available here [1]. In short, this reproduces at least a couple log recovery issues on XFS related to growfs that Christoph has tracked down and resolved. Darrick proposed a simple realtime variant in the discussion at [1], so patch 2 is a stab at that. It's basically just a copy of patch 1 with some rt related tweaks. However..
Darrick,
I believe you reproduced a problem with your customized realtime variant of the initial test. I've not been able to reproduce any test failures with patch 2 here, though I have tried to streamline the test a bit to reduce unnecessary bits (patch 1 still reproduces the original problems). I also don't tend to test much with rt, so it's possible my config is off somehow or another. Otherwise I _think_ I've included the necessary changes for rt support in the test itself.
Thoughts? I'd like to figure out what might be going on there before this should land..
Brian
[1] https://lore.kernel.org/fstests/[email protected]/
Brian Foster (2):
xfs: online grow vs. log recovery stress test
xfs: online grow vs. log recovery stress test (realtime version)
tests/xfs/609 | 69 +++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/609.out | 7 +++++ tests/xfs/610 | 71 +++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/610.out | 7 +++++ 4 files changed, 154 insertions(+) create mode 100755 tests/xfs/609 create mode 100644 tests/xfs/609.out create mode 100755 tests/xfs/610 create mode 100644 tests/xfs/610.out
Signature · 1 line
2.46.2
PATCH 1/2 xfs: online grow vs. log recovery stress test
fstests includes decent functional tests for online growfs and shrink, and decent stress tests for crash and log recovery, but no combination of the two. This test combines bits from a typical growfs stress test like xfs/104 with crash recovery cycles from a test like generic/388. As a result, this reproduces at least a couple recently fixed issues related to log recovery of online growfs operations.
--- tests/xfs/609 | 69 +++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/609.out | 7 +++++ 2 files changed, 76 insertions(+) create mode 100755 tests/xfs/609 create mode 100644 tests/xfs/609.out
tests/xfs/609 new file +69 -0
@@ -0,0 +1,69 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test No. 609 +# +# Test XFS online growfs log recovery. +# +. ./common/preamble +_begin_fstest auto growfs stress shutdown log recoveryloop + +# Import common functions. +. ./common/filter + +_stress_scratch() +{ + procs=4 + nops=999999 + # -w ensures that the only ops are ones which cause write I/O + FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \ + -n $nops $FSSTRESS_AVOID` + $FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 & +} + +_require_scratch + +_scratch_mkfs_xfs | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs +. $tmp.mkfs # extract blocksize and data size for scratch device + +endsize=`expr 550 \* 1048576` # stop after growing this big +[ `expr $endsize / $dbsize` -lt $dblocks ] || _notrun "Scratch device too small" + +nags=4 +size=`expr 125 \* 1048576` # 120 megabytes initially +sizeb=`expr $size / $dbsize` # in data blocks +logblks=$(_scratch_find_xfs_min_logblocks -dsize=${size} -dagcount=${nags}) + +_scratch_mkfs_xfs -lsize=${logblks}b -dsize=${size} -dagcount=${nags} \ + >> $seqres.full +_scratch_mount + +# Grow the filesystem in random sized chunks while stressing and performing +# shutdown and recovery. The randomization is intended to create a mix of sub-ag +# and multi-ag grows. +while [ $size -le $endsize ]; do + echo "*** stressing a ${sizeb} block filesystem" >> $seqres.full + _stress_scratch + incsize=$((RANDOM % 40 * 1048576)) + size=`expr $size + $incsize` + sizeb=`expr $size / $dbsize` # in data blocks + echo "*** growing to a ${sizeb} block filesystem" >> $seqres.full + xfs_growfs -D ${sizeb} $SCRATCH_MNT >> $seqres.full + + sleep $((RANDOM % 3)) + _scratch_shutdown + ps -e | grep fsstress > /dev/null 2>&1 + while [ $? -eq 0 ]; do + killall -9 fsstress > /dev/null 2>&1 + wait > /dev/null 2>&1 + ps -e | grep fsstress > /dev/null 2>&1 + done + _scratch_cycle_mount || _fail "cycle mount failed" +done > /dev/null 2>&1 +wait # stop for any remaining stress processes + +_scratch_unmount + +status=0 +exit
tests/xfs/609.out new file +7 -0
@@ -0,0 +1,7 @@ +QA output created by 609 +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks +data = bsize=XXX blocks=XXX, imaxpct=PCT + = sunit=XXX swidth=XXX, unwritten=X +naming =VERN bsize=XXX +log =LDEV bsize=XXX blocks=XXX +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
Signature · 1 line
2.46.2
Re: [PATCH 1/2] xfs: online grow vs. log recovery stress test
Quoted 10 lines · Brian Foster
fstests includes decent functional tests for online growfs and shrink, and decent stress tests for crash and log recovery, but no combination of the two. This test combines bits from a typical growfs stress test like xfs/104 with crash recovery cycles from a test like generic/388. As a result, this reproduces at least a couple recently fixed issues related to log recovery of online growfs operations.
Signed-off-by Brian Foster <[email protected]>---
Hi Brian,
Thanks for this new test case! Some tiny review points below :)
Quoted 29 lines
tests/xfs/609 | 69 +++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/609.out | 7 +++++ 2 files changed, 76 insertions(+) create mode 100755 tests/xfs/609 create mode 100644 tests/xfs/609.outtests/xfs/609 new file +17
-0@@ -0,0 +1,69 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test No. 609 +# +# Test XFS online growfs log recovery. +# +. ./common/preamble +_begin_fstest auto growfs stress shutdown log recoveryloop + +# Import common functions. +. ./common/filter + +_stress_scratch() +{ + procs=4
tests/xfs/609 line 27
Quoted 10 lines
tests/xfs/609 new file +10
-0+ nops=999999 + # -w ensures that the only ops are ones which cause write I/O + FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \ + -n $nops $FSSTRESS_AVOID` + $FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 & +} + +_require_scratch + +_scratch_mkfs_xfs | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
"_scratch_mkfs_xfs | _filter_mkfs >$seqres.full 2>$tmp.mkfs" can get same output as the .out file below.
tests/xfs/609 line 39
Quoted 12 lines
tests/xfs/609 new file +12
-0+. $tmp.mkfs # extract blocksize and data size for scratch device + +endsize=`expr 550 \* 1048576` # stop after growing this big +[ `expr $endsize / $dbsize` -lt $dblocks ] || _notrun "Scratch device too small" + +nags=4 +size=`expr 125 \* 1048576` # 120 megabytes initially +sizeb=`expr $size / $dbsize` # in data blocks +logblks=$(_scratch_find_xfs_min_logblocks -dsize=${size} -dagcount=${nags}) + +_scratch_mkfs_xfs -lsize=${logblks}b -dsize=${size} -dagcount=${nags} \ + >> $seqres.full
What if this mkfs (with specific options) fails? So how about || _fail "....."
tests/xfs/609 line 52
Quoted 13 lines
tests/xfs/609 new file +13
-0+_scratch_mount + +# Grow the filesystem in random sized chunks while stressing and performing +# shutdown and recovery. The randomization is intended to create a mix of sub-ag +# and multi-ag grows. +while [ $size -le $endsize ]; do + echo "*** stressing a ${sizeb} block filesystem" >> $seqres.full + _stress_scratch + incsize=$((RANDOM % 40 * 1048576)) + size=`expr $size + $incsize` + sizeb=`expr $size / $dbsize` # in data blocks + echo "*** growing to a ${sizeb} block filesystem" >> $seqres.full + xfs_growfs -D ${sizeb} $SCRATCH_MNT >> $seqres.full
_require_command "$XFS_GROWFS_PROG" xfs_growfs
Then use $XFS_GROWFS_PROG
tests/xfs/609 line 58
Quoted 6 lines
tests/xfs/609 new file +6
-0+ + sleep $((RANDOM % 3)) + _scratch_shutdown + ps -e | grep fsstress > /dev/null 2>&1 + while [ $? -eq 0 ]; do + killall -9 fsstress > /dev/null 2>&1
_require_command "$KILLALL_PROG" killall
Then use $KILLALL_PROG
tests/xfs/609 line 64
Quoted 6 lines
tests/xfs/609 new file +6
-0+ wait > /dev/null 2>&1 + ps -e | grep fsstress > /dev/null 2>&1 + done + _scratch_cycle_mount || _fail "cycle mount failed" +done > /dev/null 2>&1 +wait # stop for any remaining stress processes
If the testing be interrupted, the fsstress processes will cause later tests fail. So we deal with background processes in _cleanup().
e.g.
_cleanup()
{
$KILLALL_ALL fsstress > /dev/null 2>&1
wait
cd /
rm -f $tmp.*
}Or use a loop kill as you does above.
Quoted 11 lines
tests/xfs/609 new file +5
-0+ +_scratch_unmount + +status=0 +exittests/xfs/609.out new file +0
-0
tests/xfs/609.out line 7
Quoted 7 lines
tests/xfs/609.out new file +7
-0@@ -0,0 +1,7 @@ +QA output created by 609 +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks +data = bsize=XXX blocks=XXX, imaxpct=PCT + = sunit=XXX swidth=XXX, unwritten=X +naming =VERN bsize=XXX +log =LDEV bsize=XXX blocks=XXX +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
So what's this output in .out file for? How about "Silence is golden"?
Thanks,
Zorro
Quoted 4 lines
Signature · 1 line
2.46.2
Re: [PATCH 1/2] xfs: online grow vs. log recovery stress test
Quoted 22 lines · Zorro Lang
Quoted 10 lines · Brian Foster
fstests includes decent functional tests for online growfs and shrink, and decent stress tests for crash and log recovery, but no combination of the two. This test combines bits from a typical growfs stress test like xfs/104 with crash recovery cycles from a test like generic/388. As a result, this reproduces at least a couple recently fixed issues related to log recovery of online growfs operations.
Signed-off-by Brian Foster <[email protected]>---
Hi Brian,
Thanks for this new test case! Some tiny review points below :)
Quoted 6 lines
tests/xfs/609 | 69 +++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/609.out | 7 +++++ 2 files changed, 76 insertions(+) create mode 100755 tests/xfs/609 create mode 100644 tests/xfs/609.out
...
Quoted 7 lines
Quoted 7 lines
tests/xfs/609.out new file +1
-0@@ -0,0 +1,7 @@ +QA output created by 609
Quoted 9 lines
tests/xfs/609.out line 7
Quoted 6 lines
tests/xfs/609.out new file +6
-0+meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks +data = bsize=XXX blocks=XXX, imaxpct=PCT + = sunit=XXX swidth=XXX, unwritten=X +naming =VERN bsize=XXX +log =LDEV bsize=XXX blocks=XXX +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXXSo what's this output in .out file for? How about "Silence is golden"?
No particular reason.. this was mostly a mash and cleanup of a couple preexisting tests around growfs and crash recovery, so probably just leftover from that. All of these suggestions sound good to me. I'll apply them and post a v2. Thanks for the review!
Brian
Quoted 8 lines
Thanks,
ZorroQuoted 4 lines
Signature · 1 line
2.46.2
PATCH 2/2 xfs: online grow vs. log recovery stress test (realtime version)
In reply to Brian FosterThis is fundamentally the same as the previous growfs vs. log recovery test, with tweaks to support growing the XFS realtime volume on such configurations. Changes include using the appropriate mkfs params, growfs params, and enabling realtime inheritance on the scratch fs.
--- tests/xfs/610 | 71 +++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/610.out | 7 +++++ 2 files changed, 78 insertions(+) create mode 100755 tests/xfs/610 create mode 100644 tests/xfs/610.out
tests/xfs/610 new file +71 -0
@@ -0,0 +1,71 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test No. 610 +# +# Test XFS online growfs log recovery. +# +. ./common/preamble +_begin_fstest auto growfs stress shutdown log recoveryloop + +# Import common functions. +. ./common/filter + +_stress_scratch() +{ + procs=4 + nops=999999 + # -w ensures that the only ops are ones which cause write I/O + FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \ + -n $nops $FSSTRESS_AVOID` + $FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 & +} + +_require_scratch +_require_realtime + +_scratch_mkfs_xfs | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs +. $tmp.mkfs # extract blocksize and data size for scratch device + +endsize=`expr 550 \* 1048576` # stop after growing this big +[ `expr $endsize / $dbsize` -lt $dblocks ] || _notrun "Scratch device too small" + +nags=4 +size=`expr 125 \* 1048576` # 120 megabytes initially +sizeb=`expr $size / $dbsize` # in data blocks +logblks=$(_scratch_find_xfs_min_logblocks -rsize=${size} -dagcount=${nags}) + +_scratch_mkfs_xfs -lsize=${logblks}b -rsize=${size} -dagcount=${nags} \ + >> $seqres.full +_scratch_mount +_xfs_force_bdev realtime $SCRATCH_MNT &> /dev/null + +# Grow the filesystem in random sized chunks while stressing and performing +# shutdown and recovery. The randomization is intended to create a mix of sub-ag +# and multi-ag grows. +while [ $size -le $endsize ]; do + echo "*** stressing a ${sizeb} block filesystem" >> $seqres.full + _stress_scratch + incsize=$((RANDOM % 40 * 1048576)) + size=`expr $size + $incsize` + sizeb=`expr $size / $dbsize` # in data blocks + echo "*** growing to a ${sizeb} block filesystem" >> $seqres.full + xfs_growfs -R ${sizeb} $SCRATCH_MNT >> $seqres.full + + sleep $((RANDOM % 3)) + _scratch_shutdown + ps -e | grep fsstress > /dev/null 2>&1 + while [ $? -eq 0 ]; do + killall -9 fsstress > /dev/null 2>&1 + wait > /dev/null 2>&1 + ps -e | grep fsstress > /dev/null 2>&1 + done + _scratch_cycle_mount || _fail "cycle mount failed" +done > /dev/null 2>&1 +wait # stop for any remaining stress processes + +_scratch_unmount + +status=0 +exit
tests/xfs/610.out new file +7 -0
@@ -0,0 +1,7 @@ +QA output created by 610 +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks +data = bsize=XXX blocks=XXX, imaxpct=PCT + = sunit=XXX swidth=XXX, unwritten=X +naming =VERN bsize=XXX +log =LDEV bsize=XXX blocks=XXX +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
Signature · 1 line
2.46.2
Quoted 10 lines · Brian Foster
I believe you reproduced a problem with your customized realtime variant of the initial test. I've not been able to reproduce any test failures with patch 2 here, though I have tried to streamline the test a bit to reduce unnecessary bits (patch 1 still reproduces the original problems). I also don't tend to test much with rt, so it's possible my config is off somehow or another. Otherwise I _think_ I've included the necessary changes for rt support in the test itself.
Thoughts? I'd like to figure out what might be going on there before this should land..
Darrick mentioned that was just with his rt group patchset, which make sense as we don't have per-group metadata without that.
Anyway, the series looks good to me, and I think it supersedes my more targeted hand crafted reproducer.
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Brian Foster
I believe you reproduced a problem with your customized realtime variant of the initial test. I've not been able to reproduce any test failures with patch 2 here, though I have tried to streamline the test a bit to reduce unnecessary bits (patch 1 still reproduces the original problems). I also don't tend to test much with rt, so it's possible my config is off somehow or another. Otherwise I _think_ I've included the necessary changes for rt support in the test itself.
Quoted 6 lines
Quoted 2 lines
Thoughts? I'd like to figure out what might be going on there before this should land..
Darrick mentioned that was just with his rt group patchset, which make sense as we don't have per-group metadata without that.
Ah, that would explain it then.
Quoted 3 lines
Anyway, the series looks good to me, and I think it supersedes my more targeted hand crafted reproducer.
Ok, thanks. It would be nice if anybody who knows more about the rt group stuff could give the rt test a quick whirl and just confirm it's at least still effective in that known broken case after my tweaks. Otherwise I'll wait on any feedback on the code/test itself... thanks.
Brian
Quoted 10 lines · Brian Foster
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Brian Foster
I believe you reproduced a problem with your customized realtime variant of the initial test. I've not been able to reproduce any test failures with patch 2 here, though I have tried to streamline the test a bit to reduce unnecessary bits (patch 1 still reproduces the original problems). I also don't tend to test much with rt, so it's possible my config is off somehow or another. Otherwise I _think_ I've included the necessary changes for rt support in the test itself.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Thoughts? I'd like to figure out what might be going on there before this should land..
Darrick mentioned that was just with his rt group patchset, which make sense as we don't have per-group metadata without that.
Ah, that would explain it then.
Yep.
Quoted 8 lines
Quoted 3 lines
Anyway, the series looks good to me, and I think it supersedes my more targeted hand crafted reproducer.
Ok, thanks. It would be nice if anybody who knows more about the rt group stuff could give the rt test a quick whirl and just confirm it's at least still effective in that known broken case after my tweaks. Otherwise I'll wait on any feedback on the code/test itself... thanks.
Will do, now that I'm out of the mountains. :)
The tests look fine to me, but I guess we could wait to see what falls out when I add bfoster's tests.
--D
Quoted 3 lines
Brian
Quoted 19 lines · Brian Foster
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Brian Foster
I believe you reproduced a problem with your customized realtime variant of the initial test. I've not been able to reproduce any test failures with patch 2 here, though I have tried to streamline the test a bit to reduce unnecessary bits (patch 1 still reproduces the original problems). I also don't tend to test much with rt, so it's possible my config is off somehow or another. Otherwise I _think_ I've included the necessary changes for rt support in the test itself.
Quoted 6 lines
Quoted 2 lines
Thoughts? I'd like to figure out what might be going on there before this should land..
Darrick mentioned that was just with his rt group patchset, which make sense as we don't have per-group metadata without that.
Ah, that would explain it then.
Quoted 8 lines
Quoted 3 lines
Anyway, the series looks good to me, and I think it supersedes my more targeted hand crafted reproducer.
Ok, thanks. It would be nice if anybody who knows more about the rt group stuff could give the rt test a quick whirl and just confirm it's at least still effective in that known broken case after my tweaks. Otherwise I'll wait on any feedback on the code/test itself... thanks.
Perplexingly, I tried this out on the test fleet last night and got zero failures except for torvalds TOT.
Oh, I don't have any recoveryloop VMs that also have rt enabled, maybe that's why 610 didn't pop anywhere.
--D
Quoted 2 lines
Brian
Quoted 5 lines · Darrick J. Wong
Perplexingly, I tried this out on the test fleet last night and got zero failures except for torvalds TOT.
Oh, I don't have any recoveryloop VMs that also have rt enabled, maybe that's why 610 didn't pop anywhere.
Note that your trees already contain the fixes for AGs and RTGs, so they are not expected to fail. To Linus' tree fail is expected for AGs, and we'd need an older version of your rtgroup branch to fail for RTGs.
As far as I can tell the result is expected.