btrfs: dm-log-writes test for fstrim with fsstress
cover btrfs: dm-log-writes test for fstrim with fsstress
Just to make sure the fstrim is not trimming anything vital (like tree blocks still in use) on btrfs.
The first patch is to enhance log-writes to check each DISCARD operation.
The feature is not used in test cases, as it's too time consuming. But should be a pretty handy feature for small logwrites.
The second patch is the test case.
Which triggers fsstress and a loop of fstrim, all recorded by dm-log-writes.
Then check each FLUSH after the 'prepare' mark.
The test case in theory can go generic, but I'm using various btrfs specific hack to speed up the test and has a workaround to avoid false alert.
So far I haven't triggered a bug.
Qu Wenruo (2):
fstests: log-writes: Add new discard check point
fstests: btrfs: dm-logwrites test for fstrim and fsstress workload
src/log-writes/replay-log.c | 10 ++- tests/btrfs/197 | 131 ++++++++++++++++++++++++++++++++++++ tests/btrfs/197.out | 2 + tests/btrfs/group | 1 + 4 files changed, 142 insertions(+), 2 deletions(-) create mode 100755 tests/btrfs/197 create mode 100644 tests/btrfs/197.out
Signature · 1 line
2.23.0
PATCH 1/2 fstests: log-writes: Add new discard check point
Despite the existing <number>|fua|flush checkpoint, add a new discard check point to make sure discard is not screwing up things.
--- src/log-writes/replay-log.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
src/log-writes/replay-log.c +8 -2
@@ -64,8 +64,8 @@ static void usage(void)fprintf (stderr,"\t--no-discard - don't process discard entries\n" );fprintf (stderr,"\t--fsck - the fsck command to run, must specify " "--check\n" ); -fprintf (stderr,"\t--check [<number>|flush|fua] when to check the " -"file system, mush specify --fsck\n" ); +fprintf (stderr,"\t--check [<number>|flush|fua|discard] when to check " +"the file system, mush specify --fsck\n" );fprintf (stderr,"\t--start-sector <sector> - replay ops on region " "from <sector> onto <device>\n" ); fprintf(stderr, "\t--end-sector <sector> - replay ops on region "
@@ -115,6 +115,7 @@ enum log_replay_check_mode {CHECK_NUMBER =1 ,CHECK_FUA =2 ,CHECK_FLUSH =3 , + CHECK_DISCARD = 4, }; static int seek_to_mark(struct log *log, struct log_write_entry *entry,
@@ -253,6 +254,8 @@ int main(int argc, char **argv) check_mode =CHECK_FLUSH ; }else if (!strcmp (optarg,"fua" )) { check_mode =CHECK_FUA ; + }else if (!strcmp (optarg,"discard" )) { + check_mode =CHECK_DISCARD ; }else { check_mode =CHECK_NUMBER ; check_number =strtoull (optarg, &tmp,0 );
@@ -369,6 +372,9 @@ int main(int argc, char **argv)else if ((check_mode ==CHECK_FLUSH ) &&should_stop (entry,LOG_FLUSH_FLAG ,NULL )) ret =run_fsck (log, fsck_command); +else if ((check_mode ==CHECK_DISCARD ) && +should_stop (entry,LOG_DISCARD_FLAG ,NULL )) + ret =run_fsck (log, fsck_command);else ret =0 ;if (ret) {
Signature · 1 line
2.23.0
PATCH 2/2 fstests: btrfs: dm-logwrites test for fstrim and fsstress workload
In reply to Qu WenruoThere is a fs corruption report of a tree block in use get trimmed, and cause fs corruption.
Although I haven't found the cause from the source code, it won't hurt to add such test case.
The test case is limited to btrfs due to the replay-log --check|--fsck hack to reduce runtime.
Other fs can't go with the replay-log --check|--fsck hack as their fsck will report dirty journal as error.
---
Due to recent change in btrfs side, already trimmed tree blocks won't get trimmed again until new data is written.
This makes things safer, and I'm not sure if it's the reason why the test never fails on latest kernel.
--- tests/btrfs/197 | 131 ++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/197.out | 2 + tests/btrfs/group | 1 + 3 files changed, 134 insertions(+) create mode 100755 tests/btrfs/197 create mode 100644 tests/btrfs/197.out
tests/btrfs/197 new file +131 -0
@@ -0,0 +1,131 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. + +# FS QA Test 197 +# +# Test btrfs consistency after each DISCARD for a workload with fstrim and +# fsstress. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + kill -q $fstrim_pid &> /dev/null + "$KILLALL_PROG" -q $FSSTRESS_PROG &> /dev/null + wait + _log_writes_cleanup &> /dev/null + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/dmlogwrites + +# remove previous $seqres.full before test +rm -f $seqres.full + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_supported_os Linux +_require_command "$KILLALL_PROG" killall +_require_command "$BLKDISCARD_PROG" blkdiscard +_require_btrfs_fs_feature "no_holes" +_require_btrfs_mkfs_feature "no-holes" +_require_fstrim +_require_log_writes +_require_scratch + +runtime=30 +nr_cpus=$("$here/src/feature" -o) +# cap nr_cpus to 8 to avoid spending too much time on hosts with many cpus +if [ $nr_cpus -gt 8 ]; then + nr_cpus=8 +fi +fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 99999 -p $nr_cpus \ + $FSSTRESS_AVOID) + +fstrim_workload() +{ + trap "wait; exit" SIGTERM + + while true; do + $FSTRIM_PROG -v $SCRATCH_MNT >> $seqres.full + done +} + +# Replay and check each fua/flush (specified by $2) point. +# +# Since dm-log-writes records bio sequentially, even just replaying a range +# still needs to iterate all records before the end point. +# When number of records grows, it will be unacceptably slow, thus we need +# to use relay-log itself to trigger fsck, avoid unnecessary seek. +log_writes_fast_replay_check() +{ + local check_point=$1 + local blkdev=$2 + local fsck_command="$BTRFS_UTIL_PROG check $blkdev" + local ret + + [ -z "$check_point" -o -z "$blkdev" ] && _fail \ + "check_point and blkdev must be specified for log_writes_fast_replay_check" + + # Replay to first mark + $here/src/log-writes/replay-log --log $LOGWRITES_DEV \ + --replay $blkdev --end-mark prepare + $here/src/log-writes/replay-log --log $LOGWRITES_DEV \ + --replay $blkdev --start-mark prepare \ + --check $check_point --fsck "$fsck_command" \ + &> $tmp.full_fsck + ret=$? + tail -n 150 $tmp.full_fsck > $seqres.full + [ $ret -ne 0 ] && _fail "fsck failed during replay" +} + +_log_writes_init $SCRATCH_DEV + +# Discard the whole devices so when some tree pointer is wrong, it won't point +# to some older valid tree blocks, so we can detect it. +$BLKDISCARD_PROG $LOGWRITES_DMDEV > /dev/null 2>&1 + +# The regular workaround to avoid false alert on unexpected holes +_log_writes_mkfs -O no-holes >> $seqres.full +_log_writes_mount + +$FSTRIM_PROG -v $SCRATCH_MNT >> $seqres.full || _notrun "FSTRIM not supported" + +_log_writes_mark prepare + +fstrim_workload & +fstrim_pid=$! + +"$FSSTRESS_PROG" $fsstress_args > /dev/null & +sleep $runtime + +"$KILLALL_PROG" -q "$FSSTRESS_PROG" &> /dev/null +kill $fstrim_pid +wait + +_log_writes_unmount +_log_writes_remove + +# The best checkpoint is discard, however since there are a lot of +# discard, using discard check point is too time consuming. +# Here trade coverage for a much shorter runtime +log_writes_fast_replay_check flush "$SCRATCH_DEV" + +echo "Silence is golden" +# success, all done +status=0 +exit
tests/btrfs/197.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 197 +Silence is golden
tests/btrfs/group +1 -0
@@ -199,3 +199,4 @@ 194 auto volume 195 auto volume 196 auto metadata log volume +197 auto replay trim
Signature · 1 line
2.23.0
Re: [PATCH 2/2] fstests: btrfs: dm-logwrites test for fstrim and fsstress workload
Quoted 9 lines · Qu Wenruo
There is a fs corruption report of a tree block in use get trimmed, and cause fs corruption.
Although I haven't found the cause from the source code, it won't hurt to add such test case.
The test case is limited to btrfs due to the replay-log --check|--fsck hack to reduce runtime.
I am not sure why this is referred to as a hack?
Quoted 2 lines
Other fs can't go with the replay-log --check|--fsck hack as their fsck will report dirty journal as error.
This doesn't make the test btrfs specific.
The helper you use log_writes_fast_replay_check() is already dupliacte code from btrfs/192 and it should be in common/dmlogwrites. The helper itself would be quite generic if it didn't hardcode btrfs check tool.
I think a better solution would be to _require_fast_check_fs and it is possible that other fs will have a future flag for fsck to not fail if journal needs to be replayed, but the check whatever is possible to check in that state (e.g. the super block and the consistency of the journal).
However, regarding requiring no_holes, I am not sure how you would encode that in a generic test???
Thanks,
Amir.
Quoted 173 lines
Signed-off-by Qu Wenruo <[email protected]>---
Due to recent change in btrfs side, already trimmed tree blocks won't get trimmed again until new data is written.This makes things safer, and I'm not sure if it's the reason why the test never fails on latest kernel.
--- tests/btrfs/197 | 131 ++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/197.out | 2 + tests/btrfs/group | 1 + 3 files changed, 134 insertions(+) create mode 100755 tests/btrfs/197 create mode 100644 tests/btrfs/197.outtests/btrfs/197 new file +131
-0@@ -0,0 +1,131 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. + +# FS QA Test 197 +# +# Test btrfs consistency after each DISCARD for a workload with fstrim and +# fsstress. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + kill -q $fstrim_pid &> /dev/null + "$KILLALL_PROG" -q $FSSTRESS_PROG &> /dev/null + wait + _log_writes_cleanup &> /dev/null + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/dmlogwrites + +# remove previous $seqres.full before test +rm -f $seqres.full + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_supported_os Linux +_require_command "$KILLALL_PROG" killall +_require_command "$BLKDISCARD_PROG" blkdiscard +_require_btrfs_fs_feature "no_holes" +_require_btrfs_mkfs_feature "no-holes" +_require_fstrim +_require_log_writes +_require_scratch + +runtime=30 +nr_cpus=$("$here/src/feature" -o) +# cap nr_cpus to 8 to avoid spending too much time on hosts with many cpus +if [ $nr_cpus -gt 8 ]; then + nr_cpus=8 +fi +fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 99999 -p $nr_cpus \ + $FSSTRESS_AVOID) + +fstrim_workload() +{ + trap "wait; exit" SIGTERM + + while true; do + $FSTRIM_PROG -v $SCRATCH_MNT >> $seqres.full + done +} + +# Replay and check each fua/flush (specified by $2) point. +# +# Since dm-log-writes records bio sequentially, even just replaying a range +# still needs to iterate all records before the end point. +# When number of records grows, it will be unacceptably slow, thus we need +# to use relay-log itself to trigger fsck, avoid unnecessary seek. +log_writes_fast_replay_check() +{ + local check_point=$1 + local blkdev=$2 + local fsck_command="$BTRFS_UTIL_PROG check $blkdev" + local ret + + [ -z "$check_point" -o -z "$blkdev" ] && _fail \ + "check_point and blkdev must be specified for log_writes_fast_replay_check" + + # Replay to first mark + $here/src/log-writes/replay-log --log $LOGWRITES_DEV \ + --replay $blkdev --end-mark prepare + $here/src/log-writes/replay-log --log $LOGWRITES_DEV \ + --replay $blkdev --start-mark prepare \ + --check $check_point --fsck "$fsck_command" \ + &> $tmp.full_fsck + ret=$? + tail -n 150 $tmp.full_fsck > $seqres.full + [ $ret -ne 0 ] && _fail "fsck failed during replay" +} + +_log_writes_init $SCRATCH_DEV + +# Discard the whole devices so when some tree pointer is wrong, it won't point +# to some older valid tree blocks, so we can detect it. +$BLKDISCARD_PROG $LOGWRITES_DMDEV > /dev/null 2>&1 + +# The regular workaround to avoid false alert on unexpected holes +_log_writes_mkfs -O no-holes >> $seqres.full +_log_writes_mount + +$FSTRIM_PROG -v $SCRATCH_MNT >> $seqres.full || _notrun "FSTRIM not supported" + +_log_writes_mark prepare + +fstrim_workload & +fstrim_pid=$! + +"$FSSTRESS_PROG" $fsstress_args > /dev/null & +sleep $runtime + +"$KILLALL_PROG" -q "$FSSTRESS_PROG" &> /dev/null +kill $fstrim_pid +wait + +_log_writes_unmount +_log_writes_remove + +# The best checkpoint is discard, however since there are a lot of +# discard, using discard check point is too time consuming. +# Here trade coverage for a much shorter runtime +log_writes_fast_replay_check flush "$SCRATCH_DEV" + +echo "Silence is golden" +# success, all done +status=0 +exittests/btrfs/197.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 197 +Silence is goldentests/btrfs/group +1
-1@@ -199,3 +199,4 @@ 194 auto volume 195 auto volume 196 auto metadata log volume +197 auto replay trim --2.23.0
Re: [PATCH 2/2] fstests: btrfs: dm-logwrites test for fstrim and fsstress workload
Quoted 23 lines · Amir Goldstein
Quoted 9 lines · Qu Wenruo
There is a fs corruption report of a tree block in use get trimmed, and cause fs corruption.
Although I haven't found the cause from the source code, it won't hurt to add such test case.
The test case is limited to btrfs due to the replay-log --check|--fsck hack to reduce runtime.
I am not sure why this is referred to as a hack?
Quoted 2 lines
Other fs can't go with the replay-log --check|--fsck hack as their fsck will report dirty journal as error.
This doesn't make the test btrfs specific.
The helper you uselog_writes_fast_replay_check()is already dupliacte code from btrfs/192 and it should be in common/dmlogwrites. The helper itself would be quite generic if it didn't hardcode btrfs check tool.
Quoted 4 lines
I think a better solution would be to
_require_fast_check_fsand it is possible that other fs will have a future flag for fsck to not fail if journal needs to be replayed, but the check whatever is possible to check in that state (e.g. the super block and the consistency of the journal).
That's the best case, but we're not yet at the stage, so I'll keep the test in btrfs group for now.
Quoted 3 lines
However, regarding requiring
no_holes, I am not sure how you would encode that in a generic test???
That can be kinda worked around, as btrfs is soon to default to no-holes feature.
But for now, the explicit no-holes mkfs option is still needed.
Thanks,
Qu
Quoted 177 lines
Thanks,
Amir.Quoted 173 lines
Signed-off-by Qu Wenruo <[email protected]>---
Due to recent change in btrfs side, already trimmed tree blocks won't get trimmed again until new data is written.This makes things safer, and I'm not sure if it's the reason why the test never fails on latest kernel.
--- tests/btrfs/197 | 131 ++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/197.out | 2 + tests/btrfs/group | 1 + 3 files changed, 134 insertions(+) create mode 100755 tests/btrfs/197 create mode 100644 tests/btrfs/197.outtests/btrfs/197 new file +131
-0@@ -0,0 +1,131 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. + +# FS QA Test 197 +# +# Test btrfs consistency after each DISCARD for a workload with fstrim and +# fsstress. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + kill -q $fstrim_pid &> /dev/null + "$KILLALL_PROG" -q $FSSTRESS_PROG &> /dev/null + wait + _log_writes_cleanup &> /dev/null + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/dmlogwrites + +# remove previous $seqres.full before test +rm -f $seqres.full + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_supported_os Linux +_require_command "$KILLALL_PROG" killall +_require_command "$BLKDISCARD_PROG" blkdiscard +_require_btrfs_fs_feature "no_holes" +_require_btrfs_mkfs_feature "no-holes" +_require_fstrim +_require_log_writes +_require_scratch + +runtime=30 +nr_cpus=$("$here/src/feature" -o) +# cap nr_cpus to 8 to avoid spending too much time on hosts with many cpus +if [ $nr_cpus -gt 8 ]; then + nr_cpus=8 +fi +fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 99999 -p $nr_cpus \ + $FSSTRESS_AVOID) + +fstrim_workload() +{ + trap "wait; exit" SIGTERM + + while true; do + $FSTRIM_PROG -v $SCRATCH_MNT >> $seqres.full + done +} + +# Replay and check each fua/flush (specified by $2) point. +# +# Since dm-log-writes records bio sequentially, even just replaying a range +# still needs to iterate all records before the end point. +# When number of records grows, it will be unacceptably slow, thus we need +# to use relay-log itself to trigger fsck, avoid unnecessary seek. +log_writes_fast_replay_check() +{ + local check_point=$1 + local blkdev=$2 + local fsck_command="$BTRFS_UTIL_PROG check $blkdev" + local ret + + [ -z "$check_point" -o -z "$blkdev" ] && _fail \ + "check_point and blkdev must be specified for log_writes_fast_replay_check" + + # Replay to first mark + $here/src/log-writes/replay-log --log $LOGWRITES_DEV \ + --replay $blkdev --end-mark prepare + $here/src/log-writes/replay-log --log $LOGWRITES_DEV \ + --replay $blkdev --start-mark prepare \ + --check $check_point --fsck "$fsck_command" \ + &> $tmp.full_fsck + ret=$? + tail -n 150 $tmp.full_fsck > $seqres.full + [ $ret -ne 0 ] && _fail "fsck failed during replay" +} + +_log_writes_init $SCRATCH_DEV + +# Discard the whole devices so when some tree pointer is wrong, it won't point +# to some older valid tree blocks, so we can detect it. +$BLKDISCARD_PROG $LOGWRITES_DMDEV > /dev/null 2>&1 + +# The regular workaround to avoid false alert on unexpected holes +_log_writes_mkfs -O no-holes >> $seqres.full +_log_writes_mount + +$FSTRIM_PROG -v $SCRATCH_MNT >> $seqres.full || _notrun "FSTRIM not supported" + +_log_writes_mark prepare + +fstrim_workload & +fstrim_pid=$! + +"$FSSTRESS_PROG" $fsstress_args > /dev/null & +sleep $runtime + +"$KILLALL_PROG" -q "$FSSTRESS_PROG" &> /dev/null +kill $fstrim_pid +wait + +_log_writes_unmount +_log_writes_remove + +# The best checkpoint is discard, however since there are a lot of +# discard, using discard check point is too time consuming. +# Here trade coverage for a much shorter runtime +log_writes_fast_replay_check flush "$SCRATCH_DEV" + +echo "Silence is golden" +# success, all done +status=0 +exittests/btrfs/197.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 197 +Silence is goldentests/btrfs/group +1
-1@@ -199,3 +199,4 @@ 194 auto volume 195 auto volume 196 auto metadata log volume +197 auto replay trim --2.23.0