xfstests: new tests and fixes
cover xfstests: new tests and fixes
Pushing out my current stack:
- Fix test 016 to work with delaylog.
- Speed up 042 by using preallocation instead of writing data and files.
- Add test to exercise bmap btree corruption caused by multilevel split.
- Add simple test to exercise splice writes to catch locking issues.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 1/4 xfstests: add simple splice test
From: Dave Chinner <[email protected]>
We don't have any coverage of the splice functionality provided by the kernel in xfstests. Add a simple test that uses the sendfile operation built into xfs_io to copy a file ensure we at least execute the code path in xfstests.
--- 249 | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 249.out | 2 ++ group | 1 + 3 files changed, 65 insertions(+), 0 deletions(-) create mode 100644 249 create mode 100644 249.out
249 new file +62 -0
@@ -0,0 +1,62 @@ +#! /bin/bash +# FS QA Test No. 249 +# +# simple splice(2) test. +# +#----------------------------------------------------------------------- +# Copyright (c) 2011 Red Hat, Inc. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +#----------------------------------------------------------------------- +# +# creator [email protected] + +seq=`basename $0` +echo "QA output created by $seq" + +here=`pwd` +tmp=tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* + rm -f $SRC $DST + _cleanup_testdir +} + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +# real QA test starts here +_supported_fs generic +_supported_os Linux +_setup_testdir + +echo "Feel the serenity." + +SRC=$TEST_DIR/$seq.src +DST=$TEST_DIR/$seq.dst + +$XFS_IO_PROG -f -c "pwrite -S 0xa5a55a5a 0 32768k" -c fsync $SRC > /dev/null 2>&1 +$XFS_IO_PROG -f -c "sendfile -i $SRC 0 32768k" -c fsync $DST > /dev/null 2>&1 + +diff $SRC $DST + +status=$? +exit
249.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 249 +Feel the serenity.
group +1 -0
@@ -362,3 +362,4 @@ deprecated 246 auto quick rw 247 auto quick rw 248 auto quick rw +249 auto quick rw
Signature · 6 lines
1.7.2.3 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 1/4] xfstests: add simple splice test
Quoted 6 lines · Dave Chinner
From: Dave Chinner <[email protected]>
We don't have any coverage of the splice functionality provided by the kernel in xfstests. Add a simple test that uses the sendfile operation built into
xfs_ioto copy a file ensure we at least execute the code path in xfstests.
I'm pretty sure I reviewed this one before, but in case it's needed here it is again:
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 1/4] xfstests: add simple splice test
In reply to Dave ChinnerQuoted 8 lines · Dave Chinner
From: Dave Chinner <[email protected]>
We don't have any coverage of the splice functionality provided by the kernel in xfstests. Add a simple test that uses the sendfile operation built into
xfs_ioto copy a file ensure we at least execute the code path in xfstests.Signed-off-by Dave Chinner <[email protected]>
Looks good.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 2/4 xfstests: add test to reproduce bmap btree corruption.
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
The problem was reprted here:
https://bugzilla.redhat.com/show_bug.cgi?id=626244
With the simple test case:
# mkfs.xfs -f -d agsize=16m,size=50g <dev> # mount <dev> /mnt # xfs_io -f -c 'resvsp 0 40G' /mnt/foo
Triggering the problem. Turn this into a new xfsqa test so that we exercise the problem code and prevent future regressions.
--- 250 | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ group | 1 + 2 files changed, 98 insertions(+), 0 deletions(-) create mode 100644 250
250 new file +97 -0
@@ -0,0 +1,97 @@ +#! /bin/bash +# FS QA Test No. 250 +# +# Bmap btree corruption regression test +# +#----------------------------------------------------------------------- +# Copyright (c) 2011 Red Hat, Inc. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +#----------------------------------------------------------------------- +# +# creator [email protected] + +seq=`basename $0` +echo "QA output created by $seq" + +here=`pwd` +tmp=tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + umount $LOOP_MNT 2>/dev/null + rm -f $LOOP_DEV + rmdir $LOOP_MNT + _cleanup_testdir +} + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +# real QA test starts here +_supported_fs xfs +_supported_os Linux +_require_loop + +LOOP_DEV=$TEST_DIR/$seq.fs +LOOP_MNT=$TEST_DIR/$seq.mnt + +_filter_io() +{ + sed -e '/.* ops; /d' +} + +# real QA test starts here + +echo "*** create loop mount point" +rmdir $LOOP_MNT 2>/dev/null +mkdir -p $LOOP_MNT || _fail "cannot create loopback mount point" + +_test_loop() +{ + size=$1 + agsize=$2 + fsize=$3 + + dparam="file,name=$LOOP_DEV,size=$size" + if [ -n "$agsize" ]; then + dparam="$dparam,agsize=$agsize" + fi + + echo "*** mkfs loop file (size=$size)" + $MKFS_XFS_PROG -d $dparam \ + | _filter_mkfs # 2>/dev/null + + echo "*** mount loop filesystem" + mount -t xfs -o loop $LOOP_DEV $LOOP_MNT + + echo "*** preallocate large file" + xfs_io -f -c "resvsp 0 $fsize" $LOOP_MNT/foo | _filter_io + + echo "*** unmount loop filesystem" + umount $LOOP_MNT + + echo "*** check loop filesystem" + _check_xfs_filesystem $LOOP_DEV none none +} + +_test_loop 50g 16m 40G +echo "*** done" +status=0 +exit
group +1 -0
@@ -363,3 +363,4 @@ deprecated 247 auto quick rw 248 auto quick rw 249 auto quick rw +250 auto quick rw prealloc metadata
Signature · 6 lines
1.7.2.3 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 2/4] xfstests: add test to reproduce bmap btree corruption.
Looks good,
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 2/4] xfstests: add test to reproduce bmap btree corruption.
In reply to Dave ChinnerQuoted 8 lines · Dave Chinner
From: Dave Chinner <[email protected]>
The problem was reprted here:
https://bugzilla.redhat.com/show_bug.cgi?id=626244
With the simple test case:
Quoted 8 lines
# mkfs.xfs -f -d agsize=16m,size=50g <dev> # mount <dev> /mnt # xfs_io -f -c 'resvsp 0 40G' /mnt/fooTriggering the problem. Turn this into a new xfsqa test so that we exercise the problem code and prevent future regressions.
Signed-off-by Dave Chinner <[email protected]>
This looks good, HOWEVER, since the test crashes the target system please remove it from (at least) the "auto" group before you commit.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 2/4] xfstests: add test to reproduce bmap btree corruption.
Quoted 13 lines · Alex Elder
Quoted 12 lines · Dave Chinner
From: Dave Chinner <[email protected]>
The problem was reprted here:
https://bugzilla.redhat.com/show_bug.cgi?id=626244
With the simple test case:
# mkfs.xfs -f -d agsize=16m,size=50g <dev> # mount <dev> /mnt # xfs_io -f -c 'resvsp 0 40G' /mnt/foo
Quoted 8 lines
Quoted 4 lines
Triggering the problem. Turn this into a new xfsqa test so that we exercise the problem code and prevent future regressions.
Signed-off-by Dave Chinner <[email protected]>This looks good, HOWEVER, since the test crashes the target system please remove it from (at least) the "auto" group before you commit.
If causes an assert failure on a DEBUG kernel, a shutdown on a non-debug kernel. Either way, I'd prefer to leave it enabled as a good reminder that we need to fix the damn bug (*) sooner rather than later...
(*) This patch prevents the assert failure, but, like Lachlan, I'm not sure it is the right fix.
http://oss.sgi.com/archives/xfs/2010-09/msg00337.html
I'm working on it at the moment, along with all the other allocation things...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 3/4 xfstests: make 016 work with delaylog
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Test 016 fails with delaylog because it measures log traffic to disk and delaylog writes almost nothing to the log for the given test. TO make it work, add sync calls to the work loop to cause the log to be flushed reliably for both delaylog and nodelaylog and hence contain the same nuumber of log records.
As a result, the log space consumed by the test is not changed by the delaylog option and the test passes. The test is not significantly slowed down by the addition of the sync calls (takes 15s to run on a single SATA drive).
--- 016 | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
016 +2 -0
@@ -101,7 +101,9 @@ _log_traffic() while [ $count -ge 0 ] do touch $out + sync rm $out + sync let "count = count - 1" done
Signature · 6 lines
1.7.2.3 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 3/4] xfstests: make 016 work with delaylog
016 _log_traffic() · line 108
Quoted 9 lines
016 +2
-0@@ -101,7 +101,9 @@ _log_traffic() while [ $count -ge 0 ] do touch $out + sync rm $out + sync let "count = count - 1" done
Looks good, although it might be good to make the indentation fit the surrounding code.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 3/4] xfstests: make 016 work with delaylog
In reply to Dave ChinnerQuoted 8 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Test 016 fails with delaylog because it measures log traffic to disk and delaylog writes almost nothing to the log for the given test. TO make it work, add sync calls to the work loop to cause the log to be flushed reliably for both delaylog and nodelaylog and hence contain the same nuumber of log records.
Quoted 7 lines
As a result, the log space consumed by the test is not changed by the delaylog option and the test passes. The test is not significantly slowed down by the addition of the sync calls (takes 15s to run on a single SATA drive).
Signed-off-by Dave Chinner <[email protected]>---
Looks good.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 4/4 xfstests: Speed up test 042
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
test 042 generates a worst-case fragmented filesystem and uses it to test xfs_fsr. It uses small 4k files to generate the hole-space-hole pattern that fragments free space badly. It is much faster to generate the same pattern by creating a single large file and punching holes in it. Also, instead of writing large files to create unfragmented space, just use preallocation so we don't have to write the data to disk.
These changes reduce the runtime of the test on a single SATA drive from 106s to 27s.
--- 042 | 38 ++++++++++++++++++++++++++++++-------- 042.out | 6 +++--- 2 files changed, 33 insertions(+), 11 deletions(-)
042 +30 -8
@@ -100,16 +100,19 @@ echo "done" echo -n "Reserve 16 1Mb unfragmented regions... " for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 do - _do "dd if=/dev/zero of=$SCRATCH_MNT/hole$i bs=4096 count=256" - _do "dd if=/dev/zero of=$SCRATCH_MNT/space$i bs=4096 count=1" - _do "xfs_bmap -v $SCRATCH_MNT/hole$i" + _do "$XFS_IO_PROG -f -c \"resvsp 0 1m\" $SCRATCH_MNT/hole$i" + _do "$XFS_IO_PROG -f -c \"resvsp 0 4k\" $SCRATCH_MNT/space$i" + _do "$XFS_IO_PROG -f -c \"resvsp 0 1m\" $SCRATCH_MNT/hole$i" + _do "xfs_bmap -vp $SCRATCH_MNT/hole$i" done echo "done" # set up filesystem -echo -n "Fill filesystem with 4k files, generate manifest... " -fill_options="--verbose --seed=0 --filesize=4096 --stddev=0 --sync=1000000" -_do "src/fill2fs $fill_options --dir=$SCRATCH_MNT/fill --list=- > $tmp.manifest" +echo -n "Fill filesystem with fill file... " +for i in `seq 0 1 31`; do + _do "$XFS_IO_PROG -fs -c \"pwrite -S$i ${i}m 1m\" $SCRATCH_MNT/fill" +done +_do "xfs_bmap -vp $SCRATCH_MNT/fill" echo "done" # flush the filesystem - make sure there is no space "lost" to pre-allocation _do "umount $SCRATCH_MNT"
@@ -119,7 +122,18 @@ _do "dd if=/dev/zero of=$SCRATCH_MNT/pad bs=4096" echo "done" # create fragmented file -_do "Delete every second file" "_cull_files" +#_do "Delete every second file" "_cull_files" +echo -n "Punch every second 4k block... " +for i in `seq 0 8 32768`; do + # This generates excessive output that significantly slows down the + # test. It's not necessary for debug, so just bin it. + $XFS_IO_PROG -f -c "unresvsp ${i}k 4k" $SCRATCH_MNT/fill \ + > /dev/null 2>&1 +done +_do "xfs_bmap -vp $SCRATCH_MNT/fill" +_do "sum $SCRATCH_MNT/fill >$tmp.fillsum1" +echo "done" + echo -n "Create one very large file... " _do "src/fill2 -d nbytes=16000000,file=$SCRATCH_MNT/fragmented" echo "done"
@@ -130,7 +144,15 @@ _do "Remove other files" "rm -rf $SCRATCH_MNT/{pad,hole*}" # defragment _do "Run xfs_fsr on filesystem" "$XFS_FSR_PROG -v $SCRATCH_MNT/fragmented" _do "xfs_bmap -v $SCRATCH_MNT/fragmented" -_do "Check 4k files" "src/fill2fs_check $tmp.manifest" + +echo -n "Check fill file... " +_do "sum $SCRATCH_MNT/fill >$tmp.fillsum2" +if ! _do "diff $tmp.fillsum1 $tmp.fillsum2"; then + echo "fail" + echo "Fill file is corrupt/missing after fsr. Test failed see $seq.full" + status=1; exit +fi +echo "done" # check echo -n "Check large file... "
042.out +3 -3
@@ -1,13 +1,13 @@ QA output created by 042 Make a 48 megabyte filesystem on SCRATCH_DEV and mount... done Reserve 16 1Mb unfragmented regions... done -Fill filesystem with 4k files, generate manifest... done +Fill filesystem with fill file... done Use up any further available space using dd... done -Delete every second file... done +Punch every second 4k block... done Create one very large file... done Remove other files... done Run xfs_fsr on filesystem... done -Check 4k files... done +Check fill file... done Check large file... done Checking filesystem... done xfs_fsr tests passed.
Signature · 6 lines
1.7.2.3 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 4/4] xfstests: Speed up test 042
Quoted 10 lines · Dave Chinner
From: Dave Chinner <[email protected]>
test 042 generates a worst-case fragmented filesystem and uses it to test
xfs_fsr. It uses small 4k files to generate the hole-space-hole pattern that fragments free space badly. It is much faster to generate the same pattern by creating a single large file and punching holes in it. Also, instead of writing large files to create unfragmented space, just use preallocation so we don't have to write the data to disk.
Quoted 2 lines
These changes reduce the runtime of the test on a single SATA drive from 106s to 27s.
Looks good,
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 4/4] xfstests: Speed up test 042
In reply to Dave ChinnerQuoted 10 lines · Dave Chinner
From: Dave Chinner <[email protected]>
test 042 generates a worst-case fragmented filesystem and uses it to test
xfs_fsr. It uses small 4k files to generate the hole-space-hole pattern that fragments free space badly. It is much faster to generate the same pattern by creating a single large file and punching holes in it. Also, instead of writing large files to create unfragmented space, just use preallocation so we don't have to write the data to disk.
Quoted 4 lines
These changes reduce the runtime of the test on a single SATA drive from 106s to 27s.
Signed-off-by Dave Chinner <[email protected]>
Perhaps your test system is configured different from mine, but I don't see the speedup you see. In fact, it might have slowed it down. I really haven't experimented much though and will report back if I find anything more constructive to say.
Other than that I see no problem with the change. And the difference either way isn't a big deal to me, so:
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 4/4] xfstests: Speed up test 042
Quoted 14 lines · Alex Elder
Quoted 13 lines · Dave Chinner
From: Dave Chinner <[email protected]>
test 042 generates a worst-case fragmented filesystem and uses it to test
xfs_fsr. It uses small 4k files to generate the hole-space-hole pattern that fragments free space badly. It is much faster to generate the same pattern by creating a single large file and punching holes in it. Also, instead of writing large files to create unfragmented space, just use preallocation so we don't have to write the data to disk.These changes reduce the runtime of the test on a single SATA drive from 106s to 27s.
Quoted 7 lines
Quoted 1 line
Signed-off-by Dave Chinner <[email protected]>Perhaps your test system is configured different from mine, but I don't see the speedup you see. In fact, it might have slowed it down. I really haven't experimented much though and will report back if I find anything more constructive to say.
That may be because the resvsp call at the moment is a sync transaction. I've got a patch to make that async which should solve the problem for you. Mind you, the above numbers come from a Vm that wasn't running that patch, so it probably depends mostly on how efficient your storage is a processing barriers to the difference you see right now...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs