Skip to contentSkip to conversation

xfstests: new tests and fixes

PATCH In review Dave Chinner 3 participants 15 messages Jan 25, 2011 – Feb 2, 2011

lore
Share & tools
lore
#0 lore raw

cover xfstests: new tests and fixes

Pushing out my current stack:

- Fix test 016 to work with delaylog.
- Speed up 042 by using preallocation instead of writing data and files.
- Add test to exercise bmap btree corruption caused by multilevel split.
- Add simple test to exercise splice writes to catch locking issues.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw

PATCH 1/4 xfstests: add simple splice test

From: Dave Chinner <[email protected]>

We don't have any coverage of the splice functionality provided by the kernel in xfstests. Add a simple test that uses the sendfile operation built into xfs_io to copy a file ensure we at least execute the code path in xfstests.

---
 249     |   62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 249.out |    2 ++
 group   |    1 +
 3 files changed, 65 insertions(+), 0 deletions(-)
 create mode 100644 249
 create mode 100644 249.out
249 new file +62 -0
new file mode 100644
index 0000000..6fc972e
@@ -0,0 +1,62 @@
+#! /bin/bash
+# FS QA Test No. 249
+#
+# simple splice(2) test.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2011 Red Hat, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+# creator
[email protected]
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=tmp/$$
+status=1  # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+	rm -f $SRC $DST
+	_cleanup_testdir
+}
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_setup_testdir
+
+echo "Feel the serenity."
+
+SRC=$TEST_DIR/$seq.src
+DST=$TEST_DIR/$seq.dst
+
+$XFS_IO_PROG -f -c "pwrite -S 0xa5a55a5a 0 32768k" -c fsync $SRC > /dev/null 2>&1
+$XFS_IO_PROG -f -c "sendfile -i $SRC 0 32768k" -c fsync $DST > /dev/null 2>&1
+
+diff $SRC $DST
+
+status=$?
+exit
249.out new file +2 -0
new file mode 100644
index 0000000..592696f
@@ -0,0 +1,2 @@
+QA output created by 249
+Feel the serenity.
group +1 -0
index a40c98f..b94f543 100644
@@ -362,3 +362,4 @@ deprecated
 246 auto quick rw
 247 auto quick rw
 248 auto quick rw 
+249 auto quick rw 
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Reviewed-by on v1 1/4
#2 lore raw

Re: [PATCH 1/4] xfstests: add simple splice test

Quoted 6 lines · Dave Chinner

From: Dave Chinner <[email protected]>

We don't have any coverage of the splice functionality provided by the kernel in xfstests. Add a simple test that uses the sendfile operation built into xfs_io to copy a file ensure we at least execute the code path in xfstests.

I'm pretty sure I reviewed this one before, but in case it's needed here it is again:

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by on v1 1/4
#3 lore raw

Re: [PATCH 1/4] xfstests: add simple splice test

In reply to Dave Chinner
Quoted 8 lines · Dave Chinner

From: Dave Chinner <[email protected]>

We don't have any coverage of the splice functionality provided by the kernel in xfstests. Add a simple test that uses the sendfile operation built into xfs_io to copy a file ensure we at least execute the code path in xfstests.

Looks good.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw

PATCH 2/4 xfstests: add test to reproduce bmap btree corruption.

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

The problem was reprted here:

https://bugzilla.redhat.com/show_bug.cgi?id=626244

With the simple test case:

# mkfs.xfs -f -d agsize=16m,size=50g <dev>
# mount <dev> /mnt
# xfs_io -f -c 'resvsp 0 40G' /mnt/foo

Triggering the problem. Turn this into a new xfsqa test so that we exercise the problem code and prevent future regressions.

---
 250   |   97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 group |    1 +
 2 files changed, 98 insertions(+), 0 deletions(-)
 create mode 100644 250
250 new file +97 -0
new file mode 100644
index 0000000..cb0fa9c
@@ -0,0 +1,97 @@
+#! /bin/bash
+# FS QA Test No. 250
+#
+# Bmap btree corruption regression test
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2011 Red Hat, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+# creator
[email protected]
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=tmp/$$
+status=1  # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	umount $LOOP_MNT 2>/dev/null
+	rm -f $LOOP_DEV
+	rmdir $LOOP_MNT
+	_cleanup_testdir
+}
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+# real QA test starts here
+_supported_fs xfs
+_supported_os Linux
+_require_loop
+
+LOOP_DEV=$TEST_DIR/$seq.fs
+LOOP_MNT=$TEST_DIR/$seq.mnt
+
+_filter_io()
+{
+	sed -e '/.* ops; /d'
+}
+
+# real QA test starts here
+
+echo "*** create loop mount point"
+rmdir $LOOP_MNT 2>/dev/null
+mkdir -p $LOOP_MNT || _fail "cannot create loopback mount point"
+
+_test_loop()
+{
+	size=$1
+	agsize=$2
+	fsize=$3
+
+	dparam="file,name=$LOOP_DEV,size=$size"
+	if [ -n "$agsize" ]; then
+		dparam="$dparam,agsize=$agsize"
+	fi
+
+	echo "*** mkfs loop file (size=$size)"
+	$MKFS_XFS_PROG -d $dparam \
+		| _filter_mkfs # 2>/dev/null
+
+	echo "*** mount loop filesystem"
+	mount -t xfs -o loop $LOOP_DEV $LOOP_MNT
+
+	echo "*** preallocate large file"
+	xfs_io -f -c "resvsp 0 $fsize" $LOOP_MNT/foo | _filter_io
+
+	echo "*** unmount loop filesystem"
+	umount $LOOP_MNT
+
+	echo "*** check loop filesystem"
+	 _check_xfs_filesystem $LOOP_DEV none none
+}
+
+_test_loop 50g 16m 40G
+echo "*** done"
+status=0
+exit
group +1 -0
index b94f543..b18f2e6 100644
@@ -363,3 +363,4 @@ deprecated
 247 auto quick rw
 248 auto quick rw 
 249 auto quick rw 
+250 auto quick rw prealloc metadata
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Reviewed-by on v1 2/4
#6 lore raw

Re: [PATCH 2/4] xfstests: add test to reproduce bmap btree corruption.

In reply to Dave Chinner
Quoted 8 lines · Dave Chinner

From: Dave Chinner <[email protected]>

The problem was reprted here:

https://bugzilla.redhat.com/show_bug.cgi?id=626244

With the simple test case:

Quoted 8 lines
# mkfs.xfs -f -d agsize=16m,size=50g <dev>
# mount <dev> /mnt
# xfs_io -f -c 'resvsp 0 40G' /mnt/foo

Triggering the problem. Turn this into a new xfsqa test so that we exercise the problem code and prevent future regressions.

This looks good, HOWEVER, since the test crashes the target system please remove it from (at least) the "auto" group before you commit.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#7 lore raw

Re: [PATCH 2/4] xfstests: add test to reproduce bmap btree corruption.

Quoted 13 lines · Alex Elder
Quoted 12 lines · Dave Chinner

From: Dave Chinner <[email protected]>

The problem was reprted here:

https://bugzilla.redhat.com/show_bug.cgi?id=626244

With the simple test case:

# mkfs.xfs -f -d agsize=16m,size=50g <dev>
# mount <dev> /mnt
# xfs_io -f -c 'resvsp 0 40G' /mnt/foo
Quoted 8 lines
Quoted 4 lines

Triggering the problem. Turn this into a new xfsqa test so that we exercise the problem code and prevent future regressions.

This looks good, HOWEVER, since the test crashes the target system please remove it from (at least) the "auto" group before you commit.

If causes an assert failure on a DEBUG kernel, a shutdown on a non-debug kernel. Either way, I'd prefer to leave it enabled as a good reminder that we need to fix the damn bug (*) sooner rather than later...

(*) This patch prevents the assert failure, but, like Lachlan, I'm not sure it is the right fix.

http://oss.sgi.com/archives/xfs/2010-09/msg00337.html

I'm working on it at the moment, along with all the other allocation things...

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#8 lore raw

PATCH 3/4 xfstests: make 016 work with delaylog

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Test 016 fails with delaylog because it measures log traffic to disk and delaylog writes almost nothing to the log for the given test. TO make it work, add sync calls to the work loop to cause the log to be flushed reliably for both delaylog and nodelaylog and hence contain the same nuumber of log records.

As a result, the log space consumed by the test is not changed by the delaylog option and the test passes. The test is not significantly slowed down by the addition of the sync calls (takes 15s to run on a single SATA drive).

---
 016 |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
016 +2 -0
index 980eb0b..32d8b19 100755
@@ -101,7 +101,9 @@ _log_traffic()
     while [ $count -ge 0 ]
     do
         touch $out
+	sync
         rm $out
+	sync
         let "count = count - 1"
     done
 
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Reviewed-by on v1 3/4
#9 lore raw

Re: [PATCH 3/4] xfstests: make 016 work with delaylog

016 _log_traffic() · line 108

Quoted 9 lines
016 +2 -0
@@ -101,7 +101,9 @@ _log_traffic()
     while [ $count -ge 0 ]
     do
         touch $out
+	sync
         rm $out
+	sync
         let "count = count - 1"
     done

Looks good, although it might be good to make the indentation fit the surrounding code.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by on v1 3/4
#10 lore raw

Re: [PATCH 3/4] xfstests: make 016 work with delaylog

In reply to Dave Chinner
Quoted 8 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Test 016 fails with delaylog because it measures log traffic to disk and delaylog writes almost nothing to the log for the given test. TO make it work, add sync calls to the work loop to cause the log to be flushed reliably for both delaylog and nodelaylog and hence contain the same nuumber of log records.

Quoted 7 lines

As a result, the log space consumed by the test is not changed by the delaylog option and the test passes. The test is not significantly slowed down by the addition of the sync calls (takes 15s to run on a single SATA drive).

---

Looks good.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#11 lore raw

PATCH 4/4 xfstests: Speed up test 042

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

test 042 generates a worst-case fragmented filesystem and uses it to test xfs_fsr. It uses small 4k files to generate the hole-space-hole pattern that fragments free space badly. It is much faster to generate the same pattern by creating a single large file and punching holes in it. Also, instead of writing large files to create unfragmented space, just use preallocation so we don't have to write the data to disk.

These changes reduce the runtime of the test on a single SATA drive from 106s to 27s.

---
 042     |   38 ++++++++++++++++++++++++++++++--------
 042.out |    6 +++---
 2 files changed, 33 insertions(+), 11 deletions(-)
042 +30 -8
index 7ac5492..c583746 100755
@@ -100,16 +100,19 @@ echo "done"
 echo -n "Reserve 16 1Mb unfragmented regions... "
 for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
 do
-    _do "dd if=/dev/zero of=$SCRATCH_MNT/hole$i bs=4096 count=256"
-    _do "dd if=/dev/zero of=$SCRATCH_MNT/space$i bs=4096 count=1"
-    _do "xfs_bmap -v $SCRATCH_MNT/hole$i"
+	_do "$XFS_IO_PROG -f -c \"resvsp 0 1m\" $SCRATCH_MNT/hole$i"
+	_do "$XFS_IO_PROG -f -c \"resvsp 0 4k\" $SCRATCH_MNT/space$i"
+	_do "$XFS_IO_PROG -f -c \"resvsp 0 1m\" $SCRATCH_MNT/hole$i"
+	_do "xfs_bmap -vp $SCRATCH_MNT/hole$i"
 done
 echo "done" 
 
 # set up filesystem
-echo -n "Fill filesystem with 4k files, generate manifest... "
-fill_options="--verbose --seed=0 --filesize=4096 --stddev=0 --sync=1000000"
-_do "src/fill2fs $fill_options --dir=$SCRATCH_MNT/fill --list=- > $tmp.manifest"
+echo -n "Fill filesystem with fill file... "
+for i in `seq 0 1 31`; do
+	_do "$XFS_IO_PROG -fs -c \"pwrite -S$i ${i}m 1m\" $SCRATCH_MNT/fill"
+done
+_do "xfs_bmap -vp $SCRATCH_MNT/fill"
 echo "done"
 # flush the filesystem - make sure there is no space "lost" to pre-allocation
 _do "umount $SCRATCH_MNT"
@@ -119,7 +122,18 @@ _do "dd if=/dev/zero of=$SCRATCH_MNT/pad bs=4096"
 echo "done"
 
 # create fragmented file
-_do "Delete every second file" "_cull_files"
+#_do "Delete every second file" "_cull_files"
+echo -n "Punch every second 4k block... "
+for i in `seq 0 8 32768`; do
+	# This generates excessive output that significantly slows down the
+	# test. It's not necessary for debug, so just bin it.
+	$XFS_IO_PROG -f -c "unresvsp ${i}k 4k" $SCRATCH_MNT/fill \
+								> /dev/null 2>&1
+done
+_do "xfs_bmap -vp $SCRATCH_MNT/fill"
+_do "sum $SCRATCH_MNT/fill >$tmp.fillsum1"
+echo "done"
+
 echo -n "Create one very large file... "
 _do "src/fill2 -d nbytes=16000000,file=$SCRATCH_MNT/fragmented"
 echo "done"
@@ -130,7 +144,15 @@ _do "Remove other files" "rm -rf $SCRATCH_MNT/{pad,hole*}"
 # defragment
 _do "Run xfs_fsr on filesystem" "$XFS_FSR_PROG -v $SCRATCH_MNT/fragmented"
 _do "xfs_bmap -v $SCRATCH_MNT/fragmented"
-_do "Check 4k files" "src/fill2fs_check $tmp.manifest"
+
+echo -n "Check fill file... "
+_do "sum $SCRATCH_MNT/fill >$tmp.fillsum2"
+if ! _do "diff $tmp.fillsum1 $tmp.fillsum2"; then
+    echo "fail"
+    echo "Fill file is corrupt/missing after fsr. Test failed see $seq.full"
+    status=1; exit
+fi
+echo "done"
 
 # check
 echo -n "Check large file... "
042.out +3 -3
index 7f2e837..f5b37b3 100644
@@ -1,13 +1,13 @@
 QA output created by 042
 Make a 48 megabyte filesystem on SCRATCH_DEV and mount... done
 Reserve 16 1Mb unfragmented regions... done
-Fill filesystem with 4k files, generate manifest... done
+Fill filesystem with fill file... done
 Use up any further available space using dd... done
-Delete every second file... done
+Punch every second 4k block... done
 Create one very large file... done
 Remove other files... done
 Run xfs_fsr on filesystem... done
-Check 4k files... done
+Check fill file... done
 Check large file... done
 Checking filesystem... done
 xfs_fsr tests passed.
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Reviewed-by on v1 4/4
#12 lore raw

Re: [PATCH 4/4] xfstests: Speed up test 042

Quoted 10 lines · Dave Chinner

From: Dave Chinner <[email protected]>

test 042 generates a worst-case fragmented filesystem and uses it to test xfs_fsr. It uses small 4k files to generate the hole-space-hole pattern that fragments free space badly. It is much faster to generate the same pattern by creating a single large file and punching holes in it. Also, instead of writing large files to create unfragmented space, just use preallocation so we don't have to write the data to disk.

Quoted 2 lines

These changes reduce the runtime of the test on a single SATA drive from 106s to 27s.

Looks good,

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by on v1 4/4
#13 lore raw

Re: [PATCH 4/4] xfstests: Speed up test 042

In reply to Dave Chinner
Quoted 10 lines · Dave Chinner

From: Dave Chinner <[email protected]>

test 042 generates a worst-case fragmented filesystem and uses it to test xfs_fsr. It uses small 4k files to generate the hole-space-hole pattern that fragments free space badly. It is much faster to generate the same pattern by creating a single large file and punching holes in it. Also, instead of writing large files to create unfragmented space, just use preallocation so we don't have to write the data to disk.

Quoted 4 lines

These changes reduce the runtime of the test on a single SATA drive from 106s to 27s.

Perhaps your test system is configured different from mine, but I don't see the speedup you see. In fact, it might have slowed it down. I really haven't experimented much though and will report back if I find anything more constructive to say.

Other than that I see no problem with the change. And the difference either way isn't a big deal to me, so:

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#14 lore raw

Re: [PATCH 4/4] xfstests: Speed up test 042

Quoted 14 lines · Alex Elder
Quoted 13 lines · Dave Chinner

From: Dave Chinner <[email protected]>

test 042 generates a worst-case fragmented filesystem and uses it to test xfs_fsr. It uses small 4k files to generate the hole-space-hole pattern that fragments free space badly. It is much faster to generate the same pattern by creating a single large file and punching holes in it. Also, instead of writing large files to create unfragmented space, just use preallocation so we don't have to write the data to disk.

These changes reduce the runtime of the test on a single SATA drive from 106s to 27s.

Quoted 7 lines
Quoted 1 line

Perhaps your test system is configured different from mine, but I don't see the speedup you see. In fact, it might have slowed it down. I really haven't experimented much though and will report back if I find anything more constructive to say.

That may be because the resvsp call at the moment is a sync transaction. I've got a patch to make that async which should solve the problem for you. Mind you, the above numbers come from a Vm that wasn't running that patch, so it probably depends mostly on how efficient your storage is a processing barriers to the difference you see right now...

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs