xfs: tests extent size hint size overflows

PATCH In review Dave Chinner 2 participants 5 messages May 27, 2015 – May 29, 2015

lore

Share & tools

lore

Dave Chinner

author

2015-05-27 02:43 UTC #0 lore raw

PATCH 1/1 xfs: tests extent size hint size overflows

From: Dave Chinner <[email protected]>

in certain cases, the extent size hints can cause maximum extent size overflows resulting in extent tree corruptions. This test exercises the original reproducer, and another corner case demonstrated to expose problems on 1k block size filesystems.

Signed-off-by Dave Chinner <[email protected]>

---
 tests/xfs/074     | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/074.out |  2 ++
 tests/xfs/group   |  1 +
 3 files changed, 96 insertions(+)
 create mode 100755 tests/xfs/074
 create mode 100644 tests/xfs/074.out

tests/xfs/074 new file +93 -0

new file mode 100755
index 0000000..810a1a0
@@ -0,0 +1,93 @@
+#! /bin/bash
+# FS QA Test 074-extsz-hints-vs-maxextlen
+#
+# Check some extent size hint boundary conditions that can result in
+# MAXEXTLEN overflows.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2015 Red Hat, Inc. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+_cleanup()
+{
+	cd /
+	_destroy_loop_device $LOOP_DEV
+	rm -f $tmp.* $LOOP_FILE
+}
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs xfs
+_supported_os Linux
+
+rm -f $seqres.full
+
+_require_test
+_require_xfs_io_command "falloc"
+
+# we use loop devices for this so that we can create large files for prealloc
+# without having to care about the underlying device size.
+_require_loop
+
+LOOP_FILE=$TESTDIR/$seq.img
+LOOP_MNT=$TESTDIR/$seq.mnt
+mkdir -p $LOOP_MNT
+$XFS_IO_PROG -ft -c "truncate 1t" $LOOP_FILE 2>&1 > $seqres.full
+LOOP_DEV=`_create_loop_device $LOOP_FILE`
+
+_mkfs_dev -d size=40051712b,agcount=4 -l size=32m $LOOP_DEV
+_mount $LOOP_DEV $LOOP_MNT
+
+# Corrupt the BMBT by creating extents larger than MAXEXTLEN
+$XFS_IO_PROG -ft \
+	-c "extsize 16m" \
+	-c "falloc 0 30g" \
+	$LOOP_MNT/foo 2>&1 > $seqres.full
+
+umount $LOOP_MNT
+_check_xfs_filesystem $LOOP_DEV none none
+
+_mkfs_dev -f -l size=32m $LOOP_DEV
+_mount $LOOP_DEV $LOOP_MNT
+
+# check we trim both ends of the extent approproiately; this will fail
+# on 1k block size filesystems without the correct fixes in place.
+$XFS_IO_PROG -ft \
+	-c "extsize 1g" \
+	-c "falloc 1023m 2g" \
+	$LOOP_MNT/foo 2>&1 > $seqres.full
+
+umount $LOOP_MNT
+_check_xfs_filesystem $LOOP_DEV none none
+
+# success, all done
+echo "Silence is golden"
+status=0
+exit

tests/xfs/074.out new file +2 -0

new file mode 100644
index 0000000..81a0d24
@@ -0,0 +1,2 @@
+QA output created by 074-extsz-hints-vs-maxextlen
+Silence is golden

tests/xfs/group +1 -0

index 848a1bd..c211fc1 100644
@@ -71,6 +71,7 @@
 071 rw auto
 072 rw auto prealloc quick
 073 copy auto
+074 quick auto prealloc rw
 078 growfs auto quick
 080 rw ioctl
 081 deprecated # log logprint quota

Signature · 1 line

2.0.0

Eryu Guan

2015-05-27 03:34 UTC #1 lore raw

Quoted 72 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Signed-off-by Dave Chinner <[email protected]>

---
 tests/xfs/074     | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/074.out |  2 ++
 tests/xfs/group   |  1 +
 3 files changed, 96 insertions(+)
 create mode 100755 tests/xfs/074
 create mode 100644 tests/xfs/074.out

tests/xfs/074 new file +50 -0

new file mode 100755
index 0000000..810a1a0
@@ -0,0 +1,93 @@
+#! /bin/bash
+# FS QA Test 074-extsz-hints-vs-maxextlen
+#
+# Check some extent size hint boundary conditions that can result in
+# MAXEXTLEN overflows.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2015 Red Hat, Inc. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+_cleanup()
+{
+	cd /
+	_destroy_loop_device $LOOP_DEV
+	rm -f $tmp.* $LOOP_FILE
+}
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs xfs
+_supported_os Linux
+
+rm -f $seqres.full

tests/xfs/074 line 60

Quoted 10 lines

tests/xfs/074 new file +10 -0

+
+_require_test
+_require_xfs_io_command "falloc"
+
+# we use loop devices for this so that we can create large files for prealloc
+# without having to care about the underlying device size.
+_require_loop
+
+LOOP_FILE=$TESTDIR/$seq.img
+LOOP_MNT=$TESTDIR/$seq.mnt

This should be $TEST_DIR, TESTDIR is empty and ends up in /074.mnt

tests/xfs/074 line 62

Quoted 2 lines

tests/xfs/074 new file +2 -0

+mkdir -p $LOOP_MNT
+$XFS_IO_PROG -ft -c "truncate 1t" $LOOP_FILE 2>&1 > $seqres.full

This will leave stderr to stdout, then only stdout goes to $seqres.full I see "+ fallocate: No space left on device" when testing.

"command >$seqres.full 2>&1" will do the work.

tests/xfs/074 line 72

Quoted 10 lines

tests/xfs/074 new file +10 -0

+LOOP_DEV=`_create_loop_device $LOOP_FILE`
+
+_mkfs_dev -d size=40051712b,agcount=4 -l size=32m $LOOP_DEV
+_mount $LOOP_DEV $LOOP_MNT
+
+# Corrupt the BMBT by creating extents larger than MAXEXTLEN
+$XFS_IO_PROG -ft \
+	-c "extsize 16m" \
+	-c "falloc 0 30g" \
+	$LOOP_MNT/foo 2>&1 > $seqres.full

Here too, and should append to $seqres.full with ">>"

tests/xfs/074 line 75

Quoted 3 lines

tests/xfs/074 new file +3 -0

+
+umount $LOOP_MNT
+_check_xfs_filesystem $LOOP_DEV none none

I think '_check_scratch_fs $LOOP_DEV' is eaiser, it accepts optional device argument to check and deal with SCRATCH_LOG and SCRATCH_RT internally.

tests/xfs/074 line 85

Quoted 10 lines

tests/xfs/074 new file +10 -0

+
+_mkfs_dev -f -l size=32m $LOOP_DEV
+_mount $LOOP_DEV $LOOP_MNT
+
+# check we trim both ends of the extent approproiately; this will fail
+# on 1k block size filesystems without the correct fixes in place.
+$XFS_IO_PROG -ft \
+	-c "extsize 1g" \
+	-c "falloc 1023m 2g" \
+	$LOOP_MNT/foo 2>&1 > $seqres.full

And here.

Quoted 14 lines

tests/xfs/074 new file +8 -0

+
+umount $LOOP_MNT
+_check_xfs_filesystem $LOOP_DEV none none
+
+# success, all done
+echo "Silence is golden"
+status=0
+exit

tests/xfs/074.out new file +0 -0

new file mode 100644
index 0000000..81a0d24

tests/xfs/074.out line 1

Quoted 1 line

tests/xfs/074.out new file +1 -0

@@ -0,0 +1,2 @@
+QA output created by 074-extsz-hints-vs-maxextlen

This should be "QA output created by 074"

Thanks,
Eryu

Quoted 20 lines

tests/xfs/074.out new file +1 -0

+Silence is golden

tests/xfs/group +1 -0

index 848a1bd..c211fc1 100644
@@ -71,6 +71,7 @@
 071 rw auto
 072 rw auto prealloc quick
 073 copy auto
+074 quick auto prealloc rw
 078 growfs auto quick
 080 rw ioctl
 081 deprecated # log logprint quota

Signature · 6 lines

2.0.0

--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Dave Chinner

author

2015-05-29 01:21 UTC #2 lore raw

Quoted 9 lines · Eryu Guan

Quoted 8 lines · Dave Chinner
From: Dave Chinner <[email protected]>
in certain cases, the extent size hints can cause maximum extent size overflows resulting in extent tree corruptions. This test exercises the original reproducer, and another corner case demonstrated to expose problems on 1k block size filesystems.
Signed-off-by Dave Chinner <[email protected]>

....

Quoted 5 lines

Quoted 5 lines
tests/xfs/074 new file +5 -0
+rm -f $seqres.full
+
+_require_test
+_require_xfs_io_command "falloc"
+

Quoted 8 lines

tests/xfs/074 line 60
Quoted 6 lines
tests/xfs/074 new file +6 -0
+# we use loop devices for this so that we can create large files for prealloc
+# without having to care about the underlying device size.
+_require_loop
+
+LOOP_FILE=$TESTDIR/$seq.img
+LOOP_MNT=$TESTDIR/$seq.mnt
This should be $TEST_DIR, TESTDIR is empty and ends up in /074.mnt

Oops, will fix.

Quoted 5 lines

Quoted 2 lines
+mkdir -p $LOOP_MNT
+$XFS_IO_PROG -ft -c "truncate 1t" $LOOP_FILE 2>&1 > $seqres.full
This will leave stderr to stdout, then only stdout goes to $seqres.full I see "+ fallocate: No space left on device" when testing.

Why would the truncate command give an "fallocate" failure?

And, besides, allowing stderr to be exposed is done on purpose - an error in the execution of the truncate command will trigger a test failure because stderr is captured by the test harness...

Quoted 14 lines

"command >$seqres.full 2>&1" will do the work.
tests/xfs/074 line 72
Quoted 10 lines
tests/xfs/074 new file +10 -0
+LOOP_DEV=`_create_loop_device $LOOP_FILE`
+
+_mkfs_dev -d size=40051712b,agcount=4 -l size=32m $LOOP_DEV
+_mount $LOOP_DEV $LOOP_MNT
+
+# Corrupt the BMBT by creating extents larger than MAXEXTLEN
+$XFS_IO_PROG -ft \
+	-c "extsize 16m" \
+	-c "falloc 0 30g" \
+	$LOOP_MNT/foo 2>&1 > $seqres.full
Here too, and should append to $seqres.full with ">>"

Will fix.

Quoted 8 lines

tests/xfs/074 line 75
Quoted 4 lines
tests/xfs/074 new file +3 -0
+
+umount $LOOP_MNT
+_check_xfs_filesystem $LOOP_DEV none none
I think '_check_scratch_fs $LOOP_DEV' is eaiser, it accepts optional device argument to check and deal with SCRATCH_LOG and SCRATCH_RT internally.

But it's not the scratch device I'm using. i.e. it doesn't use SCRATCH_LOG or SCRATCH_RT because I didn't use _scratch_mkfs to create the filesystem, nor am I using _scratch_mount, so it would fail to mount if mkfs used external devices....

Quoted 6 lines

tests/xfs/074.out line 1
Quoted 4 lines
tests/xfs/074.out new file +1 -0
@@ -0,0 +1,2 @@
+QA output created by 074-extsz-hints-vs-maxextlen
This should be "QA output created by 074"

Hmm - the patch is stale then. That's strange, because I pulled it from the git tree to my test machines, and they didn't fail. I'll fix it.

Cheers,

Dave.

Signature · 2 lines

Dave Chinner
[email protected]

Eryu Guan

2015-05-29 02:51 UTC #3 lore raw

Quoted 27 lines · Dave Chinner

Quoted 9 lines · Eryu Guan
Quoted 8 lines · Dave Chinner
From: Dave Chinner <[email protected]>
in certain cases, the extent size hints can cause maximum extent size overflows resulting in extent tree corruptions. This test exercises the original reproducer, and another corner case demonstrated to expose problems on 1k block size filesystems.
Signed-off-by Dave Chinner <[email protected]>
....
Quoted 5 lines
Quoted 5 lines
tests/xfs/074 new file +5 -0
+rm -f $seqres.full
+
+_require_test
+_require_xfs_io_command "falloc"
+
Quoted 8 lines
tests/xfs/074 line 60
Quoted 6 lines
tests/xfs/074 new file +6 -0
+# we use loop devices for this so that we can create large files for prealloc
+# without having to care about the underlying device size.
+_require_loop
+
+LOOP_FILE=$TESTDIR/$seq.img
+LOOP_MNT=$TESTDIR/$seq.mnt
This should be $TEST_DIR, TESTDIR is empty and ends up in /074.mnt
Oops, will fix.

Quoted 7 lines

Quoted 5 lines
Quoted 2 lines
+mkdir -p $LOOP_MNT
+$XFS_IO_PROG -ft -c "truncate 1t" $LOOP_FILE 2>&1 > $seqres.full
This will leave stderr to stdout, then only stdout goes to $seqres.full I see "+ fallocate: No space left on device" when testing.
Why would the truncate command give an "fallocate" failure?

Oh, I mean at the "falloc" time, I should refer to the next xfs_io command below..

Quoted 4 lines

And, besides, allowing stderr to be exposed is done on purpose - an error in the execution of the truncate command will trigger a test failure because stderr is captured by the test harness...

Then I think a comment should be added, because "2>&1 >$seqres.full" is not a usual usage, usually it indicates a programming mistake. Or just ">$seqres.full" and leave stderr untouched? which indicates clearly we leave stderr unfiltered on purpose.

Quoted 13 lines

"command >$seqres.full 2>&1" will do the work.

tests/xfs/074 line 72

Quoted 10 lines

tests/xfs/074 new file +10 -0

+LOOP_DEV=`_create_loop_device $LOOP_FILE`
+
+_mkfs_dev -d size=40051712b,agcount=4 -l size=32m $LOOP_DEV
+_mount $LOOP_DEV $LOOP_MNT
+
+# Corrupt the BMBT by creating extents larger than MAXEXTLEN
+$XFS_IO_PROG -ft \
+	-c "extsize 16m" \
+	-c "falloc 0 30g" \
+	$LOOP_MNT/foo 2>&1 > $seqres.full

The "+ fallocate: No space left on device" error comes from this command. Is this on purpose too? So on patched kernel the error shouldn't be there?

Thanks,
Eryu

Dave Chinner

author

2015-05-29 03:24 UTC #4 lore raw

Quoted 40 lines · Eryu Guan

Quoted 27 lines · Dave Chinner
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Dave Chinner
From: Dave Chinner <[email protected]>
in certain cases, the extent size hints can cause maximum extent size overflows resulting in extent tree corruptions. This test exercises the original reproducer, and another corner case demonstrated to expose problems on 1k block size filesystems.
Signed-off-by Dave Chinner <[email protected]>
....
Quoted 5 lines
Quoted 5 lines
tests/xfs/074 new file +5 -0
+rm -f $seqres.full
+
+_require_test
+_require_xfs_io_command "falloc"
+
Quoted 8 lines
tests/xfs/074 line 60
Quoted 6 lines
tests/xfs/074 new file +6 -0
+# we use loop devices for this so that we can create large files for prealloc
+# without having to care about the underlying device size.
+_require_loop
+
+LOOP_FILE=$TESTDIR/$seq.img
+LOOP_MNT=$TESTDIR/$seq.mnt
This should be $TEST_DIR, TESTDIR is empty and ends up in /074.mnt
Oops, will fix.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
+mkdir -p $LOOP_MNT
+$XFS_IO_PROG -ft -c "truncate 1t" $LOOP_FILE 2>&1 > $seqres.full
This will leave stderr to stdout, then only stdout goes to $seqres.full I see "+ fallocate: No space left on device" when testing.
Why would the truncate command give an "fallocate" failure?
Oh, I mean at the "falloc" time, I should refer to the next xfs_io command below..
Quoted 1 line

Quoted 8 lines

Quoted 3 lines
And, besides, allowing stderr to be exposed is done on purpose - an error in the execution of the truncate command will trigger a test failure because stderr is captured by the test harness...
Then I think a comment should be added, because "2>&1 >$seqres.full" is not a usual usage, usually it indicates a programming mistake. Or just ">$seqres.full" and leave stderr untouched? which indicates clearly we leave stderr unfiltered on purpose.

If we had to comment on every time we are capturing stderr as the failure indication for the test, then we'd be adding comments everywhere. This is how xfstests is supposed to work - we capture *error messages* from tools because that's what is used to inform the user that there was an error. Those error messages cause golden image match failures, and that causes the test to fail.

See the comments I've made previously about why "run_check" is considered harmful - it encourages tools to be silent on error and only set exit values, because that is all it checks. Users *never* check return values - there need to be error messages when an error occurs...

Quoted 16 lines

Quoted 12 lines
Quoted 12 lines
"command >$seqres.full 2>&1" will do the work.
tests/xfs/074 line 72
Quoted 10 lines
tests/xfs/074 new file +10 -0
+LOOP_DEV=`_create_loop_device $LOOP_FILE`
+
+_mkfs_dev -d size=40051712b,agcount=4 -l size=32m $LOOP_DEV
+_mount $LOOP_DEV $LOOP_MNT
+
+# Corrupt the BMBT by creating extents larger than MAXEXTLEN
+$XFS_IO_PROG -ft \
+	-c "extsize 16m" \
+	-c "falloc 0 30g" \
+	$LOOP_MNT/foo 2>&1 > $seqres.full
The "+ fallocate: No space left on device" error comes from this command. Is this on purpose too? So on patched kernel the error shouldn't be there?

That's right - the falloc completes without error when the bug is fixed, so stderr is silent and the test passes.

Cheers,

Dave.

Signature · 2 lines

Dave Chinner
[email protected]