Skip to contentSkip to conversation

fstests: generic test for fsync after fallocate

PATCH Superseded [email protected] 3 participants 7 messages Apr 6, 2018 – Apr 9, 2018

lore
Share & tools
lore

[email protected]

author
#0 lore raw

PATCH 1/1 fstests: generic test for fsync after fallocate

From: Filipe Manana <[email protected]>

Test that fsync operations preserve extents allocated with fallocate(2) that are placed beyond a file's size.

This test is motivated by a bug found in btrfs where unwritten extents beyond the inode's i_size were not preserved after a fsync and power failure. The btrfs bug is fixed by the following patch for the linux kernel:

"Btrfs: fix loss of prealloc extents past i_size after fsync log replay"

Signed-off-by Filipe Manana <[email protected]>
---
 tests/generic/482     | 118 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/482.out |  10 +++++
 tests/generic/group   |   1 +
 3 files changed, 129 insertions(+)
 create mode 100755 tests/generic/482
 create mode 100644 tests/generic/482.out
tests/generic/482 new file +118 -0
new file mode 100755
index 00000000..43bbc913
@@ -0,0 +1,118 @@
+#! /bin/bash
+# FSQA Test No. 482
+#
+# Test that fsync operations preserve extents allocated with fallocate(2) that
+# are placed beyond a file's size.
+#
+#-----------------------------------------------------------------------
+#
+# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
+# Author: Filipe Manana <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	_cleanup_flakey
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmflakey
+. ./common/punch
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_scratch
+_require_dm_target flakey
+_require_xfs_io_command "falloc" "-k"
+_require_xfs_io_command "fiemap"
+
+rm -f $seqres.full
+
+_scratch_mkfs >>$seqres.full 2>&1
+_require_metadata_journaling $SCRATCH_DEV
+_init_flakey
+_mount_flakey
+
+# Create our test files.
+$XFS_IO_PROG -f -c "pwrite -S 0xea 0 256K" $SCRATCH_MNT/foo >/dev/null
+
+# Create a file with many extents. We later want to shrink truncate it and
+# add a prealloc extent beyond its new size.
+for ((i = 1; i <= 500; i++)); do
+	offset=$(((i - 1) * 4 * 1024))
+	$XFS_IO_PROG -f -s -c "pwrite -S 0xcf $offset 4K" \
+		$SCRATCH_MNT/bar >/dev/null
+done
+
+# A file which already has a prealloc extent beyond its size.
+# The fsync done on it is motivated by differences in the btrfs implementation
+# of fsync (first fsync has different logic from subsequent fsyncs).
+$XFS_IO_PROG -f -c "pwrite -S 0xf1 0 256K" \
+	     -c "falloc -k 256K 768K" \
+	     -c "fsync" \
+	     $SCRATCH_MNT/baz >/dev/null
+
+# Make sure everything done so far is durably persisted.
+sync
+
+# Allocate an extent beyond the size of the first test file and fsync it.
+$XFS_IO_PROG -c "falloc -k 256K 1M"\
+	     -c "fsync" \
+	     $SCRATCH_MNT/foo
+
+# Do a shrinking truncate of our test file, add a prealloc extent to it after
+# its new size and fsync it.
+$XFS_IO_PROG -c "truncate 256K" \
+	     -c "falloc -k 256K 1M"\
+	     -c "fsync" \
+	     $SCRATCH_MNT/bar
+
+# Allocate another extent beyond the size of file baz.
+$XFS_IO_PROG -c "falloc -k 1M 2M"\
+	     -c "fsync" \
+	     $SCRATCH_MNT/baz
+
+# Simulate a power failure and mount the filesystem to check that the extents
+# previously allocated were not lost.
+_flakey_drop_and_remount
+
+echo "File foo fiemap:"
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foo | _filter_fiemap
+
+echo "File bar fiemap:"
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/bar | _filter_fiemap
+
+echo "File baz fiemap:"
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/baz | _filter_fiemap
+
+_unmount_flakey
+_cleanup_flakey
+
+status=0
+exit
tests/generic/482.out new file +10 -0
new file mode 100644
index 00000000..7e3ed139
@@ -0,0 +1,10 @@
+QA output created by 482
+File foo fiemap:
+0: [0..511]: data
+1: [512..2559]: unwritten
+File bar fiemap:
+0: [0..511]: data
+1: [512..2559]: unwritten
+File baz fiemap:
+0: [0..511]: data
+1: [512..6143]: unwritten
tests/generic/group +1 -0
index e8676062..58ac1f00 100644
@@ -484,3 +484,4 @@
 479 auto quick metadata
 480 auto quick metadata
 481 auto quick log metadata
+482 auto quick log metadata
Signature · 1 line
2.11.0
#1 lore raw
Quoted 5 lines · On Thu, Apr 05, 2018 at 10:56:14PM +0100, [email protected] wrote:

From: Filipe Manana <[email protected]>

Test that fsync operations preserve extents allocated with fallocate(2) that are placed beyond a file's size.

Quoted 8 lines

This test is motivated by a bug found in btrfs where unwritten extents beyond the inode's i_size were not preserved after a fsync and power failure. The btrfs bug is fixed by the following patch for the linux kernel:

"Btrfs: fix loss of prealloc extents past i_size after fsync log replay"

Signed-off-by Filipe Manana <[email protected]>

Hmm, xfs fails this test, while ext4 passes.

# diff -u tests/generic/483.out /root/workspace/xfstests/results//xfs_4k_crc/generic/483.out.bad
--- tests/generic/483.out       2018-04-07 23:35:00.555555511 +0800
+++ /root/workspace/xfstests/results//xfs_4k_crc/generic/483.out.bad    2018-04-07 23:39:48.780659707 +0800
@@ -6,5 +6,5 @@
 0: [0..511]: data
 1: [512..2559]: unwritten
 File baz fiemap:
-0: [0..511]: data
-1: [512..6143]: unwritten
+0: [0..895]: data
+1: [896..6143]: unwritten

I'm not sure what the problem is yet, but IMHO controlling on-disk layout of a file from userspace is hard and should be avoided if possible.

Why not dumping md5sum to .out file like other dmflakey tests? I've checked the md5sum of all the three test files, and they're the same on xfs as on ext4, so the files are not corrupted on xfs.

Thanks,
Eryu

Quoted 164 lines
---
 tests/generic/482     | 118 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/482.out |  10 +++++
 tests/generic/group   |   1 +
 3 files changed, 129 insertions(+)
 create mode 100755 tests/generic/482
 create mode 100644 tests/generic/482.out
tests/generic/482 new file +118 -0
new file mode 100755
index 00000000..43bbc913
@@ -0,0 +1,118 @@
+#! /bin/bash
+# FSQA Test No. 482
+#
+# Test that fsync operations preserve extents allocated with fallocate(2) that
+# are placed beyond a file's size.
+#
+#-----------------------------------------------------------------------
+#
+# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
+# Author: Filipe Manana <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	_cleanup_flakey
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmflakey
+. ./common/punch
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_scratch
+_require_dm_target flakey
+_require_xfs_io_command "falloc" "-k"
+_require_xfs_io_command "fiemap"
+
+rm -f $seqres.full
+
+_scratch_mkfs >>$seqres.full 2>&1
+_require_metadata_journaling $SCRATCH_DEV
+_init_flakey
+_mount_flakey
+
+# Create our test files.
+$XFS_IO_PROG -f -c "pwrite -S 0xea 0 256K" $SCRATCH_MNT/foo >/dev/null
+
+# Create a file with many extents. We later want to shrink truncate it and
+# add a prealloc extent beyond its new size.
+for ((i = 1; i <= 500; i++)); do
+	offset=$(((i - 1) * 4 * 1024))
+	$XFS_IO_PROG -f -s -c "pwrite -S 0xcf $offset 4K" \
+		$SCRATCH_MNT/bar >/dev/null
+done
+
+# A file which already has a prealloc extent beyond its size.
+# The fsync done on it is motivated by differences in the btrfs implementation
+# of fsync (first fsync has different logic from subsequent fsyncs).
+$XFS_IO_PROG -f -c "pwrite -S 0xf1 0 256K" \
+	     -c "falloc -k 256K 768K" \
+	     -c "fsync" \
+	     $SCRATCH_MNT/baz >/dev/null
+
+# Make sure everything done so far is durably persisted.
+sync
+
+# Allocate an extent beyond the size of the first test file and fsync it.
+$XFS_IO_PROG -c "falloc -k 256K 1M"\
+	     -c "fsync" \
+	     $SCRATCH_MNT/foo
+
+# Do a shrinking truncate of our test file, add a prealloc extent to it after
+# its new size and fsync it.
+$XFS_IO_PROG -c "truncate 256K" \
+	     -c "falloc -k 256K 1M"\
+	     -c "fsync" \
+	     $SCRATCH_MNT/bar
+
+# Allocate another extent beyond the size of file baz.
+$XFS_IO_PROG -c "falloc -k 1M 2M"\
+	     -c "fsync" \
+	     $SCRATCH_MNT/baz
+
+# Simulate a power failure and mount the filesystem to check that the extents
+# previously allocated were not lost.
+_flakey_drop_and_remount
+
+echo "File foo fiemap:"
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foo | _filter_fiemap
+
+echo "File bar fiemap:"
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/bar | _filter_fiemap
+
+echo "File baz fiemap:"
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/baz | _filter_fiemap
+
+_unmount_flakey
+_cleanup_flakey
+
+status=0
+exit
tests/generic/482.out new file +10 -0
new file mode 100644
index 00000000..7e3ed139
@@ -0,0 +1,10 @@
+QA output created by 482
+File foo fiemap:
+0: [0..511]: data
+1: [512..2559]: unwritten
+File bar fiemap:
+0: [0..511]: data
+1: [512..2559]: unwritten
+File baz fiemap:
+0: [0..511]: data
+1: [512..6143]: unwritten
tests/generic/group +1 -0
index e8676062..58ac1f00 100644
@@ -484,3 +484,4 @@
 479 auto quick metadata
 480 auto quick metadata
 481 auto quick log metadata
+482 auto quick log metadata
Signature · 6 lines
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
author
#2 lore raw
Quoted 29 lines · Eryu Guan
Quoted 5 lines · On Thu, Apr 05, 2018 at 10:56:14PM +0100, [email protected] wrote:

From: Filipe Manana <[email protected]>

Test that fsync operations preserve extents allocated with fallocate(2) that are placed beyond a file's size.

Quoted 8 lines

This test is motivated by a bug found in btrfs where unwritten extents beyond the inode's i_size were not preserved after a fsync and power failure. The btrfs bug is fixed by the following patch for the linux kernel:

"Btrfs: fix loss of prealloc extents past i_size after fsync log replay"

Signed-off-by Filipe Manana <[email protected]>

Hmm, xfs fails this test, while ext4 passes.

# diff -u tests/generic/483.out /root/workspace/xfstests/results//xfs_4k_crc/generic/483.out.bad
root/workspace/xfstests/results//xfs_4k_crc/generic/483.out.bad 2018-04-07 23:39:48.780659707 +0800 renamed from generic/483.out 2018-04-07 23:35:00.555555511 +0800 +2 -2
@@ -6,5 +6,5 @@
 0: [0..511]: data
 1: [512..2559]: unwritten
 File baz fiemap:
-0: [0..511]: data
-1: [512..6143]: unwritten
+0: [0..895]: data
+1: [896..6143]: unwritten
Quoted 3 lines

I'm not sure what the problem is yet, but IMHO controlling on-disk layout of a file from userspace is hard and should be avoided if possible.

Well, both ext4 and f2fs give the same result, which makes sense, if userspace has only written to the first 256K of the file, only blocks 0..511 should be reported as written.
However xfs decides to report more than that was written (448K), seems like a bug to me.
The goal of fallocate(), specifically here, is precisely to create unwritten extents beyond eof.

Quoted 4 lines

Why not dumping md5sum to .out file like other dmflakey tests? I've checked the md5sum of all the three test files, and they're the same on xfs as on ext4, so the files are not corrupted on xfs.

Nop, that won't work. Because userspace can't read extents beyond eof. That would make unpatched btrfs succeed the test (or any fs with similar bug persisting prealloc extents beyond eof).
Without fiemap, how can one check that unwritten extents beyond eof exist?

Quoted 168 lines

Thanks,
Eryu

Quoted 164 lines
---
 tests/generic/482     | 118 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/482.out |  10 +++++
 tests/generic/group   |   1 +
 3 files changed, 129 insertions(+)
 create mode 100755 tests/generic/482
 create mode 100644 tests/generic/482.out
tests/generic/482 new file +118 -0
new file mode 100755
index 00000000..43bbc913
@@ -0,0 +1,118 @@
+#! /bin/bash
+# FSQA Test No. 482
+#
+# Test that fsync operations preserve extents allocated with fallocate(2) that
+# are placed beyond a file's size.
+#
+#-----------------------------------------------------------------------
+#
+# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
+# Author: Filipe Manana <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+tmp=/tmp/$$
+status=1     # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+     _cleanup_flakey
+     cd /
+     rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmflakey
+. ./common/punch
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_scratch
+_require_dm_target flakey
+_require_xfs_io_command "falloc" "-k"
+_require_xfs_io_command "fiemap"
+
+rm -f $seqres.full
+
+_scratch_mkfs >>$seqres.full 2>&1
+_require_metadata_journaling $SCRATCH_DEV
+_init_flakey
+_mount_flakey
+
+# Create our test files.
+$XFS_IO_PROG -f -c "pwrite -S 0xea 0 256K" $SCRATCH_MNT/foo >/dev/null
+
+# Create a file with many extents. We later want to shrink truncate it and
+# add a prealloc extent beyond its new size.
+for ((i = 1; i <= 500; i++)); do
+     offset=$(((i - 1) * 4 * 1024))
+     $XFS_IO_PROG -f -s -c "pwrite -S 0xcf $offset 4K" \
+             $SCRATCH_MNT/bar >/dev/null
+done
+
+# A file which already has a prealloc extent beyond its size.
+# The fsync done on it is motivated by differences in the btrfs implementation
+# of fsync (first fsync has different logic from subsequent fsyncs).
+$XFS_IO_PROG -f -c "pwrite -S 0xf1 0 256K" \
+          -c "falloc -k 256K 768K" \
+          -c "fsync" \
+          $SCRATCH_MNT/baz >/dev/null
+
+# Make sure everything done so far is durably persisted.
+sync
+
+# Allocate an extent beyond the size of the first test file and fsync it.
+$XFS_IO_PROG -c "falloc -k 256K 1M"\
+          -c "fsync" \
+          $SCRATCH_MNT/foo
+
+# Do a shrinking truncate of our test file, add a prealloc extent to it after
+# its new size and fsync it.
+$XFS_IO_PROG -c "truncate 256K" \
+          -c "falloc -k 256K 1M"\
+          -c "fsync" \
+          $SCRATCH_MNT/bar
+
+# Allocate another extent beyond the size of file baz.
+$XFS_IO_PROG -c "falloc -k 1M 2M"\
+          -c "fsync" \
+          $SCRATCH_MNT/baz
+
+# Simulate a power failure and mount the filesystem to check that the extents
+# previously allocated were not lost.
+_flakey_drop_and_remount
+
+echo "File foo fiemap:"
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foo | _filter_fiemap
+
+echo "File bar fiemap:"
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/bar | _filter_fiemap
+
+echo "File baz fiemap:"
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/baz | _filter_fiemap
+
+_unmount_flakey
+_cleanup_flakey
+
+status=0
+exit
tests/generic/482.out new file +10 -0
new file mode 100644
index 00000000..7e3ed139
@@ -0,0 +1,10 @@
+QA output created by 482
+File foo fiemap:
+0: [0..511]: data
+1: [512..2559]: unwritten
+File bar fiemap:
+0: [0..511]: data
+1: [512..2559]: unwritten
+File baz fiemap:
+0: [0..511]: data
+1: [512..6143]: unwritten
tests/generic/group +1 -1
index e8676062..58ac1f00 100644
@@ -484,3 +484,4 @@
 479 auto quick metadata
 480 auto quick metadata
 481 auto quick log metadata
+482 auto quick log metadata
--

2.11.0

--
To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
In reply to Eryu Guan
Quoted 21 lines · Eryu Guan
Quoted 5 lines · On Thu, Apr 05, 2018 at 10:56:14PM +0100, [email protected] wrote:

From: Filipe Manana <[email protected]>

Test that fsync operations preserve extents allocated with fallocate(2) that are placed beyond a file's size.

Quoted 8 lines

This test is motivated by a bug found in btrfs where unwritten extents beyond the inode's i_size were not preserved after a fsync and power failure. The btrfs bug is fixed by the following patch for the linux kernel:

"Btrfs: fix loss of prealloc extents past i_size after fsync log replay"

Signed-off-by Filipe Manana <[email protected]>

Hmm, xfs fails this test, while ext4 passes.

# diff -u tests/generic/483.out /root/workspace/xfstests/results//xfs_4k_crc/generic/483.out.bad

root/workspace/xfstests/results//xfs_4k_crc/generic/483.out.bad 2018-04-07 23:39:48.780659707 +0800 line 10

Quoted 7 lines
root/workspace/xfstests/results//xfs_4k_crc/generic/483.out.bad 2018-04-07 23:39:48.780659707 +0800 renamed from generic/483.out 2018-04-07 23:35:00.555555511 +0800 +2 -2
@@ -6,5 +6,5 @@
 0: [0..511]: data
 1: [512..2559]: unwritten
 File baz fiemap:
-0: [0..511]: data
-1: [512..6143]: unwritten
+0: [0..895]: data
+1: [896..6143]: unwritten

Perfectly valid result from the test.

Quoted 7 lines

tests/generic/482 line 79

Quoted 7 lines
tests/generic/482 new file +7 -0
+# A file which already has a prealloc extent beyond its size.
+# The fsync done on it is motivated by differences in the btrfs implementation
+# of fsync (first fsync has different logic from subsequent fsyncs).
+$XFS_IO_PROG -f -c "pwrite -S 0xf1 0 256K" \
+	     -c "falloc -k 256K 768K" \
+	     -c "fsync" \
+	     $SCRATCH_MNT/baz >/dev/null

Delayed allocation extending the file to 256k means we'll have speculative prealloc of data beyond 256k - it should be somewhere around 448k IIRC.

Now, falloc doesn't guarantee unwritten extents are allocated - it just guarantees space is allocated. unwritten extents are an optimisation to avoid needing to zero the extents pre-allocated within EOF. i.e. we can quite safely allocate data extents beyond EOF rather than unwritten extents, and this is not a bug.

In this case, the delalloc extent is completely allocated as written data, but nothing is written beyond EOF @ 256k. Hence it reports as a data extent, not an unwritten extent, and this is a perfectly valid thing to do.

Quoted 4 lines

tests/generic/482 line 99

Quoted 4 lines
tests/generic/482 new file +4 -0
+# Allocate another extent beyond the size of file baz.
+$XFS_IO_PROG -c "falloc -k 1M 2M"\
+	     -c "fsync" \
+	     $SCRATCH_MNT/baz

You also cannot assume that two separate preallocations beyond EOF are going to be contiguous (i.e. it could be two separate extents.

What you should just be checking is that there are extents allocated covering EOF to 3MB, not the exactly size, shape and type of extents are allocated.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
author
#4 lore raw
Quoted 18 lines · Dave Chinner
Quoted 17 lines · Eryu Guan
Quoted 5 lines · On Thu, Apr 05, 2018 at 10:56:14PM +0100, [email protected] wrote:

From: Filipe Manana <[email protected]>

Test that fsync operations preserve extents allocated with fallocate(2) that are placed beyond a file's size.

Quoted 8 lines

This test is motivated by a bug found in btrfs where unwritten extents beyond the inode's i_size were not preserved after a fsync and power failure. The btrfs bug is fixed by the following patch for the linux kernel:

"Btrfs: fix loss of prealloc extents past i_size after fsync log replay"

Signed-off-by Filipe Manana <[email protected]>

Hmm, xfs fails this test, while ext4 passes.

Quoted 13 lines

root/workspace/xfstests/results//xfs_4k_crc/generic/483.out.bad 2018-04-07 23:39:48.780659707 +0800 line 10

Quoted 11 lines
# diff -u tests/generic/483.out /root/workspace/xfstests/results//xfs_4k_crc/generic/483.out.bad
root/workspace/xfstests/results//xfs_4k_crc/generic/483.out.bad 2018-04-07 23:39:48.780659707 +0800 renamed from generic/483.out 2018-04-07 23:35:00.555555511 +0800 +2 -2
@@ -6,5 +6,5 @@
 0: [0..511]: data
 1: [512..2559]: unwritten
 File baz fiemap:
-0: [0..511]: data
-1: [512..6143]: unwritten
+0: [0..895]: data
+1: [896..6143]: unwritten

Perfectly valid result from the test.

Still very surprising without knowing specifics about xfs (as you describe below).

Quoted 24 lines
Quoted 8 lines

tests/generic/482 line 79

Quoted 8 lines
tests/generic/482 new file +7 -0
+# A file which already has a prealloc extent beyond its size.
+# The fsync done on it is motivated by differences in the btrfs implementation
+# of fsync (first fsync has different logic from subsequent fsyncs).
+$XFS_IO_PROG -f -c "pwrite -S 0xf1 0 256K" \
+        -c "falloc -k 256K 768K" \
+        -c "fsync" \
+        $SCRATCH_MNT/baz >/dev/null

Delayed allocation extending the file to 256k means we'll have speculative prealloc of data beyond 256k - it should be somewhere around 448k IIRC.

Now, falloc doesn't guarantee unwritten extents are allocated - it just guarantees space is allocated. unwritten extents are an optimisation to avoid needing to zero the extents pre-allocated within EOF. i.e. we can quite safely allocate data extents beyond EOF rather than unwritten extents, and this is not a bug.

In this case, the delalloc extent is completely allocated as written data, but nothing is written beyond EOF @ 256k. Hence it reports as a data extent, not an unwritten extent, and this is a perfectly valid thing to do.

Quoted 7 lines
Quoted 4 lines

tests/generic/482 line 99

Quoted 4 lines
tests/generic/482 new file +4 -0
+# Allocate another extent beyond the size of file baz.
+$XFS_IO_PROG -c "falloc -k 1M 2M"\
+        -c "fsync" \
+        $SCRATCH_MNT/baz

You also cannot assume that two separate preallocations beyond EOF are going to be contiguous (i.e. it could be two separate extents.

I really don't care about that.
Just want to check that allocated space is not lost, don't care if that space is covered by 1, 2 or more extents, or whether they are contiguous or not.

Quoted 4 lines

What you should just be checking is that there are extents allocated covering EOF to 3MB, not the exactly size, shape and type of extents are allocated.

How to do such check, for a generic test, without using fiemap?

thanks

Quoted 11 lines

Cheers,

Dave.
--
Dave Chinner
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw
Quoted 10 lines · Filipe Manana
Quoted 4 lines · Dave Chinner
Quoted 1 line · Eryu Guan

On Thu, Apr 05, 2018 at 10:56:14PM +0100, [email protected] wrote:

You also cannot assume that two separate preallocations beyond EOF are going to be contiguous (i.e. it could be two separate extents.

I really don't care about that.
Just want to check that allocated space is not lost, don't care if that space is covered by 1, 2 or more extents, or whether they are contiguous or not.

Sure, but that's not what the test does :/

Quoted 5 lines
Quoted 3 lines

What you should just be checking is that there are extents allocated covering EOF to 3MB, not the exactly size, shape and type of extents are allocated.

How to do such check, for a generic test, without using fiemap?

We already do such checks using fiemap, yes?

i.e. there's a bunch of _filter*fiemap() functions in common/punch which should show you exactly how to do this i.e. filter all the unwritten/data extents to be the same name, then they coalesce into single ranges....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
author
#6 lore raw

PATCH v2 1/1 fstests: generic test for fsync after fallocate

In reply to [email protected]

From: Filipe Manana <[email protected]>

Test that fsync operations preserve extents allocated with fallocate(2) that are placed beyond a file's size.

This test is motivated by a bug found in btrfs where unwritten extents beyond the inode's i_size were not preserved after a fsync and power failure. The btrfs bug is fixed by the following patch for the linux kernel:

"Btrfs: fix loss of prealloc extents past i_size after fsync log replay"

Signed-off-by Filipe Manana <[email protected]>

---

V2: Use different fiemap filter to not relly on written vs unwritten
differences in extents beyond eof, since has xfs a specific but valid behaviour. Also check file sizes after the power failure since the new fiemap filter now merges written and unwritten extents.

 tests/generic/482     | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/482.out |  13 ++++++
 tests/generic/group   |   1 +
 3 files changed, 138 insertions(+)
 create mode 100755 tests/generic/482
 create mode 100644 tests/generic/482.out
tests/generic/482 new file +124 -0
new file mode 100755
index 00000000..a1947693
@@ -0,0 +1,124 @@
+#! /bin/bash
+# FSQA Test No. 482
+#
+# Test that fsync operations preserve extents allocated with fallocate(2) that
+# are placed beyond a file's size.
+#
+#-----------------------------------------------------------------------
+#
+# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
+# Author: Filipe Manana <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	_cleanup_flakey
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmflakey
+. ./common/punch
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_scratch
+_require_dm_target flakey
+_require_xfs_io_command "falloc" "-k"
+_require_xfs_io_command "fiemap"
+
+rm -f $seqres.full
+
+_scratch_mkfs >>$seqres.full 2>&1
+_require_metadata_journaling $SCRATCH_DEV
+_init_flakey
+_mount_flakey
+
+# Create our test files.
+$XFS_IO_PROG -f -c "pwrite -S 0xea 0 256K" $SCRATCH_MNT/foo >/dev/null
+
+# Create a file with many extents. We later want to shrink truncate it and
+# add a prealloc extent beyond its new size.
+for ((i = 1; i <= 500; i++)); do
+	offset=$(((i - 1) * 4 * 1024))
+	$XFS_IO_PROG -f -s -c "pwrite -S 0xcf $offset 4K" \
+		$SCRATCH_MNT/bar >/dev/null
+done
+
+# A file which already has a prealloc extent beyond its size.
+# The fsync done on it is motivated by differences in the btrfs implementation
+# of fsync (first fsync has different logic from subsequent fsyncs).
+$XFS_IO_PROG -f -c "pwrite -S 0xf1 0 256K" \
+	     -c "falloc -k 256K 768K" \
+	     -c "fsync" \
+	     $SCRATCH_MNT/baz >/dev/null
+
+# Make sure everything done so far is durably persisted.
+sync
+
+# Allocate an extent beyond the size of the first test file and fsync it.
+$XFS_IO_PROG -c "falloc -k 256K 1M"\
+	     -c "fsync" \
+	     $SCRATCH_MNT/foo
+
+# Do a shrinking truncate of our test file, add a prealloc extent to it after
+# its new size and fsync it.
+$XFS_IO_PROG -c "truncate 256K" \
+	     -c "falloc -k 256K 1M"\
+	     -c "fsync" \
+	     $SCRATCH_MNT/bar
+
+# Allocate another extent beyond the size of file baz.
+$XFS_IO_PROG -c "falloc -k 1M 2M"\
+	     -c "fsync" \
+	     $SCRATCH_MNT/baz
+
+# Simulate a power failure and mount the filesystem to check that the extents
+# previously allocated were not lost and the file sizes are correct.
+_flakey_drop_and_remount
+
+echo "File foo fiemap:"
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foo | _filter_hole_fiemap
+echo "File foo size:"
+stat --format %s $SCRATCH_MNT/foo
+
+echo "File bar fiemap:"
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/bar | _filter_hole_fiemap
+echo "File bar size:"
+stat --format %s $SCRATCH_MNT/bar
+
+echo "File baz fiemap:"
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/baz | _filter_hole_fiemap
+echo "File baz size:"
+stat --format %s $SCRATCH_MNT/baz
+
+_unmount_flakey
+_cleanup_flakey
+
+status=0
+exit
tests/generic/482.out new file +13 -0
new file mode 100644
index 00000000..4848894f
@@ -0,0 +1,13 @@
+QA output created by 482
+File foo fiemap:
+0: [0..2559]: extent
+File foo size:
+262144
+File bar fiemap:
+0: [0..2559]: extent
+File bar size:
+262144
+File baz fiemap:
+0: [0..6143]: extent
+File baz size:
+262144
tests/generic/group +1 -0
index e8676062..58ac1f00 100644
@@ -484,3 +484,4 @@
 479 auto quick metadata
 480 auto quick metadata
 481 auto quick log metadata
+482 auto quick log metadata
Signature · 1 line
2.11.0