Skip to contentSkip to conversation

fstests: add a another gap extent testcase for btrfs

PATCH In review Josef Bacik 3 participants 4 messages Feb 20, 2020 – Feb 23, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fstests: add a another gap extent testcase for btrfs

This is a testcase for a corner that I missed when trying to fix gap extents for btrfs. We would end up with gaps if we hole punched past isize and then extended past the gap in a specific way. This is a simple reproducer to show the problem, and has been properly fixed by my patches now.

---
 tests/btrfs/204     | 85 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/204.out |  5 +++
 tests/btrfs/group   |  1 +
 3 files changed, 91 insertions(+)
 create mode 100755 tests/btrfs/204
 create mode 100644 tests/btrfs/204.out
tests/btrfs/204 new file +85 -0
new file mode 100755
index 00000000..0d5c4bed
@@ -0,0 +1,85 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2020 Facebook.  All Rights Reserved.
+#
+# FS QA Test 204
+#
+# Validate that without no-holes we do not get a i_size that is after a gap in
+# the file extents on disk when punching a hole past i_size.  This is fixed by
+# the following patches
+#
+#	btrfs: use the file extent tree infrastructure
+#	btrfs: replace all uses of btrfs_ordered_update_i_size
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmlogwrites
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_test
+_require_scratch
+_require_log_writes
+
+_log_writes_init $SCRATCH_DEV
+_log_writes_mkfs "-O ^no-holes" >> $seqres.full 2>&1
+
+# There's not a straightforward way to commit the transaction without also
+# flushing dirty pages, so shorten the commit interval to 1 so we're sure to get
+# a commit with our broken file
+_log_writes_mount -o commit=1
+
+# This creates a gap extent because fpunch doesn't insert hole extents past
+# i_size
+xfs_io -f -c "falloc -k 4k 8k" $SCRATCH_MNT/file
+xfs_io -f -c "fpunch 4k 4k" $SCRATCH_MNT/file
+
+# The pwrite extends the i_size to cover the gap extent, and then the truncate
+# sets the disk_i_size to 12k because it assumes everything was a-ok.
+xfs_io -f -c "pwrite 0 4k" $SCRATCH_MNT/file | _filter_xfs_io
+xfs_io -f -c "pwrite 0 8k" $SCRATCH_MNT/file | _filter_xfs_io
+xfs_io -f -c "truncate 12k" $SCRATCH_MNT/file
+
+# Wait for a transaction commit
+sleep 2
+
+_log_writes_unmount
+_log_writes_remove
+
+cur=$(_log_writes_find_next_fua 0)
+echo "cur=$cur" >> $seqres.full
+while [ ! -z "$cur" ]; do
+	_log_writes_replay_log_range $cur $SCRATCH_DEV >> $seqres.full
+
+	# We only care about the fs consistency, so just run fsck, we don't have
+	# to mount the fs to validate it
+	_check_scratch_fs
+
+	cur=$(_log_writes_find_next_fua $(($cur + 1)))
+done
+
+# success, all done
+status=0
+exit
tests/btrfs/204.out new file +5 -0
new file mode 100644
index 00000000..44c7c8ae
@@ -0,0 +1,5 @@
+QA output created by 204
+wrote 4096/4096 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 8192/8192 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
tests/btrfs/group +1 -0
index 6acc6426..7a840177 100644
@@ -206,3 +206,4 @@
 201 auto quick punch log
 202 auto quick subvol snapshot
 203 auto quick send clone
+204 auto quick log replay
Signature · 1 line
2.24.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 56 lines · Josef Bacik

This is a testcase for a corner that I missed when trying to fix gap extents for btrfs. We would end up with gaps if we hole punched past isize and then extended past the gap in a specific way. This is a simple reproducer to show the problem, and has been properly fixed by my patches now.

---
 tests/btrfs/204     | 85 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/204.out |  5 +++
 tests/btrfs/group   |  1 +
 3 files changed, 91 insertions(+)
 create mode 100755 tests/btrfs/204
 create mode 100644 tests/btrfs/204.out
tests/btrfs/204 new file +34 -0
new file mode 100755
index 00000000..0d5c4bed
@@ -0,0 +1,85 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2020 Facebook.  All Rights Reserved.
+#
+# FS QA Test 204
+#
+# Validate that without no-holes we do not get a i_size that is after a gap in
+# the file extents on disk when punching a hole past i_size.  This is fixed by
+# the following patches
+#
+#      btrfs: use the file extent tree infrastructure
+#      btrfs: replace all uses of btrfs_ordered_update_i_size
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1       # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+       cd /
+       rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmlogwrites
+
+# remove previous $seqres.full before test

tests/btrfs/204 line 44

Quoted 10 lines
tests/btrfs/204 new file +10 -0
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_test
+_require_scratch
+_require_log_writes

_require_xfs_io_command "falloc" "-k"
_require_xfs_io_command "fpunch"

Quoted 57 lines
tests/btrfs/204 new file +41 -0
+
+_log_writes_init $SCRATCH_DEV
+_log_writes_mkfs "-O ^no-holes" >> $seqres.full 2>&1
+
+# There's not a straightforward way to commit the transaction without also
+# flushing dirty pages, so shorten the commit interval to 1 so we're sure to get
+# a commit with our broken file
+_log_writes_mount -o commit=1
+
+# This creates a gap extent because fpunch doesn't insert hole extents past
+# i_size
+xfs_io -f -c "falloc -k 4k 8k" $SCRATCH_MNT/file
+xfs_io -f -c "fpunch 4k 4k" $SCRATCH_MNT/file
+
+# The pwrite extends the i_size to cover the gap extent, and then the truncate
+# sets the disk_i_size to 12k because it assumes everything was a-ok.
+xfs_io -f -c "pwrite 0 4k" $SCRATCH_MNT/file | _filter_xfs_io
+xfs_io -f -c "pwrite 0 8k" $SCRATCH_MNT/file | _filter_xfs_io
+xfs_io -f -c "truncate 12k" $SCRATCH_MNT/file
+
+# Wait for a transaction commit
+sleep 2
+
+_log_writes_unmount
+_log_writes_remove
+
+cur=$(_log_writes_find_next_fua 0)
+echo "cur=$cur" >> $seqres.full
+while [ ! -z "$cur" ]; do
+       _log_writes_replay_log_range $cur $SCRATCH_DEV >> $seqres.full
+
+       # We only care about the fs consistency, so just run fsck, we don't have
+       # to mount the fs to validate it
+       _check_scratch_fs
+
+       cur=$(_log_writes_find_next_fua $(($cur + 1)))
+done
+
+# success, all done
+status=0
+exit
tests/btrfs/204.out new file +5 -0
new file mode 100644
index 00000000..44c7c8ae
@@ -0,0 +1,5 @@
+QA output created by 204
+wrote 4096/4096 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 8192/8192 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
tests/btrfs/group +0 -0
index 6acc6426..7a840177 100644

tests/btrfs/group line 209

Quoted 4 lines
tests/btrfs/group +1 -0
@@ -206,3 +206,4 @@
 201 auto quick punch log
 202 auto quick subvol snapshot
 203 auto quick send clone
+204 auto quick log replay

"prealloc" and "punch" groups as well.

Since this just tests another variant of the same problem, maybe it could be added to btrfs/172, since that test is very recent and you authored it as well.
Anyway, I don't have a strong preference.

The test itself looks good to me, and with the _require_xfs_io_command thing added and the groups (maybe Eryu can add these at commit time):

Quoted 3 lines

--
2.24.1

Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”
#2 lore raw
Quoted 140 lines · Filipe Manana
Quoted 56 lines · Josef Bacik

This is a testcase for a corner that I missed when trying to fix gap extents for btrfs. We would end up with gaps if we hole punched past isize and then extended past the gap in a specific way. This is a simple reproducer to show the problem, and has been properly fixed by my patches now.

---
  tests/btrfs/204     | 85 +++++++++++++++++++++++++++++++++++++++++++++
  tests/btrfs/204.out |  5 +++
  tests/btrfs/group   |  1 +
  3 files changed, 91 insertions(+)

create mode 100755 tests/btrfs/204
create mode 100644 tests/btrfs/204.out

tests/btrfs/204 new file +34 -0
new file mode 100755
index 00000000..0d5c4bed
@@ -0,0 +1,85 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2020 Facebook.  All Rights Reserved.
+#
+# FS QA Test 204
+#
+# Validate that without no-holes we do not get a i_size that is after a gap in
+# the file extents on disk when punching a hole past i_size.  This is fixed by
+# the following patches
+#
+#      btrfs: use the file extent tree infrastructure
+#      btrfs: replace all uses of btrfs_ordered_update_i_size
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1       # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+       cd /
+       rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmlogwrites
+
+# remove previous $seqres.full before test

tests/btrfs/204 line 44

Quoted 10 lines
tests/btrfs/204 new file +10 -0
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_test
+_require_scratch
+_require_log_writes

_require_xfs_io_command "falloc" "-k"
_require_xfs_io_command "fpunch"

Quoted 57 lines
tests/btrfs/204 new file +41 -0
+
+_log_writes_init $SCRATCH_DEV
+_log_writes_mkfs "-O ^no-holes" >> $seqres.full 2>&1
+
+# There's not a straightforward way to commit the transaction without also
+# flushing dirty pages, so shorten the commit interval to 1 so we're sure to get
+# a commit with our broken file
+_log_writes_mount -o commit=1
+
+# This creates a gap extent because fpunch doesn't insert hole extents past
+# i_size
+xfs_io -f -c "falloc -k 4k 8k" $SCRATCH_MNT/file
+xfs_io -f -c "fpunch 4k 4k" $SCRATCH_MNT/file
+
+# The pwrite extends the i_size to cover the gap extent, and then the truncate
+# sets the disk_i_size to 12k because it assumes everything was a-ok.
+xfs_io -f -c "pwrite 0 4k" $SCRATCH_MNT/file | _filter_xfs_io
+xfs_io -f -c "pwrite 0 8k" $SCRATCH_MNT/file | _filter_xfs_io
+xfs_io -f -c "truncate 12k" $SCRATCH_MNT/file
+
+# Wait for a transaction commit
+sleep 2
+
+_log_writes_unmount
+_log_writes_remove
+
+cur=$(_log_writes_find_next_fua 0)
+echo "cur=$cur" >> $seqres.full
+while [ ! -z "$cur" ]; do
+       _log_writes_replay_log_range $cur $SCRATCH_DEV >> $seqres.full
+
+       # We only care about the fs consistency, so just run fsck, we don't have
+       # to mount the fs to validate it
+       _check_scratch_fs
+
+       cur=$(_log_writes_find_next_fua $(($cur + 1)))
+done
+
+# success, all done
+status=0
+exit
tests/btrfs/204.out new file +5 -0
new file mode 100644
index 00000000..44c7c8ae
@@ -0,0 +1,5 @@
+QA output created by 204
+wrote 4096/4096 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 8192/8192 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
tests/btrfs/group +0 -0
index 6acc6426..7a840177 100644

tests/btrfs/group line 209

Quoted 4 lines
tests/btrfs/group +1 -0
@@ -206,3 +206,4 @@
  201 auto quick punch log
  202 auto quick subvol snapshot
  203 auto quick send clone
+204 auto quick log replay

"prealloc" and "punch" groups as well.

Since this just tests another variant of the same problem, maybe it could be added to btrfs/172, since that test is very recent and you authored it as well.
Anyway, I don't have a strong preference.

Quoted 5 lines

The test itself looks good to me, and with the _require_xfs_io_command thing added and the groups (maybe Eryu can add these at commit time):

I like to keep these things discrete, if a test is testing two different things and it fails I have to go comment out one part and re-run to figure out which actually failed. Thanks,

Josef

#3 lore raw
In reply to Filipe Manana
Quoted 55 lines · Filipe Manana
Quoted 54 lines · Josef Bacik

This is a testcase for a corner that I missed when trying to fix gap extents for btrfs. We would end up with gaps if we hole punched past isize and then extended past the gap in a specific way. This is a simple reproducer to show the problem, and has been properly fixed by my patches now.

---
 tests/btrfs/204     | 85 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/204.out |  5 +++
 tests/btrfs/group   |  1 +
 3 files changed, 91 insertions(+)
 create mode 100755 tests/btrfs/204
 create mode 100644 tests/btrfs/204.out
tests/btrfs/204 new file +32 -0
new file mode 100755
index 00000000..0d5c4bed
@@ -0,0 +1,85 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2020 Facebook.  All Rights Reserved.
+#
+# FS QA Test 204
+#
+# Validate that without no-holes we do not get a i_size that is after a gap in
+# the file extents on disk when punching a hole past i_size.  This is fixed by
+# the following patches
+#
+#      btrfs: use the file extent tree infrastructure
+#      btrfs: replace all uses of btrfs_ordered_update_i_size
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1       # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+       cd /
+       rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmlogwrites
Quoted 8 lines

tests/btrfs/204 line 40

Quoted 8 lines
tests/btrfs/204 new file +8 -0
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic

I also changed 'generic' to 'btrfs'

Quoted 7 lines

tests/btrfs/204 line 44

Quoted 4 lines
tests/btrfs/204 new file +4 -0
+_supported_os Linux
+_require_test
+_require_scratch
+_require_log_writes

_require_xfs_io_command "falloc" "-k"
_require_xfs_io_command "fpunch"

Added.

Quoted 12 lines
Quoted 12 lines
tests/btrfs/204 new file +11 -0
+
+_log_writes_init $SCRATCH_DEV
+_log_writes_mkfs "-O ^no-holes" >> $seqres.full 2>&1
+
+# There's not a straightforward way to commit the transaction without also
+# flushing dirty pages, so shorten the commit interval to 1 so we're sure to get
+# a commit with our broken file
+_log_writes_mount -o commit=1
+
+# This creates a gap extent because fpunch doesn't insert hole extents past
+# i_size
Quoted 8 lines

tests/btrfs/204 line 63

Quoted 8 lines
tests/btrfs/204 new file +8 -0
+xfs_io -f -c "falloc -k 4k 8k" $SCRATCH_MNT/file
+xfs_io -f -c "fpunch 4k 4k" $SCRATCH_MNT/file
+
+# The pwrite extends the i_size to cover the gap extent, and then the truncate
+# sets the disk_i_size to 12k because it assumes everything was a-ok.
+xfs_io -f -c "pwrite 0 4k" $SCRATCH_MNT/file | _filter_xfs_io
+xfs_io -f -c "pwrite 0 8k" $SCRATCH_MNT/file | _filter_xfs_io
+xfs_io -f -c "truncate 12k" $SCRATCH_MNT/file

Changed all 'xfs_io' to '$XFS_IO_PROG'.

Quoted 36 lines
Quoted 36 lines
tests/btrfs/204 new file +22 -0
+
+# Wait for a transaction commit
+sleep 2
+
+_log_writes_unmount
+_log_writes_remove
+
+cur=$(_log_writes_find_next_fua 0)
+echo "cur=$cur" >> $seqres.full
+while [ ! -z "$cur" ]; do
+       _log_writes_replay_log_range $cur $SCRATCH_DEV >> $seqres.full
+
+       # We only care about the fs consistency, so just run fsck, we don't have
+       # to mount the fs to validate it
+       _check_scratch_fs
+
+       cur=$(_log_writes_find_next_fua $(($cur + 1)))
+done
+
+# success, all done
+status=0
+exit
tests/btrfs/204.out new file +5 -0
new file mode 100644
index 00000000..44c7c8ae
@@ -0,0 +1,5 @@
+QA output created by 204
+wrote 4096/4096 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 8192/8192 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
tests/btrfs/group +0 -1
index 6acc6426..7a840177 100644
--- a/tests/btrfs/group
Quoted 8 lines

tests/btrfs/group line 209

Quoted 6 lines

+++ b/tests/btrfs/group

tests/btrfs/group +1 -0
@@ -206,3 +206,4 @@
 201 auto quick punch log
 202 auto quick subvol snapshot
 203 auto quick send clone
+204 auto quick log replay

"prealloc" and "punch" groups as well.

Added.

Quoted 10 lines

Since this just tests another variant of the same problem, maybe it could be added to btrfs/172, since that test is very recent and you authored it as well.
Anyway, I don't have a strong preference.

The test itself looks good to me, and with the _require_xfs_io_command thing added and the groups (maybe Eryu can add these at commit time):

Thanks for the review!

Eryu

Quoted 11 lines
Quoted 5 lines

--
2.24.1

Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”