Skip to contentSkip to conversation

btrfs: test fsync after hole punching when using the no-holes feature

PATCH Superseded [email protected] 1 participant 3 messages Nov 12, 2019 – Nov 19, 2019 Btrfs file system

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 btrfs: test fsync after hole punching when using the no-holes feature

From: Filipe Manana <[email protected]>

Test that when we have the no-holes feature enabled and a specific metadata layout, if we punch a hole that starts at file offset 0 and fsync the file, after replaying the log the hole exists.

This currently fails on btrfs but is fixed by a patch for the linux kernel with the following subject:

"Btrfs: fix missing hole after hole punching and fsync when using NO_HOLES"

---
 tests/btrfs/201     | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/201.out |   5 +++
 tests/btrfs/group   |   1 +
 3 files changed, 106 insertions(+)
 create mode 100755 tests/btrfs/201
 create mode 100644 tests/btrfs/201.out
tests/btrfs/201 new file +100 -0
new file mode 100755
index 00000000..592ae23d
@@ -0,0 +1,100 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FSQA Test No. 201
+#
+# Test that when we have the no-holes feature enabled and a specific metadata
+# layout, if we punch a hole that starts at file offset 0 and fsync the file,
+# after replaying the log the hole exists.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	_cleanup_flakey
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/attr
+. ./common/filter
+. ./common/dmflakey
+
+# real QA test starts here
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_dm_target flakey
+_require_attrs
+_require_xfs_io_command "fpunch"
+_require_btrfs_fs_feature "no_holes"
+_require_btrfs_mkfs_feature "no-holes"
+_require_odirect
+
+rm -f $seqres.full
+
+# We create the filesystem with a node size of 64Kb because we need to create a
+# specific metadata layout in order to trigger the bug we are testing. At the
+# moment the node size can not be smaller then the system's page size, so given
+# that the largest possible page size is 64Kb and by default the node size is
+# set to the system's page size value, we explicitly create a filesystem with a
+# 64Kb node size, so that the test can run reliably independently of the system's
+# page size.
+_scratch_mkfs -O no-holes -n $((64 * 1024)) >>$seqres.full 2>&1
+_require_metadata_journaling $SCRATCH_DEV
+_init_flakey
+_mount_flakey
+
+# Create our first file, which is used just to fill space in a leaf. Its items
+# ocuppy most of the first leaf. We use a large xattr since it's an easy and
+# fast way to fill a lot of leaf space.
+touch $SCRATCH_MNT/foo
+$SETFATTR_PROG -n 'user.x1' -v $(printf '%0.sX' $(seq 1 63617)) $SCRATCH_MNT/foo
+
+# Create our second file, which we will use to test if fsync persists a hole
+# punch operation against it. Create several extent items for the file, all with
+# a size of 64Kb. The first extent item of this file is located within the first
+# leaf of the fs tree, as its last item, and all the remaining extent items in
+# another leaf.
+offset=0
+for ((i = 0; i <= 10; i++)); do
+	$XFS_IO_PROG -f -d -c "pwrite -S 0xab -b 64K $offset 64K" \
+		$SCRATCH_MNT/bar >/dev/null
+	offset=$(($offset + 64 * 1024))
+done
+
+# Make sure everything done so far is durably persisted. We also want to start a
+# new transaction and bump the filesystem generation. We don't fsync because we
+# want to keep the 'full sync' flag in the inode of file 'bar', so that the
+# fsync after the hole punch operation uses the slow path, which is necessary to
+# trigger the bug we are testing.
+sync
+
+# Now punch a hole that covers only the first extent item of file bar. That is
+# the only extent item in the first leaf of the first tree, so the hole punch
+# operation will drop it and will not touch the second leaf which contains the
+# remaining extent items. These conditions are necessary to trigger the bug we
+# are testing.
+$XFS_IO_PROG -c "fpunch 0 64K" -c "fsync" $SCRATCH_MNT/bar
+
+echo "File digest before power failure:"
+md5sum $SCRATCH_MNT/bar | _filter_scratch
+# Simulate a power failure and mount the filesystem to check that replaying the
+# log tree succeeds and our file bar has the expected content.
+_flakey_drop_and_remount
+echo "File digest after power failure and log replay:"
+md5sum $SCRATCH_MNT/bar | _filter_scratch
+
+_unmount_flakey
+_cleanup_flakey
+
+status=0
+exit
tests/btrfs/201.out new file +5 -0
new file mode 100644
index 00000000..66631658
@@ -0,0 +1,5 @@
+QA output created by 201
+File digest before power failure:
+e1e40ea5ab31cc36fb74830a6293eb2b  SCRATCH_MNT/bar
+File digest after power failure and log replay:
+e1e40ea5ab31cc36fb74830a6293eb2b  SCRATCH_MNT/bar
tests/btrfs/group +1 -0
index d56dcafa..d7eeb45d 100755
@@ -203,3 +203,4 @@
 198 auto quick volume
 199 auto quick trim
 200 auto quick send clone
+201 auto quick punch log
Signature · 1 line
2.11.0
#1 lore raw
Quoted 7 lines · On Tue, Nov 12, 2019 at 3:15 PM <[email protected]> wrote:

From: Filipe Manana <[email protected]>

Test that when we have the no-holes feature enabled and a specific metadata layout, if we punch a hole that starts at file offset 0 and fsync the file, after replaying the log the hole exists.

Quoted 6 lines

This currently fails on btrfs but is fixed by a patch for the linux kernel with the following subject:

"Btrfs: fix missing hole after hole punching and fsync when using NO_HOLES"

Eryu, please skip this patch for this week's update.
I have some more tests that I'll add to this test case next week, since they are all about fsync + hole punching + no-holes feature.

Thanks.

Quoted 137 lines
---
 tests/btrfs/201     | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/201.out |   5 +++
 tests/btrfs/group   |   1 +
 3 files changed, 106 insertions(+)
 create mode 100755 tests/btrfs/201
 create mode 100644 tests/btrfs/201.out
tests/btrfs/201 new file +100 -0
new file mode 100755
index 00000000..592ae23d
@@ -0,0 +1,100 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FSQA Test No. 201
+#
+# Test that when we have the no-holes feature enabled and a specific metadata
+# layout, if we punch a hole that starts at file offset 0 and fsync the file,
+# after replaying the log the hole exists.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+tmp=/tmp/$$
+status=1       # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+       _cleanup_flakey
+       cd /
+       rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/attr
+. ./common/filter
+. ./common/dmflakey
+
+# real QA test starts here
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_dm_target flakey
+_require_attrs
+_require_xfs_io_command "fpunch"
+_require_btrfs_fs_feature "no_holes"
+_require_btrfs_mkfs_feature "no-holes"
+_require_odirect
+
+rm -f $seqres.full
+
+# We create the filesystem with a node size of 64Kb because we need to create a
+# specific metadata layout in order to trigger the bug we are testing. At the
+# moment the node size can not be smaller then the system's page size, so given
+# that the largest possible page size is 64Kb and by default the node size is
+# set to the system's page size value, we explicitly create a filesystem with a
+# 64Kb node size, so that the test can run reliably independently of the system's
+# page size.
+_scratch_mkfs -O no-holes -n $((64 * 1024)) >>$seqres.full 2>&1
+_require_metadata_journaling $SCRATCH_DEV
+_init_flakey
+_mount_flakey
+
+# Create our first file, which is used just to fill space in a leaf. Its items
+# ocuppy most of the first leaf. We use a large xattr since it's an easy and
+# fast way to fill a lot of leaf space.
+touch $SCRATCH_MNT/foo
+$SETFATTR_PROG -n 'user.x1' -v $(printf '%0.sX' $(seq 1 63617)) $SCRATCH_MNT/foo
+
+# Create our second file, which we will use to test if fsync persists a hole
+# punch operation against it. Create several extent items for the file, all with
+# a size of 64Kb. The first extent item of this file is located within the first
+# leaf of the fs tree, as its last item, and all the remaining extent items in
+# another leaf.
+offset=0
+for ((i = 0; i <= 10; i++)); do
+       $XFS_IO_PROG -f -d -c "pwrite -S 0xab -b 64K $offset 64K" \
+               $SCRATCH_MNT/bar >/dev/null
+       offset=$(($offset + 64 * 1024))
+done
+
+# Make sure everything done so far is durably persisted. We also want to start a
+# new transaction and bump the filesystem generation. We don't fsync because we
+# want to keep the 'full sync' flag in the inode of file 'bar', so that the
+# fsync after the hole punch operation uses the slow path, which is necessary to
+# trigger the bug we are testing.
+sync
+
+# Now punch a hole that covers only the first extent item of file bar. That is
+# the only extent item in the first leaf of the first tree, so the hole punch
+# operation will drop it and will not touch the second leaf which contains the
+# remaining extent items. These conditions are necessary to trigger the bug we
+# are testing.
+$XFS_IO_PROG -c "fpunch 0 64K" -c "fsync" $SCRATCH_MNT/bar
+
+echo "File digest before power failure:"
+md5sum $SCRATCH_MNT/bar | _filter_scratch
+# Simulate a power failure and mount the filesystem to check that replaying the
+# log tree succeeds and our file bar has the expected content.
+_flakey_drop_and_remount
+echo "File digest after power failure and log replay:"
+md5sum $SCRATCH_MNT/bar | _filter_scratch
+
+_unmount_flakey
+_cleanup_flakey
+
+status=0
+exit
tests/btrfs/201.out new file +5 -0
new file mode 100644
index 00000000..66631658
@@ -0,0 +1,5 @@
+QA output created by 201
+File digest before power failure:
+e1e40ea5ab31cc36fb74830a6293eb2b  SCRATCH_MNT/bar
+File digest after power failure and log replay:
+e1e40ea5ab31cc36fb74830a6293eb2b  SCRATCH_MNT/bar
tests/btrfs/group +1 -1
index d56dcafa..d7eeb45d 100755
@@ -203,3 +203,4 @@
 198 auto quick volume
 199 auto quick trim
 200 auto quick send clone
+201 auto quick punch log
--

2.11.0

#2 lore raw

PATCH v2 1/1 btrfs: test fsync after hole punching when using the no-holes feature

In reply to [email protected]

From: Filipe Manana <[email protected]>

Test that when we have the no-holes feature enabled and a specific metadata layout, if we punch a hole that starts at file offset 0 and fsync the file, after replaying the log the hole exists.

This currently fails on btrfs but is fixed by a patch for the linux kernel with the following subject:

"Btrfs: fix missing hole after hole punching and fsync when using NO_HOLES"

---

V2: Include tests for other cases involving the no-holes feature, hole punching
and fsync.

 tests/btrfs/201     | 190 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/201.out |  24 +++++++
 tests/btrfs/group   |   1 +
 3 files changed, 215 insertions(+)
 create mode 100755 tests/btrfs/201
 create mode 100644 tests/btrfs/201.out
tests/btrfs/201 new file +190 -0
new file mode 100755
index 00000000..72dc1aa5
@@ -0,0 +1,190 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FSQA Test No. 201
+#
+# Test that when we have the no-holes feature enabled and a specific metadata
+# layout, if we punch a hole that starts at file offset 0 and fsync the file,
+# after replaying the log the hole exists.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	_cleanup_flakey
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/attr
+. ./common/filter
+. ./common/dmflakey
+
+# real QA test starts here
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_dm_target flakey
+_require_attrs
+_require_xfs_io_command "fpunch"
+_require_btrfs_fs_feature "no_holes"
+_require_btrfs_mkfs_feature "no-holes"
+_require_odirect
+
+rm -f $seqres.full
+
+run_test_leading_hole()
+{
+    # We create the filesystem with a node size of 64Kb because we need to
+    # create a specific metadata layout in order to trigger the bug we are
+    # testing. At the moment the node size can not be smaller then the system's
+    # page size, so given that the largest possible page size is 64Kb and by
+    # default the node size is set to the system's page size value, we explicitly
+    # create a filesystem with a 64Kb node size, so that the test can run
+    # reliably independently of the system's page size.
+    _scratch_mkfs -O no-holes -n $((64 * 1024)) >>$seqres.full 2>&1
+    _require_metadata_journaling $SCRATCH_DEV
+    _init_flakey
+    _mount_flakey
+
+    # Create our first file, which is used just to fill space in a leaf. Its
+    # items ocuppy most of the first leaf. We use a large xattr since it's an
+    # easy and fast way to fill a lot of leaf space.
+    touch $SCRATCH_MNT/foo
+    $SETFATTR_PROG -n 'user.x1' -v $(printf '%0.sX' $(seq 1 63617)) \
+        $SCRATCH_MNT/foo
+
+   # Create our second file, which we will use to test if fsync persists a hole
+   # punch operation against it. Create several extent items for the file, all with
+   # a size of 64Kb. The first extent item of this file is located within the first
+   # leaf of the fs tree, as its last item, and all the remaining extent items in
+   # another leaf.
+   local offset=0
+   for ((i = 0; i <= 10; i++)); do
+       $XFS_IO_PROG -f -d -c "pwrite -S 0xab -b 64K $offset 64K" \
+           $SCRATCH_MNT/bar >/dev/null
+       offset=$(($offset + 64 * 1024))
+   done
+
+   # Make sure everything done so far is durably persisted. We also want to start
+   # a new transaction and bump the filesystem generation. We don't fsync because
+   # we want to keep the 'full sync' flag in the inode of file 'bar', so that the
+   # fsync after the hole punch operation uses the slow path, which is necessary
+   # to trigger the bug we are testing.
+   sync
+
+   # Now punch a hole that covers only the first extent item of file bar. That
+   # is the only extent item in the first leaf of the first tree, so the hole
+   # punch operation will drop it and will not touch the second leaf which
+   # contains the remaining extent items. These conditions are necessary to
+   # trigger the bug we are testing.
+   $XFS_IO_PROG -c "fpunch 0 64K" -c "fsync" $SCRATCH_MNT/bar
+
+   echo "File digest before power failure:"
+   md5sum $SCRATCH_MNT/bar | _filter_scratch
+   # Simulate a power failure and mount the filesystem to check that replaying the
+   # log tree succeeds and our file bar has the expected content.
+   _flakey_drop_and_remount
+   echo "File digest after power failure and log replay:"
+   md5sum $SCRATCH_MNT/bar | _filter_scratch
+
+   _unmount_flakey
+   _cleanup_flakey
+}
+
+run_test_middle_hole()
+{
+    local hole_offset=$1
+    local hole_len=$2
+
+    # We create the filesystem with a node size of 64Kb because we need to
+    # create a specific metadata layout in order to trigger the bug we are
+    # testing. At the moment the node size can not be smaller then the system's
+    # page size, so given that the largest possible page size is 64Kb and by
+    # default the node size is set to the system's page size value, we explicitly
+    # create a filesystem with a 64Kb node size, so that the test can run
+    # reliably independently of the system's page size.
+    _scratch_mkfs -O no-holes -n $((64 * 1024)) >>$seqres.full 2>&1
+    _require_metadata_journaling $SCRATCH_DEV
+    _init_flakey
+    _mount_flakey
+
+    # Create our first file, which is used just to fill space in a leaf. Its
+    # items ocuppy most of the first leaf. We use a large xattr since it's an
+    # easy and fast way to fill a lot of leaf space.
+    touch $SCRATCH_MNT/foo
+    $SETFATTR_PROG -n 'user.x1' -v $(printf '%0.sX' $(seq 1 63600)) \
+        $SCRATCH_MNT/foo
+
+    # Create our second file, which we will use to test if fsync persists a hole
+    # punch operation against it. Create several extent items for the file, all
+    # with a size of 64Kb. The goal is to have the items of this file span 5
+    # btree leafs.
+    offset=0
+    for ((i = 0; i <= 2000; i++)); do
+        $XFS_IO_PROG -f -d -c "pwrite -S 0xab -b 64K $offset 64K" \
+            $SCRATCH_MNT/bar >/dev/null
+        offset=$(($offset + 64 * 1024))
+    done
+
+    # Make sure everything done so far is durably persisted. We also want to
+    # start a new transaction and bump the filesystem generation. We don't fsync
+    # because we want to keep the 'full sync' flag in the inode of file 'bar',
+    # so that the fsync after the hole punch operation uses the slow path, which
+    # is necessary to trigger the bug we are testing.
+    sync
+
+    # Now punch a hole that covers the entire 3rd leaf. This results in deleting
+    # the entire leaf, without touching the 2nd and 4th leafs. The first leaf is
+    # touched because the inode item needs to be updated (bytes used, ctime,
+    # mtime, etc). After that we modify the last extent, located at the 5th leaf,
+    # and then fsync the file. We want to verify that both the hole and the new
+    # data are correctly persisted by the fsync.
+    $XFS_IO_PROG -c "fpunch $hole_offset $hole_len" \
+                 -c "pwrite -S 0xf1 131072000 64K" \
+                 -c "fsync" $SCRATCH_MNT/bar >/dev/null
+
+    echo "File digest before power failure:"
+    md5sum $SCRATCH_MNT/bar | _filter_scratch
+    # Simulate a power failure and mount the filesystem to check that replaying
+    # the log tree succeeds and our file bar has the expected content.
+    _flakey_drop_and_remount
+    echo "File digest after power failure and log replay:"
+    md5sum $SCRATCH_MNT/bar | _filter_scratch
+
+    _unmount_flakey
+    _cleanup_flakey
+}
+
+
+echo "Testing with hole offset 0 hole length 65536"
+run_test_leading_hole
+
+# Now test cases where file extent items span many leafs and we punch a large
+# hole in the middle of the file, with the goal of getting an entire leaf
+# deleted during the punch hole operation. We test 3 different ranges because
+# depending on whether SELinux is enabled or not the layout of the items in
+# the leafs varies slightly.
+
+echo
+echo "Testing with hole offset 786432 hole length 54919168"
+run_test_middle_hole 786432 54919168
+
+echo
+echo "Testing with hole offset 720896 hole length 54919168"
+run_test_middle_hole 720896 54919168
+
+echo
+echo "Testing with hole offset 655360 hole length 54919168"
+run_test_middle_hole 655360 54919168
+
+status=0
+exit
tests/btrfs/201.out new file +24 -0
new file mode 100644
index 00000000..cd4a729e
@@ -0,0 +1,24 @@
+QA output created by 201
+Testing with hole offset 0 hole length 65536
+File digest before power failure:
+e1e40ea5ab31cc36fb74830a6293eb2b  SCRATCH_MNT/bar
+File digest after power failure and log replay:
+e1e40ea5ab31cc36fb74830a6293eb2b  SCRATCH_MNT/bar
+
+Testing with hole offset 786432 hole length 54919168
+File digest before power failure:
+8bfa78372c90f7418ce6a90ba3a92b7f  SCRATCH_MNT/bar
+File digest after power failure and log replay:
+8bfa78372c90f7418ce6a90ba3a92b7f  SCRATCH_MNT/bar
+
+Testing with hole offset 720896 hole length 54919168
+File digest before power failure:
+327a10e71854c55a0439b859665a4fdd  SCRATCH_MNT/bar
+File digest after power failure and log replay:
+327a10e71854c55a0439b859665a4fdd  SCRATCH_MNT/bar
+
+Testing with hole offset 655360 hole length 54919168
+File digest before power failure:
+2b060d70788a69e081013f0ce0e52921  SCRATCH_MNT/bar
+File digest after power failure and log replay:
+2b060d70788a69e081013f0ce0e52921  SCRATCH_MNT/bar
tests/btrfs/group +1 -0
index d56dcafa..d7eeb45d 100755
@@ -203,3 +203,4 @@
 198 auto quick volume
 199 auto quick trim
 200 auto quick send clone
+201 auto quick punch log
Signature · 1 line
2.11.0