Skip to contentSkip to conversation

generic: test number of blocks used by a file after mwrite into a hole

PATCH In review [email protected] 3 participants 6 messages Nov 4, 2020 – Nov 9, 2020 1 message not in the archive Btrfs file system

lore
Share & tools
lore
#1 lore raw

PATCH 1/2 generic: test number of blocks used by a file after mwrite into a hole

In reply to a message not in the archive

From: Filipe Manana <[email protected]>

Test that after doing a memory mapped write to an empty file, a call to stat(2) reports a non-zero number of used blocks.

This is motivated by a bug in btrfs where the number of blocks used does not change. It currenly fails on btrfs and it is fixed by a patch that has the following subject:

"btrfs: fix missing delalloc new bit for new delalloc ranges"

---
 tests/generic/614     | 50 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/614.out |  2 ++
 tests/generic/group   |  1 +
 3 files changed, 53 insertions(+)
 create mode 100755 tests/generic/614
 create mode 100644 tests/generic/614.out
tests/generic/614 new file +50 -0
new file mode 100755
index 00000000..80edf9cd
@@ -0,0 +1,50 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2020 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test No. 614
+#
+# Test that after doing a memory mapped write to an empty file, a call to
+# stat(2) reports a non-zero number of used blocks.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_scratch
+
+rm -f $seqres.full
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+$XFS_IO_PROG -f -c "truncate 64K"      \
+	     -c "mmap -w 0 64K"        \
+	     -c "mwrite -S 0xab 0 64K" \
+	     -c "munmap"               \
+	     $SCRATCH_MNT/foobar | _filter_xfs_io
+
+blocks_used=$(stat -c %b $SCRATCH_MNT/foobar)
+if [ $blocks_used -eq 0 ]; then
+    echo "error: stat(2) reported 0 used blocks"
+fi
+
+echo "Silence is golden"
+
+status=0
+exit
tests/generic/614.out new file +2 -0
new file mode 100644
index 00000000..3db70236
@@ -0,0 +1,2 @@
+QA output created by 614
+Silence is golden
tests/generic/group +1 -0
index 31057ac8..ab8ae74e 100644
@@ -616,3 +616,4 @@
 611 auto quick attr
 612 auto quick clone
 613 auto quick encrypt
+614 auto quick rw
Signature · 1 line
2.28.0
#3 lore raw

PATCH 2/2 generic: test for non-zero used blocks while writing into a file

In reply to a message not in the archive

From: Filipe Manana <[email protected]>

Test that if we keep overwriting an entire file, either with buffered writes or direct IO writes, the number of used blocks reported by stat(2) is never zero while the writes and writeback are in progress.

This is motivated by a bug in btrfs and currently fails on btrfs only. It is fixed a patchset for btrfs that has the following patches:

btrfs: fix missing delalloc new bit for new delalloc ranges
btrfs: refactor btrfs_drop_extents() to make it easier to extend
btrfs: fix race when defragging that leads to unnecessary IO
btrfs: update the number of bytes used by an inode atomically

---
 tests/generic/615     | 77 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/615.out |  3 ++
 tests/generic/group   |  1 +
 3 files changed, 81 insertions(+)
 create mode 100755 tests/generic/615
 create mode 100644 tests/generic/615.out
tests/generic/615 new file +77 -0
new file mode 100755
index 00000000..e392c4a5
@@ -0,0 +1,77 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2020 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test No. 615
+#
+# Test that if we keep overwriting an entire file, either with buffered writes
+# or direct IO writes, the number of used blocks reported by stat(2) is never
+# zero while the writes and writeback are in progress.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_scratch
+_require_odirect
+
+rm -f $seqres.full
+
+stat_loop()
+{
+	trap "wait; exit" SIGTERM
+	local filepath=$1
+	local blocks
+
+	while :; do
+		blocks=$(stat -c %b $filepath)
+		if [ $blocks -eq 0 ]; then
+		    echo "error: stat(2) reported zero blocks"
+		fi
+	done
+}
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+$XFS_IO_PROG -f -s -c "pwrite -b 64K 0 64K" $SCRATCH_MNT/foo >>$seqres.full
+
+stat_loop $SCRATCH_MNT/foo &
+loop_pid=$!
+
+echo "Testing buffered writes"
+
+# Now keep overwriting the entire file, triggering writeback after each write,
+# while another process is calling stat(2) on the file. We expect the number of
+# used blocks reported by stat(2) to be always greater than 0.
+for ((i = 0; i < 5000; i++)); do
+	$XFS_IO_PROG -s -c "pwrite -b 64K 0 64K" $SCRATCH_MNT/foo >>$seqres.full
+done
+
+echo "Testing direct IO writes"
+
+# Now similar to what we did before but for direct IO writes.
+for ((i = 0; i < 5000; i++)); do
+	$XFS_IO_PROG -d -c "pwrite -b 64K 0 64K" $SCRATCH_MNT/foo >>$seqres.full
+done
+
+kill $loop_pid &> /dev/null
+wait
+
+status=0
+exit
tests/generic/615.out new file +3 -0
new file mode 100644
index 00000000..3b549e96
@@ -0,0 +1,3 @@
+QA output created by 615
+Testing buffered writes
+Testing direct IO writes
tests/generic/group +1 -0
index ab8ae74e..fb3c638c 100644
@@ -617,3 +617,4 @@
 612 auto quick clone
 613 auto quick encrypt
 614 auto quick rw
+615 auto rw
Signature · 1 line
2.28.0
Reviewed-by on v1 1/2
#2 lore raw
In reply to [email protected]
Quoted 5 lines · On 11/4/20 6:13 AM, [email protected] wrote:

From: Filipe Manana <[email protected]>

Test that after doing a memory mapped write to an empty file, a call to stat(2) reports a non-zero number of used blocks.

Quoted 7 lines

This is motivated by a bug in btrfs where the number of blocks used does not change. It currenly fails on btrfs and it is fixed by a patch that has the following subject:

"btrfs: fix missing delalloc new bit for new delalloc ranges"

Thanks,

Josef

Reviewed-by on v1 2/2
#4 lore raw

Re: [PATCH 2/2] generic: test for non-zero used blocks while writing into a file

In reply to [email protected]
Quoted 9 lines · On 11/4/20 6:13 AM, [email protected] wrote:

From: Filipe Manana <[email protected]>

Test that if we keep overwriting an entire file, either with buffered writes or direct IO writes, the number of used blocks reported by stat(2) is never zero while the writes and writeback are in progress.

This is motivated by a bug in btrfs and currently fails on btrfs only. It is fixed a patchset for btrfs that has the following patches:

Quoted 6 lines

btrfs: fix missing delalloc new bit for new delalloc ranges
btrfs: refactor btrfs_drop_extents() to make it easier to extend
btrfs: fix race when defragging that leads to unnecessary IO
btrfs: update the number of bytes used by an inode atomically

Thanks,

Josef

#5 lore raw

Re: [PATCH 2/2] generic: test for non-zero used blocks while writing into a file

In reply to [email protected]
Quoted 83 lines · On Wed, Nov 04, 2020 at 11:13:53AM +0000, [email protected] wrote:

From: Filipe Manana <[email protected]>

Test that if we keep overwriting an entire file, either with buffered writes or direct IO writes, the number of used blocks reported by stat(2) is never zero while the writes and writeback are in progress.

This is motivated by a bug in btrfs and currently fails on btrfs only. It is fixed a patchset for btrfs that has the following patches:

btrfs: fix missing delalloc new bit for new delalloc ranges
btrfs: refactor btrfs_drop_extents() to make it easier to extend
btrfs: fix race when defragging that leads to unnecessary IO
btrfs: update the number of bytes used by an inode atomically

---
 tests/generic/615     | 77 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/615.out |  3 ++
 tests/generic/group   |  1 +
 3 files changed, 81 insertions(+)
 create mode 100755 tests/generic/615
 create mode 100644 tests/generic/615.out
tests/generic/615 new file +54 -0
new file mode 100755
index 00000000..e392c4a5
@@ -0,0 +1,77 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2020 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test No. 615
+#
+# Test that if we keep overwriting an entire file, either with buffered writes
+# or direct IO writes, the number of used blocks reported by stat(2) is never
+# zero while the writes and writeback are in progress.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_scratch
+_require_odirect
+
+rm -f $seqres.full
+
+stat_loop()
+{
+	trap "wait; exit" SIGTERM
+	local filepath=$1
+	local blocks
+
+	while :; do
+		blocks=$(stat -c %b $filepath)
+		if [ $blocks -eq 0 ]; then
+		    echo "error: stat(2) reported zero blocks"
+		fi
+	done
+}
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+$XFS_IO_PROG -f -s -c "pwrite -b 64K 0 64K" $SCRATCH_MNT/foo >>$seqres.full
+
+stat_loop $SCRATCH_MNT/foo &

tests/generic/615 line 64

Quoted 10 lines
tests/generic/615 new file +10 -0
+loop_pid=$!
+
+echo "Testing buffered writes"
+
+# Now keep overwriting the entire file, triggering writeback after each write,
+# while another process is calling stat(2) on the file. We expect the number of
+# used blocks reported by stat(2) to be always greater than 0.
+for ((i = 0; i < 5000; i++)); do
+	$XFS_IO_PROG -s -c "pwrite -b 64K 0 64K" $SCRATCH_MNT/foo >>$seqres.full
+done

Test 5000 loops of sync write takes 3 mins to 5 mins for me on my test vm, which has 2 vcpus and 8G memory.

I think we could reduce the test time by
- lower the loop count, maybe 2000, in my test it usually reproduces within 1000 iterations.
- drop "-s" option, it speeds up the test from ~180s to ~25s, and btrfs still reproduces the bug. But it seems dropping "-s" make it harder to hit the bug, perhaps 3000-5000 iteration should be fine.
- reduce the write buffer size, a 16K write still reproduce the bug on btrfs for me. But if we want to cover 64K page size env, I think 64K buffer is fine, as long as "-s" is dropped.

Further, I think we could exit early when "zero blocks" is found, e.g. call "_fail" instead of "echo" in stat_loop() and in the write loop, check for $loop_pid before every write, and break the loop if $loop_pid died. So we could report failure as soon as possible. For example

for ((i = 0; i < 5000; i++)); do
if ! kill -s 0 $loop_pid; then
echo "Bug reproduced at iteration $i"
break
fi
$XFS_IO_PROG -s -c "pwrite -b 64K 0 64K" $SCRATCH_MNT/foo >>$seqres.full
done

Thanks,
Eryu

Quoted 32 lines
tests/generic/615 new file +12 -0
+echo "Testing direct IO writes"
+
+# Now similar to what we did before but for direct IO writes.
+for ((i = 0; i < 5000; i++)); do
+	$XFS_IO_PROG -d -c "pwrite -b 64K 0 64K" $SCRATCH_MNT/foo >>$seqres.full
+done
+
+kill $loop_pid &> /dev/null
+wait
+
+status=0
+exit
tests/generic/615.out new file +3 -0
new file mode 100644
index 00000000..3b549e96
@@ -0,0 +1,3 @@
+QA output created by 615
+Testing buffered writes
+Testing direct IO writes
tests/generic/group +1 -0
index ab8ae74e..fb3c638c 100644
@@ -617,3 +617,4 @@
 612 auto quick clone
 613 auto quick encrypt
 614 auto quick rw
+615 auto rw
Signature · 1 line
2.28.0
#6 lore raw

Re: [PATCH 2/2] generic: test for non-zero used blocks while writing into a file

Quoted 90 lines · Eryu Guan
Quoted 88 lines · On Wed, Nov 04, 2020 at 11:13:53AM +0000, [email protected] wrote:

From: Filipe Manana <[email protected]>

Test that if we keep overwriting an entire file, either with buffered writes or direct IO writes, the number of used blocks reported by stat(2) is never zero while the writes and writeback are in progress.

This is motivated by a bug in btrfs and currently fails on btrfs only. It is fixed a patchset for btrfs that has the following patches:

btrfs: fix missing delalloc new bit for new delalloc ranges
btrfs: refactor btrfs_drop_extents() to make it easier to extend
btrfs: fix race when defragging that leads to unnecessary IO
btrfs: update the number of bytes used by an inode atomically

---
 tests/generic/615     | 77 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/615.out |  3 ++
 tests/generic/group   |  1 +
 3 files changed, 81 insertions(+)
 create mode 100755 tests/generic/615
 create mode 100644 tests/generic/615.out
tests/generic/615 new file +59 -0
new file mode 100755
index 00000000..e392c4a5
@@ -0,0 +1,77 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2020 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test No. 615
+#
+# Test that if we keep overwriting an entire file, either with buffered writes
+# or direct IO writes, the number of used blocks reported by stat(2) is never
+# zero while the writes and writeback are in progress.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+tmp=/tmp/$$
+status=1     # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+     cd /
+     rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_scratch
+_require_odirect
+
+rm -f $seqres.full
+
+stat_loop()
+{
+     trap "wait; exit" SIGTERM
+     local filepath=$1
+     local blocks
+
+     while :; do
+             blocks=$(stat -c %b $filepath)
+             if [ $blocks -eq 0 ]; then
+                 echo "error: stat(2) reported zero blocks"
+             fi
+     done
+}
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+$XFS_IO_PROG -f -s -c "pwrite -b 64K 0 64K" $SCRATCH_MNT/foo >>$seqres.full
+
+stat_loop $SCRATCH_MNT/foo &
+loop_pid=$!
+
+echo "Testing buffered writes"
+
+# Now keep overwriting the entire file, triggering writeback after each write,
Quoted 8 lines

tests/generic/615 line 64

Quoted 5 lines
tests/generic/615 new file +5 -0
+# while another process is calling stat(2) on the file. We expect the number of
+# used blocks reported by stat(2) to be always greater than 0.
+for ((i = 0; i < 5000; i++)); do
+     $XFS_IO_PROG -s -c "pwrite -b 64K 0 64K" $SCRATCH_MNT/foo >>$seqres.full
+done

Test 5000 loops of sync write takes 3 mins to 5 mins for me on my test vm, which has 2 vcpus and 8G memory.

That's slow yes. It takes ~60 seconds on my vms with 4 and 8 cpus (and 8Gb of ram).

Quoted 4 lines

I think we could reduce the test time by
- lower the loop count, maybe 2000, in my test it usually reproduces within 1000 iterations.

I can leave it to 2000 iterations per write type. For me sometimes it doesn't reproduce with 1000.

Quoted 3 lines

- drop "-s" option, it speeds up the test from ~180s to ~25s, and btrfs still reproduces the bug. But it seems dropping "-s" make it harder to hit the bug, perhaps 3000-5000 iteration should be fine.

The -s is important here for buffered writes.
The purpose, as the comment and test description mention, is to trigger a race between stat and writeback.
In your case, because it runs much slower, even without -s writeback is still being triggered a few times, so that's probably why you see it fail for the buffered writes case.

Quoted 3 lines

- reduce the write buffer size, a 16K write still reproduce the bug on btrfs for me. But if we want to cover 64K page size env, I think 64K buffer is fine, as long as "-s" is dropped.

Yes, the 64K is there so that it works on the 64K page size case.

Quoted 6 lines

Further, I think we could exit early when "zero blocks" is found, e.g. call "_fail" instead of "echo" in stat_loop() and in the write loop, check for $loop_pid before every write, and break the loop if $loop_pid died. So we could report failure as soon as possible. For example

Quoted 7 lines

for ((i = 0; i < 5000; i++)); do
if ! kill -s 0 $loop_pid; then
echo "Bug reproduced at iteration $i"
break
fi
$XFS_IO_PROG -s -c "pwrite -b 64K 0 64K" $SCRATCH_MNT/foo >>$seqres.full
done

I can do that, and added in v2 in fact, together with 2000 iterations instead of 5000.

I don't think however it's that much important to make it fail faster. What is more useful is to make it run faster in the success (expected) case while still being able to trigger the btrfs race frequently enough.

Thanks.

Quoted 36 lines

Thanks,
Eryu

Quoted 32 lines
tests/generic/615 new file +12 -0
+echo "Testing direct IO writes"
+
+# Now similar to what we did before but for direct IO writes.
+for ((i = 0; i < 5000; i++)); do
+     $XFS_IO_PROG -d -c "pwrite -b 64K 0 64K" $SCRATCH_MNT/foo >>$seqres.full
+done
+
+kill $loop_pid &> /dev/null
+wait
+
+status=0
+exit
tests/generic/615.out new file +3 -0
new file mode 100644
index 00000000..3b549e96
@@ -0,0 +1,3 @@
+QA output created by 615
+Testing buffered writes
+Testing direct IO writes
tests/generic/group +1 -1
index ab8ae74e..fb3c638c 100644
@@ -617,3 +617,4 @@
 612 auto quick clone
 613 auto quick encrypt
 614 auto quick rw
+615 auto rw
--

2.28.0