Skip to contentSkip to conversation

generic: test COW writeback failure when overlapping non-shared blocks

PATCH Superseded Brian Foster 2 participants 4 messages Oct 21, 2021

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 generic: test COW writeback failure when overlapping non-shared blocks

Test that COW writeback that overlaps non-shared delalloc blocks does not leave around stale delalloc blocks on I/O failure. This triggers assert failures and free space accounting corruption on XFS.

---

This test targets the problem addressed by the following patch in XFS:

https://lore.kernel.org/linux-xfs/[email protected]/

Brian

 tests/generic/651     | 53 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/651.out |  2 ++
 2 files changed, 55 insertions(+)
 create mode 100755 tests/generic/651
 create mode 100644 tests/generic/651.out
tests/generic/651 new file +53 -0
new file mode 100755
index 00000000..8d4e6728
@@ -0,0 +1,53 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test 651
+#
+# Test that COW writeback that overlaps non-shared delalloc blocks does not
+# leave around stale delalloc blocks on I/O failure. This triggers assert
+# failures and free space accounting corruption on XFS.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+_cleanup()
+{
+	_cleanup_flakey
+	cd /
+	rm -r -f $tmp.*
+}
+
+# Import common functions.
+. ./common/reflink
+. ./common/dmflakey
+
+# real QA test starts here
+_supported_fs generic
+_require_scratch_reflink
+_require_flakey_with_error_writes
+
+_scratch_mkfs >> $seqres.full
+_init_flakey
+_mount_flakey
+
+# create two files that share a single block
+$XFS_IO_PROG -fc "pwrite 4k 4k" $SCRATCH_MNT/file1 >> $seqres.full
+cp --reflink $SCRATCH_MNT/file1 $SCRATCH_MNT/file2
+
+# Perform a buffered write across the shared and non-shared blocks. On XFS, this
+# creates a COW fork extent that covers the shared block as well as the just
+# created non-shared delalloc block. Fail the writeback to verify that all
+# delayed allocation is cleaned up properly.
+_load_flakey_table $FLAKEY_ERROR_WRITES
+$XFS_IO_PROG -c "pwrite 0 8k" -c fsync $SCRATCH_MNT/file2 >> $seqres.full
+_load_flakey_table $FLAKEY_ALLOW_WRITES
+
+# Try a post-fail reflink and then unmount. Both of these are known to produce
+# errors and/or assert failures on XFS if we trip over a stale delalloc block.
+cp --reflink $SCRATCH_MNT/file2 $SCRATCH_MNT/file3
+_unmount_flakey
+
+# success, all done
+status=0
+exit
tests/generic/651.out new file +2 -0
new file mode 100644
index 00000000..bd44c80c
@@ -0,0 +1,2 @@
+QA output created by 651
+fsync: Input/output error
Signature · 1 line
2.31.1
#1 lore raw
Quoted 44 lines · Brian Foster

Test that COW writeback that overlaps non-shared delalloc blocks does not leave around stale delalloc blocks on I/O failure. This triggers assert failures and free space accounting corruption on XFS.

---

This test targets the problem addressed by the following patch in XFS:

https://lore.kernel.org/linux-xfs/[email protected]/

Brian

 tests/generic/651     | 53 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/651.out |  2 ++
 2 files changed, 55 insertions(+)
 create mode 100755 tests/generic/651
 create mode 100644 tests/generic/651.out
tests/generic/651 new file +18 -0
new file mode 100755
index 00000000..8d4e6728
@@ -0,0 +1,53 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test 651
+#
+# Test that COW writeback that overlaps non-shared delalloc blocks does not
+# leave around stale delalloc blocks on I/O failure. This triggers assert
+# failures and free space accounting corruption on XFS.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+_cleanup()
+{
+	_cleanup_flakey
+	cd /
+	rm -r -f $tmp.*

tests/generic/651 line 28

Quoted 10 lines
tests/generic/651 new file +10 -0
+}
+
+# Import common functions.
+. ./common/reflink
+. ./common/dmflakey
+
+# real QA test starts here
+_supported_fs generic
+_require_scratch_reflink
+_require_flakey_with_error_writes

_require_cp_reflink

tests/generic/651 line 35

Quoted 7 lines
tests/generic/651 new file +7 -0
+
+_scratch_mkfs >> $seqres.full
+_init_flakey
+_mount_flakey
+
+# create two files that share a single block
+$XFS_IO_PROG -fc "pwrite 4k 4k" $SCRATCH_MNT/file1 >> $seqres.full

Please use:

blksz=$(_get_file_block_size $SCRATCH_MNT)
$XFS_IO_PROG -fc "pwrite $blksz $blksz" $SCRATCH_MNT/file1 >> $seqres.full

So that this test will work properly on filesystems with bs > 4k.

tests/generic/651 line 36

Quoted 1 line
tests/generic/651 new file +1 -0
+cp --reflink $SCRATCH_MNT/file1 $SCRATCH_MNT/file2

Nit: This could be shortened to use the _cp_reflink helper, though it doesn't really matter to me if you do.

tests/generic/651 line 39

Quoted 2 lines
tests/generic/651 new file +2 -0
+# Perform a buffered write across the shared and non-shared blocks. On XFS, this
+# creates a COW fork extent that covers the shared block as well as the just

Ah, the reason why there's a cow fork extent covering the delalloc reservation is due to the default cow extent size hint, right? In that case, you need:

_require_xfs_io_command "cowextsize"
$XFS_IO_PROG -c "cowextsize 0" $SCRATCH_MNT >> $seqres.full

to ensure that the speculative cow preallocation actually gets set up. Otherwise, I think test won't reproduce the bug if the test config has -d cowextsize=1 in the mkfs options.

tests/generic/651 line 43

Quoted 4 lines
tests/generic/651 new file +4 -0
+# created non-shared delalloc block. Fail the writeback to verify that all
+# delayed allocation is cleaned up properly.
+_load_flakey_table $FLAKEY_ERROR_WRITES
+$XFS_IO_PROG -c "pwrite 0 8k" -c fsync $SCRATCH_MNT/file2 >> $seqres.full

$((2 * blksz)), not 8k

Other than that, this looks reasonable to me. I'll go look at the fix patch now. :)

--D

Quoted 21 lines
tests/generic/651 new file +10 -0
+_load_flakey_table $FLAKEY_ALLOW_WRITES
+
+# Try a post-fail reflink and then unmount. Both of these are known to produce
+# errors and/or assert failures on XFS if we trip over a stale delalloc block.
+cp --reflink $SCRATCH_MNT/file2 $SCRATCH_MNT/file3
+_unmount_flakey
+
+# success, all done
+status=0
+exit
tests/generic/651.out new file +2 -0
new file mode 100644
index 00000000..bd44c80c
@@ -0,0 +1,2 @@
+QA output created by 651
+fsync: Input/output error
Signature · 1 line
2.31.1
#2 lore raw
Quoted 68 lines · Darrick J. Wong
Quoted 44 lines · Brian Foster

Test that COW writeback that overlaps non-shared delalloc blocks does not leave around stale delalloc blocks on I/O failure. This triggers assert failures and free space accounting corruption on XFS.

---

This test targets the problem addressed by the following patch in XFS:

https://lore.kernel.org/linux-xfs/[email protected]/

Brian

 tests/generic/651     | 53 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/651.out |  2 ++
 2 files changed, 55 insertions(+)
 create mode 100755 tests/generic/651
 create mode 100644 tests/generic/651.out
tests/generic/651 new file +18 -0
new file mode 100755
index 00000000..8d4e6728
@@ -0,0 +1,53 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test 651
+#
+# Test that COW writeback that overlaps non-shared delalloc blocks does not
+# leave around stale delalloc blocks on I/O failure. This triggers assert
+# failures and free space accounting corruption on XFS.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+_cleanup()
+{
+	_cleanup_flakey
+	cd /
+	rm -r -f $tmp.*

tests/generic/651 line 28

Quoted 10 lines
tests/generic/651 new file +10 -0
+}
+
+# Import common functions.
+. ./common/reflink
+. ./common/dmflakey
+
+# real QA test starts here
+_supported_fs generic
+_require_scratch_reflink
+_require_flakey_with_error_writes

_require_cp_reflink

tests/generic/651 line 35

Quoted 7 lines
tests/generic/651 new file +7 -0
+
+_scratch_mkfs >> $seqres.full
+_init_flakey
+_mount_flakey
+
+# create two files that share a single block
+$XFS_IO_PROG -fc "pwrite 4k 4k" $SCRATCH_MNT/file1 >> $seqres.full

Please use:

Quoted 5 lines

blksz=$(_get_file_block_size $SCRATCH_MNT)
$XFS_IO_PROG -fc "pwrite $blksz $blksz" $SCRATCH_MNT/file1 >> $seqres.full

So that this test will work properly on filesystems with bs > 4k.

Yeah, I'll fix the various hardcoded sizes. Thanks.

Quoted 5 lines

tests/generic/651 line 36

Quoted 1 line
tests/generic/651 new file +1 -0
+cp --reflink $SCRATCH_MNT/file1 $SCRATCH_MNT/file2

Nit: This could be shortened to use the _cp_reflink helper, though it doesn't really matter to me if you do.

Didn't know we had it. I'll look into it.

Quoted 7 lines

tests/generic/651 line 39

Quoted 2 lines
tests/generic/651 new file +2 -0
+# Perform a buffered write across the shared and non-shared blocks. On XFS, this
+# creates a COW fork extent that covers the shared block as well as the just

Ah, the reason why there's a cow fork extent covering the delalloc reservation is due to the default cow extent size hint, right? In that case, you need:

Yeah..

Quoted 7 lines

_require_xfs_io_command "cowextsize"
$XFS_IO_PROG -c "cowextsize 0" $SCRATCH_MNT >> $seqres.full

to ensure that the speculative cow preallocation actually gets set up. Otherwise, I think test won't reproduce the bug if the test config has -d cowextsize=1 in the mkfs options.

.. but then we aren't susceptible to the problem, right?

I sometimes waffle on whether it's better for a test to create a problematic situation and test it, or run on the configuration specified by the user and test a particular scenario against that. Maybe the former makes more sense in this very specific test case, but then I suppose "cowextsize blksz*2" (or whatever large enough value) is probably more robust than "cowextsize 0" (which I assume means "default" and thus can change, right)?

Brian

Quoted 34 lines

tests/generic/651 line 43

Quoted 4 lines
tests/generic/651 new file +4 -0
+# created non-shared delalloc block. Fail the writeback to verify that all
+# delayed allocation is cleaned up properly.
+_load_flakey_table $FLAKEY_ERROR_WRITES
+$XFS_IO_PROG -c "pwrite 0 8k" -c fsync $SCRATCH_MNT/file2 >> $seqres.full

$((2 * blksz)), not 8k

Other than that, this looks reasonable to me. I'll go look at the fix patch now. :)

--D

Quoted 21 lines
tests/generic/651 new file +10 -0
+_load_flakey_table $FLAKEY_ALLOW_WRITES
+
+# Try a post-fail reflink and then unmount. Both of these are known to produce
+# errors and/or assert failures on XFS if we trip over a stale delalloc block.
+cp --reflink $SCRATCH_MNT/file2 $SCRATCH_MNT/file3
+_unmount_flakey
+
+# success, all done
+status=0
+exit
tests/generic/651.out new file +2 -0
new file mode 100644
index 00000000..bd44c80c
@@ -0,0 +1,2 @@
+QA output created by 651
+fsync: Input/output error
Signature · 1 line
2.31.1
#3 lore raw
Quoted 103 lines · Brian Foster
Quoted 68 lines · Darrick J. Wong
Quoted 44 lines · Brian Foster

Test that COW writeback that overlaps non-shared delalloc blocks does not leave around stale delalloc blocks on I/O failure. This triggers assert failures and free space accounting corruption on XFS.

---

This test targets the problem addressed by the following patch in XFS:

https://lore.kernel.org/linux-xfs/[email protected]/

Brian

 tests/generic/651     | 53 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/651.out |  2 ++
 2 files changed, 55 insertions(+)
 create mode 100755 tests/generic/651
 create mode 100644 tests/generic/651.out
tests/generic/651 new file +18 -0
new file mode 100755
index 00000000..8d4e6728
@@ -0,0 +1,53 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test 651
+#
+# Test that COW writeback that overlaps non-shared delalloc blocks does not
+# leave around stale delalloc blocks on I/O failure. This triggers assert
+# failures and free space accounting corruption on XFS.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+_cleanup()
+{
+	_cleanup_flakey
+	cd /
+	rm -r -f $tmp.*

tests/generic/651 line 28

Quoted 10 lines
tests/generic/651 new file +10 -0
+}
+
+# Import common functions.
+. ./common/reflink
+. ./common/dmflakey
+
+# real QA test starts here
+_supported_fs generic
+_require_scratch_reflink
+_require_flakey_with_error_writes

_require_cp_reflink

tests/generic/651 line 35

Quoted 7 lines
tests/generic/651 new file +7 -0
+
+_scratch_mkfs >> $seqres.full
+_init_flakey
+_mount_flakey
+
+# create two files that share a single block
+$XFS_IO_PROG -fc "pwrite 4k 4k" $SCRATCH_MNT/file1 >> $seqres.full

Please use:

Quoted 5 lines

blksz=$(_get_file_block_size $SCRATCH_MNT)
$XFS_IO_PROG -fc "pwrite $blksz $blksz" $SCRATCH_MNT/file1 >> $seqres.full

So that this test will work properly on filesystems with bs > 4k.

Yeah, I'll fix the various hardcoded sizes. Thanks.

Quoted 5 lines

tests/generic/651 line 36

Quoted 1 line
tests/generic/651 new file +1 -0
+cp --reflink $SCRATCH_MNT/file1 $SCRATCH_MNT/file2

Nit: This could be shortened to use the _cp_reflink helper, though it doesn't really matter to me if you do.

Didn't know we had it. I'll look into it.

Quoted 7 lines

tests/generic/651 line 39

Quoted 2 lines
tests/generic/651 new file +2 -0
+# Perform a buffered write across the shared and non-shared blocks. On XFS, this
+# creates a COW fork extent that covers the shared block as well as the just

Ah, the reason why there's a cow fork extent covering the delalloc reservation is due to the default cow extent size hint, right? In that case, you need:

Yeah..

Quoted 7 lines

_require_xfs_io_command "cowextsize"
$XFS_IO_PROG -c "cowextsize 0" $SCRATCH_MNT >> $seqres.full

to ensure that the speculative cow preallocation actually gets set up. Otherwise, I think test won't reproduce the bug if the test config has -d cowextsize=1 in the mkfs options.

Quoted 7 lines

.. but then we aren't susceptible to the problem, right?

I sometimes waffle on whether it's better for a test to create a problematic situation and test it, or run on the configuration specified by the user and test a particular scenario against that. Maybe the former makes more sense in this very specific test case, but then I suppose "cowextsize blksz*2" (or whatever large enough value) is

Yes, blksz*2.

Quoted 2 lines

probably more robust than "cowextsize 0" (which I assume means "default" and thus can change, right)?

Seeing as this is a reproducer, explicitly setting cowextsize seems appropriate.

Alternately, I suppose you could detect the one case where it won't work (cowextsize == 1fsb) and only then change it.

--D

Quoted 38 lines

Brian

Quoted 34 lines

tests/generic/651 line 43

Quoted 4 lines
tests/generic/651 new file +4 -0
+# created non-shared delalloc block. Fail the writeback to verify that all
+# delayed allocation is cleaned up properly.
+_load_flakey_table $FLAKEY_ERROR_WRITES
+$XFS_IO_PROG -c "pwrite 0 8k" -c fsync $SCRATCH_MNT/file2 >> $seqres.full

$((2 * blksz)), not 8k

Other than that, this looks reasonable to me. I'll go look at the fix patch now. :)

--D

Quoted 21 lines
tests/generic/651 new file +10 -0
+_load_flakey_table $FLAKEY_ALLOW_WRITES
+
+# Try a post-fail reflink and then unmount. Both of these are known to produce
+# errors and/or assert failures on XFS if we trip over a stale delalloc block.
+cp --reflink $SCRATCH_MNT/file2 $SCRATCH_MNT/file3
+_unmount_flakey
+
+# success, all done
+status=0
+exit
tests/generic/651.out new file +2 -0
new file mode 100644
index 00000000..bd44c80c
@@ -0,0 +1,2 @@
+QA output created by 651
+fsync: Input/output error
Signature · 1 line
2.31.1