Skip to contentSkip to conversation

generic: add test for missing btrfs csums in log when doing async on subpage vol

PATCH In review Mark Harmstone 5 participants 10 messages Oct 15, 2024 – Oct 29, 2024 Btrfs file system

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 generic: add test for missing btrfs csums in log when doing async on subpage vol

Adds a test for a bug we encountered on Linux 6.4 on aarch64, where a race could mean that csums weren't getting written to the log tree, leading to corruption when it was replayed.

The patches to detect log this tree corruption are in btrfs-progs 6.11.

---
This is a genericized version of the test I originally proposed as btrfs/333.

 tests/generic/757     | 71 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/757.out |  2 ++
 2 files changed, 73 insertions(+)
 create mode 100755 tests/generic/757
 create mode 100644 tests/generic/757.out
tests/generic/757 new file +71 -0
new file mode 100755
index 00000000..6ad3d01e
@@ -0,0 +1,71 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 757
+#
+# Test async dio with fsync to test a btrfs bug where a race meant that csums
+# weren't getting written to the log tree, causing corruptions on remount.
+# This can be seen on subpage FSes on Linux 6.4.
+#
+. ./common/preamble
+_begin_fstest auto quick metadata log recoveryloop
+
+_fixed_by_kernel_commit e917ff56c8e7 \
+	"btrfs: determine synchronous writers from bio or writeback control"
+
+fio_config=$tmp.fio
+
+. ./common/dmlogwrites
+
+_require_scratch
+_require_log_writes
+
+cat >$fio_config <<EOF
+[global]
+iodepth=128
+direct=1
+ioengine=libaio
+rw=randwrite
+runtime=1s
+[job0]
+rw=randwrite
+filename=$SCRATCH_MNT/file
+size=1g
+fdatasync=1
+EOF
+
+_require_fio $fio_config
+
+cat $fio_config >> $seqres.full
+
+_log_writes_init $SCRATCH_DEV
+_log_writes_mkfs >> $seqres.full 2>&1
+_log_writes_mark mkfs
+
+_log_writes_mount
+
+$FIO_PROG $fio_config > /dev/null 2>&1
+_log_writes_unmount
+
+_log_writes_remove
+
+prev=$(_log_writes_mark_to_entry_number mkfs)
+[ -z "$prev" ] && _fail "failed to locate entry mark 'mkfs'"
+cur=$(_log_writes_find_next_fua $prev)
+[ -z "$cur" ] && _fail "failed to locate next FUA write"
+
+while [ ! -z "$cur" ]; do
+	_log_writes_replay_log_range $cur $SCRATCH_DEV >> $seqres.full
+
+	_check_scratch_fs
+
+	prev=$cur
+	cur=$(_log_writes_find_next_fua $(($cur + 1)))
+	[ -z "$cur" ] && break
+done
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/generic/757.out new file +2 -0
new file mode 100644
index 00000000..dfbc8094
@@ -0,0 +1,2 @@
+QA output created by 757
+Silence is golden
Signature · 1 line
2.44.2
Reviewed-by on v1 1/1
#1 lore raw
Quoted 6 lines · Mark Harmstone

Adds a test for a bug we encountered on Linux 6.4 on aarch64, where a race could mean that csums weren't getting written to the log tree, leading to corruption when it was replayed.

The patches to detect log this tree corruption are in btrfs-progs 6.11.

This shouldn't be needed right?
Because after log replay the csums are missing and 'btrfs check' detects (IIRC) missing csums for extents referred by file extent items in a subvolume tree - if it doesn't then it should be improved.

Quoted 22 lines

---
This is a genericized version of the test I originally proposed as btrfs/333.

 tests/generic/757     | 71 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/757.out |  2 ++
 2 files changed, 73 insertions(+)
 create mode 100755 tests/generic/757
 create mode 100644 tests/generic/757.out
tests/generic/757 new file +4 -0
new file mode 100755
index 00000000..6ad3d01e
@@ -0,0 +1,71 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 757

tests/generic/757 line 14

Quoted 10 lines
tests/generic/757 new file +10 -0
+#
+# Test async dio with fsync to test a btrfs bug where a race meant that csums
+# weren't getting written to the log tree, causing corruptions on remount.
+# This can be seen on subpage FSes on Linux 6.4.
+#
+. ./common/preamble
+_begin_fstest auto quick metadata log recoveryloop
+
+_fixed_by_kernel_commit e917ff56c8e7 \
+       "btrfs: determine synchronous writers from bio or writeback control"

For generic tests what we do is:

[ $FSTYP == "btrfs" ] && _fixed_by_kernel_commit .....

As long as the failure has not been observed and fixed on other filesystems. In case one day a regression happens in another fs, we just add a corresponding line using the same logic.

Otherwise if the test one days fails on another fs and fstests suggests that that commit is missing, it would be odd.

Everything else looks good, so with that fixed (maybe Zorro can change that when picking the patch):

Thanks.

Quoted 69 lines
tests/generic/757 new file +57 -0
+
+fio_config=$tmp.fio
+
+. ./common/dmlogwrites
+
+_require_scratch
+_require_log_writes
+
+cat >$fio_config <<EOF
+[global]
+iodepth=128
+direct=1
+ioengine=libaio
+rw=randwrite
+runtime=1s
+[job0]
+rw=randwrite
+filename=$SCRATCH_MNT/file
+size=1g
+fdatasync=1
+EOF
+
+_require_fio $fio_config
+
+cat $fio_config >> $seqres.full
+
+_log_writes_init $SCRATCH_DEV
+_log_writes_mkfs >> $seqres.full 2>&1
+_log_writes_mark mkfs
+
+_log_writes_mount
+
+$FIO_PROG $fio_config > /dev/null 2>&1
+_log_writes_unmount
+
+_log_writes_remove
+
+prev=$(_log_writes_mark_to_entry_number mkfs)
+[ -z "$prev" ] && _fail "failed to locate entry mark 'mkfs'"
+cur=$(_log_writes_find_next_fua $prev)
+[ -z "$cur" ] && _fail "failed to locate next FUA write"
+
+while [ ! -z "$cur" ]; do
+       _log_writes_replay_log_range $cur $SCRATCH_DEV >> $seqres.full
+
+       _check_scratch_fs
+
+       prev=$cur
+       cur=$(_log_writes_find_next_fua $(($cur + 1)))
+       [ -z "$cur" ] && break
+done
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/generic/757.out new file +2 -1
new file mode 100644
index 00000000..dfbc8094
@@ -0,0 +1,2 @@
+QA output created by 757
+Silence is golden
--

2.44.2

#2 lore raw
Quoted 7 lines · Filipe Manana
Quoted 1 line
Quoted 5 lines · Mark Harmstone

Adds a test for a bug we encountered on Linux 6.4 on aarch64, where a race could mean that csums weren't getting written to the log tree, leading to corruption when it was replayed.

Quoted 6 lines
Quoted 1 line

The patches to detect log this tree corruption are in btrfs-progs 6.11.

This shouldn't be needed right?
Because after log replay the csums are missing and 'btrfs check' detects (IIRC) missing csums for extents referred by file extent items in a subvolume tree - if it doesn't then it should be improved.

Yes, but we're not mounting it in the tests between the log_writes calls, so the log isn't getting replayed. The patches to btrfs check make it so that it identifies filesystems that would get corrupted as soon as they're next mounted.

Quoted 45 lines
Quoted 23 lines

---
This is a genericized version of the test I originally proposed as btrfs/333.

  tests/generic/757     | 71 +++++++++++++++++++++++++++++++++++++++++++
  tests/generic/757.out |  2 ++
  2 files changed, 73 insertions(+)

create mode 100755 tests/generic/757
create mode 100644 tests/generic/757.out

tests/generic/757 new file +4 -0
new file mode 100755
index 00000000..6ad3d01e
@@ -0,0 +1,71 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 757

tests/generic/757 line 14

Quoted 10 lines
tests/generic/757 new file +10 -0
+#
+# Test async dio with fsync to test a btrfs bug where a race meant that csums
+# weren't getting written to the log tree, causing corruptions on remount.
+# This can be seen on subpage FSes on Linux 6.4.
+#
+. ./common/preamble
+_begin_fstest auto quick metadata log recoveryloop
+
+_fixed_by_kernel_commit e917ff56c8e7 \
+       "btrfs: determine synchronous writers from bio or writeback control"

For generic tests what we do is:

[ $FSTYP == "btrfs" ] && _fixed_by_kernel_commit .....

As long as the failure has not been observed and fixed on other filesystems. In case one day a regression happens in another fs, we just add a corresponding line using the same logic.

Otherwise if the test one days fails on another fs and fstests suggests that that commit is missing, it would be odd.

Quoted 7 lines

Everything else looks good, so with that fixed (maybe Zorro can change that when picking the patch):

Thanks.

Thanks Filipe. Zorro, let me know if you're happy making this change, or otherwise I'll resubmit.

Mark

#3 lore raw
Quoted 68 lines · Mark Harmstone
Quoted 7 lines · Filipe Manana
Quoted 1 line
Quoted 5 lines · Mark Harmstone

Adds a test for a bug we encountered on Linux 6.4 on aarch64, where a race could mean that csums weren't getting written to the log tree, leading to corruption when it was replayed.

Quoted 6 lines
Quoted 1 line

The patches to detect log this tree corruption are in btrfs-progs 6.11.

This shouldn't be needed right?
Because after log replay the csums are missing and 'btrfs check' detects (IIRC) missing csums for extents referred by file extent items in a subvolume tree - if it doesn't then it should be improved.

Yes, but we're not mounting it in the tests between the log_writes calls, so the log isn't getting replayed. The patches to btrfs check make it so that it identifies filesystems that would get corrupted as soon as they're next mounted.

Quoted 48 lines
Quoted 23 lines

---
This is a genericized version of the test I originally proposed as btrfs/333.

  tests/generic/757     | 71 +++++++++++++++++++++++++++++++++++++++++++
  tests/generic/757.out |  2 ++
  2 files changed, 73 insertions(+)

create mode 100755 tests/generic/757
create mode 100644 tests/generic/757.out

tests/generic/757 new file +4 -0
new file mode 100755
index 00000000..6ad3d01e
@@ -0,0 +1,71 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 757

tests/generic/757 line 14

Quoted 10 lines
tests/generic/757 new file +10 -0
+#
+# Test async dio with fsync to test a btrfs bug where a race meant that csums
+# weren't getting written to the log tree, causing corruptions on remount.
+# This can be seen on subpage FSes on Linux 6.4.
+#
+. ./common/preamble
+_begin_fstest auto quick metadata log recoveryloop
+
+_fixed_by_kernel_commit e917ff56c8e7 \
+       "btrfs: determine synchronous writers from bio or writeback control"

For generic tests what we do is:

[ $FSTYP == "btrfs" ] && _fixed_by_kernel_commit .....

As long as the failure has not been observed and fixed on other filesystems. In case one day a regression happens in another fs, we just add a corresponding line using the same logic.

Otherwise if the test one days fails on another fs and fstests suggests that that commit is missing, it would be odd.

Everything else looks good, so with that fixed (maybe Zorro can change that when picking the patch):

Quoted 7 lines
Quoted 4 lines

Thanks.

Thanks Filipe. Zorro, let me know if you're happy making this change, or otherwise I'll resubmit.

I can help to change that when I merge it. Thanks you and Filipe.

Thanks,
Zorro

Quoted 4 lines

Mark

#4 lore raw
In reply to Mark Harmstone
Quoted 58 lines · Mark Harmstone

Adds a test for a bug we encountered on Linux 6.4 on aarch64, where a race could mean that csums weren't getting written to the log tree, leading to corruption when it was replayed.

The patches to detect log this tree corruption are in btrfs-progs 6.11.

---
This is a genericized version of the test I originally proposed as btrfs/333.

 tests/generic/757     | 71 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/757.out |  2 ++
 2 files changed, 73 insertions(+)
 create mode 100755 tests/generic/757
 create mode 100644 tests/generic/757.out
tests/generic/757 new file +35 -0
new file mode 100755
index 00000000..6ad3d01e
@@ -0,0 +1,71 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 757
+#
+# Test async dio with fsync to test a btrfs bug where a race meant that csums
+# weren't getting written to the log tree, causing corruptions on remount.
+# This can be seen on subpage FSes on Linux 6.4.
+#
+. ./common/preamble
+_begin_fstest auto quick metadata log recoveryloop
+
+_fixed_by_kernel_commit e917ff56c8e7 \
+	"btrfs: determine synchronous writers from bio or writeback control"
+
+fio_config=$tmp.fio
+
+. ./common/dmlogwrites
+
+_require_scratch
+_require_log_writes
+
+cat >$fio_config <<EOF
+[global]
+iodepth=128
+direct=1
+ioengine=libaio
+rw=randwrite
+runtime=1s
+[job0]
+rw=randwrite
+filename=$SCRATCH_MNT/file
+size=1g
+fdatasync=1
+EOF

tests/generic/757 line 45

Quoted 10 lines
tests/generic/757 new file +10 -0
+
+_require_fio $fio_config
+
+cat $fio_config >> $seqres.full
+
+_log_writes_init $SCRATCH_DEV
+_log_writes_mkfs >> $seqres.full 2>&1
+_log_writes_mark mkfs
+
+_log_writes_mount

For dmlogwrites test, we generally calls _log_writes_cleanup in _cleanup, to recover the SCRATCH_DEV anyway, even if this test is killed at the middle of its testing phase, to avoid later tests failed.

I'll add below code in this case, when I merge it. If there's not objection from you.

_cleanup()

{
	cd /
	_log_writes_cleanup
	rm -f $tmp.*
}

Thanks,
Zorro

Quoted 38 lines
tests/generic/757 new file +26 -0
+
+$FIO_PROG $fio_config > /dev/null 2>&1
+_log_writes_unmount
+
+_log_writes_remove
+
+prev=$(_log_writes_mark_to_entry_number mkfs)
+[ -z "$prev" ] && _fail "failed to locate entry mark 'mkfs'"
+cur=$(_log_writes_find_next_fua $prev)
+[ -z "$cur" ] && _fail "failed to locate next FUA write"
+
+while [ ! -z "$cur" ]; do
+	_log_writes_replay_log_range $cur $SCRATCH_DEV >> $seqres.full
+
+	_check_scratch_fs
+
+	prev=$cur
+	cur=$(_log_writes_find_next_fua $(($cur + 1)))
+	[ -z "$cur" ] && break
+done
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/generic/757.out new file +2 -0
new file mode 100644
index 00000000..dfbc8094
@@ -0,0 +1,2 @@
+QA output created by 757
+Silence is golden
Signature · 1 line
2.44.2
#5 lore raw
In reply to Mark Harmstone
Quoted 8 lines · Mark Harmstone

Adds a test for a bug we encountered on Linux 6.4 on aarch64, where a race could mean that csums weren't getting written to the log tree, leading to corruption when it was replayed.

The patches to detect log this tree corruption are in btrfs-progs 6.11.

---

Sorry, more review points below. I can help to change these if you say "yes" to all :)

Quoted 16 lines

This is a genericized version of the test I originally proposed as btrfs/333.

 tests/generic/757     | 71 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/757.out |  2 ++
 2 files changed, 73 insertions(+)
 create mode 100755 tests/generic/757
 create mode 100644 tests/generic/757.out
tests/generic/757 new file +1 -0
new file mode 100755
index 00000000..6ad3d01e
@@ -0,0 +1,71 @@
+#! /bin/bash

tests/generic/757 line 11

Quoted 10 lines
tests/generic/757 new file +10 -0
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 757
+#
+# Test async dio with fsync to test a btrfs bug where a race meant that csums
+# weren't getting written to the log tree, causing corruptions on remount.
+# This can be seen on subpage FSes on Linux 6.4.
+#
+. ./common/preamble
+_begin_fstest auto quick metadata log recoveryloop
                                                      ^^^
                                                      aio
Quoted 6 lines
tests/generic/757 new file +6 -0
+
+_fixed_by_kernel_commit e917ff56c8e7 \
+	"btrfs: determine synchronous writers from bio or writeback control"
+
+fio_config=$tmp.fio
+

tests/generic/757 line 27

Quoted 10 lines
tests/generic/757 new file +10 -0
+. ./common/dmlogwrites
+
+_require_scratch
+_require_log_writes
+
+cat >$fio_config <<EOF
+[global]
+iodepth=128
+direct=1
+ioengine=libaio

_require_aiodio ?

Quoted 10 lines
tests/generic/757 new file +10 -0
+rw=randwrite
+runtime=1s
+[job0]
+rw=randwrite
+filename=$SCRATCH_MNT/file
+size=1g
+fdatasync=1
+EOF
+
+_require_fio $fio_config

tests/generic/757 line 47

Quoted 10 lines
tests/generic/757 new file +10 -0
+
+cat $fio_config >> $seqres.full
+
+_log_writes_init $SCRATCH_DEV
+_log_writes_mkfs >> $seqres.full 2>&1
+_log_writes_mark mkfs
+
+_log_writes_mount
+
+$FIO_PROG $fio_config > /dev/null 2>&1

Don't you care the output of fio running anymore? Maybe use > $seqres.full ?

And just make sure, do you want to ignore failures of fio, as you do "2>&1"?

Thanks,
Zorro

Quoted 36 lines
tests/generic/757 new file +24 -0
+_log_writes_unmount
+
+_log_writes_remove
+
+prev=$(_log_writes_mark_to_entry_number mkfs)
+[ -z "$prev" ] && _fail "failed to locate entry mark 'mkfs'"
+cur=$(_log_writes_find_next_fua $prev)
+[ -z "$cur" ] && _fail "failed to locate next FUA write"
+
+while [ ! -z "$cur" ]; do
+	_log_writes_replay_log_range $cur $SCRATCH_DEV >> $seqres.full
+
+	_check_scratch_fs
+
+	prev=$cur
+	cur=$(_log_writes_find_next_fua $(($cur + 1)))
+	[ -z "$cur" ] && break
+done
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/generic/757.out new file +2 -0
new file mode 100644
index 00000000..dfbc8094
@@ -0,0 +1,2 @@
+QA output created by 757
+Silence is golden
Signature · 1 line
2.44.2
#6 lore raw

These look good to me. Thank you for your help

Quoted 127 lines · Zorro Lang
Quoted 1 line
Quoted 8 lines · Mark Harmstone

Adds a test for a bug we encountered on Linux 6.4 on aarch64, where a race could mean that csums weren't getting written to the log tree, leading to corruption when it was replayed.

The patches to detect log this tree corruption are in btrfs-progs 6.11.

---

Sorry, more review points below. I can help to change these if you say "yes" to all :)

Quoted 16 lines

This is a genericized version of the test I originally proposed as btrfs/333.

  tests/generic/757     | 71 +++++++++++++++++++++++++++++++++++++++++++
  tests/generic/757.out |  2 ++
  2 files changed, 73 insertions(+)

create mode 100755 tests/generic/757
create mode 100644 tests/generic/757.out

tests/generic/757 new file +1 -0
new file mode 100755
index 00000000..6ad3d01e
@@ -0,0 +1,71 @@
+#! /bin/bash

tests/generic/757 line 11

Quoted 10 lines
tests/generic/757 new file +10 -0
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 757
+#
+# Test async dio with fsync to test a btrfs bug where a race meant that csums
+# weren't getting written to the log tree, causing corruptions on remount.
+# This can be seen on subpage FSes on Linux 6.4.
+#
+. ./common/preamble
+_begin_fstest auto quick metadata log recoveryloop
                                                       ^^^
                                                       aio
Quoted 6 lines
tests/generic/757 new file +6 -0
+
+_fixed_by_kernel_commit e917ff56c8e7 \
+	"btrfs: determine synchronous writers from bio or writeback control"
+
+fio_config=$tmp.fio
+

tests/generic/757 line 27

Quoted 10 lines
tests/generic/757 new file +10 -0
+. ./common/dmlogwrites
+
+_require_scratch
+_require_log_writes
+
+cat >$fio_config <<EOF
+[global]
+iodepth=128
+direct=1
+ioengine=libaio

_require_aiodio ?

Quoted 10 lines
tests/generic/757 new file +10 -0
+rw=randwrite
+runtime=1s
+[job0]
+rw=randwrite
+filename=$SCRATCH_MNT/file
+size=1g
+fdatasync=1
+EOF
+
+_require_fio $fio_config

tests/generic/757 line 47

Quoted 10 lines
tests/generic/757 new file +10 -0
+
+cat $fio_config >> $seqres.full
+
+_log_writes_init $SCRATCH_DEV
+_log_writes_mkfs >> $seqres.full 2>&1
+_log_writes_mark mkfs
+
+_log_writes_mount
+
+$FIO_PROG $fio_config > /dev/null 2>&1

Don't you care the output of fio running anymore? Maybe use > $seqres.full ?

And just make sure, do you want to ignore failures of fio, as you do "2>&1"?

Thanks,
Zorro

Quoted 36 lines
tests/generic/757 new file +24 -0
+_log_writes_unmount
+
+_log_writes_remove
+
+prev=$(_log_writes_mark_to_entry_number mkfs)
+[ -z "$prev" ] && _fail "failed to locate entry mark 'mkfs'"
+cur=$(_log_writes_find_next_fua $prev)
+[ -z "$cur" ] && _fail "failed to locate next FUA write"
+
+while [ ! -z "$cur" ]; do
+	_log_writes_replay_log_range $cur $SCRATCH_DEV >> $seqres.full
+
+	_check_scratch_fs
+
+	prev=$cur
+	cur=$(_log_writes_find_next_fua $(($cur + 1)))
+	[ -z "$cur" ] && break
+done
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/generic/757.out new file +2 -0
new file mode 100644
index 00000000..dfbc8094
@@ -0,0 +1,2 @@
+QA output created by 757
+Silence is golden
Signature · 1 line
2.44.2
#7 lore raw
In reply to Mark Harmstone
Quoted 73 lines · Mark Harmstone

Adds a test for a bug we encountered on Linux 6.4 on aarch64, where a race could mean that csums weren't getting written to the log tree, leading to corruption when it was replayed.

The patches to detect log this tree corruption are in btrfs-progs 6.11.

---
This is a genericized version of the test I originally proposed as btrfs/333.

 tests/generic/757     | 71 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/757.out |  2 ++
 2 files changed, 73 insertions(+)
 create mode 100755 tests/generic/757
 create mode 100644 tests/generic/757.out
tests/generic/757 new file +50 -0
new file mode 100755
index 00000000..6ad3d01e
@@ -0,0 +1,71 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 757
+#
+# Test async dio with fsync to test a btrfs bug where a race meant that csums
+# weren't getting written to the log tree, causing corruptions on remount.
+# This can be seen on subpage FSes on Linux 6.4.
+#
+. ./common/preamble
+_begin_fstest auto quick metadata log recoveryloop
+
+_fixed_by_kernel_commit e917ff56c8e7 \
+	"btrfs: determine synchronous writers from bio or writeback control"
+
+fio_config=$tmp.fio
+
+. ./common/dmlogwrites
+
+_require_scratch
+_require_log_writes
+
+cat >$fio_config <<EOF
+[global]
+iodepth=128
+direct=1
+ioengine=libaio
+rw=randwrite
+runtime=1s
+[job0]
+rw=randwrite
+filename=$SCRATCH_MNT/file
+size=1g
+fdatasync=1
+EOF
+
+_require_fio $fio_config
+
+cat $fio_config >> $seqres.full
+
+_log_writes_init $SCRATCH_DEV
+_log_writes_mkfs >> $seqres.full 2>&1
+_log_writes_mark mkfs
+
+_log_writes_mount
+
+$FIO_PROG $fio_config > /dev/null 2>&1
+_log_writes_unmount
+
+_log_writes_remove

tests/generic/757 line 60

Quoted 10 lines
tests/generic/757 new file +10 -0
+
+prev=$(_log_writes_mark_to_entry_number mkfs)
+[ -z "$prev" ] && _fail "failed to locate entry mark 'mkfs'"
+cur=$(_log_writes_find_next_fua $prev)
+[ -z "$cur" ] && _fail "failed to locate next FUA write"
+
+while [ ! -z "$cur" ]; do
+	_log_writes_replay_log_range $cur $SCRATCH_DEV >> $seqres.full
+
+	_check_scratch_fs

This test fails on xfs because (afaict) replaying the log to $cur results in $SCRATCH_DEV being a filesystem with a dirty log; and xfs_repair fails when it is given a filesystem with a dirty log.

I then fixed the test to mount and unmount the filesystem to recovery the dirty log before invoking xfs_repair:

	# xfs_repair won't run if the log is dirty
	if [ $FSTYP = "xfs" ]; then
		_scratch_mount
		_scratch_unmount
	fi
	_check_scratch_fs

But now the test takes a very long time to run because (on my system anyway) the fio run can initiate 17,000 FUAs, which means that this loop runs that many times. 100 iterations takes about 45 seconds, which is about two hours.

Is it necessary to iterate the loop that many times to reproduce whatever issue btrfs had?

--D

Quoted 23 lines
tests/generic/757 new file +11 -0
+
+	prev=$cur
+	cur=$(_log_writes_find_next_fua $(($cur + 1)))
+	[ -z "$cur" ] && break
+done
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/generic/757.out new file +2 -0
new file mode 100644
index 00000000..dfbc8094
@@ -0,0 +1,2 @@
+QA output created by 757
+Silence is golden
Signature · 1 line
2.44.2
#8 lore raw
Quoted 89 lines · Darrick J. Wong
Quoted 73 lines · Mark Harmstone

Adds a test for a bug we encountered on Linux 6.4 on aarch64, where a race could mean that csums weren't getting written to the log tree, leading to corruption when it was replayed.

The patches to detect log this tree corruption are in btrfs-progs 6.11.

---
This is a genericized version of the test I originally proposed as btrfs/333.

 tests/generic/757     | 71 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/757.out |  2 ++
 2 files changed, 73 insertions(+)
 create mode 100755 tests/generic/757
 create mode 100644 tests/generic/757.out
tests/generic/757 new file +50 -0
new file mode 100755
index 00000000..6ad3d01e
@@ -0,0 +1,71 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 757
+#
+# Test async dio with fsync to test a btrfs bug where a race meant that csums
+# weren't getting written to the log tree, causing corruptions on remount.
+# This can be seen on subpage FSes on Linux 6.4.
+#
+. ./common/preamble
+_begin_fstest auto quick metadata log recoveryloop
+
+_fixed_by_kernel_commit e917ff56c8e7 \
+	"btrfs: determine synchronous writers from bio or writeback control"
+
+fio_config=$tmp.fio
+
+. ./common/dmlogwrites
+
+_require_scratch
+_require_log_writes
+
+cat >$fio_config <<EOF
+[global]
+iodepth=128
+direct=1
+ioengine=libaio
+rw=randwrite
+runtime=1s
+[job0]
+rw=randwrite
+filename=$SCRATCH_MNT/file
+size=1g
+fdatasync=1
+EOF
+
+_require_fio $fio_config
+
+cat $fio_config >> $seqres.full
+
+_log_writes_init $SCRATCH_DEV
+_log_writes_mkfs >> $seqres.full 2>&1
+_log_writes_mark mkfs
+
+_log_writes_mount
+
+$FIO_PROG $fio_config > /dev/null 2>&1
+_log_writes_unmount
+
+_log_writes_remove

tests/generic/757 line 60

Quoted 10 lines
tests/generic/757 new file +10 -0
+
+prev=$(_log_writes_mark_to_entry_number mkfs)
+[ -z "$prev" ] && _fail "failed to locate entry mark 'mkfs'"
+cur=$(_log_writes_find_next_fua $prev)
+[ -z "$cur" ] && _fail "failed to locate next FUA write"
+
+while [ ! -z "$cur" ]; do
+	_log_writes_replay_log_range $cur $SCRATCH_DEV >> $seqres.full
+
+	_check_scratch_fs

This test fails on xfs because (afaict) replaying the log to $cur results in $SCRATCH_DEV being a filesystem with a dirty log; and xfs_repair fails when it is given a filesystem with a dirty log.

Quoted 8 lines

I then fixed the test to mount and unmount the filesystem to recovery the dirty log before invoking xfs_repair:

	# xfs_repair won't run if the log is dirty
	if [ $FSTYP = "xfs" ]; then
		_scratch_mount
		_scratch_unmount
	fi

Thanks Darrick, you're right.
I'm wondering can we always do a mount&unmount at here, no matter the $FSTYP, if that doesn't affect the testing of other filesystems?

Quoted 9 lines

_check_scratch_fs

But now the test takes a very long time to run because (on my system anyway) the fio run can initiate 17,000 FUAs, which means that this loop runs that many times. 100 iterations takes about 45 seconds, which is about two hours.

Is it necessary to iterate the loop that many times to reproduce whatever issue btrfs had?

Yes, it takes much long time on my side too:

 FSTYP         -- ext4
 PLATFORM      -- Linux/x86_64 dell-per750-47 6.12.0-rc4+ #1 SMP PREEMPT_DYNAMIC Fri Oct 25 14:25:45 EDT 2024
 MKFS_OPTIONS  -- -F /dev/sda4

MOUNT_OPTIONS -- -o acl,user_xattr -o context=system_u:object_r:root_t:s0 /dev/sda4 /mnt/xfstests/scratch

generic/757 4247s
Ran: generic/757
Passed all 1 tests

So better to reduce the testing time as much as possible, and remove it from the "quick" group. (Maybe we can have a tag to mark those cases need much long time too).

This patch has been merged into for-next branch, as:

cf97fa373 generic: add test for missing btrfs csums in log when doing async on subpage vol

Please send another (or other two) patch to fix above 2 problems.

Thanks,
Zorro

Quoted 27 lines

--D

Quoted 23 lines
tests/generic/757 new file +11 -0
+
+	prev=$cur
+	cur=$(_log_writes_find_next_fua $(($cur + 1)))
+	[ -z "$cur" ] && break
+done
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/generic/757.out new file +2 -0
new file mode 100644
index 00000000..dfbc8094
@@ -0,0 +1,2 @@
+QA output created by 757
+Silence is golden
Signature · 1 line
2.44.2
#9 lore raw
Quoted 119 lines · Zorro Lang
Quoted 89 lines · Darrick J. Wong
Quoted 73 lines · Mark Harmstone

Adds a test for a bug we encountered on Linux 6.4 on aarch64, where a race could mean that csums weren't getting written to the log tree, leading to corruption when it was replayed.

The patches to detect log this tree corruption are in btrfs-progs 6.11.

---
This is a genericized version of the test I originally proposed as btrfs/333.

  tests/generic/757     | 71 +++++++++++++++++++++++++++++++++++++++++++
  tests/generic/757.out |  2 ++
  2 files changed, 73 insertions(+)

create mode 100755 tests/generic/757
create mode 100644 tests/generic/757.out

tests/generic/757 new file +50 -0
new file mode 100755
index 00000000..6ad3d01e
@@ -0,0 +1,71 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test 757
+#
+# Test async dio with fsync to test a btrfs bug where a race meant that csums
+# weren't getting written to the log tree, causing corruptions on remount.
+# This can be seen on subpage FSes on Linux 6.4.
+#
+. ./common/preamble
+_begin_fstest auto quick metadata log recoveryloop
+
+_fixed_by_kernel_commit e917ff56c8e7 \
+	"btrfs: determine synchronous writers from bio or writeback control"
+
+fio_config=$tmp.fio
+
+. ./common/dmlogwrites
+
+_require_scratch
+_require_log_writes
+
+cat >$fio_config <<EOF
+[global]
+iodepth=128
+direct=1
+ioengine=libaio
+rw=randwrite
+runtime=1s
+[job0]
+rw=randwrite
+filename=$SCRATCH_MNT/file
+size=1g
+fdatasync=1
+EOF
+
+_require_fio $fio_config
+
+cat $fio_config >> $seqres.full
+
+_log_writes_init $SCRATCH_DEV
+_log_writes_mkfs >> $seqres.full 2>&1
+_log_writes_mark mkfs
+
+_log_writes_mount
+
+$FIO_PROG $fio_config > /dev/null 2>&1
+_log_writes_unmount
+
+_log_writes_remove

tests/generic/757 line 60

Quoted 10 lines
tests/generic/757 new file +10 -0
+
+prev=$(_log_writes_mark_to_entry_number mkfs)
+[ -z "$prev" ] && _fail "failed to locate entry mark 'mkfs'"
+cur=$(_log_writes_find_next_fua $prev)
+[ -z "$cur" ] && _fail "failed to locate next FUA write"
+
+while [ ! -z "$cur" ]; do
+	_log_writes_replay_log_range $cur $SCRATCH_DEV >> $seqres.full
+
+	_check_scratch_fs

This test fails on xfs because (afaict) replaying the log to $cur results in $SCRATCH_DEV being a filesystem with a dirty log; and xfs_repair fails when it is given a filesystem with a dirty log.

Quoted 8 lines

I then fixed the test to mount and unmount the filesystem to recovery the dirty log before invoking xfs_repair:

	# xfs_repair won't run if the log is dirty
	if [ $FSTYP = "xfs" ]; then
		_scratch_mount
		_scratch_unmount
	fi

Thanks Darrick, you're right.
I'm wondering can we always do a mount&unmount at here, no matter the $FSTYP, if that doesn't affect the testing of other filesystems?

Quoted 9 lines

_check_scratch_fs

But now the test takes a very long time to run because (on my system anyway) the fio run can initiate 17,000 FUAs, which means that this loop runs that many times. 100 iterations takes about 45 seconds, which is about two hours.

Is it necessary to iterate the loop that many times to reproduce whatever issue btrfs had?

Yes, it takes much long time on my side too:

  FSTYP         -- ext4
  PLATFORM      -- Linux/x86_64 dell-per750-47 6.12.0-rc4+ #1 SMP PREEMPT_DYNAMIC Fri Oct 25 14:25:45 EDT 2024
  MKFS_OPTIONS  -- -F /dev/sda4

MOUNT_OPTIONS -- -o acl,user_xattr -o context=system_u:object_r:root_t:s0 /dev/sda4 /mnt/xfstests/scratch

Quoted 7 lines

generic/757 4247s
Ran: generic/757
Passed all 1 tests

So better to reduce the testing time as much as possible, and remove it from the "quick" group. (Maybe we can have a tag to mark those cases need much long time too).

Or maybe it should be made btrfs-specific again? That's how I originally wrote it, but Filipe Manana thought it ought to be genericized.

An advantage of making it FS-specific is that you can use the --fsck option in log-writes, which is about 10 times quicker than doing the loop in bash.

Mark