Skip to contentSkip to conversation

btrfs: add a test case to verify the write behavior of large RAID5 data chunks

PATCH In review Qu Wenruo 6 participants 7 messages Jun 22, 2023 – Jul 24, 2023 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

PATCH 1/1 btrfs: add a test case to verify the write behavior of large RAID5 data chunks

There is a recent regression during v6.4 merge window, that a u32 left shift overflow can cause problems with large data chunks (over 4G) sized.

This is especially nasty for RAID56, which can lead to ASSERT() during regular writes, or corrupt memory if CONFIG_BTRFS_ASSERT is not enabled.

This is the regression test case for it.

Unlike btrfs/292, btrfs doesn't support trim inside RAID56 chunks, thus the workflow is simplified:

- Create a RAID5 or RAID6 data chunk during mkfs

- Fill the fs with 5G data and sync
For unpatched kernel, the sync would crash the kernel.

- Make sure everything is fine

---
 tests/btrfs/293     | 72 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/293.out |  2 ++
 2 files changed, 74 insertions(+)
 create mode 100755 tests/btrfs/293
 create mode 100644 tests/btrfs/293.out
tests/btrfs/293 new file +72 -0
new file mode 100755
index 00000000..68312682
@@ -0,0 +1,72 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2023 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 293
+#
+# Test btrfs write behavior with large RAID56 chunks (size beyond 4G).
+#
+. ./common/preamble
+_begin_fstest auto raid volume
+
+# Import common functions.
+. ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_require_scratch_dev_pool 8
+_fixed_by_kernel_commit a7299a18a179 \
+	"btrfs: fix u32 overflows when left shifting @stripe_nr"
+_fixed_by_kernel_commit xxxxxxxxxxxx \
+	"btrfs: use a dedicated helper to convert stripe_nr to offset"
+
+_scratch_dev_pool_get 8
+
+datasize=$((5 * 1024 * 1024 * 1024))
+
+
+workload()
+{
+	local data_profile=$1
+
+	_scratch_pool_mkfs -m raid1 -d $data_profile >> $seqres.full 2>&1
+	_scratch_mount
+	$XFS_IO_PROG -f -c "pwrite -b 1m 0 $datasize" $SCRATCH_MNT/foobar > /dev/null
+
+	# Unpatched kernel would trigger an ASSERT() or crash at writeback.
+	sync
+
+	echo "=== With initial 5G data written ($data_profile) ===" >> $seqres.full
+	$BTRFS_UTIL_PROG filesystem df $SCRATCH_MNT >> $seqres.full
+	_scratch_unmount
+
+	# Make sure we haven't corrupted anything.
+	$BTRFS_UTIL_PROG check --check-data-csum $SCRATCH_DEV >> $seqres.full 2>&1
+	if [ $? -ne 0 ]; then
+		_scratch_dev_pool_put
+		_fail "data corruption detected after initial data filling"
+	fi
+}
+
+# Make sure each device has at least 2G.
+# Btrfs has a limits on per-device stripe length of 1G.
+# Double that so that we can ensure a RAID6 data chunk with 6G size.
+for i in $SCRATCH_DEV_POOL; do
+	devsize=$(blockdev --getsize64 "$i")
+	if [ $devsize -lt $((2 * 1024 * 1024 * 1024)) ]; then
+		_scratch_dev_pool_put
+		_notrun "device $i is too small, need at least 2G"
+	fi
+done
+
+workload raid5
+workload raid6
+
+_scratch_dev_pool_put
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/btrfs/293.out new file +2 -0
new file mode 100644
index 00000000..076fc056
@@ -0,0 +1,2 @@
+QA output created by 293
+Silence is golden
Signature · 1 line
2.39.0
#1 lore raw
Quoted 8 lines · Qu Wenruo

There is a recent regression during v6.4 merge window, that a u32 left shift overflow can cause problems with large data chunks (over 4G) sized.

This is especially nasty for RAID56, which can lead to ASSERT() during regular writes, or corrupt memory if CONFIG_BTRFS_ASSERT is not enabled.

This is the regression test case for it.

I am not able to reproduce the problem with the previous fix a7299a18a179a971 applied but not the additional one so I can't validate the fix and will have to postpone sending the pull request.

#2 lore raw
Quoted 5 lines · David Sterba
Quoted 4 lines · Qu Wenruo

There is a recent regression during v6.4 merge window, that a u32 left shift overflow can cause problems with large data chunks (over 4G) sized.

Quoted 7 lines
Quoted 4 lines

This is especially nasty for RAID56, which can lead to ASSERT() during regular writes, or corrupt memory if CONFIG_BTRFS_ASSERT is not enabled.

This is the regression test case for it.

I am not able to reproduce the problem with the previous fix a7299a18a179a971 applied

Well, it's 100% reproducible here.

And the missing type cast is indeed the cause inside btrfs_map_block():

        if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK && need_raid_map &&
            (op != BTRFS_MAP_READ || mirror_num > 1)) {
                bioc->full_stripe_logical = em->start +
                        ((stripe_nr * data_stripes) <<

BTRFS_STRIPE_LEN_SHIFT);

Obviously @stripe_nr (now the full stripe number) * @data_stripes can reach stripe number beyond 4G range.

And a left shift would lead to overflow, causing a too small value.

This would trigger the ASSERT() in the raid56 code of rbio_add_bio().

I'm not sure why it didn't trigger in your case, mind to share the seqres.full?

Thanks,
Qu

Quoted 2 lines

but not the additional one so I can't validate
the fix and will have to postpone sending the pull request.

#3 lore raw
Quoted 11 lines · Qu Wenruo
Quoted 8 lines · David Sterba
Quoted 7 lines · Qu Wenruo

There is a recent regression during v6.4 merge window, that a u32 left shift overflow can cause problems with large data chunks (over 4G) sized.

This is especially nasty for RAID56, which can lead to ASSERT() during regular writes, or corrupt memory if CONFIG_BTRFS_ASSERT is not enabled.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

This is the regression test case for it.

I am not able to reproduce the problem with the previous fix a7299a18a179a971 applied

Well, it's 100% reproducible here.

So the cause was that binaries of VM fstests built in fstests got out of sync with the system and thus many things did not work. After resolving that I was able to test it properly.

Reviewed-by on v1 1/1
#4 lore raw
In reply to Qu Wenruo
Quoted 12 lines · Qu Wenruo

There is a recent regression during v6.4 merge window, that a u32 left shift overflow can cause problems with large data chunks (over 4G) sized.

This is especially nasty for RAID56, which can lead to ASSERT() during regular writes, or corrupt memory if CONFIG_BTRFS_ASSERT is not enabled.

This is the regression test case for it.

Unlike btrfs/292, btrfs doesn't support trim inside RAID56 chunks, thus the workflow is simplified:

Quoted 8 lines

- Create a RAID5 or RAID6 data chunk during mkfs

- Fill the fs with 5G data and sync
For unpatched kernel, the sync would crash the kernel.

- Make sure everything is fine

Looks good.

Thanks.

Quoted 93 lines
---
  tests/btrfs/293     | 72 +++++++++++++++++++++++++++++++++++++++++++++
  tests/btrfs/293.out |  2 ++
  2 files changed, 74 insertions(+)

create mode 100755 tests/btrfs/293
create mode 100644 tests/btrfs/293.out

tests/btrfs/293 new file +72 -0
new file mode 100755
index 00000000..68312682
@@ -0,0 +1,72 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2023 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 293
+#
+# Test btrfs write behavior with large RAID56 chunks (size beyond 4G).
+#
+. ./common/preamble
+_begin_fstest auto raid volume
+
+# Import common functions.
+. ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_require_scratch_dev_pool 8
+_fixed_by_kernel_commit a7299a18a179 \
+	"btrfs: fix u32 overflows when left shifting @stripe_nr"
+_fixed_by_kernel_commit xxxxxxxxxxxx \
+	"btrfs: use a dedicated helper to convert stripe_nr to offset"
+
+_scratch_dev_pool_get 8
+
+datasize=$((5 * 1024 * 1024 * 1024))
+
+
+workload()
+{
+	local data_profile=$1
+
+	_scratch_pool_mkfs -m raid1 -d $data_profile >> $seqres.full 2>&1
+	_scratch_mount
+	$XFS_IO_PROG -f -c "pwrite -b 1m 0 $datasize" $SCRATCH_MNT/foobar > /dev/null
+
+	# Unpatched kernel would trigger an ASSERT() or crash at writeback.
+	sync
+
+	echo "=== With initial 5G data written ($data_profile) ===" >> $seqres.full
+	$BTRFS_UTIL_PROG filesystem df $SCRATCH_MNT >> $seqres.full
+	_scratch_unmount
+
+	# Make sure we haven't corrupted anything.
+	$BTRFS_UTIL_PROG check --check-data-csum $SCRATCH_DEV >> $seqres.full 2>&1
+	if [ $? -ne 0 ]; then
+		_scratch_dev_pool_put
+		_fail "data corruption detected after initial data filling"
+	fi
+}
+
+# Make sure each device has at least 2G.
+# Btrfs has a limits on per-device stripe length of 1G.
+# Double that so that we can ensure a RAID6 data chunk with 6G size.
+for i in $SCRATCH_DEV_POOL; do
+	devsize=$(blockdev --getsize64 "$i")
+	if [ $devsize -lt $((2 * 1024 * 1024 * 1024)) ]; then
+		_scratch_dev_pool_put
+		_notrun "device $i is too small, need at least 2G"
+	fi
+done
+
+workload raid5
+workload raid6
+
+_scratch_dev_pool_put
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/btrfs/293.out new file +2 -0
new file mode 100644
index 00000000..076fc056
@@ -0,0 +1,2 @@
+QA output created by 293
+Silence is golden
Reviewed-by on v1 1/1
#5 lore raw
In reply to Qu Wenruo
Quoted 42 lines · Qu Wenruo

There is a recent regression during v6.4 merge window, that a u32 left shift overflow can cause problems with large data chunks (over 4G) sized.

This is especially nasty for RAID56, which can lead to ASSERT() during regular writes, or corrupt memory if CONFIG_BTRFS_ASSERT is not enabled.

This is the regression test case for it.

Unlike btrfs/292, btrfs doesn't support trim inside RAID56 chunks, thus the workflow is simplified:

- Create a RAID5 or RAID6 data chunk during mkfs

- Fill the fs with 5G data and sync
For unpatched kernel, the sync would crash the kernel.

- Make sure everything is fine

---
 tests/btrfs/293     | 72 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/293.out |  2 ++
 2 files changed, 74 insertions(+)
 create mode 100755 tests/btrfs/293
 create mode 100644 tests/btrfs/293.out
tests/btrfs/293 new file +9 -0
new file mode 100755
index 00000000..68312682
@@ -0,0 +1,72 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2023 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 293
+#
+# Test btrfs write behavior with large RAID56 chunks (size beyond 4G).
+#
+. ./common/preamble

tests/btrfs/293 line 19

Quoted 10 lines
tests/btrfs/293 new file +10 -0
+_begin_fstest auto raid volume
+
+# Import common functions.
+. ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_require_scratch_dev_pool 8

Could you add following lines?

# RAID5/6 not working on zoned mode.
_require_non_zoned_device "${SCRATCH_DEV}"

With that, looks good to me.

Quoted 64 lines
tests/btrfs/293 new file +53 -0
+_fixed_by_kernel_commit a7299a18a179 \
+	"btrfs: fix u32 overflows when left shifting @stripe_nr"
+_fixed_by_kernel_commit xxxxxxxxxxxx \
+	"btrfs: use a dedicated helper to convert stripe_nr to offset"
+
+_scratch_dev_pool_get 8
+
+datasize=$((5 * 1024 * 1024 * 1024))
+
+
+workload()
+{
+	local data_profile=$1
+
+	_scratch_pool_mkfs -m raid1 -d $data_profile >> $seqres.full 2>&1
+	_scratch_mount
+	$XFS_IO_PROG -f -c "pwrite -b 1m 0 $datasize" $SCRATCH_MNT/foobar > /dev/null
+
+	# Unpatched kernel would trigger an ASSERT() or crash at writeback.
+	sync
+
+	echo "=== With initial 5G data written ($data_profile) ===" >> $seqres.full
+	$BTRFS_UTIL_PROG filesystem df $SCRATCH_MNT >> $seqres.full
+	_scratch_unmount
+
+	# Make sure we haven't corrupted anything.
+	$BTRFS_UTIL_PROG check --check-data-csum $SCRATCH_DEV >> $seqres.full 2>&1
+	if [ $? -ne 0 ]; then
+		_scratch_dev_pool_put
+		_fail "data corruption detected after initial data filling"
+	fi
+}
+
+# Make sure each device has at least 2G.
+# Btrfs has a limits on per-device stripe length of 1G.
+# Double that so that we can ensure a RAID6 data chunk with 6G size.
+for i in $SCRATCH_DEV_POOL; do
+	devsize=$(blockdev --getsize64 "$i")
+	if [ $devsize -lt $((2 * 1024 * 1024 * 1024)) ]; then
+		_scratch_dev_pool_put
+		_notrun "device $i is too small, need at least 2G"
+	fi
+done
+
+workload raid5
+workload raid6
+
+_scratch_dev_pool_put
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/btrfs/293.out new file +2 -0
new file mode 100644
index 00000000..076fc056
@@ -0,0 +1,2 @@
+QA output created by 293
+Silence is golden
Signature · 1 line
2.39.0
#6 lore raw
Quoted 56 lines · Naohiro Aota
Quoted 42 lines · Qu Wenruo

There is a recent regression during v6.4 merge window, that a u32 left shift overflow can cause problems with large data chunks (over 4G) sized.

This is especially nasty for RAID56, which can lead to ASSERT() during regular writes, or corrupt memory if CONFIG_BTRFS_ASSERT is not enabled.

This is the regression test case for it.

Unlike btrfs/292, btrfs doesn't support trim inside RAID56 chunks, thus the workflow is simplified:

- Create a RAID5 or RAID6 data chunk during mkfs

- Fill the fs with 5G data and sync
For unpatched kernel, the sync would crash the kernel.

- Make sure everything is fine

---
 tests/btrfs/293     | 72 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/293.out |  2 ++
 2 files changed, 74 insertions(+)
 create mode 100755 tests/btrfs/293
 create mode 100644 tests/btrfs/293.out
tests/btrfs/293 new file +9 -0
new file mode 100755
index 00000000..68312682
@@ -0,0 +1,72 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2023 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 293
+#
+# Test btrfs write behavior with large RAID56 chunks (size beyond 4G).
+#
+. ./common/preamble

tests/btrfs/293 line 19

Quoted 10 lines
tests/btrfs/293 new file +10 -0
+_begin_fstest auto raid volume
+
+# Import common functions.
+. ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_require_scratch_dev_pool 8

Could you add following lines?

Quoted 4 lines

# RAID5/6 not working on zoned mode.
_require_non_zoned_device "${SCRATCH_DEV}"

With that, looks good to me.

Hi Naohiro,

This patch has been merged last weekend. If you feel this line is needed, please send another patch to 'fix' it.

Thanks,
Zorro

Quoted 68 lines
Quoted 64 lines
tests/btrfs/293 new file +53 -0
+_fixed_by_kernel_commit a7299a18a179 \
+	"btrfs: fix u32 overflows when left shifting @stripe_nr"
+_fixed_by_kernel_commit xxxxxxxxxxxx \
+	"btrfs: use a dedicated helper to convert stripe_nr to offset"
+
+_scratch_dev_pool_get 8
+
+datasize=$((5 * 1024 * 1024 * 1024))
+
+
+workload()
+{
+	local data_profile=$1
+
+	_scratch_pool_mkfs -m raid1 -d $data_profile >> $seqres.full 2>&1
+	_scratch_mount
+	$XFS_IO_PROG -f -c "pwrite -b 1m 0 $datasize" $SCRATCH_MNT/foobar > /dev/null
+
+	# Unpatched kernel would trigger an ASSERT() or crash at writeback.
+	sync
+
+	echo "=== With initial 5G data written ($data_profile) ===" >> $seqres.full
+	$BTRFS_UTIL_PROG filesystem df $SCRATCH_MNT >> $seqres.full
+	_scratch_unmount
+
+	# Make sure we haven't corrupted anything.
+	$BTRFS_UTIL_PROG check --check-data-csum $SCRATCH_DEV >> $seqres.full 2>&1
+	if [ $? -ne 0 ]; then
+		_scratch_dev_pool_put
+		_fail "data corruption detected after initial data filling"
+	fi
+}
+
+# Make sure each device has at least 2G.
+# Btrfs has a limits on per-device stripe length of 1G.
+# Double that so that we can ensure a RAID6 data chunk with 6G size.
+for i in $SCRATCH_DEV_POOL; do
+	devsize=$(blockdev --getsize64 "$i")
+	if [ $devsize -lt $((2 * 1024 * 1024 * 1024)) ]; then
+		_scratch_dev_pool_put
+		_notrun "device $i is too small, need at least 2G"
+	fi
+done
+
+workload raid5
+workload raid6
+
+_scratch_dev_pool_put
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/btrfs/293.out new file +2 -0
new file mode 100644
index 00000000..076fc056
@@ -0,0 +1,2 @@
+QA output created by 293
+Silence is golden
Signature · 1 line
2.39.0