btrfs: add a test case to verify the write behavior of large RAID5 data chunks
PATCH 1/1 btrfs: add a test case to verify the write behavior of large RAID5 data chunks
There is a recent regression during v6.4 merge window, that a u32 left shift overflow can cause problems with large data chunks (over 4G) sized.
This is especially nasty for RAID56, which can lead to ASSERT() during regular writes, or corrupt memory if CONFIG_BTRFS_ASSERT is not enabled.
This is the regression test case for it.
Unlike btrfs/292, btrfs doesn't support trim inside RAID56 chunks, thus the workflow is simplified:
- Create a RAID5 or RAID6 data chunk during mkfs
- Fill the fs with 5G data and sync
For unpatched kernel, the sync would crash the kernel.
- Make sure everything is fine
--- tests/btrfs/293 | 72 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/293.out | 2 ++ 2 files changed, 74 insertions(+) create mode 100755 tests/btrfs/293 create mode 100644 tests/btrfs/293.out
tests/btrfs/293 new file +72 -0
@@ -0,0 +1,72 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2023 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 293 +# +# Test btrfs write behavior with large RAID56 chunks (size beyond 4G). +# +. ./common/preamble +_begin_fstest auto raid volume + +# Import common functions. +. ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_require_scratch_dev_pool 8 +_fixed_by_kernel_commit a7299a18a179 \ + "btrfs: fix u32 overflows when left shifting @stripe_nr" +_fixed_by_kernel_commit xxxxxxxxxxxx \ + "btrfs: use a dedicated helper to convert stripe_nr to offset" + +_scratch_dev_pool_get 8 + +datasize=$((5 * 1024 * 1024 * 1024)) + + +workload() +{ + local data_profile=$1 + + _scratch_pool_mkfs -m raid1 -d $data_profile >> $seqres.full 2>&1 + _scratch_mount + $XFS_IO_PROG -f -c "pwrite -b 1m 0 $datasize" $SCRATCH_MNT/foobar > /dev/null + + # Unpatched kernel would trigger an ASSERT() or crash at writeback. + sync + + echo "=== With initial 5G data written ($data_profile) ===" >> $seqres.full + $BTRFS_UTIL_PROG filesystem df $SCRATCH_MNT >> $seqres.full + _scratch_unmount + + # Make sure we haven't corrupted anything. + $BTRFS_UTIL_PROG check --check-data-csum $SCRATCH_DEV >> $seqres.full 2>&1 + if [ $? -ne 0 ]; then + _scratch_dev_pool_put + _fail "data corruption detected after initial data filling" + fi +} + +# Make sure each device has at least 2G. +# Btrfs has a limits on per-device stripe length of 1G. +# Double that so that we can ensure a RAID6 data chunk with 6G size. +for i in $SCRATCH_DEV_POOL; do + devsize=$(blockdev --getsize64 "$i") + if [ $devsize -lt $((2 * 1024 * 1024 * 1024)) ]; then + _scratch_dev_pool_put + _notrun "device $i is too small, need at least 2G" + fi +done + +workload raid5 +workload raid6 + +_scratch_dev_pool_put +echo "Silence is golden" + +# success, all done +status=0 +exit
tests/btrfs/293.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 293 +Silence is golden
Signature · 1 line
2.39.0
Quoted 8 lines · Qu Wenruo
There is a recent regression during v6.4 merge window, that a u32 left shift overflow can cause problems with large data chunks (over 4G) sized.
This is especially nasty for RAID56, which can lead to
ASSERT()during regular writes, or corrupt memory ifCONFIG_BTRFS_ASSERTis not enabled.This is the regression test case for it.
I am not able to reproduce the problem with the previous fix a7299a18a179a971 applied but not the additional one so I can't validate the fix and will have to postpone sending the pull request.
Quoted 5 lines · David Sterba
Quoted 4 lines · Qu Wenruo
There is a recent regression during v6.4 merge window, that a u32 left shift overflow can cause problems with large data chunks (over 4G) sized.
Quoted 7 lines
Quoted 4 lines
This is especially nasty for RAID56, which can lead to
ASSERT()during regular writes, or corrupt memory ifCONFIG_BTRFS_ASSERTis not enabled.This is the regression test case for it.
I am not able to reproduce the problem with the previous fix
a7299a18a179a971applied
Well, it's 100% reproducible here.
And the missing type cast is indeed the cause inside btrfs_map_block():
if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK && need_raid_map &&
(op != BTRFS_MAP_READ || mirror_num > 1)) {
bioc->full_stripe_logical = em->start +
((stripe_nr * data_stripes) <<BTRFS_STRIPE_LEN_SHIFT);
Obviously @stripe_nr (now the full stripe number) * @data_stripes can reach stripe number beyond 4G range.
And a left shift would lead to overflow, causing a too small value.
This would trigger the ASSERT() in the raid56 code of rbio_add_bio().
I'm not sure why it didn't trigger in your case, mind to share the seqres.full?
Thanks,
Qu
Quoted 2 lines
but not the additional one so I can't validate
the fix and will have to postpone sending the pull request.
Quoted 11 lines · Qu Wenruo
Quoted 8 lines · David Sterba
Quoted 7 lines · Qu Wenruo
There is a recent regression during v6.4 merge window, that a u32 left shift overflow can cause problems with large data chunks (over 4G) sized.
This is especially nasty for RAID56, which can lead to
ASSERT()during regular writes, or corrupt memory ifCONFIG_BTRFS_ASSERTis not enabled.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
This is the regression test case for it.
I am not able to reproduce the problem with the previous fix
a7299a18a179a971appliedWell, it's 100% reproducible here.
So the cause was that binaries of VM fstests built in fstests got out of sync with the system and thus many things did not work. After resolving that I was able to test it properly.
Quoted 12 lines · Qu Wenruo
There is a recent regression during v6.4 merge window, that a u32 left shift overflow can cause problems with large data chunks (over 4G) sized.
This is especially nasty for RAID56, which can lead to
ASSERT()during regular writes, or corrupt memory ifCONFIG_BTRFS_ASSERTis not enabled.This is the regression test case for it.
Unlike btrfs/292, btrfs doesn't support trim inside RAID56 chunks, thus the workflow is simplified:
Quoted 8 lines
- Create a RAID5 or RAID6 data chunk during mkfs
- Fill the fs with 5G data and sync
For unpatched kernel, the sync would crash the kernel.- Make sure everything is fine
Signed-off-by Qu Wenruo <[email protected]>
Looks good.
Thanks.
Quoted 93 lines
--- tests/btrfs/293 | 72 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/293.out | 2 ++ 2 files changed, 74 insertions(+)create mode 100755 tests/btrfs/293
create mode 100644 tests/btrfs/293.outtests/btrfs/293 new file +72
-0@@ -0,0 +1,72 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2023 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 293 +# +# Test btrfs write behavior with large RAID56 chunks (size beyond 4G). +# +. ./common/preamble +_begin_fstest auto raid volume + +# Import common functions. +. ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_require_scratch_dev_pool 8 +_fixed_by_kernel_commit a7299a18a179 \ + "btrfs: fix u32 overflows when left shifting @stripe_nr" +_fixed_by_kernel_commit xxxxxxxxxxxx \ + "btrfs: use a dedicated helper to convert stripe_nr to offset" + +_scratch_dev_pool_get 8 + +datasize=$((5 * 1024 * 1024 * 1024)) + + +workload() +{ + local data_profile=$1 + + _scratch_pool_mkfs -m raid1 -d $data_profile >> $seqres.full 2>&1 + _scratch_mount + $XFS_IO_PROG -f -c "pwrite -b 1m 0 $datasize" $SCRATCH_MNT/foobar > /dev/null + + # Unpatched kernel would trigger an ASSERT() or crash at writeback. + sync + + echo "=== With initial 5G data written ($data_profile) ===" >> $seqres.full + $BTRFS_UTIL_PROG filesystem df $SCRATCH_MNT >> $seqres.full + _scratch_unmount + + # Make sure we haven't corrupted anything. + $BTRFS_UTIL_PROG check --check-data-csum $SCRATCH_DEV >> $seqres.full 2>&1 + if [ $? -ne 0 ]; then + _scratch_dev_pool_put + _fail "data corruption detected after initial data filling" + fi +} + +# Make sure each device has at least 2G. +# Btrfs has a limits on per-device stripe length of 1G. +# Double that so that we can ensure a RAID6 data chunk with 6G size. +for i in $SCRATCH_DEV_POOL; do + devsize=$(blockdev --getsize64 "$i") + if [ $devsize -lt $((2 * 1024 * 1024 * 1024)) ]; then + _scratch_dev_pool_put + _notrun "device $i is too small, need at least 2G" + fi +done + +workload raid5 +workload raid6 + +_scratch_dev_pool_put +echo "Silence is golden" + +# success, all done +status=0 +exittests/btrfs/293.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 293 +Silence is golden
Quoted 42 lines · Qu Wenruo
There is a recent regression during v6.4 merge window, that a u32 left shift overflow can cause problems with large data chunks (over 4G) sized.
This is especially nasty for RAID56, which can lead to
ASSERT()during regular writes, or corrupt memory ifCONFIG_BTRFS_ASSERTis not enabled.This is the regression test case for it.
Unlike btrfs/292, btrfs doesn't support trim inside RAID56 chunks, thus the workflow is simplified:
- Create a RAID5 or RAID6 data chunk during mkfs
- Fill the fs with 5G data and sync
For unpatched kernel, the sync would crash the kernel.- Make sure everything is fine
Signed-off-by Qu Wenruo <[email protected]>--- tests/btrfs/293 | 72 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/293.out | 2 ++ 2 files changed, 74 insertions(+) create mode 100755 tests/btrfs/293 create mode 100644 tests/btrfs/293.outtests/btrfs/293 new file +9
-0@@ -0,0 +1,72 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2023 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 293 +# +# Test btrfs write behavior with large RAID56 chunks (size beyond 4G). +# +. ./common/preamble
tests/btrfs/293 line 19
Quoted 10 lines
tests/btrfs/293 new file +10
-0+_begin_fstest auto raid volume + +# Import common functions. +. ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_require_scratch_dev_pool 8
Could you add following lines?
# RAID5/6 not working on zoned mode._require_non_zoned_device "${SCRATCH_DEV}"
With that, looks good to me.
Quoted 64 lines
tests/btrfs/293 new file +53
-0+_fixed_by_kernel_commit a7299a18a179 \ + "btrfs: fix u32 overflows when left shifting @stripe_nr" +_fixed_by_kernel_commit xxxxxxxxxxxx \ + "btrfs: use a dedicated helper to convert stripe_nr to offset" + +_scratch_dev_pool_get 8 + +datasize=$((5 * 1024 * 1024 * 1024)) + + +workload() +{ + local data_profile=$1 + + _scratch_pool_mkfs -m raid1 -d $data_profile >> $seqres.full 2>&1 + _scratch_mount + $XFS_IO_PROG -f -c "pwrite -b 1m 0 $datasize" $SCRATCH_MNT/foobar > /dev/null + + # Unpatched kernel would trigger an ASSERT() or crash at writeback. + sync + + echo "=== With initial 5G data written ($data_profile) ===" >> $seqres.full + $BTRFS_UTIL_PROG filesystem df $SCRATCH_MNT >> $seqres.full + _scratch_unmount + + # Make sure we haven't corrupted anything. + $BTRFS_UTIL_PROG check --check-data-csum $SCRATCH_DEV >> $seqres.full 2>&1 + if [ $? -ne 0 ]; then + _scratch_dev_pool_put + _fail "data corruption detected after initial data filling" + fi +} + +# Make sure each device has at least 2G. +# Btrfs has a limits on per-device stripe length of 1G. +# Double that so that we can ensure a RAID6 data chunk with 6G size. +for i in $SCRATCH_DEV_POOL; do + devsize=$(blockdev --getsize64 "$i") + if [ $devsize -lt $((2 * 1024 * 1024 * 1024)) ]; then + _scratch_dev_pool_put + _notrun "device $i is too small, need at least 2G" + fi +done + +workload raid5 +workload raid6 + +_scratch_dev_pool_put +echo "Silence is golden" + +# success, all done +status=0 +exittests/btrfs/293.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 293 +Silence is goldenSignature · 1 line
2.39.0
Quoted 56 lines · Naohiro Aota
Quoted 42 lines · Qu Wenruo
There is a recent regression during v6.4 merge window, that a u32 left shift overflow can cause problems with large data chunks (over 4G) sized.
This is especially nasty for RAID56, which can lead to
ASSERT()during regular writes, or corrupt memory ifCONFIG_BTRFS_ASSERTis not enabled.This is the regression test case for it.
Unlike btrfs/292, btrfs doesn't support trim inside RAID56 chunks, thus the workflow is simplified:
- Create a RAID5 or RAID6 data chunk during mkfs
- Fill the fs with 5G data and sync
For unpatched kernel, the sync would crash the kernel.- Make sure everything is fine
Signed-off-by Qu Wenruo <[email protected]>--- tests/btrfs/293 | 72 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/293.out | 2 ++ 2 files changed, 74 insertions(+) create mode 100755 tests/btrfs/293 create mode 100644 tests/btrfs/293.outtests/btrfs/293 new file +9
-0@@ -0,0 +1,72 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2023 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 293 +# +# Test btrfs write behavior with large RAID56 chunks (size beyond 4G). +# +. ./common/preambletests/btrfs/293 line 19
Quoted 10 lines
tests/btrfs/293 new file +10
-0+_begin_fstest auto raid volume + +# Import common functions. +. ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_require_scratch_dev_pool 8Could you add following lines?
Quoted 4 lines
# RAID5/6 not working on zoned mode.
_require_non_zoned_device"${SCRATCH_DEV}"With that, looks good to me.
Hi Naohiro,
This patch has been merged last weekend. If you feel this line is needed, please send another patch to 'fix' it.
Thanks,
Zorro
Quoted 68 lines
Reviewed-by Naohiro Aota <[email protected]>Quoted 64 lines
tests/btrfs/293 new file +53
-0+_fixed_by_kernel_commit a7299a18a179 \ + "btrfs: fix u32 overflows when left shifting @stripe_nr" +_fixed_by_kernel_commit xxxxxxxxxxxx \ + "btrfs: use a dedicated helper to convert stripe_nr to offset" + +_scratch_dev_pool_get 8 + +datasize=$((5 * 1024 * 1024 * 1024)) + + +workload() +{ + local data_profile=$1 + + _scratch_pool_mkfs -m raid1 -d $data_profile >> $seqres.full 2>&1 + _scratch_mount + $XFS_IO_PROG -f -c "pwrite -b 1m 0 $datasize" $SCRATCH_MNT/foobar > /dev/null + + # Unpatched kernel would trigger an ASSERT() or crash at writeback. + sync + + echo "=== With initial 5G data written ($data_profile) ===" >> $seqres.full + $BTRFS_UTIL_PROG filesystem df $SCRATCH_MNT >> $seqres.full + _scratch_unmount + + # Make sure we haven't corrupted anything. + $BTRFS_UTIL_PROG check --check-data-csum $SCRATCH_DEV >> $seqres.full 2>&1 + if [ $? -ne 0 ]; then + _scratch_dev_pool_put + _fail "data corruption detected after initial data filling" + fi +} + +# Make sure each device has at least 2G. +# Btrfs has a limits on per-device stripe length of 1G. +# Double that so that we can ensure a RAID6 data chunk with 6G size. +for i in $SCRATCH_DEV_POOL; do + devsize=$(blockdev --getsize64 "$i") + if [ $devsize -lt $((2 * 1024 * 1024 * 1024)) ]; then + _scratch_dev_pool_put + _notrun "device $i is too small, need at least 2G" + fi +done + +workload raid5 +workload raid6 + +_scratch_dev_pool_put +echo "Silence is golden" + +# success, all done +status=0 +exittests/btrfs/293.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 293 +Silence is goldenSignature · 1 line
2.39.0