fstests: various RAID56 related fixes for btrfs
v2 cover fstests: various RAID56 related fixes for btrfs
v1: https://lore.kernel.org/linux-btrfs/[email protected]/
v1->v2:
- Reworked the golden output of btrfs/197 and btrfs/198 to handle running some of the RAID profiles.
- Added a new way to require RAID profiles in the tests, updated all the different tests.
- Changed btrfs/197, btrfs/198, and btrfs/297 to skip any RAID profiles that aren't supported by our RAID profile settings.
--- Original email ---
Hello,
While trying to get CI setup internally I noticed that we were sometimes failing raid56 tests even though I had specified BTRFS_PROFILE_CONFIGS without raid56 in them.
This is because tests where we require raid56 to work only check to see if it's enabled by the kernel, not to check and see if we're configured to use it with our profile configs.
One test needed to be updated to skip any configurations that weren't in our profile configs, and then a few tests didn't use the
_require_btrfs_fs_feature raid56
check in the top of their test. This series fixes everything up and honors the user settings which makes my internal CI runs clean where we don't want to test raid56. Thanks,
Josef
Josef Bacik (3):
fstests: change btrfs/197 and btrfs/198 golden output
fstests: change how we test for supported raid configs
fstests: update tests to skip unsupported raid profile types
common/btrfs | 29 ++++++++++++++++++++++++----- tests/btrfs/125 | 2 +- tests/btrfs/148 | 3 ++- tests/btrfs/157 | 2 +- tests/btrfs/158 | 2 +- tests/btrfs/197 | 17 +++++++++++++---- tests/btrfs/197.out | 25 +------------------------ tests/btrfs/198 | 17 +++++++++++++---- tests/btrfs/198.out | 25 +------------------------ tests/btrfs/297 | 10 ++++++++++ 10 files changed, 67 insertions(+), 65 deletions(-)
Signature · 1 line
2.43.0
PATCH v2 1/3 fstests: change btrfs/197 and btrfs/198 golden output
Both btrfs/197 and btrfs/198 check several raid types. We may not have support for raid5/6 for our available profiles, but we'd like to be able to test the other profiles. In order to enable this, update the golden output to have no output, and simply have the test check for the device we removed to see if it still exists in the device list output. This will allow us to add a check to skip unsupported raid configurations in our config.
--- tests/btrfs/197 | 7 +++++-- tests/btrfs/197.out | 25 +------------------------ tests/btrfs/198 | 7 +++++-- tests/btrfs/198.out | 25 +------------------------ 4 files changed, 12 insertions(+), 52 deletions(-)
tests/btrfs/197 +5 -2
@@ -38,7 +38,7 @@ workout() raid=$1 device_nr=$2 - echo $raid + echo $raid >> $seqres.full _scratch_dev_pool_get $device_nr _spare_dev_get
@@ -62,7 +62,9 @@ workout() _mount -o degraded $device_2 $SCRATCH_MNT # Check if missing device is reported as in the .out $BTRFS_UTIL_PROG filesystem show -m $SCRATCH_MNT | \ - _filter_btrfs_filesystem_show + _filter_btrfs_filesystem_show > $tmp.output 2>&1 + cat $tmp.output >> $seqres.full + grep -q "$device_1" $tmp.output && _fail "found stale device" $BTRFS_UTIL_PROG device remove "$device_1" "$TEST_DIR/$seq.mnt" $UMOUNT_PROG $TEST_DIR/$seq.mnt
@@ -77,5 +79,6 @@ workout "raid6" "4" workout "raid10" "4" # success, all done +echo "Silence is golden" status=0 exit
tests/btrfs/197.out +1 -24
@@ -1,25 +1,2 @@ QA output created by 197 -raid1 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - -raid5 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - -raid6 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - -raid10 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - +Silence is golden
tests/btrfs/198 +5 -2
@@ -28,7 +28,7 @@ workout() raid=$1 device_nr=$2 - echo $raid + echo $raid >> $seqres.full _scratch_dev_pool_get $device_nr _scratch_pool_mkfs "-d$raid -m$raid" >> $seqres.full 2>&1 || \
@@ -46,7 +46,9 @@ workout() _mount -o degraded $device_2 $SCRATCH_MNT # Check if missing device is reported as in the 196.out $BTRFS_UTIL_PROG filesystem show -m $SCRATCH_MNT | \ - _filter_btrfs_filesystem_show + _filter_btrfs_filesystem_show > $tmp.output 2>&1 + cat $tmp.output >> $seqres.full + grep -q "$device_1" $tmp.output && _fail "found stale device" _scratch_unmount _scratch_dev_pool_put
@@ -58,5 +60,6 @@ workout "raid6" "4" workout "raid10" "4" # success, all done +echo "Silence is golden" status=0 exit
tests/btrfs/198.out +1 -24
@@ -1,25 +1,2 @@ QA output created by 198 -raid1 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - -raid5 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - -raid6 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - -raid10 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - +Silence is golden
Signature · 1 line
2.43.0
PATCH v2 2/3 fstests: change how we test for supported raid configs
In reply to Josef BacikIn btrfs there's a few ways we limit the RAID profiles we'll use. We have the raid56 feature that can be compiled out, zoned devices don't support certain raid configurations, and you can manually set BTRFS_PROFILE_CONFIGS to limit what you're testing.
To handle all of these different scenarios in the same way, update _btrfs_get_profile_configs() to check for RAID56 support and remove it if it is not there, and then add _require_btrfs_raid_type and _check_btrfs_raid_type to get all the settings and then check if the requested raid type is available.
From there I've updated all of the existing tests that use
_require_btrfs_fs_feature raid56
to use
_require_btrfs_raid_type <type>
where appropriate.
--- common/btrfs | 29 ++++++++++++++++++++++++----- tests/btrfs/125 | 2 +- tests/btrfs/148 | 3 ++- tests/btrfs/157 | 2 +- tests/btrfs/158 | 2 +- 5 files changed, 29 insertions(+), 9 deletions(-)
common/btrfs +24 -5
@@ -98,11 +98,6 @@ _require_btrfs_fs_feature() modprobe btrfs > /dev/null 2>&1 [ -e /sys/fs/btrfs/features/$feat ] || \ _notrun "Feature $feat not supported by the available btrfs version" - - if [ $feat = "raid56" ]; then - # Zoned btrfs only supports SINGLE profile - _require_non_zoned_device "${SCRATCH_DEV}" - fi } _require_btrfs_fs_sysfs()
@@ -218,6 +213,8 @@ _btrfs_get_profile_configs() return fi + modprobe btrfs > /dev/null 2>&1 + local unsupported=() if [ "$1" == "replace" ]; then # We can't do replace with these profiles because they
@@ -237,6 +234,14 @@ _btrfs_get_profile_configs() ) fi + if [ ! -e /sys/fs/btrfs/features/raid56 ]; then + # We don't have raid56 compiled in, remove them + unsupported+=( + "raid5" + "raid6" + ) + fi + if _scratch_btrfs_is_zoned; then # Zoned btrfs only supports SINGLE profile unsupported+=(
@@ -792,3 +797,17 @@ _has_btrfs_sysfs_feature_attr() test -e /sys/fs/btrfs/features/$feature_attr } + +_check_btrfs_raid_type() +{ + _btrfs_get_profile_configs + if [[ ! "${_btrfs_profile_configs[@]}" =~ "$1" ]]; then + return 1 + fi + return 0 +} + +_require_btrfs_raid_type() +{ + _check_btrfs_raid_type $1 || _notrun "$1 isn't supported by the profile config" +}
tests/btrfs/125 +1 -1
@@ -41,7 +41,7 @@ _supported_fs btrfs _require_scratch_dev_pool 3 _test_unmount _require_btrfs_forget_or_module_loadable -_require_btrfs_fs_feature raid56 +_require_btrfs_raid_type raid5 _scratch_dev_pool_get 3
tests/btrfs/148 +2 -1
@@ -17,7 +17,8 @@ _supported_fs btrfs _require_scratch _require_scratch_dev_pool 4 _require_odirect -_require_btrfs_fs_feature raid56 +_require_btrfs_raid_type raid5 +_require_btrfs_raid_type raid6 _scratch_dev_pool_get 4
tests/btrfs/157 +1 -1
@@ -32,7 +32,7 @@ _begin_fstest auto quick raid read_repair _supported_fs btrfs _require_scratch_dev_pool 4 _require_btrfs_command inspect-internal dump-tree -_require_btrfs_fs_feature raid56 +_require_btrfs_raid_type raid6 get_physical() {
tests/btrfs/158 +1 -1
@@ -24,7 +24,7 @@ _begin_fstest auto quick raid scrub _supported_fs btrfs _require_scratch_dev_pool 4 _require_btrfs_command inspect-internal dump-tree -_require_btrfs_fs_feature raid56 +_require_btrfs_raid_type raid5 get_physical() {
Signature · 1 line
2.43.0
PATCH v2 3/3 fstests: update tests to skip unsupported raid profile types
In reply to Josef BacikTests btrfs/197, btrfs/198, and btrfs/297 test multiple raid types in their workout() function. We may not support some of the raid types, so add a check in the workout() function to skip any incompatible raid profiles.
--- tests/btrfs/197 | 10 ++++++++-- tests/btrfs/198 | 10 ++++++++-- tests/btrfs/297 | 10 ++++++++++ 3 files changed, 26 insertions(+), 4 deletions(-)
tests/btrfs/197 +8 -2
@@ -30,14 +30,20 @@ _supported_fs btrfs _require_test _require_scratch _require_scratch_dev_pool 5 -# Zoned btrfs only supports SINGLE profile -_require_non_zoned_device ${SCRATCH_DEV} +# We require at least one raid setup, raid1 is the easiest, use this to gate on +# wether or not we run this test +_require_btrfs_raid_type raid1 workout() { raid=$1 device_nr=$2 + if ! _check_btrfs_raid_type $raid; then + echo "$raid isn't supported, skipping" >> $seqres.full + return + fi + echo $raid >> $seqres.full _scratch_dev_pool_get $device_nr _spare_dev_get
tests/btrfs/198 +8 -2
@@ -18,8 +18,9 @@ _supported_fs btrfs _require_command "$WIPEFS_PROG" wipefs _require_scratch _require_scratch_dev_pool 4 -# Zoned btrfs only supports SINGLE profile -_require_non_zoned_device ${SCRATCH_DEV} +# We require at least one raid setup, raid1 is the easiest, use this to gate on +# wether or not we run this test +_require_btrfs_raid_type raid1 _fixed_by_kernel_commit 96c2e067ed3e3e \ "btrfs: skip devices without magic signature when mounting"
@@ -28,6 +29,11 @@ workout() raid=$1 device_nr=$2 + if ! _check_btrfs_raid_type $raid; then + echo "$raid isn't supported, skipping" >> $seqres.full + return + fi + echo $raid >> $seqres.full _scratch_dev_pool_get $device_nr
tests/btrfs/297 +10 -0
@@ -18,11 +18,21 @@ _require_scratch_dev_pool 3 _fixed_by_kernel_commit 486c737f7fdc \ "btrfs: raid56: always verify the P/Q contents for scrub" +# If neither raid5 or raid6 are supported do _notrun +if ! _check_btrfs_raid_type raid5 && ! _check_btrfs_raid_type raid6; then + _notrun "requires either raid5 or raid6 support" +fi + workload() { local profile=$1 local nr_devs=$2 + if ! _check_btrfs_raid_type $profile; then + echo "$profile isn't supported, skipping" >> $seqres.full + return + fi + echo "=== Testing $nr_devs devices $profile ===" >> $seqres.full _scratch_dev_pool_get $nr_devs
Signature · 1 line
2.43.0
Re: [PATCH v2 1/3] fstests: change btrfs/197 and btrfs/198 golden output
In reply to Josef BacikQuoted 9 lines · Josef Bacik
Both btrfs/197 and btrfs/198 check several raid types. We may not have support for raid5/6 for our available profiles, but we'd like to be able to test the other profiles. In order to enable this, update the golden output to have no output, and simply have the test check for the device we removed to see if it still exists in the device list output. This will allow us to add a check to skip unsupported raid configurations in our config.
Signed-off-by Josef Bacik <[email protected]>
Thx, Anand
Quoted 131 lines
--- tests/btrfs/197 | 7 +++++-- tests/btrfs/197.out | 25 +------------------------ tests/btrfs/198 | 7 +++++-- tests/btrfs/198.out | 25 +------------------------ 4 files changed, 12 insertions(+), 52 deletions(-)tests/btrfs/197 +5
-2@@ -38,7 +38,7 @@ workout() raid=$1 device_nr=$2 - echo $raid + echo $raid >> $seqres.full _scratch_dev_pool_get $device_nr _spare_dev_get@@ -62,7 +62,9 @@ workout() _mount -o degraded $device_2 $SCRATCH_MNT # Check if missing device is reported as in the .out $BTRFS_UTIL_PROG filesystem show -m $SCRATCH_MNT | \ - _filter_btrfs_filesystem_show + _filter_btrfs_filesystem_show > $tmp.output 2>&1 + cat $tmp.output >> $seqres.full + grep -q "$device_1" $tmp.output && _fail "found stale device" $BTRFS_UTIL_PROG device remove "$device_1" "$TEST_DIR/$seq.mnt" $UMOUNT_PROG $TEST_DIR/$seq.mnt@@ -77,5 +79,6 @@ workout "raid6" "4" workout "raid10" "4" # success, all done +echo "Silence is golden" status=0 exittests/btrfs/197.out +1
-24@@ -1,25 +1,2 @@ QA output created by 197 -raid1 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - -raid5 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - -raid6 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - -raid10 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - +Silence is goldentests/btrfs/198 +5
-2@@ -28,7 +28,7 @@ workout() raid=$1 device_nr=$2 - echo $raid + echo $raid >> $seqres.full _scratch_dev_pool_get $device_nr _scratch_pool_mkfs "-d$raid -m$raid" >> $seqres.full 2>&1 || \@@ -46,7 +46,9 @@ workout() _mount -o degraded $device_2 $SCRATCH_MNT # Check if missing device is reported as in the 196.out $BTRFS_UTIL_PROG filesystem show -m $SCRATCH_MNT | \ - _filter_btrfs_filesystem_show + _filter_btrfs_filesystem_show > $tmp.output 2>&1 + cat $tmp.output >> $seqres.full + grep -q "$device_1" $tmp.output && _fail "found stale device" _scratch_unmount _scratch_dev_pool_put@@ -58,5 +60,6 @@ workout "raid6" "4" workout "raid10" "4" # success, all done +echo "Silence is golden" status=0 exittests/btrfs/198.out +1
-24@@ -1,25 +1,2 @@ QA output created by 198 -raid1 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - -raid5 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - -raid6 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - -raid10 -Label: none uuid: <UUID> - Total devices <NUM> FS bytes used <SIZE> - devid <DEVID> size <SIZE> used <SIZE> path SCRATCH_DEV - *** Some devices missing - +Silence is golden
Re: [PATCH v2 2/3] fstests: change how we test for supported raid configs
In reply to Josef BacikQuoted 78 lines · Josef Bacik
In btrfs there's a few ways we limit the RAID profiles we'll use. We have the raid56 feature that can be compiled out, zoned devices don't support certain raid configurations, and you can manually set
BTRFS_PROFILE_CONFIGSto limit what you're testing.To handle all of these different scenarios in the same way, update
_btrfs_get_profile_configs()to check for RAID56 support and remove it if it is not there, and then add_require_btrfs_raid_typeand_check_btrfs_raid_typeto get all the settings and then check if the requested raid type is available.From there I've updated all of the existing tests that use
_require_btrfs_fs_featureraid56to use
_require_btrfs_raid_type<type>where appropriate.
Signed-off-by Josef Bacik <[email protected]>--- common/btrfs | 29 ++++++++++++++++++++++++----- tests/btrfs/125 | 2 +- tests/btrfs/148 | 3 ++- tests/btrfs/157 | 2 +- tests/btrfs/158 | 2 +- 5 files changed, 29 insertions(+), 9 deletions(-)common/btrfs +14
-5@@ -98,11 +98,6 @@ _require_btrfs_fs_feature() modprobe btrfs > /dev/null 2>&1 [ -e /sys/fs/btrfs/features/$feat ] || \ _notrun "Feature $feat not supported by the available btrfs version" - - if [ $feat = "raid56" ]; then - # Zoned btrfs only supports SINGLE profile - _require_non_zoned_device "${SCRATCH_DEV}" - fi } _require_btrfs_fs_sysfs()@@ -218,6 +213,8 @@ _btrfs_get_profile_configs() return fi + modprobe btrfs > /dev/null 2>&1 + local unsupported=() if [ "$1" == "replace" ]; then # We can't do replace with these profiles because they@@ -237,6 +234,14 @@ _btrfs_get_profile_configs() ) fi + if [ ! -e /sys/fs/btrfs/features/raid56 ]; then + # We don't have raid56 compiled in, remove them + unsupported+=( + "raid5" + "raid6" + ) + fi + if _scratch_btrfs_is_zoned; then # Zoned btrfs only supports SINGLE profile unsupported+=(@@ -792,3 +797,17 @@ _has_btrfs_sysfs_feature_attr() test -e /sys/fs/btrfs/features/$feature_attr } + +_check_btrfs_raid_type() +{ + _btrfs_get_profile_configs
common/btrfs _has_btrfs_sysfs_feature_attr() · line 813
Quoted 10 lines
common/btrfs +10
-0+ if [[ ! "${_btrfs_profile_configs[@]}" =~ "$1" ]]; then + return 1 + fi + return 0 +} + +_require_btrfs_raid_type() +{ + _check_btrfs_raid_type $1 || _notrun "$1 isn't supported by the profile config" +}
We also call notrun for zoned devices.
I suggest tweaking the notrun message to:
_notrun "$1 isn't supported by the profile config or scratch device"
I have made this change and applied it.
Thanks, Anand
Quoted 53 lines
tests/btrfs/125 +1
-1@@ -41,7 +41,7 @@ _supported_fs btrfs _require_scratch_dev_pool 3 _test_unmount _require_btrfs_forget_or_module_loadable -_require_btrfs_fs_feature raid56 +_require_btrfs_raid_type raid5 _scratch_dev_pool_get 3tests/btrfs/148 +2
-1@@ -17,7 +17,8 @@ _supported_fs btrfs _require_scratch _require_scratch_dev_pool 4 _require_odirect -_require_btrfs_fs_feature raid56 +_require_btrfs_raid_type raid5 +_require_btrfs_raid_type raid6 _scratch_dev_pool_get 4tests/btrfs/157 +1
-1@@ -32,7 +32,7 @@ _begin_fstest auto quick raid read_repair _supported_fs btrfs _require_scratch_dev_pool 4 _require_btrfs_command inspect-internal dump-tree -_require_btrfs_fs_feature raid56 +_require_btrfs_raid_type raid6 get_physical() {tests/btrfs/158 +1
-1@@ -24,7 +24,7 @@ _begin_fstest auto quick raid scrub _supported_fs btrfs _require_scratch_dev_pool 4 _require_btrfs_command inspect-internal dump-tree -_require_btrfs_fs_feature raid56 +_require_btrfs_raid_type raid5 get_physical() {
Re: [PATCH v2 3/3] fstests: update tests to skip unsupported raid profile types
In reply to Josef BacikQuoted 6 lines · Josef Bacik
Tests btrfs/197, btrfs/198, and btrfs/297 test multiple raid types in their
workout()function. We may not support some of the raid types, so add a check in theworkout()function to skip any incompatible raid profiles.Signed-off-by Josef Bacik <[email protected]>
Thx
Anand
Quoted 87 lines
--- tests/btrfs/197 | 10 ++++++++-- tests/btrfs/198 | 10 ++++++++-- tests/btrfs/297 | 10 ++++++++++ 3 files changed, 26 insertions(+), 4 deletions(-)tests/btrfs/197 +8
-2@@ -30,14 +30,20 @@ _supported_fs btrfs _require_test _require_scratch _require_scratch_dev_pool 5 -# Zoned btrfs only supports SINGLE profile -_require_non_zoned_device ${SCRATCH_DEV} +# We require at least one raid setup, raid1 is the easiest, use this to gate on +# wether or not we run this test +_require_btrfs_raid_type raid1 workout() { raid=$1 device_nr=$2 + if ! _check_btrfs_raid_type $raid; then + echo "$raid isn't supported, skipping" >> $seqres.full + return + fi + echo $raid >> $seqres.full _scratch_dev_pool_get $device_nr _spare_dev_gettests/btrfs/198 +8
-2@@ -18,8 +18,9 @@ _supported_fs btrfs _require_command "$WIPEFS_PROG" wipefs _require_scratch _require_scratch_dev_pool 4 -# Zoned btrfs only supports SINGLE profile -_require_non_zoned_device ${SCRATCH_DEV} +# We require at least one raid setup, raid1 is the easiest, use this to gate on +# wether or not we run this test +_require_btrfs_raid_type raid1 _fixed_by_kernel_commit 96c2e067ed3e3e \ "btrfs: skip devices without magic signature when mounting"@@ -28,6 +29,11 @@ workout() raid=$1 device_nr=$2 + if ! _check_btrfs_raid_type $raid; then + echo "$raid isn't supported, skipping" >> $seqres.full + return + fi + echo $raid >> $seqres.full _scratch_dev_pool_get $device_nrtests/btrfs/297 +10
-0@@ -18,11 +18,21 @@ _require_scratch_dev_pool 3 _fixed_by_kernel_commit 486c737f7fdc \ "btrfs: raid56: always verify the P/Q contents for scrub" +# If neither raid5 or raid6 are supported do _notrun +if ! _check_btrfs_raid_type raid5 && ! _check_btrfs_raid_type raid6; then + _notrun "requires either raid5 or raid6 support" +fi + workload() { local profile=$1 local nr_devs=$2 + if ! _check_btrfs_raid_type $profile; then + echo "$profile isn't supported, skipping" >> $seqres.full + return + fi + echo "=== Testing $nr_devs devices $profile ===" >> $seqres.full _scratch_dev_pool_get $nr_devs