fstests: add tests for btrfs' raid-stripe-tree feature
v8 cover fstests: add tests for btrfs' raid-stripe-tree feature
Changes in v8:
- The RAID stripe tree dump filter has been updated; now, it also handles trailing whitespace and version filtering.
- Patches re-ordered.
- Link to v7: https://lore.kernel.org/all/[email protected]
Changes in v7:
- Fixed trailing whitespace in the .out files
- Fixed the following test statement in 30[4-8]:
test _get_page_size -eq 4096
- Link to v6: https://lore.kernel.org/r/[email protected]
--- original cover page from Johannes ----
Add tests for btrfs' raid-stripe-tree feature. All of these test work by writing a specific pattern to a newly created filesystem and afterwards using btrfs inspect-internal -t raid-stripe $SCRATCH_DEV_POOL to verify the placement and the layout of the metadata.
The md5sum of each file will be compared as well after a re-mount of the filesystem.
---
Changes in v6:
- require 4k pagesize for all tests as output depends on page size
- Add Filipe's Reviewed-by
- Link to v5: https://lore.kernel.org/r/[email protected]
Changes in v5:
- add _require_btrfs_free_space_tree helper and use in tests
- Link to v4: https://lore.kernel.org/r/[email protected]
Changes in v4:
- add _require_btrfs_no_compress to all tests
- add _require_btrfs_no_nodatacow helper and add to btrfs/308
- add _require_btrfs_feature "free_space_tree" to all tests
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- added 'raid-stripe-tree' to mkfs options, as only zoned raid gets it automatically
- Rename test cases as btrfs/302 and btrfs/303 already exist upstream
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Re-ordered series so the newly introduced group is added before the tests
- Changes Filipe requested to the tests.
- Link to v1: https://lore.kernel.org/r/[email protected]
Subject: [PATCH v8 00/10] *** SUBJECT HERE ***
*** BLURB HERE ***
Anand Jain (1):
common: add _filter_trailing_whitespace
Johannes Thumshirn (9):
fstests: doc: add new raid-stripe-tree group
common: add _require_btrfs_no_nodatacow helper
common: add _require_btrfs_free_space_tree
common: add filter for btrfs raid-stripe dump
btrfs: add fstest for stripe-tree metadata with 4k write
btrfs: add fstest for 8k write spanning two stripes on
raid-stripe-tree
btrfs: add fstest for writing to a file at an offset with RST
btrfs: add fstests to write 128k to a RST filesystem
btrfs: add fstest for overwriting a file partially with RST
common/btrfs | 17 +++++++ common/filter | 5 +++ common/filter.btrfs | 15 +++++++ doc/group-names.txt | 1 + tests/btrfs/304 | 58 ++++++++++++++++++++++++ tests/btrfs/304.out | 58 ++++++++++++++++++++++++ tests/btrfs/305 | 63 ++++++++++++++++++++++++++ tests/btrfs/305.out | 82 ++++++++++++++++++++++++++++++++++ tests/btrfs/306 | 61 +++++++++++++++++++++++++ tests/btrfs/306.out | 75 +++++++++++++++++++++++++++++++ tests/btrfs/307 | 58 ++++++++++++++++++++++++ tests/btrfs/307.out | 65 +++++++++++++++++++++++++++ tests/btrfs/308 | 62 ++++++++++++++++++++++++++ tests/btrfs/308.out | 106 ++++++++++++++++++++++++++++++++++++++++++++ 14 files changed, 726 insertions(+) create mode 100755 tests/btrfs/304 create mode 100644 tests/btrfs/304.out create mode 100755 tests/btrfs/305 create mode 100644 tests/btrfs/305.out create mode 100755 tests/btrfs/306 create mode 100644 tests/btrfs/306.out create mode 100755 tests/btrfs/307 create mode 100644 tests/btrfs/307.out create mode 100755 tests/btrfs/308 create mode 100644 tests/btrfs/308.out
Signature · 1 line
2.38.1
PATCH v8 1/10 fstests: doc: add new raid-stripe-tree group
From: Johannes Thumshirn <[email protected]>
Add a new test group for testing the raid-stripe-tree feature of btrfs with fstests.
--- doc/group-names.txt | 1 + 1 file changed, 1 insertion(+)
doc/group-names.txt +1 -0
@@ -94,6 +94,7 @@ punch fallocate FALLOC_FL_PUNCH_HOLE qgroup btrfs qgroup feature quota filesystem usage quotas raid btrfs RAID +raid-stripe-tree btrfs raid-stripe-tree feature read_repair btrfs error correction on read failure realtime XFS realtime volumes recoveryloop crash recovery loops
Signature · 1 line
2.38.1
PATCH v8 2/10 common: add _require_btrfs_no_nodatacow helper
In reply to Anand JainFrom: Johannes Thumshirn <[email protected]>
--- common/btrfs | 7 +++++++ 1 file changed, 7 insertions(+)
common/btrfs +7 -0
@@ -120,6 +120,13 @@ _require_btrfs_no_compress() fi } +_require_btrfs_no_nodatacow() +{ + if _normalize_mount_options "$MOUNT_OPTIONS" | grep -q "nodatacow"; then + _notrun "This test requires no nodatacow enabled" + fi +} + _check_btrfs_filesystem() { device=$1
Signature · 1 line
2.38.1
PATCH v8 3/10 common: add _require_btrfs_free_space_tree
In reply to Anand JainFrom: Johannes Thumshirn <[email protected]>
--- common/btrfs | 10 ++++++++++ 1 file changed, 10 insertions(+)
common/btrfs +10 -0
@@ -127,6 +127,16 @@ _require_btrfs_no_nodatacow() fi } +_require_btrfs_free_space_tree() +{ + _scratch_mkfs > /dev/null 2>&1 + if ! $BTRFS_UTIL_PROG inspect-internal dump-super $SCRATCH_DEV | \ + grep -q "FREE_SPACE_TREE" + then + _notrun "This test requires a free-space-tree" + fi +} + _check_btrfs_filesystem() { device=$1
Signature · 1 line
2.38.1
PATCH v8 4/10 common: add _filter_trailing_whitespace
In reply to Anand JainThe command 'btrfs inspect-internal dump-tree -t raid_stripe' introduces trailing whitespace in its output.
Apply a filter to remove it. Used in btrfs/30[4-8][.out].
--- common/filter | 5 +++++ 1 file changed, 5 insertions(+)
common/filter +5 -0
@@ -651,5 +651,10 @@ _filter_bash() sed -e "s/^bash: line 1: /bash: /" } +_filter_trailing_whitespace() +{ + sed -E -e "s/\s+$//" +} + # make sure this script returns success /bin/true
Signature · 1 line
2.38.1
PATCH v8 5/10 common: add filter for btrfs raid-stripe dump
In reply to Anand JainFrom: Johannes Thumshirn <[email protected]>
--- common/filter.btrfs | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
common/filter.btrfs +15 -0
@@ -126,5 +126,20 @@ _filter_btrfs_cloner_error() sed -e "s/\(clone failed:\) Operation not supported/\1 Invalid argument/g" } +# filter output of "btrfs inspect-internal dump-tree -t raid-stripe" +_filter_stripe_tree() +{ + _filter_trailing_whitespace | _filter_btrfs_version |\ + sed -E -e "s/leaf [0-9]+ items [0-9]+ free space [0-9]+ generation [0-9]+ owner RAID_STRIPE_TREE/leaf XXXXXXXXX items X free space XXXXX generation X owner RAID_STRIPE_TREE/" \ + -e "s/leaf [0-9]+ flags 0x1\(WRITTEN\) backref revision 1/leaf XXXXXXXXX flags 0x1\(WRITTEN\) backref revision 1/" \ + -e "s/checksum stored [0-9a-f]+/checksum stored <CHECKSUM>/" \ + -e "s/checksum calced [0-9a-f]+/checksum calced <CHECKSUM>/" \ + -e "s/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/<UUID>/" \ + -e "s/item ([0-9]+) key \([0-9]+ RAID_STRIPE ([0-9]+)\) itemoff [0-9]+ itemsize ([0-9]+)/item \1 key \(XXXXXX RAID_STRIPE \2\) itemoff XXXXX itemsize \3/" \ + -e "s/stripe ([0-9]+) devid ([0-9]+) physical [0-9]+/stripe \1 devid \2 physical XXXXXXXXX/" \ + -e "s/total bytes [0-9]+/total bytes XXXXXXXX/" \ + -e "s/bytes used [0-9]+/bytes used XXXXXX/" +} + # make sure this script returns success /bin/true
Signature · 1 line
2.38.1
PATCH v8 6/10 btrfs: add fstest for stripe-tree metadata with 4k write
In reply to Anand JainFrom: Johannes Thumshirn <[email protected]>
Test a simple 4k write on all RAID profiles currently supported with the raid-stripe-tree.
--- tests/btrfs/304 | 58 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/304.out | 58 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 116 insertions(+) create mode 100755 tests/btrfs/304 create mode 100644 tests/btrfs/304.out
tests/btrfs/304 new file +58 -0
@@ -0,0 +1,58 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Western Digital Cooperation. All Rights Reserved. +# +# FS QA Test 304 +# +# Test on-disk layout of RAID Stripe Tree Metadata writing 4k to a new file on +# a pristine file system. +# +. ./common/preamble +_begin_fstest auto quick raid remount volume raid-stripe-tree + +. ./common/filter +. ./common/filter.btrfs + +_supported_fs btrfs +_require_btrfs_command inspect-internal dump-tree +_require_btrfs_mkfs_feature "raid-stripe-tree" +_require_scratch_dev_pool 4 +_require_btrfs_fs_feature "raid_stripe_tree" +_require_btrfs_fs_feature "free_space_tree" +_require_btrfs_free_space_tree +_require_btrfs_no_compress + +test $(_get_page_size) -eq 4096 || _notrun "this tests requires 4k pagesize" + +test_4k_write() +{ + local profile=$1 + local ndevs=$2 + + _scratch_dev_pool_get $ndevs + + echo "==== Testing $profile ====" + _scratch_pool_mkfs -d $profile -m $profile -O raid-stripe-tree + _scratch_mount + + $XFS_IO_PROG -fc "pwrite 0 4k" "$SCRATCH_MNT/foo" | _filter_xfs_io + + _scratch_cycle_mount + md5sum "$SCRATCH_MNT/foo" | _filter_scratch + + _scratch_unmount + + $BTRFS_UTIL_PROG inspect-internal dump-tree -t raid_stripe $SCRATCH_DEV_POOL |\ + _filter_stripe_tree + + _scratch_dev_pool_put +} + +echo "= Test basic 4k write =" +test_4k_write raid0 2 +test_4k_write raid1 2 +test_4k_write raid10 4 + +# success, all done +status=0 +exit
tests/btrfs/304.out new file +58 -0
58 lines not on this page · raw message
Signature · 1 line
2.38.1
PATCH v8 7/10 btrfs: add fstest for 8k write spanning two stripes on raid-stripe-tree
In reply to Anand JainFrom: Johannes Thumshirn <[email protected]>
Add a test-case writing 8k to a raid-stripe-tree formatted filesystem with one stripe pre-filled to 60k so the 8k are split into a 4k write finishing stripe 1 and a 4k write starting the next stripe.
--- tests/btrfs/305 | 63 ++++++++++++++++++++++++++++++++++ tests/btrfs/305.out | 82 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 145 insertions(+) create mode 100755 tests/btrfs/305 create mode 100644 tests/btrfs/305.out
tests/btrfs/305 new file +63 -0
@@ -0,0 +1,63 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Western Digital Cooperation. All Rights Reserved. +# +# FS QA Test 305 +# +# Test on-disk layout of RAID Stripe Tree Metadata by writing 8k to a new file +# with a filesystem prepropulated, so that 4k of the write are written to the +# 1st stripe and 4k start a new stripe. +# +. ./common/preamble +_begin_fstest auto quick raid remount volume raid-stripe-tree + +. ./common/filter +. ./common/filter.btrfs + +_supported_fs btrfs +_require_btrfs_command inspect-internal dump-tree +_require_btrfs_mkfs_feature "raid-stripe-tree" +_require_scratch_dev_pool 4 +_require_btrfs_fs_feature "raid_stripe_tree" +_require_btrfs_fs_feature "free_space_tree" +_require_btrfs_free_space_tree +_require_btrfs_no_compress + +test $(_get_page_size) -eq 4096 || _notrun "this tests requires 4k pagesize" + +test_8k_new_stripe() +{ + local profile=$1 + local ndevs=$2 + + _scratch_dev_pool_get $ndevs + + echo "==== Testing $profile ====" + _scratch_pool_mkfs -d $profile -m $profile -O raid-stripe-tree + _scratch_mount + + # Fill the first stripe up to 64k - 4k + $XFS_IO_PROG -fc "pwrite 0 60k" -c fsync "$SCRATCH_MNT/bar" | _filter_xfs_io + + # The actual 8k write + $XFS_IO_PROG -fc "pwrite 0 8k" "$SCRATCH_MNT/foo" | _filter_xfs_io + + _scratch_cycle_mount + md5sum "$SCRATCH_MNT/foo" | _filter_scratch + + _scratch_unmount + + $BTRFS_UTIL_PROG inspect-internal dump-tree -t raid_stripe $SCRATCH_DEV_POOL |\ + _filter_stripe_tree + + _scratch_dev_pool_put +} + +echo "= Test 8k write to a new file so that 4k start a new stripe =" +test_8k_new_stripe raid0 2 +test_8k_new_stripe raid1 2 +test_8k_new_stripe raid10 4 + +# success, all done +status=0 +exit
tests/btrfs/305.out new file +82 -0
82 lines not on this page · raw message
Signature · 1 line
2.38.1
PATCH v8 8/10 btrfs: add fstest for writing to a file at an offset with RST
In reply to Anand JainFrom: Johannes Thumshirn <[email protected]>
Add a fstest writing 4k at offset 64k to a file with one RAID tripe already pre-filled for a raid-stripe-tree formatted file system.
--- tests/btrfs/306 | 61 ++++++++++++++++++++++++++++++++++++ tests/btrfs/306.out | 75 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+) create mode 100755 tests/btrfs/306 create mode 100644 tests/btrfs/306.out
tests/btrfs/306 new file +61 -0
@@ -0,0 +1,61 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Western Digital Cooperation. All Rights Reserved. +# +# FS QA Test 306 +# +# Test on-disk layout of RAID Stripe Tree Metadata by writing 4k to an emppty +# file at offset 64k with one stripe pre-filled on an otherwise pristine +# filesystem. +# +. ./common/preamble +_begin_fstest auto quick raid remount volume raid-stripe-tree + +. ./common/filter +. ./common/filter.btrfs + +_supported_fs btrfs +_require_btrfs_command inspect-internal dump-tree +_require_btrfs_mkfs_feature "raid-stripe-tree" +_require_scratch_dev_pool 4 +_require_btrfs_fs_feature "raid_stripe_tree" +_require_btrfs_fs_feature "free_space_tree" +_require_btrfs_free_space_tree +_require_btrfs_no_compress + +test $(_get_page_size) -eq 4096 || _notrun "this tests requires 4k pagesize" + +test_4k_write_64koff() +{ + local profile=$1 + local ndevs=$2 + + _scratch_dev_pool_get $ndevs + + echo "==== Testing $profile ====" + _scratch_pool_mkfs -d $profile -m $profile -O raid-stripe-tree + _scratch_mount + + # precondition one stripe + $XFS_IO_PROG -fc "pwrite 0 64k" "$SCRATCH_MNT/bar" | _filter_xfs_io + + $XFS_IO_PROG -fc "pwrite 64k 4k" "$SCRATCH_MNT/foo" | _filter_xfs_io + + _scratch_cycle_mount + md5sum "$SCRATCH_MNT/foo" | _filter_scratch + + _scratch_unmount + + $BTRFS_UTIL_PROG inspect-internal dump-tree -t raid_stripe $SCRATCH_DEV_POOL |\ + _filter_stripe_tree + + _scratch_dev_pool_put +} +echo "= Test 4k write to an empty file at offset 64k with one stripe prefilled =" +test_4k_write_64koff raid0 2 +test_4k_write_64koff raid1 2 +test_4k_write_64koff raid10 4 + +# success, all done +status=0 +exit
tests/btrfs/306.out new file +75 -0
75 lines not on this page · raw message
Signature · 1 line
2.38.1
PATCH v8 9/10 btrfs: add fstests to write 128k to a RST filesystem
In reply to Anand JainFrom: Johannes Thumshirn <[email protected]>
Add a test writing 128k to a file on an empty filesystem formatted with a raid-stripe-tree.
--- tests/btrfs/307 | 58 ++++++++++++++++++++++++++++++++++++++++ tests/btrfs/307.out | 65 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 123 insertions(+) create mode 100755 tests/btrfs/307 create mode 100644 tests/btrfs/307.out
tests/btrfs/307 new file +58 -0
@@ -0,0 +1,58 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Western Digital Cooperation. All Rights Reserved. +# +# FS QA Test 307 +# +# Test on-disk layout of RAID Stripe Tree Metadata by writing 128k to a new +# file on a pristine filesystem +# +. ./common/preamble +_begin_fstest auto quick raid remount volume raid-stripe-tree + +. ./common/filter +. ./common/filter.btrfs + +_supported_fs btrfs +_require_btrfs_command inspect-internal dump-tree +_require_btrfs_mkfs_feature "raid-stripe-tree" +_require_scratch_dev_pool 4 +_require_btrfs_fs_feature "raid_stripe_tree" +_require_btrfs_fs_feature "free_space_tree" +_require_btrfs_free_space_tree +_require_btrfs_no_compress + +test $(_get_page_size) -eq 4096 || _notrun "this tests requires 4k pagesize" + +test_128k_write() +{ + local profile=$1 + local ndevs=$2 + + _scratch_dev_pool_get $ndevs + + echo "==== Testing $profile ====" + _scratch_pool_mkfs -d $profile -m $profile -O raid-stripe-tree + _scratch_mount + + $XFS_IO_PROG -fc "pwrite 0 128k" "$SCRATCH_MNT/foo" | _filter_xfs_io + + _scratch_cycle_mount + md5sum "$SCRATCH_MNT/foo" | _filter_scratch + + _scratch_unmount + + $BTRFS_UTIL_PROG inspect-internal dump-tree -t raid_stripe $SCRATCH_DEV_POOL |\ + _filter_stripe_tree + + _scratch_dev_pool_put +} + +echo "= Test 128k write to empty file =" +test_128k_write raid0 2 +test_128k_write raid1 2 +test_128k_write raid10 4 + +# success, all done +status=0 +exit
tests/btrfs/307.out new file +65 -0
65 lines not on this page · raw message
Signature · 1 line
2.38.1
PATCH v8 10/10 btrfs: add fstest for overwriting a file partially with RST
In reply to Anand JainFrom: Johannes Thumshirn <[email protected]>
Add a test writing 128k to an empty file with one stripe already pre-filled on-disk. Then overwrite a portion of the file in the middle.
--- tests/btrfs/308 | 62 ++++++++++++++++++++++++++ tests/btrfs/308.out | 106 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 168 insertions(+) create mode 100755 tests/btrfs/308 create mode 100644 tests/btrfs/308.out
tests/btrfs/308 new file +62 -0
@@ -0,0 +1,62 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Western Digital Cooperation. All Rights Reserved. +# +# FS QA Test 308 +# +# Test on-disk layout of RAID Stripe Tree Metadata by writing 128k to an empty +# file on a filesystem that has one stripe already pre-filled. Afterwards +# overwrite a portion of the file. +# +. ./common/preamble +_begin_fstest auto quick raid remount volume raid-stripe-tree + +. ./common/filter +. ./common/filter.btrfs + +_supported_fs btrfs +_require_btrfs_command inspect-internal dump-tree +_require_btrfs_mkfs_feature "raid-stripe-tree" +_require_scratch_dev_pool 4 +_require_btrfs_fs_feature "raid_stripe_tree" +_require_btrfs_fs_feature "free_space_tree" +_require_btrfs_free_space_tree +_require_btrfs_no_compress +_require_btrfs_no_nodatacow + +test $(_get_page_size) -eq 4096 || _notrun "this tests requires 4k pagesize" + +test_128k_write_overwrite() +{ + local profile=$1 + local ndevs=$2 + + _scratch_dev_pool_get $ndevs + + echo "==== Testing $profile ====" + _scratch_pool_mkfs -d $profile -m $profile -O raid-stripe-tree + _scratch_mount + + $XFS_IO_PROG -fc "pwrite -W 0 32k" "$SCRATCH_MNT/bar" | _filter_xfs_io + $XFS_IO_PROG -fc "pwrite -W 0 128k" "$SCRATCH_MNT/foo" | _filter_xfs_io + $XFS_IO_PROG -fc "pwrite -W 64k 8k" "$SCRATCH_MNT/foo" | _filter_xfs_io + + _scratch_cycle_mount + md5sum "$SCRATCH_MNT/foo" | _filter_scratch + + _scratch_unmount + + $BTRFS_UTIL_PROG inspect-internal dump-tree -t raid_stripe $SCRATCH_DEV_POOL |\ + _filter_stripe_tree + + _scratch_dev_pool_put +} + +echo "= Test 128k write to empty file with 1st stripe partially prefilled then overwrite =" +test_128k_write_overwrite raid0 2 +test_128k_write_overwrite raid1 2 +test_128k_write_overwrite raid10 4 + +# success, all done +status=0 +exit
tests/btrfs/308.out new file +106 -0
106 lines not on this page · raw message
Signature · 1 line
2.38.1