btrfs: simple quotas fstests
cover btrfs: simple quotas fstests
Add new tests (btrfs/400 for now, but intend to grab a real number before the last version)
Also made modifications for passing existing qgroups tests when possible and for passing all tests with simple quota enabled via mkfs and with squota-aware btrfs check.
btrfs/400 depends on the kernel patchset:
https://lore.kernel.org/linux-btrfs/[email protected]/T/#t
and the btrfs-progs patchset:
https://lore.kernel.org/linux-btrfs/[email protected]/T/#t
(and config appropriate binaries to use squota-aware versions)
Boris Burkov (5):
btrfs/400: new test for simple quotas
common/btrfs: quota mode helpers
common/btrfs: quota rescan helpers
btrfs: use new rescan wrapper
btrfs: skip squota incompatible tests
common/btrfs | 68 +++++++ tests/btrfs/017 | 1 + tests/btrfs/022 | 1 + tests/btrfs/028 | 2 +- tests/btrfs/057 | 1 + tests/btrfs/091 | 3 +- tests/btrfs/104 | 2 +- tests/btrfs/123 | 2 +- tests/btrfs/126 | 2 +- tests/btrfs/139 | 2 +- tests/btrfs/153 | 2 +- tests/btrfs/171 | 6 +- tests/btrfs/179 | 2 +- tests/btrfs/180 | 2 +- tests/btrfs/190 | 2 +- tests/btrfs/193 | 2 +- tests/btrfs/210 | 2 +- tests/btrfs/224 | 6 +- tests/btrfs/230 | 2 +- tests/btrfs/232 | 2 +- tests/btrfs/400 | 444 ++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/400.out | 2 + 22 files changed, 538 insertions(+), 20 deletions(-) create mode 100755 tests/btrfs/400 create mode 100644 tests/btrfs/400.out
Signature · 1 line
2.41.0
PATCH 1/5 btrfs/400: new test for simple quotas
Test some interesting basic and edge cases of simple quotas.
To some extent, this is redundant with the alternate testing strategy of using MKFS_OPTIONS to enable simple quotas, running the full suite and relying on kernel warnings and fsck to surface issues.
--- tests/btrfs/400 | 439 ++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/400.out | 2 + 2 files changed, 441 insertions(+) create mode 100755 tests/btrfs/400 create mode 100644 tests/btrfs/400.out
tests/btrfs/400 new file +439 -0
@@ -0,0 +1,439 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Meta Platforms, Inc. All Rights Reserved. +# +# FS QA Test 400 +# +# Test common btrfs simple quotas scenarios involving sharing extents and +# removing them in various orders. +# +. ./common/preamble +_begin_fstest auto quick qgroup copy_range snapshot + +# Import common functions. +# . ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_require_scratch + +SUBV=$SCRATCH_MNT/subv +NESTED=$SCRATCH_MNT/subv/nested +SNAP=$SCRATCH_MNT/snap +K=1024 +M=$(($K * $K)) +NR_FILL=1024 +FILL_SZ=$((8 * $K)) +TOTAL_FILL=$(($NR_FILL * $FILL_SZ)) +EB_SZ=$((16 * $K)) +EXT_SZ=$((128 * M)) +LIMIT_NR=8 +LIMIT=$(($EXT_SZ * $LIMIT_NR)) + +get_qgroup_usage() +{ + local qgroupid=$1 + + $BTRFS_UTIL_PROG qgroup show --sync --raw $SCRATCH_MNT | grep "$qgroupid" | $AWK_PROG '{print $3}' +} + +get_subvol_usage() +{ + local subvolid=$1 + get_qgroup_usage "0/$subvolid" +} + +count_subvol_owned_metadata() +{ + local subvolid=$1 + # find nodes and leaves owned by the subvol, then get unique offsets + # to account for snapshots sharing metadata. + count=$($BTRFS_UTIL_PROG inspect-internal dump-tree $SCRATCH_DEV | \ + grep "owner $subvolid" | awk '{print $2}' | sort | uniq | wc -l) + # output bytes rather than number of metadata blocks + echo $(($count * $EB_SZ)) +} + +check_qgroup_usage() +{ + local qgroupid=$1 + local expected=$2 + local actual=$(get_qgroup_usage $qgroupid) + + [ $expected -eq $actual ] || _fail "qgroup $qgroupid mismatched usage $actual vs $expected" +} + +check_subvol_usage() +{ + local subvolid=$1 + local expected_data=$2 + # need to sync to see updated usage numbers. + # could probably improve by placing syncs only where they are strictly + # needed after actual operations, but it is more error prone. + sync + + local expected_meta=$(count_subvol_owned_metadata $subvolid) + local actual=$(get_subvol_usage $subvolid) + local expected=$(($expected_data + $expected_meta)) + + [ $expected -eq $actual ] || _fail "subvol $subvolid mismatched usage $actual vs $expected (expected data $expected_data expected meta $expected_meta diff $(($actual - $expected)))" + echo "OK $subvolid $expected_data $expected_meta $actual" >> $seqres.full +} + +set_subvol_limit() +{ + local subvolid=$1 + local limit=$2 + + $BTRFS_UTIL_PROG qgroup limit $2 0/$1 $SCRATCH_MNT +} + +sync_check_subvol_usage() +{ + sync + check_subvol_usage $@ +} + +trigger_cleaner() +{ + echo "trigger cleaner" > /dev/kmsg + $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT + sleep 1 + $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT + echo "cleaner triggered" > /dev/kmsg +} + +cycle_mount_check_subvol_usage() +{ + echo "cycle mounting" > /dev/kmsg + _scratch_cycle_mount + check_subvol_usage $@ + echo "cycle mount done" > /dev/kmsg +} + + +do_write() +{ + local file=$1 + local sz=$2 + + echo "write" > /dev/kmsg + $XFS_IO_PROG -fc "pwrite -q 0 $sz" $file + local ret=$? + echo "write done" > /dev/kmsg + return $ret +} + +do_enospc_write() +{ + local file=$1 + local sz=$2 + + do_write $file $sz 2>/dev/null && _fail "write expected enospc" +} + +do_falloc() +{ + local file=$1 + local sz=$2 + + $XFS_IO_PROG -fc "falloc 0 $sz" $file +} + +do_enospc_falloc() +{ + local file=$1 + local sz=$2 + + do_falloc $file $sz 2>/dev/null && _fail "falloc expected enospc" +} + +enable_quota() +{ + local mode=$1 + + [ $mode == "n" ] && return + arg=$([ $mode == "s" ] && echo "--simple") + + $BTRFS_UTIL_PROG quota enable $arg $SCRATCH_MNT +} + +prepare() +{ + echo "preparing" > /dev/kmsg + _scratch_mkfs >> $seqres.full + _scratch_mount + enable_quota "s" + $BTRFS_UTIL_PROG subvolume create $SUBV >> $seqres.full + set_subvol_limit 256 $LIMIT + check_subvol_usage 256 0 + + echo "filling" > /dev/kmsg + # Create a bunch of little filler files to generate several levels in + # the btree, to make snapshotting sharing scenarios complex enough. + $FIO_PROG --name=filler --directory=$SUBV --rw=randwrite --nrfiles=$NR_FILL --filesize=$FILL_SZ >/dev/null 2>&1 + echo "filled" > /dev/kmsg + check_subvol_usage 256 $TOTAL_FILL + + # Create a single file whose extents we will explicitly share/unshare. + do_write $SUBV/f $EXT_SZ + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + echo "prepared" > /dev/kmsg +} + +prepare_snapshotted() +{ + echo "prepare snapshotted" > /dev/kmsg + prepare + $BTRFS_UTIL_PROG subvolume snapshot $SUBV $SNAP >> $seqres.full + echo "snapshot" >> $seqres.full + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + check_subvol_usage 257 0 + echo "prepared snapshotted" > /dev/kmsg +} + +prepare_nested() +{ + echo "prepare nested" > /dev/kmsg + prepare + $BTRFS_UTIL_PROG qgroup create 1/100 $SCRATCH_MNT + $BTRFS_UTIL_PROG qgroup assign 0/256 1/100 $SCRATCH_MNT >> $seqres.full + $BTRFS_UTIL_PROG subvolume create $NESTED >> $seqres.full + do_write $NESTED/f $EXT_SZ + check_subvol_usage 257 $EXT_SZ + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + local subv_usage=$(get_subvol_usage 256) + local nested_usage=$(get_subvol_usage 257) + check_qgroup_usage 1/100 $(($subv_usage + $nested_usage)) + echo "prepared nested" > /dev/kmsg +} + +basic_accounting() +{ + echo "basic" > /dev/kmsg + prepare + echo "basic" >> $seqres.full + echo "delete file" >> $seqres.full + rm $SUBV/f + check_subvol_usage 256 $TOTAL_FILL + cycle_mount_check_subvol_usage 256 $TOTAL_FILL + do_write $SUBV/tmp 512M + rm $SUBV/tmp + do_write $SUBV/tmp 512M + rm $SUBV/tmp + do_enospc_falloc $SUBV/large_falloc 2G + do_enospc_write $SUBV/large 2G + _scratch_unmount +} + +reservation_accounting() +{ + echo "rsv" > /dev/kmsg + prepare + for i in $(seq 10); do + do_write $SUBV/tmp 512M + rm $SUBV/tmp + done + do_enospc_write $SUBV/large 2G + _scratch_unmount +} + +snapshot_accounting() +{ + echo "snap" > /dev/kmsg + prepare_snapshotted + echo "unshare snapshot metadata" >> $seqres.full + touch $SNAP/f + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + check_subvol_usage 257 0 + echo "unshare snapshot data" >> $seqres.full + do_write $SNAP/f $EXT_SZ + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + check_subvol_usage 257 $EXT_SZ + echo "delete snapshot file" >> $seqres.full + rm $SNAP/f + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + check_subvol_usage 257 0 + echo "delete original file" >> $seqres.full + rm $SUBV/f + check_subvol_usage 256 $TOTAL_FILL + check_subvol_usage 257 0 + cycle_mount_check_subvol_usage 256 $TOTAL_FILL + check_subvol_usage 257 0 + _scratch_unmount +} + +delete_subvol_file() +{ + echo "del sv ref" > /dev/kmsg + prepare_snapshotted + echo "delete subvol file first" >> $seqres.full + rm $SUBV/f + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + check_subvol_usage 257 0 + rm $SNAP/f + trigger_cleaner + check_subvol_usage 256 $TOTAL_FILL + check_subvol_usage 257 0 + cycle_mount_check_subvol_usage 256 $TOTAL_FILL + check_subvol_usage 257 0 + _scratch_unmount +} + +delete_snapshot_file() +{ + echo "del snap ref" > /dev/kmsg + prepare_snapshotted + echo "delete snapshot file first" >> $seqres.full + rm $SNAP/f + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + check_subvol_usage 257 0 + rm $SUBV/f + check_subvol_usage 256 $TOTAL_FILL + check_subvol_usage 257 0 + cycle_mount_check_subvol_usage 256 $TOTAL_FILL + check_subvol_usage 257 0 + _scratch_unmount +} + +delete_subvol() +{ + echo "del sv" > /dev/ksmg + prepare_snapshotted + echo "delete subvol first" >> $seqres.full + $BTRFS_UTIL_PROG subvolume delete $SUBV >> $seqres.full + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + check_subvol_usage 257 0 + rm $SNAP/f + trigger_cleaner + check_subvol_usage 256 $TOTAL_FILL + check_subvol_usage 257 0 + $BTRFS_UTIL_PROG subvolume delete $SNAP >> $seqres.full + trigger_cleaner + check_subvol_usage 256 0 + check_subvol_usage 257 0 + cycle_mount_check_subvol_usage 256 0 + check_subvol_usage 257 0 + _scratch_unmount +} + +delete_snapshot() +{ + echo "del snap" > /dev/ksmg + prepare_snapshotted + echo "delete snapshot first" >> $seqres.full + $BTRFS_UTIL_PROG subvolume delete $SNAP >> $seqres.full + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + check_subvol_usage 257 0 + $BTRFS_UTIL_PROG subvolume delete $SUBV >> $seqres.full + trigger_cleaner + check_subvol_usage 256 0 + check_subvol_usage 257 0 + _scratch_unmount +} + +nested_accounting() +{ + echo "nested" > /dev/ksmg + prepare_nested + echo "nested" >> $seqres.full + echo "delete file" >> $seqres.full + rm $SUBV/f + check_subvol_usage 256 $TOTAL_FILL + check_subvol_usage 257 $EXT_SZ + local subv_usage=$(get_subvol_usage 256) + local nested_usage=$(get_subvol_usage 257) + check_qgroup_usage 1/100 $(($subv_usage + $nested_usage)) + rm $NESTED/f + check_subvol_usage 256 $TOTAL_FILL + check_subvol_usage 257 0 + subv_usage=$(get_subvol_usage 256) + nested_usage=$(get_subvol_usage 257) + check_qgroup_usage 1/100 $(($subv_usage + $nested_usage)) + _scratch_unmount +} + +enable_mature() +{ + echo "mature" > /dev/ksmg + _scratch_mkfs >> $seqres.full + _scratch_mount + $BTRFS_UTIL_PROG subvolume create $SUBV >> $seqres.full + do_write $SUBV/f $EXT_SZ + sync + enable_quota "s" + set_subvol_limit 256 $LIMIT + _scratch_cycle_mount + usage=$(get_subvol_usage 256) + [ $usage -lt $EXT_SZ ] || _fail "captured usage from before enable $usage" + do_write $SUBV/g $EXT_SZ + usage=$(get_subvol_usage 256) + [ $usage -lt $EXT_SZ ] && "failed to capture usage after enable $usage" + check_subvol_usage 256 $EXT_SZ + rm $SUBV/f + check_subvol_usage 256 $EXT_SZ + _scratch_cycle_mount + rm $SUBV/g + check_subvol_usage 256 0 + _scratch_unmount +} + +reflink_accounting() +{ + echo "reflink" > /dev/kmsg + prepare + # do more reflinks than would fit + for i in $(seq $((NR_LIMIT * 2))); do + cp --reflink=always $SUBV/f $SUBV/f.i + done + # no additional data usage from reflinks + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + _scratch_unmount +} + +delete_link() +{ + echo "delete link first" > /dev/kmsg + prepare + cp --reflink=always $SUBV/f $SUBV/f.link + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + rm $SUBV/f.link + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + rm $SUBV/f + check_subvol_usage 256 $(($TOTAL_FILL)) + _scratch_unmount +} + +delete_linked() +{ + echo "delete linked first" > /dev/kmsg + prepare + cp --reflink=always $SUBV/f $SUBV/f.link + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + rm $SUBV/f + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + rm $SUBV/f.link + check_subvol_usage 256 $(($TOTAL_FILL)) + _scratch_unmount +} + +basic_accounting +reservation_accounting +snapshot_accounting +delete_subvol_file +delete_snapshot_file +delete_subvol +delete_snapshot +nested_accounting +enable_mature +reflink_accounting +delete_link +delete_linked + +echo "Silence is golden" + +# success, all done +status=0 +exit
tests/btrfs/400.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 400 +Silence is golden
Signature · 1 line
2.41.0
Re: [PATCH 1/5] btrfs/400: new test for simple quotas
Quoted 30 lines · Boris Burkov
Test some interesting basic and edge cases of simple quotas.
To some extent, this is redundant with the alternate testing strategy of using
MKFS_OPTIONSto enable simple quotas, running the full suite and relying on kernel warnings and fsck to surface issues.Signed-off-by Boris Burkov <[email protected]>--- tests/btrfs/400 | 439 ++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/400.out | 2 + 2 files changed, 441 insertions(+)create mode 100755 tests/btrfs/400
create mode 100644 tests/btrfs/400.outtests/btrfs/400 new file +10
-0@@ -0,0 +1,439 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Meta Platforms, Inc. All Rights Reserved. +# +# FS QA Test 400 +# +# Test common btrfs simple quotas scenarios involving sharing extents and +# removing them in various orders. +# +. ./common/preamble
tests/btrfs/400 line 20
Quoted 10 lines
tests/btrfs/400 new file +10
-0+_begin_fstest auto quick qgroup copy_range snapshot + +# Import common functions. +# . ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_require_scratch
I don't see any prerequisite checking and call of notrun() on the systems without the kernel or progs simple-quota support. Is it not required?
tests/btrfs/400 line 33
Quoted 13 lines
tests/btrfs/400 new file +13
-0+ +SUBV=$SCRATCH_MNT/subv +NESTED=$SCRATCH_MNT/subv/nested +SNAP=$SCRATCH_MNT/snap +K=1024 +M=$(($K * $K)) +NR_FILL=1024 +FILL_SZ=$((8 * $K)) +TOTAL_FILL=$(($NR_FILL * $FILL_SZ)) +EB_SZ=$((16 * $K)) +EXT_SZ=$((128 * M)) +LIMIT_NR=8 +LIMIT=$(($EXT_SZ * $LIMIT_NR))
Style consistency requires the use of lowercase for test local variables.
tests/btrfs/400 line 165
Quoted 4 lines
tests/btrfs/400 new file +4
-0+ +prepare() +{ + echo "preparing" > /dev/kmsg
Please use $seqres.full or stdout for debugging purpose.
Quoted 15 lines
tests/btrfs/400 new file +15
-0+ _scratch_mkfs >> $seqres.full + _scratch_mount + enable_quota "s" + $BTRFS_UTIL_PROG subvolume create $SUBV >> $seqres.full + set_subvol_limit 256 $LIMIT + check_subvol_usage 256 0 + + echo "filling" > /dev/kmsg + # Create a bunch of little filler files to generate several levels in + # the btree, to make snapshotting sharing scenarios complex enough. + $FIO_PROG --name=filler --directory=$SUBV --rw=randwrite --nrfiles=$NR_FILL --filesize=$FILL_SZ >/dev/null 2>&1 + echo "filled" > /dev/kmsg + check_subvol_usage 256 $TOTAL_FILL + + # Create a single file whose extents we will explicitly share/unshare.
tests/btrfs/400
Quoted 10 lines
tests/btrfs/400 new file +7
-0+ do_write $SUBV/f $EXT_SZ + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + echo "prepared" > /dev/kmsg +} + + +echo "Silence is golden"
We can have the echo part, like (echo 'prepared' > /dev/kmsg), directed to stdout; this will be useful for verification and debugging as well.
Thanks, Anand
Quoted 12 lines
tests/btrfs/400 new file +4
-0+ +# success, all done +status=0 +exittests/btrfs/400.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 400 +Silence is golden
Re: [PATCH 1/5] btrfs/400: new test for simple quotas
Quoted 44 lines · Anand Jain
Quoted 38 lines · Boris Burkov
Test some interesting basic and edge cases of simple quotas.
To some extent, this is redundant with the alternate testing strategy of using
MKFS_OPTIONSto enable simple quotas, running the full suite and relying on kernel warnings and fsck to surface issues.Signed-off-by Boris Burkov <[email protected]>---
tests/btrfs/400 | 439 ++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/400.out | 2 +2 files changed, 441 insertions(+)create mode 100755 tests/btrfs/400
create mode 100644 tests/btrfs/400.outtests/btrfs/400 new file +7
-0@@ -0,0 +1,439 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Meta Platforms, Inc. All Rights Reserved. +# +# FS QA Test 400 +# +# Test common btrfs simple quotas scenarios involving sharing extentsand
tests/btrfs/400 new file +10
-0+# removing them in various orders. +# +. ./common/preamble +_begin_fstest auto quick qgroup copy_range snapshot + +# Import common functions. +# . ./common/filter + +# real QA test starts here +
Quoted 9 lines
tests/btrfs/400 line 20
Quoted 3 lines
tests/btrfs/400 new file +3
-0+# Modify as appropriate. +_supported_fs btrfs +_require_scratchI don't see any prerequisite checking and call of
notrun()on the systems without the kernel or progs simple-quota support. Is it not required?
_require_scratch_qgroup() in patch 2/5 does that; So at this patch, we are expecting the testcase to run fine on both with and without kernel and progs simple quota patches.
Thanks, Anand
Quoted 76 lines
Quoted 4 lines
tests/btrfs/400 new file +3
-0+ +SUBV=$SCRATCH_MNT/subv +NESTED=$SCRATCH_MNT/subv/nestedtests/btrfs/400 line 33
Quoted 10 lines
tests/btrfs/400 new file +10
-0+SNAP=$SCRATCH_MNT/snap +K=1024 +M=$(($K * $K)) +NR_FILL=1024 +FILL_SZ=$((8 * $K)) +TOTAL_FILL=$(($NR_FILL * $FILL_SZ)) +EB_SZ=$((16 * $K)) +EXT_SZ=$((128 * M)) +LIMIT_NR=8 +LIMIT=$(($EXT_SZ * $LIMIT_NR))Style consistency requires the use of lowercase for test local variables.
tests/btrfs/400 line 165
Quoted 4 lines
tests/btrfs/400 new file +4
-0+ +prepare() +{ + echo "preparing" > /dev/kmsgPlease use $seqres.full or stdout for debugging purpose.
Quoted 21 lines
+
_scratch_mkfs>> $seqres.full
+_scratch_mount
+enable_quota"s"
+ $BTRFS_UTIL_PROG subvolume create $SUBV >> $seqres.full +set_subvol_limit256 $LIMIT
+check_subvol_usage256 0
+
+ echo "filling" >/dev/kmsg
+ # Create a bunch of little filler files to generate several levels in
+ # the btree, to make snapshotting sharing scenarios complex enough. + $FIO_PROG --name=filler --directory=$SUBV --rw=randwrite --nrfiles=$NR_FILL --filesize=$FILL_SZ >/dev/null 2>&1
+ echo "filled" >/dev/kmsg
+check_subvol_usage256 $TOTAL_FILL
+
+ # Create a single file whose extents we will explicitly share/unshare.
+do_write$SUBV/f $EXT_SZ
+check_subvol_usage256 $(($TOTAL_FILL + $EXT_SZ))
+ echo "prepared" >/dev/kmsgtests/btrfs/400 line 435
Quoted 7 lines
tests/btrfs/400 new file +4
-0+} + + +echo "Silence is golden"We can have the echo part, like (echo 'prepared' >
/dev/kmsg), directed to stdout; this will be useful for verification and debugging as well.Thanks, Anand
Quoted 12 lines
tests/btrfs/400 new file +4
-0+ +# success, all done +status=0 +exittests/btrfs/400.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 400 +Silence is golden
Re: [PATCH 1/5] btrfs/400: new test for simple quotas
In reply to Anand JainQuoted 53 lines · Anand Jain
Quoted 30 lines · Boris Burkov
Test some interesting basic and edge cases of simple quotas.
To some extent, this is redundant with the alternate testing strategy of using
MKFS_OPTIONSto enable simple quotas, running the full suite and relying on kernel warnings and fsck to surface issues.Signed-off-by Boris Burkov <[email protected]>--- tests/btrfs/400 | 439 ++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/400.out | 2 + 2 files changed, 441 insertions(+)create mode 100755 tests/btrfs/400
create mode 100644 tests/btrfs/400.outtests/btrfs/400 new file +10
-0@@ -0,0 +1,439 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Meta Platforms, Inc. All Rights Reserved. +# +# FS QA Test 400 +# +# Test common btrfs simple quotas scenarios involving sharing extents and +# removing them in various orders. +# +. ./common/preambletests/btrfs/400 line 20
Quoted 10 lines
tests/btrfs/400 new file +10
-0+_begin_fstest auto quick qgroup copy_range snapshot + +# Import common functions. +# . ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_require_scratchI don't see any prerequisite checking and call of
notrun()on the systems without the kernel or progs simple-quota support. Is it not required?
Quoted 17 lines
tests/btrfs/400 line 33
Quoted 13 lines
tests/btrfs/400 new file +13
-0+ +SUBV=$SCRATCH_MNT/subv +NESTED=$SCRATCH_MNT/subv/nested +SNAP=$SCRATCH_MNT/snap +K=1024 +M=$(($K * $K)) +NR_FILL=1024 +FILL_SZ=$((8 * $K)) +TOTAL_FILL=$(($NR_FILL * $FILL_SZ)) +EB_SZ=$((16 * $K)) +EXT_SZ=$((128 * M)) +LIMIT_NR=8 +LIMIT=$(($EXT_SZ * $LIMIT_NR))Style consistency requires the use of lowercase for test local variables.
Is this documented? There are lots of counterexamples. Will fix, of course.
Quoted 8 lines
tests/btrfs/400 line 165
Quoted 5 lines
tests/btrfs/400 new file +4
-0+ +prepare() +{ + echo "preparing" > /dev/kmsgPlease use $seqres.full or stdout for debugging purpose.
These messages are only helpful in dmesg, as they help to sequence test actions with kernel logs while debugging test failures.
I personally find it to be a great pattern for writing debuggable tests that don't end up with annoying fragile test.out files, so I would encourage you to play with the pattern and see what you think. Especially in vm-like environments there's really no downside, that I can see, it's just super nice to have the test actions in dmesg.
However, I appreciate that I'm the one doing the weird thing, so I'll remove the messages or throw them in seqres.full.
Quoted 47 lines
Quoted 17 lines
tests/btrfs/400 new file +15
-0+ _scratch_mkfs >> $seqres.full + _scratch_mount + enable_quota "s" + $BTRFS_UTIL_PROG subvolume create $SUBV >> $seqres.full + set_subvol_limit 256 $LIMIT + check_subvol_usage 256 0 + + echo "filling" > /dev/kmsg + # Create a bunch of little filler files to generate several levels in + # the btree, to make snapshotting sharing scenarios complex enough. + $FIO_PROG --name=filler --directory=$SUBV --rw=randwrite --nrfiles=$NR_FILL --filesize=$FILL_SZ >/dev/null 2>&1 + echo "filled" > /dev/kmsg + check_subvol_usage 256 $TOTAL_FILL + + # Create a single file whose extents we will explicitly share/unshare.tests/btrfs/400
Quoted 10 lines
tests/btrfs/400 new file +7
-0+ do_write $SUBV/f $EXT_SZ + check_subvol_usage 256 $(($TOTAL_FILL + $EXT_SZ)) + echo "prepared" > /dev/kmsg +} + + +echo "Silence is golden"We can have the echo part, like (echo 'prepared' >
/dev/kmsg), directed to stdout; this will be useful for verification and debugging as well.Thanks, Anand
Quoted 12 lines
tests/btrfs/400 new file +4
-0+ +# success, all done +status=0 +exittests/btrfs/400.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 400 +Silence is golden
PATCH 2/5 common/btrfs: quota mode helpers
In reply to Boris BurkovTo facilitate skipping tests depending on the qgroup mode after mkfs, add support for figuring out the mode. This cannot just rely on the new sysfs file, since it might not be present on older kernels.
--- common/btrfs | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
common/btrfs +43 -0
@@ -680,3 +680,46 @@ _require_btrfs_scratch_logical_resolve_v2() fi _scratch_unmount } + +_qgroup_mode_file() +{ + local mnt=$1 + + uuid=$(findmnt -n -o UUID $mnt) + echo /sys/fs/btrfs/${uuid}/qgroups/mode +} + +_qgroup_enabled_file() +{ + local mnt=$1 + + uuid=$(findmnt -n -o UUID $mnt) + echo /sys/fs/btrfs/${uuid}/qgroups/enabled +} + +_qgroup_mode() +{ + local mnt=$1 + + if [ ! -f "$(_qgroup_enabled_file $mnt)" ]; then + echo "disabled" + return + fi + + if [ -f "$(_qgroup_mode_file $mnt)" ]; then + cat $(_qgroup_mode_file $mnt) + elif [ $(cat $(_qgroup_enabled_file $mnt)) -eq "1" ]; then + echo "qgroup" + else + echo "disabled" # should not be reachable, the enabled file won't exist. + fi +} + +_require_scratch_qgroup() +{ + _scratch_mkfs >>$seqres.full 2>&1 + _scratch_mount + _run_btrfs_util_prog quota enable $SCRATCH_MNT + _check_regular_qgroup $SCRATCH_MNT || _notrun "not running normal qgroups" + _scratch_unmount +}
Signature · 1 line
2.41.0
Re: [PATCH 2/5] common/btrfs: quota mode helpers
Quoted 42 lines · Boris Burkov
To facilitate skipping tests depending on the qgroup mode after mkfs, add support for figuring out the mode. This cannot just rely on the new sysfs file, since it might not be present on older kernels.
Signed-off-by Boris Burkov <[email protected]>--- common/btrfs | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)common/btrfs +19
-0@@ -680,3 +680,46 @@ _require_btrfs_scratch_logical_resolve_v2() fi _scratch_unmount } + +_qgroup_mode_file() +{ + local mnt=$1 + + uuid=$(findmnt -n -o UUID $mnt) + echo /sys/fs/btrfs/${uuid}/qgroups/mode +} + +_qgroup_enabled_file() +{ + local mnt=$1 + + uuid=$(findmnt -n -o UUID $mnt) + echo /sys/fs/btrfs/${uuid}/qgroups/enabled +} + +_qgroup_mode() +{
common/btrfs _require_btrfs_scratch_logical_resolve_v2() · line 711
Quoted 10 lines
common/btrfs +10
-0+ local mnt=$1 + + if [ ! -f "$(_qgroup_enabled_file $mnt)" ]; then + echo "disabled" + return + fi + + if [ -f "$(_qgroup_mode_file $mnt)" ]; then + cat $(_qgroup_mode_file $mnt) + elif [ $(cat $(_qgroup_enabled_file $mnt)) -eq "1" ]; then
/> + echo "qgroup"
common/btrfs _require_btrfs_scratch_logical_resolve_v2() · line 725
Quoted 13 lines
common/btrfs +13
-0+ else + echo "disabled" # should not be reachable, the enabled file won't exist. + fi +} + +_require_scratch_qgroup() +{ + _scratch_mkfs >>$seqres.full 2>&1 + _scratch_mount + _run_btrfs_util_prog quota enable $SCRATCH_MNT + _check_regular_qgroup $SCRATCH_MNT || _notrun "not running normal qgroups" + _scratch_unmount +}
Can you add _has_scratch_fs_sysfs() helper for scratch? See for example _has_fs_sysfs().
So that you can do something like..
if [ _has_scratch_fs_sysfs qgroups/mode ]; then
if [ $(_get_fs_sysfs_attr $mnt qgroups/mode) == 1 ]; then echo qgroup else echo something
else
_notrun "qgroup unsupported"
fi
Thanks, Anand
PATCH 3/5 common/btrfs: quota rescan helpers
In reply to Boris BurkovMany btrfs tests explicitly trigger quota rescan. This is not a meaningful operation for simple quotas, so we wrap it in a helper that doesn't blow up quite so badly and lets us run those tests where the rescan is a qgroup detail.
--- common/btrfs | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
common/btrfs +25 -0
@@ -715,6 +715,31 @@ _qgroup_mode() fi } +_check_regular_qgroup() +{ + local mnt=$1 + + _qgroup_mode $mnt | grep -q 'qgroup' +} + +_qgroup_rescan() +{ + local mnt=$1 + + _check_regular_qgroup $mnt || return 1 + _run_btrfs_util_prog quota rescan -w $mnt +} + +_require_qgroup_rescan() +{ + _scratch_mkfs >>$seqres.full 2>&1 + _scratch_mount + _run_btrfs_util_prog quota enable $SCRATCH_MNT + $BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT || \ + _notrun "not able to run quota rescan" + _scratch_unmount +} + _require_scratch_qgroup() { _scratch_mkfs >>$seqres.full 2>&1
Signature · 1 line
2.41.0
Re: [PATCH 3/5] common/btrfs: quota rescan helpers
Quoted 32 lines · Boris Burkov
Many btrfs tests explicitly trigger quota rescan. This is not a meaningful operation for simple quotas, so we wrap it in a helper that doesn't blow up quite so badly and lets us run those tests where the rescan is a qgroup detail.
Signed-off-by Boris Burkov <[email protected]>--- common/btrfs | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)common/btrfs +14
-0@@ -715,6 +715,31 @@ _qgroup_mode() fi } +_check_regular_qgroup() +{ + local mnt=$1 + + _qgroup_mode $mnt | grep -q 'qgroup' +} + +_qgroup_rescan() +{ + local mnt=$1 + + _check_regular_qgroup $mnt || return 1 + _run_btrfs_util_prog quota rescan -w $mnt +}
common/btrfs _qgroup_mode() · line 741
Quoted 10 lines
common/btrfs +10
-0+ +_require_qgroup_rescan() +{ + _scratch_mkfs >>$seqres.full 2>&1 + _scratch_mount + _run_btrfs_util_prog quota enable $SCRATCH_MNT + $BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT || \ + _notrun "not able to run quota rescan" + _scratch_unmount +}
Looks like whitespace errors here. Thanks,
Josef
PATCH 4/5 btrfs: use new rescan wrapper
In reply to Boris BurkovThese tests can pass in simple quota mode if we skip the rescans via the wrapper.
--- tests/btrfs/022 | 1 + tests/btrfs/028 | 2 +- tests/btrfs/104 | 2 +- tests/btrfs/123 | 2 +- tests/btrfs/126 | 2 +- tests/btrfs/139 | 2 +- tests/btrfs/153 | 2 +- tests/btrfs/171 | 6 +++--- tests/btrfs/179 | 2 +- tests/btrfs/180 | 2 +- tests/btrfs/190 | 2 +- tests/btrfs/193 | 2 +- tests/btrfs/210 | 2 +- tests/btrfs/224 | 6 +++--- tests/btrfs/230 | 2 +- tests/btrfs/232 | 2 +- 16 files changed, 20 insertions(+), 19 deletions(-)
tests/btrfs/022 +1 -0
@@ -14,6 +14,7 @@ _begin_fstest auto qgroup limit _supported_fs btrfs _require_scratch +_require_qgroup_rescan _require_btrfs_qgroup_report # Test to make sure we can actually turn it on and it makes sense
tests/btrfs/028 +1 -1
@@ -25,7 +25,7 @@ _scratch_mkfs >/dev/null _scratch_mount _run_btrfs_util_prog quota enable $SCRATCH_MNT -_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT +_qgroup_rescan $SCRATCH_MNT # Increase the probability of generating de-refer extent, and decrease # other.
tests/btrfs/104 +1 -1
@@ -94,7 +94,7 @@ _explode_fs_tree 1 $SCRATCH_MNT/snap2/files-snap2 # Enable qgroups now that we have our filesystem prepared. This # will kick off a scan which we will have to wait for. _run_btrfs_util_prog quota enable $SCRATCH_MNT -_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT +_qgroup_rescan $SCRATCH_MNT # Remount to clear cache, force everything to disk _scratch_cycle_mount
tests/btrfs/123 +1 -1
@@ -39,7 +39,7 @@ sync # enable quota and rescan to get correct number _run_btrfs_util_prog quota enable $SCRATCH_MNT -_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT +_qgroup_rescan $SCRATCH_MNT # now balance data block groups to corrupt qgroup _run_btrfs_balance_start -d $SCRATCH_MNT >> $seqres.full
tests/btrfs/126 +1 -1
@@ -28,7 +28,7 @@ _scratch_mkfs >/dev/null _scratch_mount "-o enospc_debug" _run_btrfs_util_prog quota enable $SCRATCH_MNT -_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT +_qgroup_rescan $SCRATCH_MNT _run_btrfs_util_prog qgroup limit 512K 0/5 $SCRATCH_MNT # The amount of written data may change due to different nodesize at mkfs time,
tests/btrfs/139 +1 -1
@@ -30,7 +30,7 @@ SUBVOL=$SCRATCH_MNT/subvol _run_btrfs_util_prog subvolume create $SUBVOL _run_btrfs_util_prog quota enable $SCRATCH_MNT -_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT +_qgroup_rescan $SCRATCH_MNT _run_btrfs_util_prog qgroup limit -e 1G $SUBVOL # Write and delete files within 1G limits, multiple times
tests/btrfs/153 +1 -1
@@ -24,7 +24,7 @@ _scratch_mkfs >/dev/null _scratch_mount _run_btrfs_util_prog quota enable $SCRATCH_MNT -_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT +_qgroup_rescan $SCRATCH_MNT _run_btrfs_util_prog qgroup limit 100M 0/5 $SCRATCH_MNT testfile1=$SCRATCH_MNT/testfile1
tests/btrfs/171 +3 -3
@@ -35,7 +35,7 @@ $BTRFS_UTIL_PROG subvolume snapshot "$SCRATCH_MNT/subvol" \ "$SCRATCH_MNT/snapshot" > /dev/null $BTRFS_UTIL_PROG quota enable "$SCRATCH_MNT" > /dev/null -$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null +_qgroup_rescan $SCRATCH_MNT > /dev/null # Create high level qgroup $BTRFS_UTIL_PROG qgroup create 1/0 "$SCRATCH_MNT"
@@ -45,7 +45,7 @@ $BTRFS_UTIL_PROG qgroup assign "$SCRATCH_MNT/snapshot" 1/0 "$SCRATCH_MNT" \ # Above assignment will mark qgroup inconsistent due to the shared extents # between subvol/snapshot/high level qgroup, do rescan here. -$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null +_qgroup_rescan $SCRATCH_MNT > /dev/null # Now remove the qgroup relationship and make 1/0 childless # Due to the shared extent outside of 1/0, we will mark qgroup inconsistent
@@ -54,7 +54,7 @@ $BTRFS_UTIL_PROG qgroup remove "$SCRATCH_MNT/snapshot" 1/0 "$SCRATCH_MNT" \ 2>&1 | _filter_btrfs_qgroup_assign_warnings # Above removal also marks qgroup inconsistent, rescan again -$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null +_qgroup_rescan $SCRATCH_MNT > /dev/null # After the test, btrfs check will verify qgroup numbers to catch any # corruption.
tests/btrfs/179 +1 -1
@@ -33,7 +33,7 @@ _scratch_mount mkdir -p "$SCRATCH_MNT/snapshots" $BTRFS_UTIL_PROG subvolume create "$SCRATCH_MNT/src" > /dev/null $BTRFS_UTIL_PROG quota enable "$SCRATCH_MNT" > /dev/null -$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null +_qgroup_rescan "$SCRATCH_MNT" > /dev/null fill_workload() {
tests/btrfs/180 +1 -1
@@ -27,7 +27,7 @@ _scratch_mkfs > /dev/null _scratch_mount $BTRFS_UTIL_PROG quota enable "$SCRATCH_MNT" > /dev/null -$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null +_qgroup_rescan "$SCRATCH_MNT" > /dev/null $BTRFS_UTIL_PROG qgroup limit -e 1G "$SCRATCH_MNT" $XFS_IO_PROG -f -c "falloc 0 900M" "$SCRATCH_MNT/padding"
tests/btrfs/190 +1 -1
@@ -30,7 +30,7 @@ _log_writes_mkfs >> $seqres.full 2>&1 _log_writes_mount $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT >> $seqres.full -$BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT >> $seqres.full +_qgroup_rescan $SCRATCH_MNT >> $seqres.full # Create enough metadata for later balance for ((i = 0; i < $nr_files; i++)); do
tests/btrfs/193 +1 -1
@@ -26,7 +26,7 @@ _scratch_mkfs > /dev/null _scratch_mount $BTRFS_UTIL_PROG quota enable "$SCRATCH_MNT" > /dev/null -$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null +_qgroup_rescan "$SCRATCH_MNT" > /dev/null $BTRFS_UTIL_PROG qgroup limit -e 256M "$SCRATCH_MNT" # Create a file with the following layout:
tests/btrfs/210 +1 -1
@@ -29,7 +29,7 @@ _pwrite_byte 0xcd 0 16M "$SCRATCH_MNT/src/file" > /dev/null # by qgroup sync $BTRFS_UTIL_PROG quota enable "$SCRATCH_MNT" -$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null +_qgroup_rescan "$SCRATCH_MNT" > /dev/null $BTRFS_UTIL_PROG qgroup create 1/0 "$SCRATCH_MNT" # Create a snapshot with qgroup inherit
tests/btrfs/224 +3 -3
@@ -30,7 +30,7 @@ assign_shared_test() echo "=== qgroup assign shared test ===" >> $seqres.full $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT - $BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT >> $seqres.full + _qgroup_rescan $SCRATCH_MNT >> $seqres.full $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/a >> $seqres.full $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/b >> $seqres.full
@@ -54,7 +54,7 @@ assign_no_shared_test() echo "=== qgroup assign no shared test ===" >> $seqres.full $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT - $BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT >> $seqres.full + _qgroup_rescan $SCRATCH_MNT >> $seqres.full $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/a >> $seqres.full $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/b >> $seqres.full
@@ -75,7 +75,7 @@ snapshot_test() echo "=== qgroup snapshot test ===" >> $seqres.full $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT - $BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT >> $seqres.full + _qgroup_rescan $SCRATCH_MNT >> $seqres.full $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/a >> $seqres.full _ddt of="$SCRATCH_MNT"/a/file1 bs=1M count=1 >> $seqres.full 2>&1
tests/btrfs/230 +1 -1
@@ -31,7 +31,7 @@ _pwrite_byte 0xcd 0 1G $SCRATCH_MNT/file >> $seqres.full sync $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT -$BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT >> $seqres.full +_qgroup_rescan $SCRATCH_MNT >> $seqres.full # Set the limit to just 512MiB, which is way below the existing usage $BTRFS_UTIL_PROG qgroup limit 512M 0/5 $SCRATCH_MNT
tests/btrfs/232 +1 -1
@@ -46,7 +46,7 @@ _pwrite_byte 0xcd 0 900m $SCRATCH_MNT/file >> $seqres.full sync $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT -$BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT >> $seqres.full +_qgroup_rescan $SCRATCH_MNT >> $seqres.full # set the limit to 1 g, leaving us just 100mb of slack space $BTRFS_UTIL_PROG qgroup limit 1G 0/5 $SCRATCH_MNT
Signature · 1 line
2.41.0
PATCH 5/5 btrfs: skip squota incompatible tests
In reply to Boris BurkovThese tests cannot succeed if mkfs enable squotas, as they either test the specifics of qgroups behavior or they test *enabling* squotas. Skip these in squota mode.
--- tests/btrfs/017 | 1 + tests/btrfs/057 | 1 + tests/btrfs/091 | 3 ++- tests/btrfs/400 | 5 +++++ 4 files changed, 9 insertions(+), 1 deletion(-)
tests/btrfs/017 +1 -0
@@ -22,6 +22,7 @@ _begin_fstest auto quick qgroup _supported_fs btrfs _require_scratch +_require_scratch_qgroup _require_cloner # Currently in btrfs the node/leaf size can not be smaller than the page
tests/btrfs/057 +1 -0
@@ -15,6 +15,7 @@ _begin_fstest auto quick # real QA test starts here _supported_fs btrfs _require_scratch +_require_qgroup_rescan _scratch_mkfs_sized $((1024 * 1024 * 1024)) >> $seqres.full 2>&1
tests/btrfs/091 +2 -1
@@ -19,6 +19,7 @@ _begin_fstest auto quick qgroup _supported_fs btrfs _require_scratch _require_cp_reflink +_require_scratch_qgroup # use largest node/leaf size (64K) to allow the test to be run on arch with # page size > 4k.
@@ -35,7 +36,7 @@ _run_btrfs_util_prog subvolume create $SCRATCH_MNT/subv2 _run_btrfs_util_prog subvolume create $SCRATCH_MNT/subv3 _run_btrfs_util_prog quota enable $SCRATCH_MNT -_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT +_qgroup_rescan $SCRATCH_MNT $XFS_IO_PROG -f -c "pwrite 0 256K" $SCRATCH_MNT/subv1/file1 | _filter_xfs_io cp --reflink $SCRATCH_MNT/subv1/file1 $SCRATCH_MNT/subv2/file1
tests/btrfs/400 +5 -0
@@ -32,6 +32,11 @@ EXT_SZ=$((128 * M)) LIMIT_NR=8 LIMIT=$(($EXT_SZ * $LIMIT_NR)) +_scratch_mkfs >> $seqres.full +_scratch_mount +_qgroup_mode $SCRATCH_MNT | grep 'squota' && _notrun "test relies on starting without simple quotas enabled" +_scratch_unmount + get_qgroup_usage() { local qgroupid=$1
Signature · 1 line
2.41.0