fstests: btrfs: Test if balance takes too long time or return -ENOSPC unexpected
RFC PATCH 1/1 fstests: btrfs: Test if balance takes too long time or return -ENOSPC unexpected
Kernel commit 64403612b73a ("btrfs: rework btrfs_check_space_for_delayed_refs") is introducing a regression for btrfs balance performance.
Originally, the workload used in the test case only takes seconds for balance on v4.20 while now it takes over 400 seconds for balance.
During that 400 seconds balance, it commits over 2000 transactions just for nothing, compare to original several transactions.
Add test cases to detect such regression.
---
Extra note:
In fact, without the snapshots created in the test case, it would return -ENOSPC even we have enough unallocated space.
--- tests/btrfs/180 | 80 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/180.out | 2 ++ tests/btrfs/group | 1 + 3 files changed, 83 insertions(+) create mode 100755 tests/btrfs/180 create mode 100644 tests/btrfs/180.out
tests/btrfs/180 new file +80 -0
@@ -0,0 +1,80 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 180 +# +# Test if metadata balance would take forever to finish or return ENOSPC even +# there there are tons of space. +# +# This is regression caused by upstream commit 64403612b73a +# ("btrfs: rework btrfs_check_space_for_delayed_refs") +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# remove previous $seqres.full before test +rm -f $seqres.full + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_xfs_io_command falloc + +i=0 +loop=16384 + +_scratch_mkfs > /dev/null 2>&1 +_scratch_mount + +$BTRFS_UTIL_PROG subvolume create "$SCRATCH_MNT/src" > /dev/null + +while [ $i -le $loop ]; do + # Use small file to create inline extent + _pwrite_byte 0x00 0 2K "$SCRATCH_MNT/src/inline_$i" > /dev/null + #$XFS_IO_PROG -f -c "falloc 0 4K" "$SCRATCH_MNT/src/regular_$i" > /dev/null + i=$((i + 1)) +done + +# Create enough snapshots so at space reservation part of relocation, we could +# generate enough space pressure +for i in $(seq -w 0 16); do + $BTRFS_UTIL_PROG subvolume snapshot "$SCRATCH_MNT/src" \ + "$SCRATCH_MNT/snapshot_$i" > /dev/null + # touch random files to create some new tree blocks + for j in $(seq -w 0 16); do + victim="$(ls $SCRATCH_MNT/snapshot_$i | sort -R | head -n1)" + touch "$SCRATCH_MNT/snapshot_$i/$victim" + done +done + +# Balancing metadata shouldn't be too time consuming, as the amount of metadata +# is less than 8M, thus normally it should finish very quick. +# +# However with that offending commit, it will take forever to finish or return +# ENOSPC after a long wait. +$BTRFS_UTIL_PROG balance start -m "$SCRATCH_MNT" > /dev/null + +echo "Silence is golden" +# success, all done +status=0 +exit
tests/btrfs/180.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 180 +Silence is golden
tests/btrfs/group +1 -0
@@ -182,3 +182,4 @@ 177 auto quick swap balance 178 auto quick send 179 auto qgroup dangerous +180 auto balance dangerous
Signature · 1 line
2.18.0
Quoted 11 lines · Qu Wenruo
Kernel commit
64403612b73a("btrfs: reworkbtrfs_check_space_for_delayed_refs") is introducing a regression for btrfs balance performance.Originally, the workload used in the test case only takes seconds for balance on v4.20 while now it takes over 400 seconds for balance.
During that 400 seconds balance, it commits over 2000 transactions just for nothing, compare to original several transactions.
Add test cases to detect such regression.
How do you detect that the test regressed, other than comparing the runtime recorded between runs of the testsuite? I don't actually know how this could be done inside the script, as there can't be a fixed number of seconds to compare with. The run time depends on the underlying storage.
I takes 14 seconds on a physical machine on a rotational disk and 9 seconds on a SSD.
Other than the, I've used this test to validate the fix, for that
It would be good to have the test in the testsuite so possibly some large timeout can be set and the test will print a message, this should catch attention at least and let us decide.
Quoted 63 lines
Signed-off-by Qu Wenruo <[email protected]>---
Extra note:
In fact, without the snapshots created in the test case, it would return(quoted diff) +0
-2-ENOSPC even we have enough unallocated space. --- tests/btrfs/180 | 80 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/180.out | 2 ++ tests/btrfs/group | 1 + 3 files changed, 83 insertions(+) create mode 100755 tests/btrfs/180 create mode 100644 tests/btrfs/180.outtests/btrfs/180 new file +44
-0@@ -0,0 +1,80 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 180 +# +# Test if metadata balance would take forever to finish or return ENOSPC even +# there there are tons of space. +# +# This is regression caused by upstream commit 64403612b73a +# ("btrfs: rework btrfs_check_space_for_delayed_refs") +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# remove previous $seqres.full before test +rm -f $seqres.full + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_xfs_io_command falloc + +i=0 +loop=16384
tests/btrfs/180 line 54
Quoted 10 lines
tests/btrfs/180 new file +10
-0+ +_scratch_mkfs > /dev/null 2>&1 +_scratch_mount + +$BTRFS_UTIL_PROG subvolume create "$SCRATCH_MNT/src" > /dev/null + +while [ $i -le $loop ]; do + # Use small file to create inline extent + _pwrite_byte 0x00 0 2K "$SCRATCH_MNT/src/inline_$i" > /dev/null + #$XFS_IO_PROG -f -c "falloc 0 4K" "$SCRATCH_MNT/src/regular_$i" > /dev/null
You can delete the commented command
Quoted 39 lines
tests/btrfs/180 new file +26
-0+ i=$((i + 1)) +done + +# Create enough snapshots so at space reservation part of relocation, we could +# generate enough space pressure +for i in $(seq -w 0 16); do + $BTRFS_UTIL_PROG subvolume snapshot "$SCRATCH_MNT/src" \ + "$SCRATCH_MNT/snapshot_$i" > /dev/null + # touch random files to create some new tree blocks + for j in $(seq -w 0 16); do + victim="$(ls $SCRATCH_MNT/snapshot_$i | sort -R | head -n1)" + touch "$SCRATCH_MNT/snapshot_$i/$victim" + done +done + +# Balancing metadata shouldn't be too time consuming, as the amount of metadata +# is less than 8M, thus normally it should finish very quick. +# +# However with that offending commit, it will take forever to finish or return +# ENOSPC after a long wait. +$BTRFS_UTIL_PROG balance start -m "$SCRATCH_MNT" > /dev/null + +echo "Silence is golden" +# success, all done +status=0 +exittests/btrfs/180.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 180 +Silence is goldentests/btrfs/group +0
-0
tests/btrfs/group line 185
Quoted 4 lines
tests/btrfs/group +1
-0@@ -182,3 +182,4 @@ 177 auto quick swap balance 178 auto quick send 179 auto qgroup dangerous +180 auto balance dangerous
I don't think it falls to the dangerous category, it does not crash the machine, only that the test could take long.
Quoted 11 lines · David Sterba
Quoted 10 lines · Qu Wenruo
Kernel commit
64403612b73a("btrfs: reworkbtrfs_check_space_for_delayed_refs") is introducing a regression for btrfs balance performance.Originally, the workload used in the test case only takes seconds for balance on v4.20 while now it takes over 400 seconds for balance.
During that 400 seconds balance, it commits over 2000 transactions just for nothing, compare to original several transactions.
Quoted 7 lines
Quoted 1 line
Add test cases to detect such regression.
How do you detect that the test regressed, other than comparing the runtime recorded between runs of the testsuite? I don't actually know how this could be done inside the script, as there can't be a fixed number of seconds to compare with. The run time depends on the underlying storage.
I have a better idea for detecting this, by using super generation as failure criteria.
So then we could have a simple FAIL/PASS check. Although, we still need to take a long time to run if we hit that regression.
Thanks,
Qu
Quoted 135 lines
I takes 14 seconds on a physical machine on a rotational disk and 9 seconds on a SSD.
Other than the, I've used this test to validate the fix, for that
Tested-by David Sterba <[email protected]>It would be good to have the test in the testsuite so possibly some large timeout can be set and the test will print a message, this should catch attention at least and let us decide.
Quoted 63 lines
Signed-off-by Qu Wenruo <[email protected]>---
Extra note:
In fact, without the snapshots created in the test case, it would return(quoted diff) +0
-2-ENOSPC even we have enough unallocated space. --- tests/btrfs/180 | 80 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/180.out | 2 ++ tests/btrfs/group | 1 + 3 files changed, 83 insertions(+) create mode 100755 tests/btrfs/180 create mode 100644 tests/btrfs/180.outtests/btrfs/180 new file +44
-0@@ -0,0 +1,80 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 180 +# +# Test if metadata balance would take forever to finish or return ENOSPC even +# there there are tons of space. +# +# This is regression caused by upstream commit 64403612b73a +# ("btrfs: rework btrfs_check_space_for_delayed_refs") +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# remove previous $seqres.full before test +rm -f $seqres.full + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_xfs_io_command falloc + +i=0 +loop=16384tests/btrfs/180 line 54
Quoted 10 lines
tests/btrfs/180 new file +10
-0+ +_scratch_mkfs > /dev/null 2>&1 +_scratch_mount + +$BTRFS_UTIL_PROG subvolume create "$SCRATCH_MNT/src" > /dev/null + +while [ $i -le $loop ]; do + # Use small file to create inline extent + _pwrite_byte 0x00 0 2K "$SCRATCH_MNT/src/inline_$i" > /dev/null + #$XFS_IO_PROG -f -c "falloc 0 4K" "$SCRATCH_MNT/src/regular_$i" > /dev/nullYou can delete the commented command
Quoted 39 lines
tests/btrfs/180 new file +26
-0+ i=$((i + 1)) +done + +# Create enough snapshots so at space reservation part of relocation, we could +# generate enough space pressure +for i in $(seq -w 0 16); do + $BTRFS_UTIL_PROG subvolume snapshot "$SCRATCH_MNT/src" \ + "$SCRATCH_MNT/snapshot_$i" > /dev/null + # touch random files to create some new tree blocks + for j in $(seq -w 0 16); do + victim="$(ls $SCRATCH_MNT/snapshot_$i | sort -R | head -n1)" + touch "$SCRATCH_MNT/snapshot_$i/$victim" + done +done + +# Balancing metadata shouldn't be too time consuming, as the amount of metadata +# is less than 8M, thus normally it should finish very quick. +# +# However with that offending commit, it will take forever to finish or return +# ENOSPC after a long wait. +$BTRFS_UTIL_PROG balance start -m "$SCRATCH_MNT" > /dev/null + +echo "Silence is golden" +# success, all done +status=0 +exittests/btrfs/180.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 180 +Silence is goldentests/btrfs/group +0
-0tests/btrfs/group line 185
Quoted 4 lines
tests/btrfs/group +1
-0@@ -182,3 +182,4 @@ 177 auto quick swap balance 178 auto quick send 179 auto qgroup dangerous +180 auto balance dangerousI don't think it falls to the dangerous category, it does not crash the machine, only that the test could take long.