generic/747: handle ENOSPC gracefully during write/delete cycles
PATCH 1/1 generic/747: handle ENOSPC gracefully during write/delete cycles
generic/747 consistently fails on btrfs in my fstests setup, with an ~88% failure rate across multiple runs on kernels ranging from v6.9 to v7.0-rc5. This is not a regression but a pre-existing issue since the test was added.
The test fills a filesystem to 95% then does mixed write/delete cycles, using statfs to decide whether to write or delete. However, statfs f_bavail may overestimate the actual available space. On btrfs, the statfs implementation documents its estimate as "a close approximation" (fs/btrfs/super.c). At high fill levels the discrepancy between what statfs reports and what the filesystem can actually allocate becomes significant, causing dd to hit ENOSPC even though statfs indicated there was room.
This is not a filesystem bug. The filesystem correctly rejects the write when it cannot reserve space. The test's purpose is to stress garbage collection through write/delete churn, not to validate space accounting.
Handle ENOSPC by cleaning up the partial file and making room:
In _direct_fillup: break out of the fill loop (we're full enough). In _mixed_write_delete: delete a file to free space and retry. If writes fail 10 consecutive times, _fail the test as that indicates a real filesystem issue rather than a transient statfs discrepancy.
Redirect dd stderr to seqres.full so errors are preserved for debugging without polluting the expected output.
--- tests/generic/747 | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-)
tests/generic/747 +19 -9
@@ -35,11 +35,7 @@ _create_file() { POSIXLY_CORRECT=yes dd if=/dev/zero of=${file_name} \ bs=${bs} count=$(( $file_sz / ${bs} )) \ - status=none $dd_extra 2>&1 - - if [ $? -ne 0 ]; then - _fail "Failed writing $file_name" - fi + status=none $dd_extra 2>>$seqres.full } _total_M() {
@@ -69,7 +65,10 @@ _direct_fillup () { while [ $(_used_percent) -lt $fill_percent ]; do local fsz=$(_get_random_fsz) - _create_file $testseq $fsz "oflag=direct conv=fsync" + if ! _create_file $testseq $fsz "oflag=direct conv=fsync"; then + rm ${SCRATCH_MNT}/data_${testseq} + break + fi testseq=$((${testseq} + 1)) done }
@@ -79,14 +78,25 @@ _mixed_write_delete() { local total_M=$(_total_M) local to_write_M=$(( ${overwrite_percentage} * ${total_M} / 100 )) local written_M=0 + local enospc_retries=0 + local max_enospc_retries=10 while [ $written_M -lt $to_write_M ]; do if [ $(_used_percent) -lt $fill_percent ]; then local fsz=$(_get_random_fsz) - _create_file $testseq $fsz "$dd_extra" - written_M=$((${written_M} + ${fsz}/${M})) - testseq=$((${testseq} + 1)) + if ! _create_file $testseq $fsz "$dd_extra"; then + rm ${SCRATCH_MNT}/data_${testseq} + _delete_random_file + enospc_retries=$((enospc_retries + 1)) + if [ $enospc_retries -ge $max_enospc_retries ]; then + _fail "failed to write after $max_enospc_retries consecutive ENOSPC attempts" + fi + else + written_M=$((${written_M} + ${fsz}/${M})) + testseq=$((${testseq} + 1)) + enospc_retries=0 + fi else _delete_random_file fi
Signature · 1 line
2.52.0
On Tue, 24 Mar 2026 16:33:26 -0700 Leo Martins <[email protected]> wrote:
Forgot to CC fstests mailing list.
Quoted 94 lines
generic/747 consistently fails on btrfs in my fstests setup, with an ~88% failure rate across multiple runs on kernels ranging from v6.9 to v7.0-rc5. This is not a regression but a pre-existing issue since the test was added.
The test fills a filesystem to 95% then does mixed write/delete cycles, using statfs to decide whether to write or delete. However, statfs
f_bavailmay overestimate the actual available space. On btrfs, the statfs implementation documents its estimate as "a close approximation" (fs/btrfs/super.c). At high fill levels the discrepancy between what statfs reports and what the filesystem can actually allocate becomes significant, causing dd to hit ENOSPC even though statfs indicated there was room.This is not a filesystem bug. The filesystem correctly rejects the write when it cannot reserve space. The test's purpose is to stress garbage collection through write/delete churn, not to validate space accounting.
Handle ENOSPC by cleaning up the partial file and making room:
In
_direct_fillup: break out of the fill loop (we're full enough). In_mixed_write_delete: delete a file to free space and retry. If writes fail 10 consecutive times,_failthe test as that indicates a real filesystem issue rather than a transient statfs discrepancy.Redirect dd stderr to seqres.full so errors are preserved for debugging without polluting the expected output.
Signed-off-by Leo Martins <[email protected]>--- tests/generic/747 | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-)tests/generic/747 +19
-9@@ -35,11 +35,7 @@ _create_file() { POSIXLY_CORRECT=yes dd if=/dev/zero of=${file_name} \ bs=${bs} count=$(( $file_sz / ${bs} )) \ - status=none $dd_extra 2>&1 - - if [ $? -ne 0 ]; then - _fail "Failed writing $file_name" - fi + status=none $dd_extra 2>>$seqres.full } _total_M() {@@ -69,7 +65,10 @@ _direct_fillup () { while [ $(_used_percent) -lt $fill_percent ]; do local fsz=$(_get_random_fsz) - _create_file $testseq $fsz "oflag=direct conv=fsync" + if ! _create_file $testseq $fsz "oflag=direct conv=fsync"; then + rm ${SCRATCH_MNT}/data_${testseq} + break + fi testseq=$((${testseq} + 1)) done }@@ -79,14 +78,25 @@ _mixed_write_delete() { local total_M=$(_total_M) local to_write_M=$(( ${overwrite_percentage} * ${total_M} / 100 )) local written_M=0 + local enospc_retries=0 + local max_enospc_retries=10 while [ $written_M -lt $to_write_M ]; do if [ $(_used_percent) -lt $fill_percent ]; then local fsz=$(_get_random_fsz) - _create_file $testseq $fsz "$dd_extra" - written_M=$((${written_M} + ${fsz}/${M})) - testseq=$((${testseq} + 1)) + if ! _create_file $testseq $fsz "$dd_extra"; then + rm ${SCRATCH_MNT}/data_${testseq} + _delete_random_file + enospc_retries=$((enospc_retries + 1)) + if [ $enospc_retries -ge $max_enospc_retries ]; then + _fail "failed to write after $max_enospc_retries consecutive ENOSPC attempts" + fi + else + written_M=$((${written_M} + ${fsz}/${M})) + testseq=$((${testseq} + 1)) + enospc_retries=0 + fi else _delete_random_file fiSignature · 1 line
2.52.0
Quoted 4 lines · Leo Martins
Quoted 4 lines
The test fills a filesystem to 95% then does mixed write/delete cycles, using statfs to decide whether to write or delete. However, statfs
f_bavailmay overestimate the actual available space. On btrfs, the statfs implementation documents its estimate as "a close approximation"
Quoted 8 lines
Quoted 8 lines
(
fs/btrfs/super.c). At high fill levels the discrepancy between what statfs reports and what the filesystem can actually allocate becomes significant, causing dd to hit ENOSPC even though statfs indicated there was room.
This is not a filesystem bug. The filesystem correctly rejects the write when it cannot reserve space. The test's purpose is to stress garbage collection through write/delete churn, not to validate space accounting.
f_bavail overestimating is a btrfs implementation bug that needs fixing. It can be wrong, but it needs to be too low in doubt. Without this you're going to cause unexpected ENOSPC for real life applictions as well (and btrfs does have a bit of a reputation for that, so fixing this properly will benefit your users!)
Quoted 4 lines · Leo Martins
generic/747 consistently fails on btrfs in my fstests setup, with an ~88% failure rate across multiple runs on kernels ranging from v6.9 to v7.0-rc5. This is not a regression but a pre-existing issue since the test was added.
Glad we finally have someone trying to fix this.
Quoted 4 lines
The test fills a filesystem to 95% then does mixed write/delete cycles, using statfs to decide whether to write or delete. However, statfs
f_bavailmay overestimate the actual available space.
If this is the problem, then I believe we should fix our f_bavail calculation.
Quoted 6 lines
On btrfs, the
statfs implementation documents its estimate as "a close approximation" (fs/btrfs/super.c). At high fill levels the discrepancy between what statfs reports and what the filesystem can actually allocate becomes significant, causing dd to hit ENOSPC even though statfs indicated there was room.
From what I can see, there are several different ways that f_bavail is overestimated:
- btrfs_calc_available_data_space()
That functions goes through each device and try to find unallocated space.
This can not handle uneven space (e.g. 1G + 10G RAID0), and it should go through the per-profile estimation, which handles uneven disks way better.
Normally it shouldn't affect single device, but we can still over- estimate by taking the reserved 1M bytes as available.
- total_free_data calculation
It's just on-disk size minus the on-disk used space, which doesn't take things like reserved space into consideration.
Thus I guess the problem is mostly from this part.
Mind to give this idea a try to fix the long failing g747?
Thanks,
Qu
Quoted 79 lines
This is not a filesystem bug. The filesystem correctly rejects the write when it cannot reserve space. The test's purpose is to stress garbage collection through write/delete churn, not to validate space accounting.
Handle ENOSPC by cleaning up the partial file and making room:
In
_direct_fillup: break out of the fill loop (we're full enough). In_mixed_write_delete: delete a file to free space and retry. If writes fail 10 consecutive times,_failthe test as that indicates a real filesystem issue rather than a transient statfs discrepancy.Redirect dd stderr to seqres.full so errors are preserved for debugging without polluting the expected output.
Signed-off-by Leo Martins <[email protected]>--- tests/generic/747 | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-)tests/generic/747 +19
-9@@ -35,11 +35,7 @@ _create_file() { POSIXLY_CORRECT=yes dd if=/dev/zero of=${file_name} \ bs=${bs} count=$(( $file_sz / ${bs} )) \ - status=none $dd_extra 2>&1 - - if [ $? -ne 0 ]; then - _fail "Failed writing $file_name" - fi + status=none $dd_extra 2>>$seqres.full } _total_M() {@@ -69,7 +65,10 @@ _direct_fillup () { while [ $(_used_percent) -lt $fill_percent ]; do local fsz=$(_get_random_fsz) - _create_file $testseq $fsz "oflag=direct conv=fsync" + if ! _create_file $testseq $fsz "oflag=direct conv=fsync"; then + rm ${SCRATCH_MNT}/data_${testseq} + break + fi testseq=$((${testseq} + 1)) done }@@ -79,14 +78,25 @@ _mixed_write_delete() { local total_M=$(_total_M) local to_write_M=$(( ${overwrite_percentage} * ${total_M} / 100 )) local written_M=0 + local enospc_retries=0 + local max_enospc_retries=10 while [ $written_M -lt $to_write_M ]; do if [ $(_used_percent) -lt $fill_percent ]; then local fsz=$(_get_random_fsz) - _create_file $testseq $fsz "$dd_extra" - written_M=$((${written_M} + ${fsz}/${M})) - testseq=$((${testseq} + 1)) + if ! _create_file $testseq $fsz "$dd_extra"; then + rm ${SCRATCH_MNT}/data_${testseq} + _delete_random_file + enospc_retries=$((enospc_retries + 1)) + if [ $enospc_retries -ge $max_enospc_retries ]; then + _fail "failed to write after $max_enospc_retries consecutive ENOSPC attempts" + fi + else + written_M=$((${written_M} + ${fsz}/${M})) + testseq=$((${testseq} + 1)) + enospc_retries=0 + fi else _delete_random_file fi