Skip to contentSkip to conversation

generic/747: handle ENOSPC gracefully during write/delete cycles

PATCH In review Leo Martins 3 participants 4 messages Mar 24 – Mar 25 Btrfs file system

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 generic/747: handle ENOSPC gracefully during write/delete cycles

generic/747 consistently fails on btrfs in my fstests setup, with an ~88% failure rate across multiple runs on kernels ranging from v6.9 to v7.0-rc5. This is not a regression but a pre-existing issue since the test was added.

The test fills a filesystem to 95% then does mixed write/delete cycles, using statfs to decide whether to write or delete. However, statfs f_bavail may overestimate the actual available space. On btrfs, the statfs implementation documents its estimate as "a close approximation" (fs/btrfs/super.c). At high fill levels the discrepancy between what statfs reports and what the filesystem can actually allocate becomes significant, causing dd to hit ENOSPC even though statfs indicated there was room.

This is not a filesystem bug. The filesystem correctly rejects the write when it cannot reserve space. The test's purpose is to stress garbage collection through write/delete churn, not to validate space accounting.

Handle ENOSPC by cleaning up the partial file and making room:

In _direct_fillup: break out of the fill loop (we're full enough). In _mixed_write_delete: delete a file to free space and retry. If writes fail 10 consecutive times, _fail the test as that indicates a real filesystem issue rather than a transient statfs discrepancy.

Redirect dd stderr to seqres.full so errors are preserved for debugging without polluting the expected output.

---
 tests/generic/747 | 28 +++++++++++++++++++---------
 1 file changed, 19 insertions(+), 9 deletions(-)
tests/generic/747 +19 -9
index 44834186..35de3ccb 100755
@@ -35,11 +35,7 @@ _create_file() {
 
 	POSIXLY_CORRECT=yes dd if=/dev/zero of=${file_name} \
 		bs=${bs} count=$(( $file_sz / ${bs} )) \
-		status=none $dd_extra  2>&1
-
-	if [ $? -ne 0 ]; then
-		_fail "Failed writing $file_name"
-	fi
+		status=none $dd_extra  2>>$seqres.full
 }
 
 _total_M() {
@@ -69,7 +65,10 @@ _direct_fillup () {
 	while [ $(_used_percent) -lt $fill_percent ]; do
 		local fsz=$(_get_random_fsz)
 
-		_create_file $testseq $fsz "oflag=direct conv=fsync"
+		if ! _create_file $testseq $fsz "oflag=direct conv=fsync"; then
+			rm ${SCRATCH_MNT}/data_${testseq}
+			break
+		fi
 		testseq=$((${testseq} + 1))
 	done
 }
@@ -79,14 +78,25 @@ _mixed_write_delete() {
 	local total_M=$(_total_M)
 	local to_write_M=$(( ${overwrite_percentage} * ${total_M} / 100 ))
 	local written_M=0
+	local enospc_retries=0
+	local max_enospc_retries=10
 
 	while [ $written_M -lt $to_write_M ]; do
 		if [ $(_used_percent) -lt $fill_percent ]; then
 			local fsz=$(_get_random_fsz)
 
-			_create_file $testseq $fsz "$dd_extra"
-			written_M=$((${written_M} + ${fsz}/${M}))
-			testseq=$((${testseq} + 1))
+			if ! _create_file $testseq $fsz "$dd_extra"; then
+				rm ${SCRATCH_MNT}/data_${testseq}
+				_delete_random_file
+				enospc_retries=$((enospc_retries + 1))
+				if [ $enospc_retries -ge $max_enospc_retries ]; then
+					_fail "failed to write after $max_enospc_retries consecutive ENOSPC attempts"
+				fi
+			else
+				written_M=$((${written_M} + ${fsz}/${M}))
+				testseq=$((${testseq} + 1))
+				enospc_retries=0
+			fi
 		else
 			_delete_random_file
 		fi
Signature · 1 line
2.52.0
#1 lore raw

On Tue, 24 Mar 2026 16:33:26 -0700 Leo Martins <[email protected]> wrote:

Forgot to CC fstests mailing list.

Quoted 94 lines

generic/747 consistently fails on btrfs in my fstests setup, with an ~88% failure rate across multiple runs on kernels ranging from v6.9 to v7.0-rc5. This is not a regression but a pre-existing issue since the test was added.

The test fills a filesystem to 95% then does mixed write/delete cycles, using statfs to decide whether to write or delete. However, statfs f_bavail may overestimate the actual available space. On btrfs, the statfs implementation documents its estimate as "a close approximation" (fs/btrfs/super.c). At high fill levels the discrepancy between what statfs reports and what the filesystem can actually allocate becomes significant, causing dd to hit ENOSPC even though statfs indicated there was room.

This is not a filesystem bug. The filesystem correctly rejects the write when it cannot reserve space. The test's purpose is to stress garbage collection through write/delete churn, not to validate space accounting.

Handle ENOSPC by cleaning up the partial file and making room:

In _direct_fillup: break out of the fill loop (we're full enough). In _mixed_write_delete: delete a file to free space and retry. If writes fail 10 consecutive times, _fail the test as that indicates a real filesystem issue rather than a transient statfs discrepancy.

Redirect dd stderr to seqres.full so errors are preserved for debugging without polluting the expected output.

---
 tests/generic/747 | 28 +++++++++++++++++++---------
 1 file changed, 19 insertions(+), 9 deletions(-)
tests/generic/747 +19 -9
index 44834186..35de3ccb 100755
@@ -35,11 +35,7 @@ _create_file() {
 
 	POSIXLY_CORRECT=yes dd if=/dev/zero of=${file_name} \
 		bs=${bs} count=$(( $file_sz / ${bs} )) \
-		status=none $dd_extra  2>&1
-
-	if [ $? -ne 0 ]; then
-		_fail "Failed writing $file_name"
-	fi
+		status=none $dd_extra  2>>$seqres.full
 }
 
 _total_M() {
@@ -69,7 +65,10 @@ _direct_fillup () {
 	while [ $(_used_percent) -lt $fill_percent ]; do
 		local fsz=$(_get_random_fsz)
 
-		_create_file $testseq $fsz "oflag=direct conv=fsync"
+		if ! _create_file $testseq $fsz "oflag=direct conv=fsync"; then
+			rm ${SCRATCH_MNT}/data_${testseq}
+			break
+		fi
 		testseq=$((${testseq} + 1))
 	done
 }
@@ -79,14 +78,25 @@ _mixed_write_delete() {
 	local total_M=$(_total_M)
 	local to_write_M=$(( ${overwrite_percentage} * ${total_M} / 100 ))
 	local written_M=0
+	local enospc_retries=0
+	local max_enospc_retries=10
 
 	while [ $written_M -lt $to_write_M ]; do
 		if [ $(_used_percent) -lt $fill_percent ]; then
 			local fsz=$(_get_random_fsz)
 
-			_create_file $testseq $fsz "$dd_extra"
-			written_M=$((${written_M} + ${fsz}/${M}))
-			testseq=$((${testseq} + 1))
+			if ! _create_file $testseq $fsz "$dd_extra"; then
+				rm ${SCRATCH_MNT}/data_${testseq}
+				_delete_random_file
+				enospc_retries=$((enospc_retries + 1))
+				if [ $enospc_retries -ge $max_enospc_retries ]; then
+					_fail "failed to write after $max_enospc_retries consecutive ENOSPC attempts"
+				fi
+			else
+				written_M=$((${written_M} + ${fsz}/${M}))
+				testseq=$((${testseq} + 1))
+				enospc_retries=0
+			fi
 		else
 			_delete_random_file
 		fi
Signature · 1 line
2.52.0
#2 lore raw
Quoted 4 lines · Leo Martins
Quoted 4 lines

The test fills a filesystem to 95% then does mixed write/delete cycles, using statfs to decide whether to write or delete. However, statfs f_bavail may overestimate the actual available space. On btrfs, the statfs implementation documents its estimate as "a close approximation"

Quoted 8 lines
Quoted 8 lines

(fs/btrfs/super.c). At high fill levels the discrepancy between what statfs reports and what the filesystem can actually allocate becomes significant, causing dd to hit ENOSPC even though statfs indicated there was room.
This is not a filesystem bug. The filesystem correctly rejects the write when it cannot reserve space. The test's purpose is to stress garbage collection through write/delete churn, not to validate space accounting.

f_bavail overestimating is a btrfs implementation bug that needs fixing. It can be wrong, but it needs to be too low in doubt. Without this you're going to cause unexpected ENOSPC for real life applictions as well (and btrfs does have a bit of a reputation for that, so fixing this properly will benefit your users!)

#3 lore raw
In reply to Leo Martins
Quoted 4 lines · Leo Martins

generic/747 consistently fails on btrfs in my fstests setup, with an ~88% failure rate across multiple runs on kernels ranging from v6.9 to v7.0-rc5. This is not a regression but a pre-existing issue since the test was added.

Glad we finally have someone trying to fix this.

Quoted 4 lines

The test fills a filesystem to 95% then does mixed write/delete cycles, using statfs to decide whether to write or delete. However, statfs f_bavail may overestimate the actual available space.

If this is the problem, then I believe we should fix our f_bavail calculation.

Quoted 6 lines

On btrfs, the
statfs implementation documents its estimate as "a close approximation" (fs/btrfs/super.c). At high fill levels the discrepancy between what statfs reports and what the filesystem can actually allocate becomes significant, causing dd to hit ENOSPC even though statfs indicated there was room.

From what I can see, there are several different ways that f_bavail is overestimated:

- btrfs_calc_available_data_space()
That functions goes through each device and try to find unallocated space.
This can not handle uneven space (e.g. 1G + 10G RAID0), and it should go through the per-profile estimation, which handles uneven disks way better.

Normally it shouldn't affect single device, but we can still over- estimate by taking the reserved 1M bytes as available.

- total_free_data calculation
It's just on-disk size minus the on-disk used space, which doesn't take things like reserved space into consideration.

Thus I guess the problem is mostly from this part.

Mind to give this idea a try to fix the long failing g747?

Thanks,
Qu

Quoted 79 lines

This is not a filesystem bug. The filesystem correctly rejects the write when it cannot reserve space. The test's purpose is to stress garbage collection through write/delete churn, not to validate space accounting.

Handle ENOSPC by cleaning up the partial file and making room:

In _direct_fillup: break out of the fill loop (we're full enough). In _mixed_write_delete: delete a file to free space and retry. If writes fail 10 consecutive times, _fail the test as that indicates a real filesystem issue rather than a transient statfs discrepancy.

Redirect dd stderr to seqres.full so errors are preserved for debugging without polluting the expected output.

---
  tests/generic/747 | 28 +++++++++++++++++++---------
  1 file changed, 19 insertions(+), 9 deletions(-)
tests/generic/747 +19 -9
index 44834186..35de3ccb 100755
@@ -35,11 +35,7 @@ _create_file() {
 
  	POSIXLY_CORRECT=yes dd if=/dev/zero of=${file_name} \
  		bs=${bs} count=$(( $file_sz / ${bs} )) \
-		status=none $dd_extra  2>&1
-
-	if [ $? -ne 0 ]; then
-		_fail "Failed writing $file_name"
-	fi
+		status=none $dd_extra  2>>$seqres.full
  }
 
  _total_M() {
@@ -69,7 +65,10 @@ _direct_fillup () {
  	while [ $(_used_percent) -lt $fill_percent ]; do
  		local fsz=$(_get_random_fsz)
 
-		_create_file $testseq $fsz "oflag=direct conv=fsync"
+		if ! _create_file $testseq $fsz "oflag=direct conv=fsync"; then
+			rm ${SCRATCH_MNT}/data_${testseq}
+			break
+		fi
  		testseq=$((${testseq} + 1))
  	done
  }
@@ -79,14 +78,25 @@ _mixed_write_delete() {
  	local total_M=$(_total_M)
  	local to_write_M=$(( ${overwrite_percentage} * ${total_M} / 100 ))
  	local written_M=0
+	local enospc_retries=0
+	local max_enospc_retries=10
 
  	while [ $written_M -lt $to_write_M ]; do
  		if [ $(_used_percent) -lt $fill_percent ]; then
  			local fsz=$(_get_random_fsz)
 
-			_create_file $testseq $fsz "$dd_extra"
-			written_M=$((${written_M} + ${fsz}/${M}))
-			testseq=$((${testseq} + 1))
+			if ! _create_file $testseq $fsz "$dd_extra"; then
+				rm ${SCRATCH_MNT}/data_${testseq}
+				_delete_random_file
+				enospc_retries=$((enospc_retries + 1))
+				if [ $enospc_retries -ge $max_enospc_retries ]; then
+					_fail "failed to write after $max_enospc_retries consecutive ENOSPC attempts"
+				fi
+			else
+				written_M=$((${written_M} + ${fsz}/${M}))
+				testseq=$((${testseq} + 1))
+				enospc_retries=0
+			fi
  		else
  			_delete_random_file
  		fi