Skip to contentSkip to conversation

generic/558: avoid forkbombs on filesystems with many free inodes

PATCH In review Darrick J. Wong 3 participants 5 messages Jul 14, 2023 – Jul 18, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 generic/558: avoid forkbombs on filesystems with many free inodes

From: Darrick J. Wong <[email protected]>

Mikulas reported that this test became a forkbomb on his system when he tested it with bcachefs. Unlike XFS and ext4, which have large inodes consuming hundreds of bytes, bcachefs has very tiny ones. Therefore, it reports a large number of free inodes on a freshly mounted 1GB fs (~15 million), which causes this test to try to create 15000 processes.

There's really no reason to do that -- all this test wanted to do was to exhaust the number of inodes as quickly as possible using all available CPUs, and then it ran xfs_repair to try to reproduce a bug. Set the number of subshells to 4x the CPU count and spread the work among them instead of forking thousands of processes.

---
 tests/generic/558 |   18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
tests/generic/558 +12 -6
index 4e22ce656b..de5c28d00d 100755
@@ -39,15 +39,21 @@ _scratch_mkfs_sized $((1024 * 1024 * 1024)) >>$seqres.full 2>&1
 _scratch_mount
 
 i=0
-free_inode=`_get_free_inode $SCRATCH_MNT`
-file_per_dir=1000
-loop=$((free_inode / file_per_dir + 1))
+free_inodes=$(_get_free_inode $SCRATCH_MNT)
+nr_cpus=$(( $($here/src/feature -o) * 4 * LOAD_FACTOR ))
+echo "free inodes: $free_inodes nr_cpus: $nr_cpus" >> $seqres.full
+
+if ((free_inodes <= nr_cpus)); then
+	nr_cpus=1
+	files_per_dir=$free_inodes
+else
+	files_per_dir=$(( (free_inodes + nr_cpus - 1) / nr_cpus ))
+fi
 mkdir -p $SCRATCH_MNT/testdir
 
 echo "Create $((loop * file_per_dir)) files in $SCRATCH_MNT/testdir" >>$seqres.full
-while [ $i -lt $loop ]; do
-	create_file $SCRATCH_MNT/testdir $file_per_dir $i >>$seqres.full 2>&1 &
-	let i=$i+1
+for ((i = 0; i < nr_cpus; i++)); do
+	create_file $SCRATCH_MNT/testdir $files_per_dir $i >>$seqres.full 2>&1 &
 done
 wait
 
Reviewed-by on v1 1/1
#1 lore raw
Quoted 14 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Mikulas reported that this test became a forkbomb on his system when he tested it with bcachefs. Unlike XFS and ext4, which have large inodes consuming hundreds of bytes, bcachefs has very tiny ones. Therefore, it reports a large number of free inodes on a freshly mounted 1GB fs (~15 million), which causes this test to try to create 15000 processes.

There's really no reason to do that -- all this test wanted to do was to exhaust the number of inodes as quickly as possible using all available CPUs, and then it ran xfs_repair to try to reproduce a bug. Set the number of subshells to 4x the CPU count and spread the work among them instead of forking thousands of processes.

Quoted 3 lines
Quoted 37 lines
---
 tests/generic/558 |   18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
tests/generic/558 +12 -6
index 4e22ce656b..de5c28d00d 100755
@@ -39,15 +39,21 @@ _scratch_mkfs_sized $((1024 * 1024 * 1024)) >>$seqres.full 2>&1
 _scratch_mount
 
 i=0
-free_inode=`_get_free_inode $SCRATCH_MNT`
-file_per_dir=1000
-loop=$((free_inode / file_per_dir + 1))
+free_inodes=$(_get_free_inode $SCRATCH_MNT)
+nr_cpus=$(( $($here/src/feature -o) * 4 * LOAD_FACTOR ))
+echo "free inodes: $free_inodes nr_cpus: $nr_cpus" >> $seqres.full
+
+if ((free_inodes <= nr_cpus)); then
+	nr_cpus=1
+	files_per_dir=$free_inodes
+else
+	files_per_dir=$(( (free_inodes + nr_cpus - 1) / nr_cpus ))
+fi
 mkdir -p $SCRATCH_MNT/testdir
 
 echo "Create $((loop * file_per_dir)) files in $SCRATCH_MNT/testdir" >>$seqres.full
-while [ $i -lt $loop ]; do
-	create_file $SCRATCH_MNT/testdir $file_per_dir $i >>$seqres.full 2>&1 &
-	let i=$i+1
+for ((i = 0; i < nr_cpus; i++)); do
+	create_file $SCRATCH_MNT/testdir $files_per_dir $i >>$seqres.full 2>&1 &
 done
 wait
 
#2 lore raw
In reply to Darrick J. Wong
Quoted 35 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Mikulas reported that this test became a forkbomb on his system when he tested it with bcachefs. Unlike XFS and ext4, which have large inodes consuming hundreds of bytes, bcachefs has very tiny ones. Therefore, it reports a large number of free inodes on a freshly mounted 1GB fs (~15 million), which causes this test to try to create 15000 processes.

There's really no reason to do that -- all this test wanted to do was to exhaust the number of inodes as quickly as possible using all available CPUs, and then it ran xfs_repair to try to reproduce a bug. Set the number of subshells to 4x the CPU count and spread the work among them instead of forking thousands of processes.

---
 tests/generic/558 |   18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
tests/generic/558 +3 -3
index 4e22ce656b..de5c28d00d 100755
@@ -39,15 +39,21 @@ _scratch_mkfs_sized $((1024 * 1024 * 1024)) >>$seqres.full 2>&1
 _scratch_mount
 
 i=0
-free_inode=`_get_free_inode $SCRATCH_MNT`
-file_per_dir=1000
-loop=$((free_inode / file_per_dir + 1))
+free_inodes=$(_get_free_inode $SCRATCH_MNT)
+nr_cpus=$(( $($here/src/feature -o) * 4 * LOAD_FACTOR ))
+echo "free inodes: $free_inodes nr_cpus: $nr_cpus" >> $seqres.full

tests/generic/558 _scratch_mkfs_sized $((1024 * 1024 * 1024)) >>$seqres.full 2>&1 · line 54

Quoted 10 lines
tests/generic/558 +7 -0
+
+if ((free_inodes <= nr_cpus)); then
+	nr_cpus=1
+	files_per_dir=$free_inodes
+else
+	files_per_dir=$(( (free_inodes + nr_cpus - 1) / nr_cpus ))
+fi
 mkdir -p $SCRATCH_MNT/testdir
 
 echo "Create $((loop * file_per_dir)) files in $SCRATCH_MNT/testdir" >>$seqres.full

Has the $loop been removed?

Quoted 9 lines
tests/generic/558 +2 -3
-while [ $i -lt $loop ]; do
-	create_file $SCRATCH_MNT/testdir $file_per_dir $i >>$seqres.full 2>&1 &
-	let i=$i+1
+for ((i = 0; i < nr_cpus; i++)); do
+	create_file $SCRATCH_MNT/testdir $files_per_dir $i >>$seqres.full 2>&1 &
 done
 wait
 
#3 lore raw
Quoted 45 lines · Zorro Lang
Quoted 44 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Mikulas reported that this test became a forkbomb on his system when he tested it with bcachefs. Unlike XFS and ext4, which have large inodes consuming hundreds of bytes, bcachefs has very tiny ones. Therefore, it reports a large number of free inodes on a freshly mounted 1GB fs (~15 million), which causes this test to try to create 15000 processes.

There's really no reason to do that -- all this test wanted to do was to exhaust the number of inodes as quickly as possible using all available CPUs, and then it ran xfs_repair to try to reproduce a bug. Set the number of subshells to 4x the CPU count and spread the work among them instead of forking thousands of processes.

---
 tests/generic/558 |   18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
tests/generic/558 +10 -3
index 4e22ce656b..de5c28d00d 100755
@@ -39,15 +39,21 @@ _scratch_mkfs_sized $((1024 * 1024 * 1024)) >>$seqres.full 2>&1
 _scratch_mount
 
 i=0
-free_inode=`_get_free_inode $SCRATCH_MNT`
-file_per_dir=1000
-loop=$((free_inode / file_per_dir + 1))
+free_inodes=$(_get_free_inode $SCRATCH_MNT)
+nr_cpus=$(( $($here/src/feature -o) * 4 * LOAD_FACTOR ))
+echo "free inodes: $free_inodes nr_cpus: $nr_cpus" >> $seqres.full
+
+if ((free_inodes <= nr_cpus)); then
+	nr_cpus=1
+	files_per_dir=$free_inodes
+else
+	files_per_dir=$(( (free_inodes + nr_cpus - 1) / nr_cpus ))
+fi
 mkdir -p $SCRATCH_MNT/testdir
 
Quoted 3 lines

tests/generic/558 _scratch_mkfs_sized $((1024 * 1024 * 1024)) >>$seqres.full 2>&1 · line 54

Quoted 1 line
tests/generic/558 +0 -0
 echo "Create $((loop * file_per_dir)) files in $SCRATCH_MNT/testdir" >>$seqres.full

Has the $loop been removed?

DOH. v3 on the way. I hate bash.

--D

Quoted 10 lines
Quoted 9 lines
tests/generic/558 +2 -3
-while [ $i -lt $loop ]; do
-	create_file $SCRATCH_MNT/testdir $file_per_dir $i >>$seqres.full 2>&1 &
-	let i=$i+1
+for ((i = 0; i < nr_cpus; i++)); do
+	create_file $SCRATCH_MNT/testdir $files_per_dir $i >>$seqres.full 2>&1 &
 done
 wait
 
#4 lore raw
In reply to Darrick J. Wong
Quoted 26 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Mikulas reported that this test became a forkbomb on his system when he tested it with bcachefs. Unlike XFS and ext4, which have large inodes consuming hundreds of bytes, bcachefs has very tiny ones. Therefore, it reports a large number of free inodes on a freshly mounted 1GB fs (~15 million), which causes this test to try to create 15000 processes.

There's really no reason to do that -- all this test wanted to do was to exhaust the number of inodes as quickly as possible using all available CPUs, and then it ran xfs_repair to try to reproduce a bug. Set the number of subshells to 4x the CPU count and spread the work among them instead of forking thousands of processes.

---
 tests/generic/558 |   18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
tests/generic/558 +0 -0
index 4e22ce656b..de5c28d00d 100755

tests/generic/558 _scratch_mkfs_sized $((1024 * 1024 * 1024)) >>$seqres.full 2>&1 · line 44

Quoted 6 lines
tests/generic/558 +0 -3
@@ -39,15 +39,21 @@ _scratch_mkfs_sized $((1024 * 1024 * 1024)) >>$seqres.full 2>&1
 _scratch_mount
 
 i=0
-free_inode=`_get_free_inode $SCRATCH_MNT`
-file_per_dir=1000
-loop=$((free_inode / file_per_dir + 1))

NAK. Here, the old code effectively does:

loop = howmany(free_inode, 1000);
for i in 0...loop:

	create_file ... 1000files... &

IOWs, it rounds the number of files to create up to the nearest 1000, which I overlooked because I was overloaded and words are easier than resurrecting mathematical concepts from raw formulae.

If, say, the 1G fs claims to have 524,288 free inodes, the test will start *525* create_file subshells to create 1000 files each, or 525,000 files.

The /new/ code does this instead:

nr_cpus=(cpu count * 4)
files_per_dir = howmany(free_inodes, nr_cpus)
for i in 0..nr_cpus:

	create_file ... files_per_dir... &

If nr_cpu is a factor of free_inodes, we don't do /any/ roundup at all. 524,288 free inodes with 4 CPUs gets you 16 threads and 32768 files per thread.

Apparently this is significant somehow, because on a lark I decided to revert the referenced commit and the new code doesn't reliably reproduce the failure when parent pointers are enabled.

Reintroducing the "rounding free_inodes up to the nearest 1000" does make it trip, though.

Sooooo... I'll have a new version out tomorrow after some testing. Please do not apply this patch until then, unless you are testing bcachefs.

--D

Quoted 21 lines
tests/generic/558 +12 -3
+free_inodes=$(_get_free_inode $SCRATCH_MNT)
+nr_cpus=$(( $($here/src/feature -o) * 4 * LOAD_FACTOR ))
+echo "free inodes: $free_inodes nr_cpus: $nr_cpus" >> $seqres.full
+
+if ((free_inodes <= nr_cpus)); then
+	nr_cpus=1
+	files_per_dir=$free_inodes
+else
+	files_per_dir=$(( (free_inodes + nr_cpus - 1) / nr_cpus ))
+fi
 mkdir -p $SCRATCH_MNT/testdir
 
 echo "Create $((loop * file_per_dir)) files in $SCRATCH_MNT/testdir" >>$seqres.full
-while [ $i -lt $loop ]; do
-	create_file $SCRATCH_MNT/testdir $file_per_dir $i >>$seqres.full 2>&1 &
-	let i=$i+1
+for ((i = 0; i < nr_cpus; i++)); do
+	create_file $SCRATCH_MNT/testdir $files_per_dir $i >>$seqres.full 2>&1 &
 done
 wait