Skip to contentSkip to conversation

generic/551: prevent OOM on NFS on systems with no swap memory

PATCH In review Yongcheng Yang 3 participants 3 messages Mar 25

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 generic/551: prevent OOM on NFS on systems with no swap memory

From: Scott Mayhew <[email protected]>

We have frequently observed the oom-killer killing aio-dio-write-verify when generic/551 is run on NFS filesystems on virtual machines in AWS.

Virtual machines in AWS typically don't have a swap partition, so check for that condition when testing NFS and only use 90% of available memory when generating the list of write operations passed to aio-dio-write-verify.

---
 common/rc         | 5 +++++
 tests/generic/551 | 7 +++++++
 2 files changed, 12 insertions(+)
common/rc +5 -0
index 92cb6982..f2a4496f 100644
@@ -1201,6 +1201,11 @@ _available_memory_bytes()
 	fi
 }
 
+_total_swap_bytes()
+{
+	free -b | awk '/^Swap/ { print $2 }'
+}
+
 _check_minimal_fs_size()
 {
 	local fssize=$1
tests/generic/551 +7 -0
index 267c57ec..9c5ec4b2 100755
@@ -38,11 +38,18 @@ do_test()
 	local truncsize
 	local total_size=0
 	local avail_mem=`_available_memory_bytes`
+	local total_swap=`_total_swap_bytes`
 
 	# To avoid OOM on tmpfs, subtract the amount of available memory
 	# allocated for the tmpfs
 	[ "$FSTYP" = "tmpfs" ] && avail_mem=$((avail_mem - free_size_k * 1024))
 
+	# To avoid OOM on NFS on systems with no swap memory, only use 90%
+	# of available memory when generating the list of write operations
+	if [ "$FSTYP" = "nfs" -a $total_swap -eq 0 ]; then
+		avail_mem=$((avail_mem - avail_mem / 10))
+	fi
+
 	# the number of AIO write operation
 	num_oper=$((RANDOM % 64 + 1))
 
Signature · 1 line
2.52.0
#1 lore raw
Quoted 9 lines · Yongcheng Yang

From: Scott Mayhew <[email protected]>

We have frequently observed the oom-killer killing aio-dio-write-verify when generic/551 is run on NFS filesystems on virtual machines in AWS.

Virtual machines in AWS typically don't have a swap partition, so check for that condition when testing NFS and only use 90% of available memory when generating the list of write operations passed to aio-dio-write-verify.

I don't think this is a good idea. The proper fix is reduce whatever crazy large memory allocations this workloads causes in NFS. I suspect it might be page lists or similar, and just breaking them into somewhat smaller chunks and/or using potentially failing allocations to dynamically adjust would help.

#2 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · Yongcheng Yang

From: Scott Mayhew <[email protected]>

We have frequently observed the oom-killer killing aio-dio-write-verify when generic/551 is run on NFS filesystems on virtual machines in AWS.

Virtual machines in AWS typically don't have a swap partition, so check for that condition when testing NFS and only use 90% of available memory when generating the list of write operations passed to aio-dio-write-verify.

Quoted 5 lines

I don't think this is a good idea. The proper fix is reduce whatever crazy large memory allocations this workloads causes in NFS. I suspect it might be page lists or similar, and just breaking them into somewhat smaller chunks and/or using potentially failing allocations to dynamically adjust would help.

I run fstests on XFS every night on a fleets of VMs with no swap and never hit OOM.

Or at least I didn't until IT mandated CrowdStrike last week and now it's anyone's guess if the test results are valid. <grumble>

--D