generic/551: prevent OOM on NFS on systems with no swap memory
Subsystems (2)
PATCH 1/1 generic/551: prevent OOM on NFS on systems with no swap memory
From: Scott Mayhew <[email protected]>
We have frequently observed the oom-killer killing aio-dio-write-verify when generic/551 is run on NFS filesystems on virtual machines in AWS.
Virtual machines in AWS typically don't have a swap partition, so check for that condition when testing NFS and only use 90% of available memory when generating the list of write operations passed to aio-dio-write-verify.
--- common/rc | 5 +++++ tests/generic/551 | 7 +++++++ 2 files changed, 12 insertions(+)
common/rc +5 -0
@@ -1201,6 +1201,11 @@ _available_memory_bytes() fi } +_total_swap_bytes() +{ + free -b | awk '/^Swap/ { print $2 }' +} + _check_minimal_fs_size() { local fssize=$1
tests/generic/551 +7 -0
@@ -38,11 +38,18 @@ do_test() local truncsize local total_size=0 local avail_mem=`_available_memory_bytes` + local total_swap=`_total_swap_bytes` # To avoid OOM on tmpfs, subtract the amount of available memory # allocated for the tmpfs [ "$FSTYP" = "tmpfs" ] && avail_mem=$((avail_mem - free_size_k * 1024)) + # To avoid OOM on NFS on systems with no swap memory, only use 90% + # of available memory when generating the list of write operations + if [ "$FSTYP" = "nfs" -a $total_swap -eq 0 ]; then + avail_mem=$((avail_mem - avail_mem / 10)) + fi + # the number of AIO write operation num_oper=$((RANDOM % 64 + 1))
Signature · 1 line
2.52.0
Quoted 9 lines · Yongcheng Yang
From: Scott Mayhew <[email protected]>
We have frequently observed the oom-killer killing aio-dio-write-verify when generic/551 is run on NFS filesystems on virtual machines in AWS.
Virtual machines in AWS typically don't have a swap partition, so check for that condition when testing NFS and only use 90% of available memory when generating the list of write operations passed to aio-dio-write-verify.
I don't think this is a good idea. The proper fix is reduce whatever crazy large memory allocations this workloads causes in NFS. I suspect it might be page lists or similar, and just breaking them into somewhat smaller chunks and/or using potentially failing allocations to dynamically adjust would help.
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · Yongcheng Yang
From: Scott Mayhew <[email protected]>
We have frequently observed the oom-killer killing aio-dio-write-verify when generic/551 is run on NFS filesystems on virtual machines in AWS.
Virtual machines in AWS typically don't have a swap partition, so check for that condition when testing NFS and only use 90% of available memory when generating the list of write operations passed to aio-dio-write-verify.
Quoted 5 lines
I don't think this is a good idea. The proper fix is reduce whatever crazy large memory allocations this workloads causes in NFS. I suspect it might be page lists or similar, and just breaking them into somewhat smaller chunks and/or using potentially failing allocations to dynamically adjust would help.
I run fstests on XFS every night on a fleets of VMs with no swap and never hit OOM.
Or at least I didn't until IT mandated CrowdStrike last week and now it's anyone's guess if the test results are valid. <grumble>
--D