Skip to contentSkip to conversation

Fix tst_pollute_memory() safety margin for huge systems

PATCH New Martin Doucha 4 participants 6 messages Jan 27, 2021 Library code

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Fix tst_pollute_memory() safety margin for huge systems

tst_pollute_memory() still has OOM issues on system with huge amounts of RAM. Set safety margin to the largest value of:
- 2 * min_free_kbytes
- 128MB
- 4096 pages
- Free RAM / 128 (to account for memory allocation overhead)

---
 lib/tst_memutils.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
lib/tst_memutils.c +4 -1
index dd09db490..7cdb3dbe0 100644
@@ -20,8 +20,11 @@ void tst_pollute_memory(size_t maxsize, int fillchar)
 	struct sysinfo info;
 
 	SAFE_SYSINFO(&info);
-	safety = MAX(4096 * SAFE_SYSCONF(_SC_PAGESIZE), 128 * 1024 * 1024);
+	SAFE_FILE_SCANF("/proc/sys/vm/min_free_kbytes", "%zd", &safety);
+	safety = MAX(2048 * safety, 128 * 1024 * 1024);
+	safety = MAX(safety, 4096 * SAFE_SYSCONF(_SC_PAGESIZE));
 	safety /= info.mem_unit;
+	safety = MAX(safety, info.freeram / 128);
 
 	if (info.freeswap > safety)
 		safety = 0;
Signature · 1 line
2.29.2
#1 lore raw

PATCH 1/1 Fix tst_pollute_memory() safety margin for huge systems

Hi!
Actually I've been thinking about this one and since this is a strictly best effort action I think we should be more aggressive about the safety margin.

I also started to talk to kernel developers about this and generally about tests that try to allocate maximal amount of memory without triggering OOM since we have a few in the tree. I will send the results of this discussion once I have any here as well.

Signature · 2 lines
Cyril Hrubis
[email protected]
#2 lore raw

PATCH 1/1 Fix tst_pollute_memory() safety margin for huge systems

According my test,malloc amount greater than available memory,then invoke the oom-killer. #man free
Estimation of how much memory is available for starting new applications, without swapping.

    [root@liuxp mywork]# head /proc/meminfo
    MemTotal:       198101744 kB
    MemFree:        189303148 kB
    MemAvailable:   188566732 kB

Set safety margin to larger value, may be not sensible.

From: Cyril Hrubis
Date: 2021-01-27 20:03
To: Martin Doucha
CC: liuxp11; ltp
Subject: Re: [LTP] [PATCH] Fix tst_pollute_memory() safety margin for huge systems Hi!
Actually I've been thinking about this one and since this is a strictly best effort action I think we should be more aggressive about the safety margin.

I also started to talk to kernel developers about this and generally about tests that try to allocate maximal amount of memory without triggering OOM since we have a few in the tree. I will send the results of this discussion once I have any here as well.

Signature · 5 lines
Cyril Hrubis
[email protected]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20210127/ddde8d39/attachment.htm>
#3 lore raw

PATCH 1/1 Fix tst_pollute_memory() safety margin for huge systems

Hi!

Quoted 1 line

According my test,malloc amount greater than available memory,then invoke the oom-killer.

However that does not mean that allocating MemAvailable or slightly less than MemAvailable is safe. In fact it's not, at least that I've been told be kernel developers. I will try to figure out something safe enough with their help.

Signature · 2 lines
Cyril Hrubis
[email protected]
#4 lore raw

PATCH 1/1 Fix tst_pollute_memory() safety margin for huge systems

OK,I also wanna know how to fix it.
Thanks!

From: Cyril Hrubis
Date: 2021-01-27 20:28
To: [email protected]
CC: mdoucha; ltp
Subject: Re: Re: [LTP] [PATCH] Fix tst_pollute_memory() safety margin for huge systems Hi!

Quoted 1 line

According my test,malloc amount greater than available memory,then invoke the oom-killer.

However that does not mean that allocating MemAvailable or slightly less than MemAvailable is safe. In fact it's not, at least that I've been told be kernel developers. I will try to figure out something safe enough with their help.

Signature · 5 lines
Cyril Hrubis
[email protected]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20210127/ed42b7e7/attachment.htm>
#5 lore raw

PATCH 1/1 Fix tst_pollute_memory() safety margin for huge systems

In reply to Cyril Hrubis
Quoted 9 lines · Cyril Hrubis

Hi!

Quoted 1 line

According my test,malloc amount greater than available memory,then

invoke the oom-killer.

However that does not mean that allocating MemAvailable or slightly less than MemAvailable is safe. In fact it's not, at least that I've been told be kernel developers. I will try to figure out something safe enough with their help.

If we only hope to avoid trigger OOM during allocating until the MAP_FAILED, I think the knob 'proc/sys/vm/overcommit_memory' maybe helpful.

e.g.

set overcommit_memory = 2, overcommit_ratio = 85 or 90; ... memory allocating as much as you can ... restore the value of overcommit_*

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20210127/dbf6179c/attachment-0001.htm>