Skip to contentSkip to conversation

Should calculation of vm.overcommit_ratio be changed?

DISCUSSION Dirk Geschke 1 participant 1 message Apr 20, 2010

lore
Share & tools
lore
#0 lore raw

Hi all,

I am not on the mailing list and a friend pointed me to this thread...

Probably we had the same problem: We had a linux computer with 16 GB of RAM without swap. There was only one big job running on it which did a lot of I/O. This program failed to allocate much memory, we thought that this was due to the high amount of cached memory use. To avoid problems with overcommit we had set overcommit_memory to 2.

Now I have seen this thread and now it gets clear: The default value of overcommit_ratio is 50, therefore one program can not allocate more than 8GB of memory at all.

After reading this thread I wrote a little programm to allocate memory in 512MB blocks and fill it with zeros. My test system has 4GB of RAM and so I started:

qfix:~# free
             total       used       free     shared    buffers     cached
Mem:       4052376     338124    3714252          0          0      17992
-/+ buffers/cache:     320132    3732244
Swap:            0          0          0
geschke@qfix:~$ ./a.out
got 1 * 512MB
got 2 * 512MB
got 3 * 512MB
malloc failure after 3 * 512 MB

So 1.5 GB are ok, 2 GB of possible 4GB not. I guess some memory of the 4GB are not useable at all and therefore the limit is slightly below 2GB with an overcommit_ratio=50.

Next step is to set overcommit_ratio=100:

qfix:~# echo 100 >/proc/sys/vm/overcommit_ratio

and run the porgram again:

geschke@qfix:~$ ./a.out
got 1 * 512MB
got 2 * 512MB
got 3 * 512MB
got 4 * 512MB
got 5 * 512MB
got 6 * 512MB
malloc failure after 6 * 512 MB

That are more than 3 GB but I would have expected to get at least 3.5GB:

geschke@qfix:~$ free
             total       used       free     shared    buffers     cached
Mem:       4052376     344976    3707400          0          0      22472
-/+ buffers/cache:     322504    3729872
Swap:            0          0          0

Maybe this is due to a reserved percentage for the root user?

However, if I set overcommit_ratio=110 I get more than 3.5 GB:

geschke@qfix:~$ ./a.out
got 1 * 512MB
got 2 * 512MB
got 3 * 512MB
got 4 * 512MB
got 5 * 512MB
got 6 * 512MB
got 7 * 512MB
malloc failure after 7 * 512 MB

However, now I tested this with a high usage of cached memory, I die a lot of I/O:

geschke@qfix:~$ free
             total       used       free     shared    buffers     cached
Mem:       4052376    1945512    2106864          0          0    1621200
-/+ buffers/cache:     324312    3728064
Swap:            0          0          0

A new run gives:

geschke@qfix:~$ ./a.out
got 1 * 512MB
got 2 * 512MB
got 3 * 512MB
got 4 * 512MB
got 5 * 512MB
got 6 * 512MB
got 7 * 512MB
malloc failure after 7 * 512 MB

and:

qfix:~# free
             total       used       free     shared    buffers     cached
Mem:       4052376     346928    3705448          0          0      26008
-/+ buffers/cache:     320920    3731456
Swap:            0          0          0

So the cached memory is not really a problem for malloc.

But since I am testing, I tried what will happens if a lot of memory is already in use. So I opened a large file with "vi":

geschke@qfix:~$ free
             total       used       free     shared    buffers     cached
Mem:       4052376    1597168    2455208          0          0     391364
-/+ buffers/cache:    1205804    2846572
Swap:            0          0          0

Now I start the program again:

geschke@qfix:~$ ./a.out
got 1 * 512MB
got 2 * 512MB
got 3 * 512MB
got 4 * 512MB
got 5 * 512MB
malloc failure after 5 * 512 MB

Fine: It seems that there is not really a problem to increase to overcommit_ratio=100 if there is no swap in the system and one has set overcommit_memory=2.

So I think, it is not really a problem to run with these settings.

Best regards

Dirk

Signature · 5 lines
+----------------------------------------------------------------------+
| Dr. Dirk Geschke       / Plankensteinweg 61    / 85435 Erding        |
| Telefon: 08122-559448  / Mobil: 0176-96906350 / Fax: 08122-9818106   |
| [email protected] / [email protected]  / [email protected] | 
+----------------------------------------------------------------------+