Should calculation of vm.overcommit_ratio be changed?
Hi all,
I am not on the mailing list and a friend pointed me to this thread...
Probably we had the same problem: We had a linux computer with 16 GB of RAM without swap. There was only one big job running on it which did a lot of I/O. This program failed to allocate much memory, we thought that this was due to the high amount of cached memory use. To avoid problems with overcommit we had set overcommit_memory to 2.
Now I have seen this thread and now it gets clear: The default value of overcommit_ratio is 50, therefore one program can not allocate more than 8GB of memory at all.
After reading this thread I wrote a little programm to allocate memory in 512MB blocks and fill it with zeros. My test system has 4GB of RAM and so I started:
qfix:~# free
total used free shared buffers cached
Mem: 4052376 338124 3714252 0 0 17992
-/+ buffers/cache: 320132 3732244
Swap: 0 0 0geschke@qfix:~$ ./a.out got 1 * 512MB got 2 * 512MB got 3 * 512MB malloc failure after 3 * 512 MB
So 1.5 GB are ok, 2 GB of possible 4GB not. I guess some memory of the 4GB are not useable at all and therefore the limit is slightly below 2GB with an overcommit_ratio=50.
Next step is to set overcommit_ratio=100:
qfix:~# echo 100 >/proc/sys/vm/overcommit_ratio
and run the porgram again:
geschke@qfix:~$ ./a.out got 1 * 512MB got 2 * 512MB got 3 * 512MB got 4 * 512MB got 5 * 512MB got 6 * 512MB malloc failure after 6 * 512 MB
That are more than 3 GB but I would have expected to get at least 3.5GB:
geschke@qfix:~$ free
total used free shared buffers cached
Mem: 4052376 344976 3707400 0 0 22472
-/+ buffers/cache: 322504 3729872
Swap: 0 0 0Maybe this is due to a reserved percentage for the root user?
However, if I set overcommit_ratio=110 I get more than 3.5 GB:
geschke@qfix:~$ ./a.out got 1 * 512MB got 2 * 512MB got 3 * 512MB got 4 * 512MB got 5 * 512MB got 6 * 512MB got 7 * 512MB malloc failure after 7 * 512 MB
However, now I tested this with a high usage of cached memory, I die a lot of I/O:
geschke@qfix:~$ free
total used free shared buffers cached
Mem: 4052376 1945512 2106864 0 0 1621200
-/+ buffers/cache: 324312 3728064
Swap: 0 0 0A new run gives:
geschke@qfix:~$ ./a.out got 1 * 512MB got 2 * 512MB got 3 * 512MB got 4 * 512MB got 5 * 512MB got 6 * 512MB got 7 * 512MB malloc failure after 7 * 512 MB
and:
qfix:~# free
total used free shared buffers cached
Mem: 4052376 346928 3705448 0 0 26008
-/+ buffers/cache: 320920 3731456
Swap: 0 0 0So the cached memory is not really a problem for malloc.
But since I am testing, I tried what will happens if a lot of memory is already in use. So I opened a large file with "vi":
geschke@qfix:~$ free
total used free shared buffers cached
Mem: 4052376 1597168 2455208 0 0 391364
-/+ buffers/cache: 1205804 2846572
Swap: 0 0 0Now I start the program again:
geschke@qfix:~$ ./a.out got 1 * 512MB got 2 * 512MB got 3 * 512MB got 4 * 512MB got 5 * 512MB malloc failure after 5 * 512 MB
Fine: It seems that there is not really a problem to increase to overcommit_ratio=100 if there is no swap in the system and one has set overcommit_memory=2.
So I think, it is not really a problem to run with these settings.
Best regards
Dirk
Signature · 5 lines
+----------------------------------------------------------------------+ | Dr. Dirk Geschke / Plankensteinweg 61 / 85435 Erding | | Telefon: 08122-559448 / Mobil: 0176-96906350 / Fax: 08122-9818106 | | [email protected] / [email protected] / [email protected] | +----------------------------------------------------------------------+