strict VM overcommit accounting for 2.4.32/2.4.33-pre1
Quoted 4 lines · Barry K. Nathan
This patch adds strict VM overcommit accounting to the mainline 2.4 kernel, thus allowing overcommit to be truly disabled. This feature has been in 2.4-ac, Red Hat Enterprise Linux 3 (RHEL 3) vendor kernels, and 2.6 for a long while.
Thanks a lot!
Review
Quoted 4 lines
+3 - (NEW) paranoid overcommit The total address space commit
(quoted diff) +3
-0+ for the system is not permitted to exceed swap. The machine + will never kill a process accessing pages it has mapped + except due to a bug (ie report it!)
This one isn't in 2.6, which is critical for a stable system.
Thanks!
--
Al
Quoted 8 lines · Al Boldi
Thanks a lot!
Review
Quoted 4 lines
+3 - (NEW) paranoid overcommit The total address space commit
(quoted diff) +3
-0+ for the system is not permitted to exceed swap. The machine + will never kill a process accessing pages it has mapped + except due to a bug (ie report it!)This one isn't in 2.6, which is critical for a stable system.
I mentioned in my original post (maybe I wasn't clear enough) that this is only in the documentation and not in the actual code (i.e. the code has no advantage over 2.6 in this regard). This error comes from the 2.4-ac/pac kernels from which this documentation originally comes. Yes, I need to fix the documentation, I didn't get to do that yet.
I think you can get paranoid overcommit with either my patch or 2.6 by setting /proc/sys/vm/overcommit_mode to 2 *and* /proc/sys/vm/overcommit_ratio to 0, however.
--
-Barry K. Nathan <[email protected]>
Quoted 8 lines · Barry K. Nathan
Quoted 7 lines · Al Boldi
Thanks a lot!
Quoted 4 lines
+3 - (NEW) paranoid overcommit The total address space commit
(quoted diff) +3
-0+ for the system is not permitted to exceed swap. The machine + will never kill a process accessing pages it has mapped + except due to a bug (ie report it!)
Quoted 5 lines
Quoted 1 line
This one isn't in 2.6, which is critical for a stable system.
I think you can get paranoid overcommit with either my patch or 2.6 by setting
/proc/sys/vm/overcommit_modeto 2 *and*/proc/sys/vm/overcommit_ratioto 0, however.
Not really in 2.6.
And even if this were made to work, what would it imply to a system running w/o swap?
Thanks!
--
Al
Quoted 11 lines · Al Boldi
Quoted 10 lines · Barry K. Nathan
Quoted 8 lines · Al Boldi
Thanks a lot!
Review
Quoted 4 lines
+3 - (NEW) paranoid overcommit The total address space commit
(quoted diff) +3
-0+ for the system is not permitted to exceed swap. The machine + will never kill a process accessing pages it has mapped + except due to a bug (ie report it!)This one isn't in 2.6, which is critical for a stable system.
Quoted 7 lines
Quoted 3 lines
I think you can get paranoid overcommit with either my patch or 2.6 by setting
/proc/sys/vm/overcommit_modeto 2 *and*/proc/sys/vm/overcommit_ratioto 0, however.Not really in 2.6.
And even if this were made to work, what would it imply to a system running w/o swap?
I can clearly reply on this one :
root@pcw:vm# swapoff -a
root@pcw:vm# free
total used free shared buffers cached
Mem: 1031752 84992 946760 0 3144 38564
-/+ buffers/cache: 43284 988468
Swap: 0 0 0
root@pcw:vm# echo 2 > overcommit_memory
root@pcw:vm# echo 0 > overcommit_ratio
root@pcw:vm# uname -a
bash: fork: Cannot allocate memory
root@pcw:vm# echo 10 > overcommit_ratio
root@pcw:vm# uname -a
Linux pcw 2.4.33-pre1 #1 SMP Fri Dec 30 21:52:06 CET 2005 i686 unknown
root@pcw:vm#So as one could expect, if you're limited to allocation of (swap_size)+0% of RAM, then you have a limit of 0 kB, so that malloc() always fails. I'll do some crash tests with mmap, shm and 100 ratio to see how it behaves, but at the moment it's clearly good. Multi-process malloc cannot allocate more than specified and that's good.
Quoted 4 lines
Thanks!
--
Al
Regards,
Willy
Quoted 6 lines · Al Boldi
Review
Quoted 4 lines
+3 - (NEW) paranoid overcommit The total address space commit
(quoted diff) +3
-0+ for the system is not permitted to exceed swap. The machine + will never kill a process accessing pages it has mapped + except due to a bug (ie report it!)This one isn't in 2.6, which is critical for a stable system.
Actually it is
In the 2.4 case we took "50% RAM + swap" as the safe sane world 'never OOM kill' and to all intents and purposes it works. We also had a 100% paranoia mode.
When it was ported to 2.6 (not by me) whoever did it very sensibly made the percentage tunable and removed "mode 3" since its mode 2 0% ram and can be set that way.
Alan
Quoted 10 lines · Alan Cox
Quoted 6 lines · Al Boldi
Review
Quoted 4 lines
+3 - (NEW) paranoid overcommit The total address space commit
(quoted diff) +3
-0+ for the system is not permitted to exceed swap. The machine + will never kill a process accessing pages it has mapped + except due to a bug (ie report it!)This one isn't in 2.6, which is critical for a stable system.
Actually it is
Quoted 7 lines
In the 2.4 case we took "50% RAM + swap" as the safe sane world 'never OOM kill' and to all intents and purposes it works. We also had a 100% paranoia mode.
When it was ported to 2.6 (not by me) whoever did it very sensibly made the percentage tunable and removed "mode 3" since its mode 2 0% ram and can be set that way.
Only, doesn't this imply that you cannot control overcommit unless backed by swap? i.e Without swap the kernel cannot use all of ram, because it would overcommit no-matter what, thus invoking OOM-killer.
Which raises an important question: What's overcommit to do with limiting access to physical RAM?
Thanks!
--
Al
Quoted 19 lines · Al Boldi
Quoted 10 lines · Alan Cox
Quoted 6 lines · Al Boldi
Review
Quoted 4 lines
+3 - (NEW) paranoid overcommit The total address space commit
(quoted diff) +3
-0+ for the system is not permitted to exceed swap. The machine + will never kill a process accessing pages it has mapped + except due to a bug (ie report it!)This one isn't in 2.6, which is critical for a stable system.
Actually it is
Quoted 7 lines
In the 2.4 case we took "50% RAM + swap" as the safe sane world 'never OOM kill' and to all intents and purposes it works. We also had a 100% paranoia mode.
When it was ported to 2.6 (not by me) whoever did it very sensibly made the percentage tunable and removed "mode 3" since its mode 2 0% ram and can be set that way.
Quoted 6 lines
Only, doesn't this imply that you cannot control overcommit unless backed by swap? i.e Without swap the kernel cannot use all of ram, because it would overcommit no-matter what, thus invoking OOM-killer.
Which raises an important question: What's overcommit to do with limiting access to physical RAM?
As shown in my previous mail, it allows malloc() to return NULL. I've also successfully verified that it allows mmap() to fail if there is not enough memory. I disabled swap, and set the overcommit_ratio to 95 and could not kill the system. Above this, it becomes tricky. At 97, I see the last malloc() calls take a very long time, and at 98, the system still hangs. But 95% without swap seems stable here.
Quoted 4 lines
Thanks!
--
Al
Cheers,
Willy
Quoted 27 lines · Willy Tarreau
Quoted 19 lines · Al Boldi
Quoted 10 lines · Alan Cox
Quoted 6 lines · Al Boldi
Review
Quoted 4 lines
+3 - (NEW) paranoid overcommit The total address space commit
(quoted diff) +3
-0+ for the system is not permitted to exceed swap. The machine + will never kill a process accessing pages it has mapped + except due to a bug (ie report it!)This one isn't in 2.6, which is critical for a stable system.
Actually it is
Quoted 7 lines
In the 2.4 case we took "50% RAM + swap" as the safe sane world 'never OOM kill' and to all intents and purposes it works. We also had a 100% paranoia mode.
When it was ported to 2.6 (not by me) whoever did it very sensibly made the percentage tunable and removed "mode 3" since its mode 2 0% ram and can be set that way.
Quoted 6 lines
Only, doesn't this imply that you cannot control overcommit unless backed by swap? i.e Without swap the kernel cannot use all of ram, because it would overcommit no-matter what, thus invoking OOM-killer.
Which raises an important question: What's overcommit to do with limiting access to physical RAM?
Quoted 6 lines
As shown in my previous mail, it allows
malloc()to return NULL. I've also successfully verified that it allowsmmap()to fail if there is not enough memory. I disabled swap, and set theovercommit_ratioto 95 and could not kill the system. Above this, it becomes tricky. At 97, I see the lastmalloc()calls take a very long time, and at 98, the system still hangs. But 95% without swap seems stable here.
Thanks, for confirming this! And I agree that this patch and 2.6 offer an important and necessary workaround to inhibit OOM-killer, but it's no more than a workaround.
And so the question remains: Why should overcommit come into play at all when dealing with physical RAM only?
Shouldn't it be possible to disable overcommit completely, thus giving kswapd a break from running wild trying to find something to swap/page, which is the reason why the system gets unstable going over 95% in your example.
Thanks!
--
Al
Quoted 1 line · Al Boldi
Willy Tarreau wrote:
(...)
Quoted 7 lines
Quoted 6 lines
As shown in my previous mail, it allows
malloc()to return NULL. I've also successfully verified that it allowsmmap()to fail if there is not enough memory. I disabled swap, and set theovercommit_ratioto 95 and could not kill the system. Above this, it becomes tricky. At 97, I see the lastmalloc()calls take a very long time, and at 98, the system still hangs. But 95% without swap seems stable here.
Quoted 6 lines
Thanks, for confirming this! And I agree that this patch and 2.6 offer an important and necessary workaround to inhibit OOM-killer, but it's no more than a workaround.
And so the question remains: Why should overcommit come into play at all when dealing with physical RAM only?
It is very important when you have many processes wasting lots of unused memory. For instance, when firefox allocates 10 MB and uses only 3, then the remaining 7 MB can be used by another process. But if finally firefox tries to use them, and there is no memory left, then chances are that some processes will be killed. I believe the same problem happens after a fork() because data gets COWed but I'm not certain about this.
I'd bet that using a heavy GUI under X with no swap an overcommit_ratio set around 95%, you could get occasionnal malloc() failures. But once again, I may be wrong.
Quoted 3 lines
Shouldn't it be possible to disable overcommit completely, thus giving kswapd a break from running wild trying to find something to swap/page, which is the reason why the system gets unstable going over 95% in your example.
I think it's going unstable above 95% because there are lots of other areas consuming memory. I don't know for example if dentry caches, network buffers, various hash tables, etc... are accounted for.
Quoted 4 lines
Thanks!
--
Al
Willy
Quoted 3 lines · Al Boldi
Shouldn't it be possible to disable overcommit completely, thus giving kswapd a break from running wild trying to find something to swap/page, which is the reason why the system gets unstable going over 95% in your example.
shared mappings make this impractical. To disable overcommit completely, each process would need to account for all its own shared libraries, eg each process gets glibc added etc. You'll find that on any non-extremely-stripped system you then end up with much more memory needed than you have ram.
Quoted 11 lines · Arjan van de Ven
Quoted 3 lines · Al Boldi
Shouldn't it be possible to disable overcommit completely, thus giving kswapd a break from running wild trying to find something to swap/page, which is the reason why the system gets unstable going over 95% in your example.
shared mappings make this impractical. To disable overcommit completely, each process would need to account for all its own shared libraries, eg each process gets glibc added etc. You'll find that on any non-extremely-stripped system you then end up with much more memory needed than you have ram.
Arjan, is this true even for read-only mappings such as shared libs ? It seems to me that those ones can precisely be mapped once because they are areas the process cannot extend. Am I wrong ?
Willy
Quoted 7 lines · Willy Tarreau
Quoted 6 lines · Arjan van de Ven
Quoted 3 lines · Al Boldi
Shouldn't it be possible to disable overcommit completely, thus giving kswapd a break from running wild trying to find something to swap/page, which is the reason why the system gets unstable going over 95% in your example.
Quoted 7 lines
Quoted 5 lines
shared mappings make this impractical. To disable overcommit completely, each process would need to account for all its own shared libraries, eg each process gets glibc added etc. You'll find that on any non-extremely-stripped system you then end up with much more memory needed than you have ram.
Arjan, is this true even for read-only mappings such as shared libs ?
shared libs aren't read only! they have all the relocations applied for example. (the fact that glibc mprotects it read only again afterwards doesn't change things.. part of the memory is already written to). and mprotect can be called again.... what would mprotect need to do if it's asked to make memory writable and the overcommit says "no space" ....
Quoted 11 lines · Arjan van de Ven
Quoted 4 lines · Al Boldi
Shouldn't it be possible to disable overcommit completely, thus giving kswapd a break from running wild trying to find something to swap/page, which is the reason why the system gets unstable going over 95% in your example.
shared mappings make this impractical. To disable overcommit completely, each process would need to account for all its own shared libraries, eg each process gets glibc added etc. You'll find that on any non-extremely-stripped system you then end up with much more memory needed than you have ram.
Are you implying shared maps are implemented by way of overcommitting?
Really, overcommit is an add-on feature like swapping, only overcommit is free because it's a lier. So removing an add-on feature should not affect the underlying system in any way, such as shared mappings or swapping.
It should be possible to allow swapping to handle all memory requests exceeding physical RAM. OverCommit should be a tuning option for those who like to live on the edge, because it really is a gamble.
In the case where swap = physical RAM and overcommit_ratio = 0, the kernel is in effect hiding the fact that it is overcommitting.
Can you see the overhead involved here?
Thanks for your input!
--
Al
Quoted 7 lines · Al Boldi
Quoted 6 lines · Arjan van de Ven
Quoted 4 lines · Al Boldi
Shouldn't it be possible to disable overcommit completely, thus giving kswapd a break from running wild trying to find something to swap/page, which is the reason why the system gets unstable going over 95% in your example.
Quoted 7 lines
Quoted 5 lines
shared mappings make this impractical. To disable overcommit completely, each process would need to account for all its own shared libraries, eg each process gets glibc added etc. You'll find that on any non-extremely-stripped system you then end up with much more memory needed than you have ram.
Are you implying shared maps are implemented by way of overcommitting?
yes. Using only 1 page shared is already overcommitting because in principle each user can cause a COW on that page and cause a memory allocation for it in the future. (just like "traditional" overcommit can cause a pagefault with an allocation)
Quoted 3 lines
Really, overcommit is an add-on feature like swapping, only overcommit is free because it's a lier. So removing an add-on feature should not affect the underlying system in any way, such as shared mappings or swapping.
then I think you're misunderstanding how the linux VM works
Quoted 3 lines
It should be possible to allow swapping to handle all memory requests exceeding physical RAM. OverCommit should be a tuning option for those who like to live on the edge, because it really is a gamble.
but it's a worthwhile gamble to not need GOBBLES of memory you're not using 99.9999999999999% of the time
Quoted 3 lines
In the case where swap = physical RAM and
overcommit_ratio= 0, the kernel is in effect hiding the fact that it is overcommitting.
swap==ram is not really relevant; you'll need a LOT more to cover the shared maps...