Protecting processes from the OOM killer
For a while now, I've been trying to figure out how to make the oom killer not kill important processes.
How about rewarding processes that have an
RSS limit if they stay well below it?
The operator can then mark processes that are important by using 'ulimit -m'.
(This is orthogonal to Rik's recent patch.)
oom_kill.c renamed from oom_kill.c.orig +12 -0
@@ -86,6 +90,18 @@ points *=2 ;/* +* Processes which *have* an RSS limit, but which are under half of it, +* are behaving well, so halve their badness points. +* Do it again if they're under a quarter of their RSS limit. +*/ +if (p->rlim[RLIMIT_RSS ].rlim_max !=ULONG_MAX ) { +if (p->mm->rss < (p->rlim[RLIMIT_RSS ].rlim_max >> (PAGE_SHIFT +1 ))) + points /=2 ; +if (p->mm->rss < (p->rlim[RLIMIT_RSS ].rlim_max >> (PAGE_SHIFT +2 ))) + points /=2 ; + } + +/* * Superuser processes are usually more important, so we make it * less likely that we kill those. */
Signature · 3 lines
Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045
Quoted 2 lines · Dan Kegel
For a while now, I've been trying to figure out how to make the oom killer not kill important processes.
How about by not allowing your system to excessively overcommit. Everything else is armwaving "works half the time" stuff. By the time the OOM kicks in the game is already over. The rlimit one doesnt deal with things like fork explosions where you have lots of processes all under 1/4 of the rlimit range who cumulatively overcommit. In fact you now pick harder on other tasks...
Quoted 3 lines · you [Alan Cox]
How about by not allowing your system to excessively overcommit. Everything else is armwaving "works half the time" stuff.
Which invites the question: the strict overcommit stuff from -ac (the 'echo {2,3} > /proc/sys/vm/overcommit_memory' stuff) hasn't found it's way to mainline yet, has it? I wonder if it would be compatible with up-to-date -aa vm...
-- v --
Quoted 9 lines · Ville Herva
Quoted 3 lines · you [Alan Cox]
How about by not allowing your system to excessively overcommit. Everything else is armwaving "works half the time" stuff.
Which invites the question: the strict overcommit stuff from -ac (the 'echo {2,3} >
/proc/sys/vm/overcommit_memory' stuff) hasn't found it's way to mainline yet, has it? I wonder if it would be compatible with up-to-date -aa vm...
Marcelo didn't want it for base. Its in 2.5 and in -ac. There is no longer any rmap requirement on the code so it should "just work" with the -aa changes too
Quoted 7 lines · Alan Cox
Quoted 2 lines · Dan Kegel
For a while now, I've been trying to figure out how to make the oom killer not kill important processes.
How about by not allowing your system to excessively overcommit.
(I'm using 2.4.18; is
http://www.kernel.org/pub/linux/kernel/people/rml/vm/strict-overcommit/v2.4/vm-strict-overcommit-rml-2.4.18-1.patch
still the approprate patch for that?)
Quoted 2 lines
Everything else is armwaving "works half the time" stuff. By the time the OOM kicks in the game is already over.
Even with overcommit disallowed, the OOM killer is going to run when my users try to run too big a job, so I would still like the OOM killer to behave "well".
Quoted 4 lines
The rlimit one doesnt deal
with things like fork explosions where you have lots of processes all under 1/4 of the rlimit range who cumulatively overcommit. In fact you now pick harder on other tasks...
We do not see fork explosions in our workload, but if we did, we could abuse the RSS limit for now by setting it to zero except for the processes we wanted to protect from the OOM killer.
If that works in practice the same idea could be done without the abuse; the RSS limit is just a handy knob.
- Dan
Signature · 3 lines
Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045
Quoted 9 lines · Dan Kegel
Quoted 4 lines · Alan Cox
On Fri, 2003-02-28 at 01:21, Dan Kegel wrote:
Everything else is armwaving "works half the time" stuff. By the time the OOM kicks in the game is already over.
Even with overcommit disallowed, the OOM killer is going to run when my users try to run too big a job, so I would still like the OOM killer to behave "well".
If OOM is called you've overcommitted memory, so this isn't true
... no overcommit == NULL from malloc() etc.
Signature · 4 lines
# James Antill -- [email protected] :0: * ^From: .*james@and\.org /dev/null
Quoted 1 line · Dan Kegel
Alan Cox wrote:
snip
Quoted 6 lines
Quoted 2 lines
Everything else is armwaving "works half the time" stuff. By the time the OOM kicks in the game is already over.
Even with overcommit disallowed, the OOM killer is going to run when my users try to run too big a job, so I would still like the OOM killer to behave "well".
Shouldn't - the process the user tries to run will not be started since it must reserve the space first. malloc will fail immediately, allowing the process to handle the even gracefully and exit.
Anything else is a bug in the application.
Signature · 5 lines
------------------------------------------------------------------------- Jesse I Pollard, II Email: [email protected] Any opinions expressed are solely my own.
Quoted 5 lines · Jesse Pollard
Shouldn't - the process the user tries to run will not be started since it must reserve the space first. malloc will fail immediately, allowing the process to handle the even gracefully and exit.
Anything else is a bug in the application.
The one case you can't cover cleanly in C is a stack grow exceeding memory usage. At that point it requires a tiny bit of magic. You can do it, but the overcommit blocker has to armwave a little for the kernel and other things so I've never seen it happen in a normal situation
Quoted 9 lines · Jesse Pollard
Quoted 1 line · Dan Kegel
Alan Cox wrote:
snip
Quoted 3 lines
Quoted 2 lines
Everything else is armwaving "works half the time" stuff. By the time the OOM kicks in the game is already over.
Quoted 8 lines
Quoted 3 lines
Even with overcommit disallowed, the OOM killer is going to run when my users try to run too big a job, so I would still like the OOM killer to behave "well".
Shouldn't - the process the user tries to run will not be started since it must reserve the space first. malloc will fail immediately, allowing the process to handle the even gracefully and exit.
I thought of that about five minutes after I hit 'send'.
I have a feeling that there might still be a few cases not perfectly covered by the strict overcommit patch. Say, memory allocations due to incoming network traffic. I guess if memory runs out during incoming traffic, the kernel should simply drop the traffic. Until all those situations are nicely ironed out, there's still some chance the OOM killer might run even on a strict overcommit system.
But enough talking; I need to go try it.
- Dan
Signature · 3 lines
Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045