OOM killer???
Subsystems (3)
Just a quick bug-report:
One of our machines just started spewing:
Out of Memory: Killed process 117 (sendmail).
Out of Memory: Killed process 117 (sendmail).
Out of Memory: Killed process 117 (sendmail).
Out of Memory: Killed process 117 (sendmail).
Out of Memory: Killed process 117 (sendmail).
Out of Memory: Killed process 117 (sendmail).
Out of Memory: Killed process 117 (sendmail).
Out of Memory: Killed process 117 (sendmail).
Out of Memory: Killed process 117 (sendmail).
What we did to run it out of memory, I don't know. But I do know that it shouldn't be killing one process more than once... (the process should not exist after one try...)
Kernel 2.4.0 .
Roger.
Signature · 4 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2137555 ** *-- BitWizard writes Linux device drivers for any device you may have! --* * There are old pilots, and there are bold pilots. * There are also old, bald pilots.
Quoted 5 lines
Out of Memory: Killed process 117 (sendmail).
What we did to run it out of memory, I don't know. But I do know that it shouldn't be killing one process more than once... (the process should not exist after one try...)
This is a known bug in the Out-of-Memory handler, where it does not count the buffer and cache memory as "free" (it should), causing premature OOM killing. It is, however, normal for the OOM killer to attempt to kill a process more than once - it takes a few scheduler cycles for the SIGKILL to actually reach the process and take effect.
Also, it probably shouldn't have killed Sendmail, since that is usually a long-running, low-UID (and important) process. The OOM-kill selector is another thing that wants fixing, and my patch contains a *very rough* beginning to this.
The following patch should solve your problem for now, until a more detailed fix (which also clears up many other problems) is available in the stable kernel.
Alan and/or Linus may wish to apply this patch too...
(excerpt from my original patch from Saturday follows)
--- start ---
mm/oom_kill.c Sat Mar 24 20:35:20 2001 renamed from mm/oom_kill.c Tue Nov 14 18:56:46 2000 +17 -3
@@ -76,7 +76,9 @@ run_time = (jiffies - p->start_time) >> (SHIFT_HZ + 10); points /= int_sqrt(cpu_time); - points /= int_sqrt(int_sqrt(run_time)); + + /* Long-running processes are *very* important, so don't take the 4th root */ + points /= run_time; /* * Niced processes are most likely less important, so double
@@ -93,6 +95,10 @@ p->uid == 0 || p->euid == 0) points /= 4; + /* Much the same goes for processes with low UIDs */ + if(p->uid < 100 || p->euid < 100) + points /= 2; + /* * We don't want to kill a process with direct hardware access. * Not only could that mess up the hardware, but usually users
@@ -192,12 +198,20 @@ int out_of_memory(void) { struct sysinfo swp_info; + long free; /* Enough free memory? Not OOM. */ - if (nr_free_pages() > freepages.min) + free = nr_free_pages(); + if (free > freepages.min) + return 0; + + if (free + nr_inactive_clean_pages() > freepages.low) return 0; - if (nr_free_pages() + nr_inactive_clean_pages() > freepages.low) + /* Buffers and caches can be freed up (Jonathan "Chromatix" Morton) */ + free += atomic_read(&buffermem_pages); + free += atomic_read(&page_cache_size); + if (free > freepages.low) return 0; /* Enough swap space left? Not OOM. */
--- end ---
--------------------------------------------------------------
from: Jonathan "Chromatix" Morton mail: [email protected] (not for attachments) big-mail: [email protected] uni-mail: [email protected]
The key to knowledge is not to rely on people to teach you it.
Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/
-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*) -----END GEEK CODE BLOCK-----
Quoted 33 lines · Jonathan Morton
Quoted 6 lines
Out of Memory: Killed process 117 (sendmail).
What we did to run it out of memory, I don't know. But I do know that it shouldn't be killing one process more than once... (the process should not exist after one try...)
This is a known bug in the Out-of-Memory handler, where it does not count the buffer and cache memory as "free" (it should), causing premature OOM killing. It is, however, normal for the OOM killer to attempt to kill a process more than once - it takes a few scheduler cycles for the SIGKILL to actually reach the process and take effect.
Also, it probably shouldn't have killed Sendmail, since that is usually a long-running, low-UID (and important) process. The OOM-kill selector is another thing that wants fixing, and my patch contains a *very rough* beginning to this.
The following patch should solve your problem for now, until a more detailed fix (which also clears up many other problems) is available in the stable kernel.
Alan and/or Linus may wish to apply this patch too...
(excerpt from my original patch from Saturday follows)
--- start ---
diff -ulinux-2.4.1.orig/mm/oom_kill.clinux/mm/oom_kill.cmm/oom_kill.c Sat Mar 24 20:35:20 2001 renamed from mm/oom_kill.c Tue Nov 14 18:56:46 2000 +3
-1@@ -76,7 +76,9 @@ run_time = (jiffies - p->start_time) >> (SHIFT_HZ + 10); points /= int_sqrt(cpu_time); - points /= int_sqrt(int_sqrt(run_time)); + + /* Long-running processes are *very* important, so don't take the 4th root */ + points /= run_time; /* * Niced processes are most likely less important, so double
mm/oom_kill.c Sat Mar 24 20:35:20 2001 line 101
Quoted 7 lines
mm/oom_kill.c Sat Mar 24 20:35:20 2001 renamed from mm/oom_kill.c Tue Nov 14 18:56:46 2000 +4
-0@@ -93,6 +95,10 @@ p->uid == 0 || p->euid == 0) points /= 4; + /* Much the same goes for processes with low UIDs */ + if(p->uid < 100 || p->euid < 100) + points /= 2; +
Plase change to 100 to 500 - this would make it consistant with the useradd command, which starts adding new users at the UID 500
Quoted 2 lines
Plase change to 100 to 500 - this would make it consistant with the useradd command, which starts adding new users at the UID 500
Depends on which distribution you're using. In my experience, almost all the really important stuff happens below 100. In any case, the OOM-kill-selection algorithm in this patch is *not* final. See my accompanying mail.
--------------------------------------------------------------
from: Jonathan "Chromatix" Morton mail: [email protected] (not for attachments) big-mail: [email protected] uni-mail: [email protected]
The key to knowledge is not to rely on people to teach you it.
Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/
-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*) -----END GEEK CODE BLOCK-----
Quoted 2 lines · Martin Dalecki
Plase change to 100 to 500 - this would make it consistant with the useradd command, which starts adding new users at the UID 500
It's probably best to keep it somewhere <500, so that one can have "static" (<500) UIDs of either flavor: OOM-killable or not. 100 seems like "enough" non-killable users to me, but that may be a lack of imagination on my part.
Signature · 1 line
/Jonathan Lundell.
Config bug? In 2.2.19 CONFIG_RTL8139 depends on CONFIG_EXPERIMENTAL
In 2.2.19 CONFIG_RTL8139 depends on CONFIG_EXPERIMENTAL. The RTL8139 driver is not labelled as experimental. Is this an error?
- Greg
Re: Config bug? In 2.2.19 CONFIG_RTL8139 depends on CONFIG_EXPERIMENTAL
Quoted 3 lines · Greg Ingram
In 2.2.19
CONFIG_RTL8139depends onCONFIG_EXPERIMENTAL. The RTL8139 driver is not labelled as experimental. Is this an error?
Yeah, add '(EXPERIMENTAL)' to the text. Send a patch to Alan if you want...
Signature · 3 lines
Jeff Garzik | May you have warm words on a cold evening, Building 1024 | a full moon on a dark night, MandrakeSoft | and a smooth road all the way to your door.
PATCH 1/1 2.2.19 drivers/net/Config.in
Quoted 7 lines · Jeff Garzik
Quoted 3 lines · Greg Ingram
In 2.2.19
CONFIG_RTL8139depends onCONFIG_EXPERIMENTAL. The RTL8139 driver is not labelled as experimental. Is this an error?Yeah, add '(EXPERIMENTAL)' to the text. Send a patch to Alan if you want...
Okay. I really thought it would be the other way around, that is, that the driver is no longer experimental. Anyway, I also tagged the 8139too driver as experimental. Patch follows.
- Greg
drivers/net/Config.in renamed from drivers/net/Config.in.orig +2 -2
@@ -98,10 +98,10 @@ tristate 'NI6510 support' CONFIG_NI65 fi if [ "$CONFIG_EXPERIMENTAL" = "y" ]; then - tristate 'RealTek 8129/8139 (not 8019/8029!) support' CONFIG_RTL8139 + tristate 'RealTek 8129/8139 (not 8019/8029!) support (EXPERIMENTAL)' CONFIG_RTL8139 fi if [ "$CONFIG_EXPERIMENTAL" = "y" ]; then - tristate 'Alternative RealTek 8139 driver (8139too) support' CONFIG_RTL8139TOO + tristate 'Alternative RealTek 8139 driver (8139too) support (EXPERIMENTAL)' CONFIG_RTL8139TOO if [ "$CONFIG_RTL8139TOO" != "n" ]; then bool ' Use PIO instead of MMIO' CONFIG_8139TOO_PIO bool ' Support for automatic channel equalization' CONFIG_8139TOO_TUNE_TWISTER
Quoted 11 lines · Martin Dalecki
mm/oom_kill.c Sat Mar 24 20:35:20 2001 line 101
Quoted 8 lines
mm/oom_kill.c Sat Mar 24 20:35:20 2001 renamed from mm/oom_kill.c Tue Nov 14 18:56:46 2000 +4
-0@@ -93,6 +95,10 @@ p->uid == 0 || p->euid == 0) points /= 4; + /* Much the same goes for processes with low UIDs */ + if(p->uid < 100 || p->euid < 100) + points /= 2; +Plase change to 100 to 500 - this would make it consistant with the useradd command, which starts adding new users at the UID 500
No, useradd reads usally the /etc/login.defs to select the range. The oom-killer should have configurables for that, to allow the policy decisions in USER space -- where it belongs -- not in KERNEL space
If we use my OOM killer API, this patch would be a module and could have module parameters to select that.
Johnathan: I URGE you to apply my patch before adding OOM killer
stuff. What's wrong with it, that you cannot use it? ;-)
It is easy to add configurables to a module and play with them WITHOUT recompiling.
Dynamic sysctl tables would also be possible, IF we had an value that is DEFINED to be invalid for sysctrl(2) and only valid for /proc.
It is also better to include the egid into the decision. There are deamons, that I defintely want to be killed on a workstation, but not on a server.
e.g. My important matlab calculation, which runs in user mode should not be killed. But killing a local webserver, which serves my help system is ok (because I will not loose work, and might get it over the net, if there is a problem).
So as Rik stated: The OOM killer cannot suit all people, so it has to be configurable, to be OOM kill, not overkill ;-)
Thanks & Regards
Ingo Oeser
Signature · 2 lines
10.+11.03.2001 - 3. Chemnitzer LinuxTag <http://www.tu-chemnitz.de/linux/tag>
<<<<<<<<<<<< been there and had much fun >>>>>>>>>>>>Quoted 8 lines
If we use my OOM killer API, this patch would be a module and could have module parameters to select that.
Johnathan: I URGE you to apply my patch before adding OOM killer
stuff. What's wrong with it, that you cannot use it? ;-)It is easy to add configurables to a module and play with them WITHOUT recompiling.
Thanks for reminding me - I'll look into it on the plane and see what I can do with it.
Quoted 7 lines
e.g. My important matlab calculation, which runs in user mode should not be killed. But killing a local webserver, which serves my help system is ok (because I will not loose work, and might get it over the net, if there is a problem).
So as Rik stated: The OOM killer cannot suit all people, so it has to be configurable, to be OOM kill, not overkill ;-)
Yes, configurability is probably a very good idea. However, it would be best to include a good set of general parameters in the kernel itself, so the set of average systems needs as little tweaking as possible. One cannot expect every sysadmin to be familiar with these arcane (and rarely actually used) parameters, so being able to select "server", "batch", "workstation", "embedded" and so on would help massively.
--------------------------------------------------------------
from: Jonathan "Chromatix" Morton mail: [email protected] (not for attachments) big-mail: [email protected] uni-mail: [email protected]
The key to knowledge is not to rely on people to teach you it.
Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/
-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*) -----END GEEK CODE BLOCK-----
Quoted 14 lines · Ingo Oeser
Quoted 11 lines · Martin Dalecki
mm/oom_kill.c Sat Mar 24 20:35:20 2001 line 101
Quoted 8 lines
mm/oom_kill.c Sat Mar 24 20:35:20 2001 renamed from mm/oom_kill.c Tue Nov 14 18:56:46 2000 +4
-0@@ -93,6 +95,10 @@ p->uid == 0 || p->euid == 0) points /= 4; + /* Much the same goes for processes with low UIDs */ + if(p->uid < 100 || p->euid < 100) + points /= 2; +Plase change to 100 to 500 - this would make it consistant with the useradd command, which starts adding new users at the UID 500
Quoted 3 lines
No, useradd reads usally the
/etc/login.defsto select the range. The oom-killer should have configurables for that, to allow the policy decisions in USER space -- where it belongs -- not in KERNEL space
OK sysctl would be more appripriate.
Quoted 8 lines
If we use my OOM killer API, this patch would be a module and could have module parameters to select that.
Johnathan: I URGE you to apply my patch before adding OOM killer
stuff. What's wrong with it, that you cannot use it? ;-)It is easy to add configurables to a module and play with them WITHOUT recompiling.
It's total overkill and therefore not a good design.
Quoted 7 lines
Dynamic sysctl tables would also be possible, IF we had an value that is DEFINED to be invalid for sysctrl(2) and only valid for /proc.
It is also better to include the egid into the decision. There are deamons, that I defintely want to be killed on a workstation, but not on a server.
Quoted 7 lines
e.g. My important matlab calculation, which runs in user mode should not be killed. But killing a local webserver, which serves my help system is ok (because I will not loose work, and might get it over the net, if there is a problem).
So as Rik stated: The OOM killer cannot suit all people, so it has to be configurable, to be OOM kill, not overkill ;-)
Irony: Why then not store this information permanently - inside the UID of the application?
Quoted 6 lines · Martin Dalecki
Quoted 2 lines · Ingo Oeser
So as Rik stated: The OOM killer cannot suit all people, so it has to be configurable, to be OOM kill, not overkill ;-)
Irony: Why then not store this information permanently - inside the UID of the application?
Because in some cases (large companies and such) the UID is centrally controlled across all machines in the company, so there are > 100 (or 500 or 1000) "system" UIDs. At one company I did work for, there were dozens (maybe > 100) oracle instances alone (each with different UID and passwords for security), and lots more "system" application UIDs, each unique.
Encoding more information into the UID is getting back to the bad old days of "uid 0" is can do anything, rather than the capability model we are working towards. Even so, encoding process killability info in the UID is _still_ not putting policy in user space, because if you don't like how the OOM killer works you still need to recompile and reboot.
Having a configurable OOM killer is not overkill, IMHO, because it is only called in very rare cases (i.e. OOM is hopefully a rare event), so it is definitely not on the fast path. I'm sure people will agree that spending a few extra cycles to kill the correct process is far better than killing a lot of incorrect processes quickly.
Every time this subject comes up, I point to AIX and SIGDANGER - a signal sent to processes when the system gets OOM. If the process has registered a SIGDANGER handler, it has a chance to free cache and such (or do a clean shutdown), otherwise the default signal handler will kill the process.
SIGDANGER would fix the original problem (killing numerical methods application running for weeks) perfectly - the application can freely allocate cache memory to speed up the calculations. When system gets OOM (for whatever reason), it sends SIGDANGER to applications first and they can free buffers or do safe shutdown, and this may get system out of OOM case without having to kill anything.
Granted, I'm not against fixing the VM to reducing OOM conditions in the first place. Having SIGDANGER still gives the application a chance to save itself before it is killed, which none of the OOM changes have addressed at all. It is _still_ possible to get a system into OOM from network buffers and such, regardless of whether an application is calling malloc() returns NULL or not.
Also, having a SIGDANGER handler _could_ reduce a process "badness" value when looking for processes to SIGKILL, when calling all of the SIGDANGER handlers has not freed enough memory to get out of OOM. This assumes that programs which register SIGDANGER handlers are important, rather than malicious (in which case your system has other problems).
Cheers, Andreas
Signature · 3 lines
Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto,
\ would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- DogbertQuoted 4 lines · --On Tuesday, March 27, 2001 12:55:50 -0700 Andreas Dilger…
Every time this subject comes up, I point to AIX and SIGDANGER - a signal sent to processes when the system gets OOM. If the process has registered a SIGDANGER handler, it has a chance to free cache and such (or do a clean shutdown), otherwise the default signal handler will kill the process.
Having a SIGDANGER would be a fine thing, but this will need patching in all current daemons and there has to be a possibility to configure the behaviour of the process when recieving a SIGDANGER. i.e. it is a good idea to kill apache on a workstation, but a very bad idea to kill apache on a webserver. Generally I'd like to see such an implementation, but wouldn't it be better to have a pre-seclction of the processes getting SIGDANGER?
For example: if OOM occours, send SIGDANGER to all non-root-processes with a nice-level of n or higher (where n should be discussed).
This would make it easy to "configure" SIGDANGER-unaware Applications - in the meantime, until all applications are SIGDANGER-aware - to deal with OOM-situations. You just do an "nice -n -1 httpd" and one's httpd won't
get killed when OOM occours.
IMO this would dramatically improve the OOM-Problems right now.
--
Andreas Rogge <[email protected]>
Available on IRCnet:#linux.de as Dyson