Andrea VM changes
I spent way too long tweaking the OOM killer before I realized it was hopeless.
The fact that incoming network traffic can be what causes the OOM condition makes it Really Hard to decide which app deserves the axe.
In the test-and-measurement system I'm developing, it turned out the sanest thing to do with OOM conditions was to consider them user errors, and to handle them by dumping memory usage info about processes and slab caches, then halt. It's been very helpful because it turns flaky conditions into rock-solid failures. Too bad this drastic approach isn't appropriate for general use.
- Dan
Signature · 3 lines
Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045
Quoted 8 lines · Dan Kegel
In the test-and-measurement system I'm developing, it turned out the sanest thing to do with OOM conditions was to consider them user errors, and to handle them by dumping memory usage info about processes and slab caches, then halt. It's been very helpful because it turns flaky conditions into rock-solid failures. Too bad this drastic approach isn't appropriate for general use.
Sound interesting. Can you send a patch for the interested and unafraid?
Jörn
Signature · 3 lines
A quarrel is quickly settled when deserted by one party; there is no battle unless there be two. -- Seneca
Quoted 11 lines · Jörn Engel
Quoted 7 lines · Dan Kegel
In the test-and-measurement system I'm developing, it turned out the sanest thing to do with OOM conditions was to consider them user errors, and to handle them by dumping memory usage info about processes and slab caches, then halt. It's been very helpful because it turns flaky conditions into rock-solid failures. Too bad this drastic approach isn't appropriate for general use.
Quoted 2 lines
Sound interesting. Can you send a patch for the interested and unafraid?
This is against 2.4.21 or so.
oom_kill.c +64 -0
@@ -20,9 +20,13 @@#include <linux/swap.h> #include <linux/swapctl.h> #include <linux/timex.h> +#include <asm/uaccess.h> /* #define DEBUG */ +#define CONFIG_OOM_HALT +#ifndef CONFIG_OOM_HALT +/** * int_sqrt - oom_kill.c internal function, rough approximation to sqrt * @x: integer of which to calculate the sqrt
@@ -193,6 +197,62 @@return ; } +#else + +/** +* oom_halt - log out of memory condition, then halt system. +* +* For embedded systems which can't tolerate the chance that +* the oom killer will kill the wrong process, and would rather +* simply log the event in detail and halt. +*/ +static void +oom_halt (void ) +{ +struct task_struct *p; +struct file *file; +int ret; + +printk (KERN_EMERG "oom: Out of memory!\n" ); + +printk (KERN_EMERG "oom: VM and RSS in KB, pid, and mm ptr for each task:\n" ); +read_lock (&tasklist_lock); +for_each_task (p) { +if (p->mm) +printk (KERN_EMERG "oom> vm %5d rss %5d pid %5d mm %p (%s)\n" , + p->mm->total_vm * (PAGE_SIZE /1024 ), + p->mm->rss * (PAGE_SIZE /1024 ), p->pid, p->mm, p->comm); + } +read_unlock (&tasklist_lock); + + file =filp_open ("/proc/slabinfo" ,O_RDONLY ,0 ); +if (IS_ERR (file) || !file->f_op || !file->f_op->read) + goto out; +printk (KERN_EMERG "oom: Contents of /proc/slabinfo:\n" ); +do { +char buf[128 ]; +int pos; +mm_segment_t fs =get_fs (); +/* read one line */ +for (pos =0 ; pos <sizeof (buf); pos++) { +set_fs (KERNEL_DS ); + ret = file->f_op->read (file, buf + pos,1 , &file->f_pos); +set_fs (fs); +if (ret !=1 || buf[pos] =='\n' ) +break ; + } + buf[pos] =0 ; +printk (KERN_EMERG "oom> %s\n" , buf); + }while (ret ==1 ); +/* filp_close(file, NULL); */ +out: +printk (KERN_EMERG "oom: Halting.\n" ); +cli (); +machine_halt (); +} + +#endif +/** * out_of_memory - is the system out of memory? */
@@ -237,7 +297,11 @@/* * Ok, really out of memory. Kill something. */ lastkill = now; +#ifdef CONFIG_OOM_HALT +oom_halt (); +#else oom_kill (); +#endif reset:
first = now;
Signature · 3 lines
Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045
Quoted 7 lines · Dan Kegel
In the test-and-measurement system I'm developing, it turned out the sanest thing to do with OOM conditions was to consider them user errors, and to handle them by dumping memory usage info about processes and slab caches, then halt. It's been very helpful because it turns flaky conditions into rock-solid failures. Too bad this drastic approach isn't appropriate for general use.
Likewise in an HA environment, if you've got a standby node available, we prefer to fail over on an oom condition or (or an oops, for that matter) than to try to continue running in some randomly crippled way. The node in question can then reboot and return to service as a standby.
Ideally, we'd have a notifier that would be triggered for every unanticipated process kill (oom, oops, whatever).
Signature · 1 line
/Jonathan Lundell.
Quoted 5 lines · Dan Kegel
I spent way too long tweaking the OOM killer before I realized it was hopeless.
The fact that incoming network traffic can be what causes the OOM condition makes it Really Hard to decide which app deserves the axe.
This may be a little off topic, but is there a way to manually select this? I can see having a mode where everything stops thrashing for a while, in order to let the admin calmly kill off the offending process, as a useful feature.
It would be useless in an environment where OOM is actually needed (can't wait for a human admin to show up), but cool for those that like to bring their machines back from the edge.
- Chris
Quoted 8 lines · Chris Frey
Quoted 3 lines
The fact that incoming network traffic can be what causes the OOM condition makes it Really Hard to decide which app deserves the axe.
This may be a little off topic, but is there a way to manually select this? I can see having a mode where everything stops thrashing for a while, in order to let the admin calmly kill off the offending process, as a useful feature.
I'd love to be able to select which app _doesn't_ deserve the axe. I.e. not sshd, and then not httpd.
I once ran GCC on a box out there in netland, on a short bit of code, and it was a surprise memory hog due to the usual GCC surprises.
It totally crippled the machine, for 18 hours until I was able to get someone to reboot it. No ssh, no http, no nothing except TCP initial handshakes, and ping. Not good.
When that happens I'd like the VM to notice that my most important tasks (sshd and its subshells) aren't making progress and start killing off other tasks.
The obvious answer is to turn off swap, but I like to have some swap to hold static data that isn't much used, to free up some RAM.
-- Jamie
Quoted 2 lines · Jamie Lokier
I once ran GCC on a box out there in netland, on a short bit of code, and it was a surprise memory hog due to the usual GCC surprises.
Run -ac on remote boxes and turn on no overcommit. Paranoid people also run watchdog drivers set to NOWAYOUT and monitor a list of apps to be sure they are there - if the app is gone, or the watchdog app dies the box will reboot.