Skip to contentSkip to conversation

Andrea VM changes

DISCUSSION Dan Kegel 6 participants 7 messages Aug 31, 2003 – Sep 1, 2003

lore
Share & tools
lore

Dan Kegel

author
#0 lore raw

I spent way too long tweaking the OOM killer before I realized it was hopeless.
The fact that incoming network traffic can be what causes the OOM condition makes it Really Hard to decide which app deserves the axe.

In the test-and-measurement system I'm developing, it turned out the sanest thing to do with OOM conditions was to consider them user errors, and to handle them by dumping memory usage info about processes and slab caches, then halt. It's been very helpful because it turns flaky conditions into rock-solid failures. Too bad this drastic approach isn't appropriate for general use.
- Dan

Signature · 3 lines
Dan Kegel
http://www.kegel.com
http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045
#1 lore raw
Quoted 8 lines · Dan Kegel

In the test-and-measurement system I'm developing, it turned out the sanest thing to do with OOM conditions was to consider them user errors, and to handle them by dumping memory usage info about processes and slab caches, then halt. It's been very helpful because it turns flaky conditions into rock-solid failures. Too bad this drastic approach isn't appropriate for general use.

Sound interesting. Can you send a patch for the interested and unafraid?

Jörn

Signature · 3 lines
A quarrel is quickly settled when deserted by one party; there is
no battle unless there be two.
-- Seneca
author
#2 lore raw
Quoted 11 lines · Jörn Engel
Quoted 7 lines · Dan Kegel

In the test-and-measurement system I'm developing, it turned out the sanest thing to do with OOM conditions was to consider them user errors, and to handle them by dumping memory usage info about processes and slab caches, then halt. It's been very helpful because it turns flaky conditions into rock-solid failures. Too bad this drastic approach isn't appropriate for general use.

Quoted 2 lines

Sound interesting. Can you send a patch for the interested and unafraid?

This is against 2.4.21 or so.

oom_kill.c +64 -0
@@ -20,9 +20,13 @@
 #include <linux/swap.h>
 #include <linux/swapctl.h>
 #include <linux/timex.h>
+#include <asm/uaccess.h>
 
 /* #define DEBUG */
 
+#define CONFIG_OOM_HALT
+#ifndef CONFIG_OOM_HALT
+
 /**
  * int_sqrt - oom_kill.c internal function, rough approximation to sqrt
  * @x: integer of which to calculate the sqrt
@@ -193,6 +197,62 @@
 	return;
 }
 
+#else
+
+/**
+ * oom_halt - log out of memory condition, then halt system.
+ *
+ * For embedded systems which can't tolerate the chance that
+ * the oom killer will kill the wrong process, and would rather
+ * simply log the event in detail and halt.
+ */
+static void
+oom_halt(void)
+{
+	struct task_struct *p;
+	struct file *file;
+	int ret;
+
+	printk(KERN_EMERG "oom: Out of memory!\n");
+
+	printk(KERN_EMERG "oom: VM and RSS in KB, pid, and mm ptr for each task:\n");
+	read_lock(&tasklist_lock);
+	for_each_task(p) {
+		if (p->mm)
+			printk(KERN_EMERG "oom> vm %5d rss %5d pid %5d mm %p (%s)\n",
+			   p->mm->total_vm * (PAGE_SIZE / 1024),
+			   p->mm->rss * (PAGE_SIZE / 1024), p->pid, p->mm, p->comm);
+	}
+	read_unlock(&tasklist_lock);
+
+	file = filp_open("/proc/slabinfo", O_RDONLY, 0);
+	if (IS_ERR(file) || !file->f_op || !file->f_op->read)
+		goto out;
+	printk(KERN_EMERG "oom: Contents of /proc/slabinfo:\n");
+	do {
+		char buf[128];
+		int pos;
+		mm_segment_t fs = get_fs();
+		/* read one line */
+		for (pos = 0; pos < sizeof (buf); pos++) {
+			set_fs(KERNEL_DS);
+			ret = file->f_op->read(file, buf + pos, 1, &file->f_pos);
+			set_fs(fs);
+			if (ret != 1 || buf[pos] == '\n')
+				break;
+		}
+		buf[pos] = 0;
+		printk(KERN_EMERG "oom> %s\n", buf);
+	} while (ret == 1);
+	/* filp_close(file, NULL); */
+out:
+	printk(KERN_EMERG "oom: Halting.\n");
+	cli();
+	machine_halt();
+}
+
+#endif
+
 /**
  * out_of_memory - is the system out of memory?
  */
@@ -237,7 +297,11 @@
 	/*
 	 * Ok, really out of memory. Kill something.
 	 */
 	lastkill = now;
+#ifdef CONFIG_OOM_HALT
+	oom_halt();
+#else
 	oom_kill();
+#endif
 
 reset:

first = now;

Signature · 3 lines
Dan Kegel
http://www.kegel.com
http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045
#3 lore raw
In reply to Dan Kegel
Quoted 7 lines · Dan Kegel

In the test-and-measurement system I'm developing, it turned out the sanest thing to do with OOM conditions was to consider them user errors, and to handle them by dumping memory usage info about processes and slab caches, then halt. It's been very helpful because it turns flaky conditions into rock-solid failures. Too bad this drastic approach isn't appropriate for general use.

Likewise in an HA environment, if you've got a standby node available, we prefer to fail over on an oom condition or (or an oops, for that matter) than to try to continue running in some randomly crippled way. The node in question can then reboot and return to service as a standby.

Ideally, we'd have a notifier that would be triggered for every unanticipated process kill (oom, oops, whatever).

Signature · 1 line
/Jonathan Lundell.
#4 lore raw
In reply to Dan Kegel
Quoted 5 lines · Dan Kegel

I spent way too long tweaking the OOM killer before I realized it was hopeless.
The fact that incoming network traffic can be what causes the OOM condition makes it Really Hard to decide which app deserves the axe.

This may be a little off topic, but is there a way to manually select this? I can see having a mode where everything stops thrashing for a while, in order to let the admin calmly kill off the offending process, as a useful feature.

It would be useless in an environment where OOM is actually needed (can't wait for a human admin to show up), but cool for those that like to bring their machines back from the edge.

- Chris

#5 lore raw
Quoted 8 lines · Chris Frey
Quoted 3 lines

The fact that incoming network traffic can be what causes the OOM condition makes it Really Hard to decide which app deserves the axe.

This may be a little off topic, but is there a way to manually select this? I can see having a mode where everything stops thrashing for a while, in order to let the admin calmly kill off the offending process, as a useful feature.

I'd love to be able to select which app _doesn't_ deserve the axe. I.e. not sshd, and then not httpd.

I once ran GCC on a box out there in netland, on a short bit of code, and it was a surprise memory hog due to the usual GCC surprises.

It totally crippled the machine, for 18 hours until I was able to get someone to reboot it. No ssh, no http, no nothing except TCP initial handshakes, and ping. Not good.

When that happens I'd like the VM to notice that my most important tasks (sshd and its subshells) aren't making progress and start killing off other tasks.

The obvious answer is to turn off swap, but I like to have some swap to hold static data that isn't much used, to free up some RAM.

-- Jamie

#6 lore raw
Quoted 2 lines · Jamie Lokier

I once ran GCC on a box out there in netland, on a short bit of code, and it was a surprise memory hog due to the usual GCC surprises.

Run -ac on remote boxes and turn on no overcommit. Paranoid people also run watchdog drivers set to NOWAYOUT and monitor a list of apps to be sure they are there - if the app is gone, or the watchdog app dies the box will reboot.