Skip to contentSkip to conversation

oom killer in 2.4.23

DISCUSSION Kristian Peters 5 participants 10 messages Dec 5, 2003 – Dec 9, 2003 3 messages not in the archive

Subsystems (5)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 5 lines · Andrea Arcangeli

Marcelo asked me to to make it configurable at runtime so you could go in the deadlock prone stautus of 2.4.22 on demand, but I'm not going to add more features to 2.4 today unless they're blocker bugs (even if that would be simple to implement), actually it's not even my choice so don't ask me for that sorry.

Andrea, your vm does not work correctly in any cases.

It's so simple. I've tried to fill up my memory with that crappy khexedit that comes with kde2. You'll see how my memory fills with the contents of the whole file I load. When I have started 2 or 3 instances of khexedit my memory was nearly completely filled. Than I tried to start another khexedit (with a file that should nearly fit into memory), and the pain began.

See:

Dec  5 13:33:52 adlib kernel: __alloc_pages: 2-order allocation failed (gfp=0x1f0/0)
Dec  5 13:33:52 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:33:59 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0)
Dec  5 13:33:59 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:34:00 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Dec  5 13:34:01 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:34:02 adlib last message repeated 3 times

-------> kernel killed wmfire without saying

Dec  5 13:34:02 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0)
Dec  5 13:34:03 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0)
Dec  5 13:34:18 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:34:18 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:34:22 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0)
Dec  5 13:34:22 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:34:28 adlib last message repeated 3 times

-------> kernel killed xosview without mentioning

Dec  5 13:34:29 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0)
Dec  5 13:34:29 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:34:32 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0)
Dec  5 13:34:41 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Dec  5 13:34:41 adlib kernel: VM: killing process khexedit

-------> that was intended

Ok. That still is acceptable but when I tried to start mozilla, it got even worse:

Dec  5 13:37:26 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:37:27 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Dec  5 13:37:27 adlib kernel: VM: killing process mozilla-bin
Dec  5 13:37:27 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0)
Dec  5 13:37:27 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:37:28 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:37:30 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0)
Dec  5 13:37:30 adlib last message repeated 2 times
Dec  5 13:37:30 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:37:40 adlib last message repeated 3 times
Dec  5 13:37:56 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Dec  5 13:37:56 adlib kernel: VM: killing process mozilla-bin

-------> that was intended too
-------> but not the killing of another xosview and aterm

Dec  5 13:37:57 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:37:57 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0)
Dec  5 13:37:58 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:40:32 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Dec  5 13:40:32 adlib kernel: VM: killing process XFree86

-------> ouch ...

Ok. Could you please describe what your vm really does here in my specific case ? Rick's old vm worked better. It'd have killed the task that had last allocated memory.

PS: If you need more details it should be no problem to do this again.

*Kristian

   _o)
   /\\
  _\_V
#4 lore raw

Thus spake Kristian Peters ([email protected]):

Quoted 17 lines
Quoted 5 lines · Andrea Arcangeli

Marcelo asked me to to make it configurable at runtime so you could go in the deadlock prone stautus of 2.4.22 on demand, but I'm not going to add more features to 2.4 today unless they're blocker bugs (even if that would be simple to implement), actually it's not even my choice so don't ask me for that sorry.

Andrea, your vm does not work correctly in any cases.

It's so simple. I've tried to fill up my memory with that crappy khexedit that comes with kde2. You'll see how my memory fills with the contents of the whole file I load. When I have started 2 or 3 instances of khexedit my memory was nearly completely filled. Than I tried to start another khexedit (with a file that should nearly fit into memory), and the pain began.

See:

Dec  5 13:33:52 adlib kernel: __alloc_pages: 2-order allocation failed (gfp=0x1f0/0)
Dec  5 13:33:52 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:33:59 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0)
Dec  5 13:33:59 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)

<snip>

Quoted 5 lines
Dec  5 13:37:57 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0)
Dec  5 13:37:58 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
Dec  5 13:40:32 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Dec  5 13:40:32 adlib kernel: VM: killing process XFree86
Quoted 9 lines

-------> ouch ...

Ok. Could you please describe what your vm really does here in my specific case ? Rick's old vm worked better. It'd have killed the task that had last allocated memory.

PS: If you need more details it should be no problem to do this again.

I'm see'ing similar except it's killing random apps such as CRON, named and some others.

How far do I have to roll back to get the previous oomkiller? Trying to roll 2.4.23 out.

:wq!
---------------------------------------------------------------------------
Robert L. Harris | GPG Key ID: E344DA3B

                                         @ x-hkp://pgp.mit.edu

DISCLAIMER:
These are MY OPINIONS ALONE. I speak for no-one else.

Life is not a destination, it's a journey.
Microsoft produces 15 car pileups on the highway.
Don't stop traffic to stand and gawk at the tragedy.

#5 lore raw
In reply to Kristian Peters
Quoted 8 lines · Kristian Peters
Quoted 5 lines · Andrea Arcangeli

Marcelo asked me to to make it configurable at runtime so you could go in the deadlock prone stautus of 2.4.22 on demand, but I'm not going to add more features to 2.4 today unless they're blocker bugs (even if that would be simple to implement), actually it's not even my choice so don't ask me for that sorry.

Andrea, your vm does not work correctly in any cases.

what you're complaining is the 'selection of the task to be killed'. That's not solvable. the kernel can't read your brain period. Only if the kernel could read the brain of the adminstrator then you would be happy, there is no way the kernel can know which is the task you really want to have killed first.

Quoted 6 lines

It's so simple. I've tried to fill up my memory with that crappy khexedit that comes with kde2. You'll see how my memory fills with the contents of the whole file I load. When I have started 2 or 3 instances of khexedit my memory was nearly completely filled. Than I tried to start another khexedit (with a file that should nearly fit into memory), and the pain began.

The kernel can't know what is a pain for you and what is pain for other people.

Measuring the page fault rate seems to get the closest heuristic that may not be a pain for most people, all current oom killers are a pain for somebody, desktop users where pretty much fine with 2.4.22, server users had pain with 2.4.22 and should have less pain with 2.4.23. There is no way to make everybody happy in 2.4.

Quoted 1 line

Rick's old vm worked better. It'd have killed the task that had last allocated memory.

it was the biggest one that's why. the old omm killer gets your desktop scenario always correctly, true, (as far as your biggest task doesn't get stuck on nfs etc..)

Quoted 1 line

PS: If you need more details it should be no problem to do this again.

I'm aware about those issues, that's a feature not a bug.

#6 lore raw
Quoted 5 lines · Andrea Arcangeli

what you're complaining is the 'selection of the task to be killed'. That's not solvable. the kernel can't read your brain period. Only if the kernel could read the brain of the adminstrator then you would be happy, there is no way the kernel can know which is the task you really want to have killed first.

I agree. On a server the most likely application to be killed would be the service with the most pages in memory. And those services tend to be the important ones.

However, for a simple desktop-linux that statistical approach seems to be wrong. Your vm has even killed /sbin/getty sometimes, so that I can't login via a simple console.

Re-enabling the oom-killer gives a good result for me:

Dec  6 09:46:19 adlib kernel: Out of Memory: Killed process 643 (khexedit).
Dec  6 09:48:42 adlib kernel: Out of Memory: Killed process 645 (khexedit).

What I complain is that your vm kills some processes without mentioning in the logs. How can I determine what processes the kernel has killed ?

I hope that fairly simple patch does things right for all people that want the old behaviour. It's already a year ago I last hacked on the kernel.

include/linux/sched.h Sat Dec 6 09:57:04 2003 renamed from include/linux/sched.h Fri Nov 28 19:26:21 2003 +1 -0
diff -rauN linux-2.4.23/include/linux/sched.h linux-2.4.23-kp1/include/linux/sched.h
@@ -429,6 +429,7 @@
 #define PF_DUMPCORE    0x00000200      /* dumped core */
 #define PF_SIGNALED    0x00000400      /* killed by a signal */
 #define PF_MEMALLOC    0x00000800      /* Allocating memory */
+#define PF_MEMDIE      0x00001000      /* Killed for out-of-memory */
 #define PF_FREE_PAGES  0x00002000      /* per process page freeing */
 #define PF_NOIO                0x00004000      /* avoid generating further I/O */
 
mm/oom_kill.c Fri Dec 5 20:31:39 2003 renamed from mm/oom_kill.c Fri Nov 28 19:26:21 2003 +0 -4
diff -rauN linux-2.4.23/mm/oom_kill.c linux-2.4.23-kp1/mm/oom_kill.c
@@ -21,8 +21,6 @@
 #include <linux/swapctl.h>
 #include <linux/timex.h>
 
-#if 0          /* Nothing in this file is used */
-
 /* #define DEBUG */
 
 /**
@@ -257,5 +255,3 @@
        first = now;
        count = 0;
 }
-
-#endif /* Unused file */
mm/vmscan.c Sat Dec 6 10:21:55 2003 renamed from mm/vmscan.c Fri Nov 28 19:26:21 2003 +1 -7
diff -rauN linux-2.4.23/mm/vmscan.c linux-2.4.23-kp1/mm/vmscan.c
@@ -649,13 +649,7 @@
                                failed_swapout = !swap_out(classzone);
                } while (--tries);
 
-       if (likely(current->pid != 1))
-               break;
-       if (!check_classzone_need_balance(classzone))
-               break;
-
-       __set_current_state(TASK_RUNNING);
-       yield();
+       out_of_memory();
        }
 
        return 0;

*Kristian

     _o)
     /\\
    _\_V
#7 lore raw
Quoted 7 lines · Kristian Peters
Quoted 5 lines · Andrea Arcangeli

what you're complaining is the 'selection of the task to be killed'. That's not solvable. the kernel can't read your brain period. Only if the kernel could read the brain of the adminstrator then you would be happy, there is no way the kernel can know which is the task you really want to have killed first.

Quoted 5 lines

I agree. On a server the most likely application to be killed would be the service with the most pages in memory. And those services tend to be the important ones.

However, for a simple desktop-linux that statistical approach seems to be wrong. Your vm has even killed /sbin/getty sometimes, so that I can't login via a simple console.

I see this problem, for most desktop users the biggest task plus some other random bit like the nicelevel and length of runtime etc.. actually result in a reasonable estimate of the best task to kill.

Quoted 8 lines

Re-enabling the oom-killer gives a good result for me:

Dec  6 09:46:19 adlib kernel: Out of Memory: Killed process 643 (khexedit).
Dec  6 09:48:42 adlib kernel: Out of Memory: Killed process 645 (khexedit).

What I complain is that your vm kills some processes without mentioning in the logs. How can I determine what processes the kernel has killed ?

no this is is not the case. It is always mentioned in the logs. Grep for VM and you'll find all of them, no difference from before. I agree it would be very bad if the admin couldn't know what was killed exactly to restart everything properly (with the bad app under rlimit the next time ;)

Quoted 41 lines

I hope that fairly simple patch does things right for all people that want the old behaviour. It's already a year ago I last hacked on the kernel.

diff -rauN linux-2.4.23/include/linux/sched.h linux-2.4.23-kp1/include/linux/sched.h

include/linux/sched.h Sat Dec 6 09:57:04 2003 renamed from include/linux/sched.h Fri Nov 28 19:26:21 2003 +1 -0
@@ -429,6 +429,7 @@
 #define PF_DUMPCORE    0x00000200      /* dumped core */
 #define PF_SIGNALED    0x00000400      /* killed by a signal */
 #define PF_MEMALLOC    0x00000800      /* Allocating memory */
+#define PF_MEMDIE      0x00001000      /* Killed for out-of-memory */
 #define PF_FREE_PAGES  0x00002000      /* per process page freeing */
 #define PF_NOIO                0x00004000      /* avoid generating further I/O */
 

diff -rauN linux-2.4.23/mm/oom_kill.c linux-2.4.23-kp1/mm/oom_kill.c

mm/oom_kill.c Fri Dec 5 20:31:39 2003 renamed from mm/oom_kill.c Fri Nov 28 19:26:21 2003 +0 -4
@@ -21,8 +21,6 @@
 #include <linux/swapctl.h>
 #include <linux/timex.h>
 
-#if 0          /* Nothing in this file is used */
-
 /* #define DEBUG */
 
 /**
@@ -257,5 +255,3 @@
        first = now;
        count = 0;
 }
-
-#endif /* Unused file */

diff -rauN linux-2.4.23/mm/vmscan.c linux-2.4.23-kp1/mm/vmscan.c

mm/vmscan.c Sat Dec 6 10:21:55 2003 renamed from mm/vmscan.c Fri Nov 28 19:26:21 2003 +0 -1
@@ -649,13 +649,7 @@
                                failed_swapout = !swap_out(classzone);
                } while (--tries);
 
-       if (likely(current->pid != 1))

mm/vmscan.c Sat Dec 6 10:21:55 2003 line 655

Quoted 10 lines
mm/vmscan.c Sat Dec 6 10:21:55 2003 renamed from mm/vmscan.c Fri Nov 28 19:26:21 2003 +1 -6
-               break;
-       if (!check_classzone_need_balance(classzone))
-               break;
-
-       __set_current_state(TASK_RUNNING);
-       yield();
+       out_of_memory();
        }
 
        return 0;

this should go back to the 2.4.22 deadlock prone oom killer yes. you can also leave the yield() in the main loop (after out_of_memory()) (__set_current_state(TASK_RUNNING) isn't needed these days just yield is safe enough).

#8 lore raw
Quoted 8 lines · Andrea Arcangeli
Quoted 7 lines · Kristian Peters
Quoted 5 lines · Andrea Arcangeli

what you're complaining is the 'selection of the task to be killed'. That's not solvable. the kernel can't read your brain period. Only if the kernel could read the brain of the adminstrator then you would be happy, there is no way the kernel can know which is the task you really want to have killed first.

Quoted 6 lines
Quoted 5 lines

I agree. On a server the most likely application to be killed would be the service with the most pages in memory. And those services tend to be the important ones.

However, for a simple desktop-linux that statistical approach seems to be wrong. Your vm has even killed /sbin/getty sometimes, so that I can't login via a simple console.

Not true! Killing a getty or agetty or any other login program cannot prevent a login! Init will just start another one if there is enough RAM to fork()/exec(). If there isn't, it has nothing to do with killing the getties and everything to do with being completely out of RAM. In that case, you need to enable quotas and not let fork-bombs or other deliberate crashers be executed by root.

Script started on Tue Dec 9 09:42:06 2003

# ps
  PID TTY STAT  TIME COMMAND
  478   3 S    0:00 /sbin/agetty 38400 tty3
  479   4 S    0:00 /sbin/agetty 38400 tty4
  480   5 S    0:00 /sbin/agetty 38400 tty5
  481   6 S    0:00 /sbin/agetty 38400 tty6
 4404   1 S    0:00 -bash
 6102   2 S    0:00 -bash
 8772   1 S    0:00 pine
 8778   2 S    0:00 script
 8779   2 S    0:00 script
 8780  p0 S    0:00 bash -i
 8784  p0 R    0:00 ps
# killall -9 agetty
# ps
  PID TTY STAT  TIME COMMAND
 4404   1 S    0:00 -bash
 6102   2 S    0:00 -bash
 8772   1 S    0:00 pine
 8778   2 S    0:00 script
 8779   2 S    0:00 script
 8780  p0 S    0:00 bash -i
 8787   3 S    0:00 /sbin/agetty 38400 tty3
 8788   4 S    0:00 /sbin/agetty 38400 tty4
 8789   5 S    0:00 /sbin/agetty 38400 tty5
 8790   6 S    0:00 /sbin/agetty 38400 tty6
 8791  p0 R    0:00 ps
# exit
exit

Script done on Tue Dec 9 09:42:41 2003

The above shows 4 getty's being killed. They are immediately restarted by init. These getties become new shell-tasks in the following order: exec /sbin/login, exec /bin/bash. There are no additional forks so, in principle, you have nearly all the resources needed for a shell as soon as a getty is started.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (797.90 BogoMips).
Note 96.31% of all statistics are fiction.

#9 lore raw
Quoted 12 lines · Richard B. Johnson
Quoted 10 lines · Andrea Arcangeli
Quoted 9 lines · Kristian Peters
Quoted 5 lines · Andrea Arcangeli

what you're complaining is the 'selection of the task to be killed'. That's not solvable. the kernel can't read your brain period. Only if the kernel could read the brain of the adminstrator then you would be happy, there is no way the kernel can know which is the task you really want to have killed first.

I agree. On a server the most likely application to be killed would be the service with the most pages in memory. And those services tend to be the important ones.

Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

However, for a simple desktop-linux that statistical approach seems to be wrong. Your vm has even killed /sbin/getty sometimes, so that I can't login via a simple console.

Not true! Killing a getty or agetty or any other login program cannot prevent a login! Init will just start another

killing getty is a very very lucky scenario indeed. the way I read it was that agetty can hardly be a mem eater, and in turn the same way agetty was killed, it could have been ssh or X to be killed. That's true, but the same issues on the desktop will happen if you have no swap if you enable the old oom killer, the vm will go nuts even if you don't run oom, and there will be all other sort of troubles mentioned a few times already.

#10 lore raw
Quoted 7 lines · Andrea Arcangeli

killing getty is a very very lucky scenario indeed. the way I read it was that agetty can hardly be a mem eater, and in turn the same way agetty was killed, it could have been ssh or X to be killed. That's true, but the same issues on the desktop will happen if you have no swap if you enable the old oom killer, the vm will go nuts even if you don't run oom, and there will be all other sort of troubles mentioned a few times already.

I think getty was respawned but the console was screwed somehow. The screen was all black as with broken framebuffer and I couldn't type anything (even not the "blind" way.). Only X was there.

Thanks for all your answers.

*Kristian

#11 lore raw
In reply to Kristian Peters
Quoted 4 lines · Kristian Peters
Dec  5 13:34:41 adlib kernel: VM: killing process khexedit
Dec  5 13:37:27 adlib kernel: VM: killing process mozilla-bin
Dec  5 13:37:56 adlib kernel: VM: killing process mozilla-bin
Dec  5 13:40:32 adlib kernel: VM: killing process XFree86

This is with 2.4.23?

Why is the VM killing anything if the oom-killer is removed?

#12 lore raw
Quoted 10 lines · Mike Fedyk
Quoted 4 lines · Kristian Peters
Dec  5 13:34:41 adlib kernel: VM: killing process khexedit
Dec  5 13:37:27 adlib kernel: VM: killing process mozilla-bin
Dec  5 13:37:56 adlib kernel: VM: killing process mozilla-bin
Dec  5 13:40:32 adlib kernel: VM: killing process XFree86

This is with 2.4.23?

Why is the VM killing anything if the oom-killer is removed?

the 2.4.23 kernel will kill the task that triggered the oom condition, it has to kill something of course, and the task that triggered the oom during the page fault is the only one we can kill synchronously easily, in turn guaranteeing that the machine won't deadlock in omm.

the oom killer normally is meant as the heuristc that chooses a special task to kill, instead of the one that triggered the oom condition. But choosing a different task and not the one that triggered the oom in the page fault, isn't math safe w.r.t deadlocks.