oom killer in 2.4.23
Subsystems (5)
Quoted 5 lines · Andrea Arcangeli
Marcelo asked me to to make it configurable at runtime so you could go in the deadlock prone stautus of 2.4.22 on demand, but I'm not going to add more features to 2.4 today unless they're blocker bugs (even if that would be simple to implement), actually it's not even my choice so don't ask me for that sorry.
Andrea, your vm does not work correctly in any cases.
It's so simple. I've tried to fill up my memory with that crappy khexedit that comes with kde2. You'll see how my memory fills with the contents of the whole file I load. When I have started 2 or 3 instances of khexedit my memory was nearly completely filled. Than I tried to start another khexedit (with a file that should nearly fit into memory), and the pain began.
See:
Dec 5 13:33:52 adlib kernel: __alloc_pages: 2-order allocation failed (gfp=0x1f0/0) Dec 5 13:33:52 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:33:59 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) Dec 5 13:33:59 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:34:00 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 5 13:34:01 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:34:02 adlib last message repeated 3 times
-------> kernel killed wmfire without saying
Dec 5 13:34:02 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) Dec 5 13:34:03 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) Dec 5 13:34:18 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:34:18 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:34:22 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) Dec 5 13:34:22 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:34:28 adlib last message repeated 3 times
-------> kernel killed xosview without mentioning
Dec 5 13:34:29 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) Dec 5 13:34:29 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:34:32 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) Dec 5 13:34:41 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 5 13:34:41 adlib kernel: VM: killing process khexedit
-------> that was intended
Ok. That still is acceptable but when I tried to start mozilla, it got even worse:
Dec 5 13:37:26 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:37:27 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 5 13:37:27 adlib kernel: VM: killing process mozilla-bin Dec 5 13:37:27 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) Dec 5 13:37:27 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:37:28 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:37:30 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) Dec 5 13:37:30 adlib last message repeated 2 times Dec 5 13:37:30 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:37:40 adlib last message repeated 3 times Dec 5 13:37:56 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 5 13:37:56 adlib kernel: VM: killing process mozilla-bin
-------> that was intended too
-------> but not the killing of another xosview and aterm
Dec 5 13:37:57 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:37:57 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) Dec 5 13:37:58 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:40:32 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 5 13:40:32 adlib kernel: VM: killing process XFree86
-------> ouch ...
Ok. Could you please describe what your vm really does here in my specific case ? Rick's old vm worked better. It'd have killed the task that had last allocated memory.
PS: If you need more details it should be no problem to do this again.
*Kristian
_o) /\\ _\_V
Thus spake Kristian Peters ([email protected]):
Quoted 17 lines
Quoted 5 lines · Andrea Arcangeli
Marcelo asked me to to make it configurable at runtime so you could go in the deadlock prone stautus of 2.4.22 on demand, but I'm not going to add more features to 2.4 today unless they're blocker bugs (even if that would be simple to implement), actually it's not even my choice so don't ask me for that sorry.
Andrea, your vm does not work correctly in any cases.
It's so simple. I've tried to fill up my memory with that crappy khexedit that comes with kde2. You'll see how my memory fills with the contents of the whole file I load. When I have started 2 or 3 instances of khexedit my memory was nearly completely filled. Than I tried to start another khexedit (with a file that should nearly fit into memory), and the pain began.
See:
Dec 5 13:33:52 adlib kernel: __alloc_pages: 2-order allocation failed (gfp=0x1f0/0) Dec 5 13:33:52 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:33:59 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) Dec 5 13:33:59 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
<snip>
Quoted 5 lines
Dec 5 13:37:57 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0xf0/0) Dec 5 13:37:58 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) Dec 5 13:40:32 adlib kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Dec 5 13:40:32 adlib kernel: VM: killing process XFree86
Quoted 9 lines
-------> ouch ...
Ok. Could you please describe what your vm really does here in my specific case ? Rick's old vm worked better. It'd have killed the task that had last allocated memory.
PS: If you need more details it should be no problem to do this again.
I'm see'ing similar except it's killing random apps such as CRON, named and some others.
How far do I have to roll back to get the previous oomkiller? Trying to roll 2.4.23 out.
:wq!
---------------------------------------------------------------------------
Robert L. Harris | GPG Key ID: E344DA3B
@ x-hkp://pgp.mit.edu
DISCLAIMER:
These are MY OPINIONS ALONE. I speak for no-one else.
Life is not a destination, it's a journey.
Microsoft produces 15 car pileups on the highway.
Don't stop traffic to stand and gawk at the tragedy.
Quoted 8 lines · Kristian Peters
Quoted 5 lines · Andrea Arcangeli
Marcelo asked me to to make it configurable at runtime so you could go in the deadlock prone stautus of 2.4.22 on demand, but I'm not going to add more features to 2.4 today unless they're blocker bugs (even if that would be simple to implement), actually it's not even my choice so don't ask me for that sorry.
Andrea, your vm does not work correctly in any cases.
what you're complaining is the 'selection of the task to be killed'. That's not solvable. the kernel can't read your brain period. Only if the kernel could read the brain of the adminstrator then you would be happy, there is no way the kernel can know which is the task you really want to have killed first.
Quoted 6 lines
It's so simple. I've tried to fill up my memory with that crappy khexedit that comes with kde2. You'll see how my memory fills with the contents of the whole file I load. When I have started 2 or 3 instances of khexedit my memory was nearly completely filled. Than I tried to start another khexedit (with a file that should nearly fit into memory), and the pain began.
The kernel can't know what is a pain for you and what is pain for other people.
Measuring the page fault rate seems to get the closest heuristic that may not be a pain for most people, all current oom killers are a pain for somebody, desktop users where pretty much fine with 2.4.22, server users had pain with 2.4.22 and should have less pain with 2.4.23. There is no way to make everybody happy in 2.4.
Quoted 1 line
Rick's old vm worked better. It'd have killed the task that had last allocated memory.
it was the biggest one that's why. the old omm killer gets your desktop scenario always correctly, true, (as far as your biggest task doesn't get stuck on nfs etc..)
Quoted 1 line
PS: If you need more details it should be no problem to do this again.
I'm aware about those issues, that's a feature not a bug.
Quoted 4 lines · Kristian Peters
Dec 5 13:34:41 adlib kernel: VM: killing process khexedit Dec 5 13:37:27 adlib kernel: VM: killing process mozilla-bin Dec 5 13:37:56 adlib kernel: VM: killing process mozilla-bin Dec 5 13:40:32 adlib kernel: VM: killing process XFree86
This is with 2.4.23?
Why is the VM killing anything if the oom-killer is removed?
Quoted 10 lines · Mike Fedyk
Quoted 4 lines · Kristian Peters
Dec 5 13:34:41 adlib kernel: VM: killing process khexedit Dec 5 13:37:27 adlib kernel: VM: killing process mozilla-bin Dec 5 13:37:56 adlib kernel: VM: killing process mozilla-bin Dec 5 13:40:32 adlib kernel: VM: killing process XFree86This is with 2.4.23?
Why is the VM killing anything if the oom-killer is removed?
the 2.4.23 kernel will kill the task that triggered the oom condition, it has to kill something of course, and the task that triggered the oom during the page fault is the only one we can kill synchronously easily, in turn guaranteeing that the machine won't deadlock in omm.
the oom killer normally is meant as the heuristc that chooses a special task to kill, instead of the one that triggered the oom condition. But choosing a different task and not the one that triggered the oom in the page fault, isn't math safe w.r.t deadlocks.
Quoted 5 lines · Andrea Arcangeli
what you're complaining is the 'selection of the task to be killed'. That's not solvable. the kernel can't read your brain period. Only if the kernel could read the brain of the adminstrator then you would be happy, there is no way the kernel can know which is the task you really want to have killed first.
I agree. On a server the most likely application to be killed would be the service with the most pages in memory. And those services tend to be the important ones.
However, for a simple desktop-linux that statistical approach seems to be wrong. Your vm has even killed /sbin/getty sometimes, so that I can't login via a simple console.
Re-enabling the oom-killer gives a good result for me:
Dec 6 09:46:19 adlib kernel: Out of Memory: Killed process 643 (khexedit). Dec 6 09:48:42 adlib kernel: Out of Memory: Killed process 645 (khexedit).
What I complain is that your vm kills some processes without mentioning in the logs. How can I determine what processes the kernel has killed ?
I hope that fairly simple patch does things right for all people that want the old behaviour. It's already a year ago I last hacked on the kernel.
include/linux/sched.h Sat Dec 6 09:57:04 2003 renamed from include/linux/sched.h Fri Nov 28 19:26:21 2003 +1 -0
@@ -429,6 +429,7 @@ #define PF_DUMPCORE 0x00000200 /* dumped core */ #define PF_SIGNALED 0x00000400 /* killed by a signal */ #define PF_MEMALLOC 0x00000800 /* Allocating memory */ +#define PF_MEMDIE 0x00001000 /* Killed for out-of-memory */ #define PF_FREE_PAGES 0x00002000 /* per process page freeing */ #define PF_NOIO 0x00004000 /* avoid generating further I/O */
mm/oom_kill.c Fri Dec 5 20:31:39 2003 renamed from mm/oom_kill.c Fri Nov 28 19:26:21 2003 +0 -4
@@ -21,8 +21,6 @@ #include <linux/swapctl.h> #include <linux/timex.h> -#if 0 /* Nothing in this file is used */ - /* #define DEBUG */ /**
@@ -257,5 +255,3 @@ first = now; count = 0; } - -#endif /* Unused file */
mm/vmscan.c Sat Dec 6 10:21:55 2003 renamed from mm/vmscan.c Fri Nov 28 19:26:21 2003 +1 -7
@@ -649,13 +649,7 @@ failed_swapout = !swap_out(classzone); } while (--tries); - if (likely(current->pid != 1)) - break; - if (!check_classzone_need_balance(classzone)) - break; - - __set_current_state(TASK_RUNNING); - yield(); + out_of_memory(); } return 0;
*Kristian
_o)
/\\
_\_VQuoted 7 lines · Kristian Peters
Quoted 5 lines · Andrea Arcangeli
what you're complaining is the 'selection of the task to be killed'. That's not solvable. the kernel can't read your brain period. Only if the kernel could read the brain of the adminstrator then you would be happy, there is no way the kernel can know which is the task you really want to have killed first.
Quoted 5 lines
I agree. On a server the most likely application to be killed would be the service with the most pages in memory. And those services tend to be the important ones.
However, for a simple desktop-linux that statistical approach seems to be wrong. Your vm has even killed
/sbin/gettysometimes, so that I can't login via a simple console.
I see this problem, for most desktop users the biggest task plus some other random bit like the nicelevel and length of runtime etc.. actually result in a reasonable estimate of the best task to kill.
Quoted 8 lines
Re-enabling the oom-killer gives a good result for me:
Dec 6 09:46:19 adlib kernel: Out of Memory: Killed process 643 (khexedit). Dec 6 09:48:42 adlib kernel: Out of Memory: Killed process 645 (khexedit).What I complain is that your vm kills some processes without mentioning in the logs. How can I determine what processes the kernel has killed ?
no this is is not the case. It is always mentioned in the logs. Grep for VM and you'll find all of them, no difference from before. I agree it would be very bad if the admin couldn't know what was killed exactly to restart everything properly (with the bad app under rlimit the next time ;)
Quoted 41 lines
I hope that fairly simple patch does things right for all people that want the old behaviour. It's already a year ago I last hacked on the kernel.
diff -rauN
linux-2.4.23/include/linux/sched.hlinux-2.4.23-kp1/include/linux/sched.hinclude/linux/sched.h Sat Dec 6 09:57:04 2003 renamed from include/linux/sched.h Fri Nov 28 19:26:21 2003 +1
-0@@ -429,6 +429,7 @@ #define PF_DUMPCORE 0x00000200 /* dumped core */ #define PF_SIGNALED 0x00000400 /* killed by a signal */ #define PF_MEMALLOC 0x00000800 /* Allocating memory */ +#define PF_MEMDIE 0x00001000 /* Killed for out-of-memory */ #define PF_FREE_PAGES 0x00002000 /* per process page freeing */ #define PF_NOIO 0x00004000 /* avoid generating further I/O */diff -rauN
linux-2.4.23/mm/oom_kill.clinux-2.4.23-kp1/mm/oom_kill.cmm/oom_kill.c Fri Dec 5 20:31:39 2003 renamed from mm/oom_kill.c Fri Nov 28 19:26:21 2003 +0
-4@@ -21,8 +21,6 @@ #include <linux/swapctl.h> #include <linux/timex.h> -#if 0 /* Nothing in this file is used */ - /* #define DEBUG */ /**@@ -257,5 +255,3 @@ first = now; count = 0; } - -#endif /* Unused file */diff -rauN
linux-2.4.23/mm/vmscan.clinux-2.4.23-kp1/mm/vmscan.cmm/vmscan.c Sat Dec 6 10:21:55 2003 renamed from mm/vmscan.c Fri Nov 28 19:26:21 2003 +0
-1@@ -649,13 +649,7 @@ failed_swapout = !swap_out(classzone); } while (--tries); - if (likely(current->pid != 1))
mm/vmscan.c Sat Dec 6 10:21:55 2003 line 655
Quoted 10 lines
mm/vmscan.c Sat Dec 6 10:21:55 2003 renamed from mm/vmscan.c Fri Nov 28 19:26:21 2003 +1
-6- break; - if (!check_classzone_need_balance(classzone)) - break; - - __set_current_state(TASK_RUNNING); - yield(); + out_of_memory(); } return 0;
this should go back to the 2.4.22 deadlock prone oom killer yes. you can also leave the yield() in the main loop (after out_of_memory()) (__set_current_state(TASK_RUNNING) isn't needed these days just yield is safe enough).
Quoted 8 lines · Andrea Arcangeli
Quoted 7 lines · Kristian Peters
Quoted 5 lines · Andrea Arcangeli
what you're complaining is the 'selection of the task to be killed'. That's not solvable. the kernel can't read your brain period. Only if the kernel could read the brain of the adminstrator then you would be happy, there is no way the kernel can know which is the task you really want to have killed first.
Quoted 6 lines
Quoted 5 lines
I agree. On a server the most likely application to be killed would be the service with the most pages in memory. And those services tend to be the important ones.
However, for a simple desktop-linux that statistical approach seems to be wrong. Your vm has even killed
/sbin/gettysometimes, so that I can't login via a simple console.
Not true! Killing a getty or agetty or any other login program cannot prevent a login! Init will just start another one if there is enough RAM to fork()/exec(). If there isn't, it has nothing to do with killing the getties and everything to do with being completely out of RAM. In that case, you need to enable quotas and not let fork-bombs or other deliberate crashers be executed by root.
Script started on Tue Dec 9 09:42:06 2003
# ps PID TTY STAT TIME COMMAND 478 3 S 0:00 /sbin/agetty 38400 tty3 479 4 S 0:00 /sbin/agetty 38400 tty4 480 5 S 0:00 /sbin/agetty 38400 tty5 481 6 S 0:00 /sbin/agetty 38400 tty6 4404 1 S 0:00 -bash 6102 2 S 0:00 -bash 8772 1 S 0:00 pine 8778 2 S 0:00 script 8779 2 S 0:00 script 8780 p0 S 0:00 bash -i 8784 p0 R 0:00 ps # killall -9 agetty # ps PID TTY STAT TIME COMMAND 4404 1 S 0:00 -bash 6102 2 S 0:00 -bash 8772 1 S 0:00 pine 8778 2 S 0:00 script 8779 2 S 0:00 script 8780 p0 S 0:00 bash -i 8787 3 S 0:00 /sbin/agetty 38400 tty3 8788 4 S 0:00 /sbin/agetty 38400 tty4 8789 5 S 0:00 /sbin/agetty 38400 tty5 8790 6 S 0:00 /sbin/agetty 38400 tty6 8791 p0 R 0:00 ps # exit exit
Script done on Tue Dec 9 09:42:41 2003
The above shows 4 getty's being killed. They are immediately restarted by init. These getties become new shell-tasks in the following order: exec /sbin/login, exec /bin/bash. There are no additional forks so, in principle, you have nearly all the resources needed for a shell as soon as a getty is started.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (797.90 BogoMips).
Note 96.31% of all statistics are fiction.
Quoted 12 lines · Richard B. Johnson
Quoted 10 lines · Andrea Arcangeli
Quoted 9 lines · Kristian Peters
Quoted 5 lines · Andrea Arcangeli
what you're complaining is the 'selection of the task to be killed'. That's not solvable. the kernel can't read your brain period. Only if the kernel could read the brain of the adminstrator then you would be happy, there is no way the kernel can know which is the task you really want to have killed first.
I agree. On a server the most likely application to be killed would be the service with the most pages in memory. And those services tend to be the important ones.
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
However, for a simple desktop-linux that statistical approach seems to be wrong. Your vm has even killed
/sbin/gettysometimes, so that I can't login via a simple console.Not true! Killing a getty or agetty or any other login program cannot prevent a login! Init will just start another
killing getty is a very very lucky scenario indeed. the way I read it was that agetty can hardly be a mem eater, and in turn the same way agetty was killed, it could have been ssh or X to be killed. That's true, but the same issues on the desktop will happen if you have no swap if you enable the old oom killer, the vm will go nuts even if you don't run oom, and there will be all other sort of troubles mentioned a few times already.
Quoted 7 lines · Andrea Arcangeli
killing getty is a very very lucky scenario indeed. the way I read it was that agetty can hardly be a mem eater, and in turn the same way agetty was killed, it could have been ssh or X to be killed. That's true, but the same issues on the desktop will happen if you have no swap if you enable the old oom killer, the vm will go nuts even if you don't run oom, and there will be all other sort of troubles mentioned a few times already.
I think getty was respawned but the console was screwed somehow. The screen was all black as with broken framebuffer and I couldn't type anything (even not the "blind" way.). Only X was there.
Thanks for all your answers.
*Kristian