oom killer not triggering in 2.6.9-rc3
Subsystems (3)
I have an Xserve running 2.6.9-rc3 and patched to run the ppc kernel rather than the ppc64 kernel. It's configured with 2GB of memory, no swap.
If I run one instance of the following program, it allocates all but about 3MB of memory, and the memory hog spins with 100% of the cpu.
If I run two instances of the program, the machine locks up, doesn't respond to pings, and is basically dead to the world.
Shouldn't the oom-killer be kicking in?
Chris
#include <stdlib.h>
#include <unistd.h>
#define PAGES 1000
#define BLOCKSIZE (pgsz * PAGES)
int main()
{
int pgsz = sysconf(_SC_PAGESIZE);
while(1) {
char *p = (char *)malloc(BLOCKSIZE);
if (p)
for (int i=0;i<PAGES;i++)
*(p+(i*pgsz)) = 1;
}
return 0;
}Quoted 11 lines · * Chris Friesen ([email protected]) wrote:
I have an Xserve running 2.6.9-rc3 and patched to run the ppc kernel rather than the ppc64 kernel. It's configured with 2GB of memory, no swap.
If I run one instance of the following program, it allocates all but about 3MB of memory, and the memory hog spins with 100% of the cpu.
If I run two instances of the program, the machine locks up, doesn't respond to pings, and is basically dead to the world.
Shouldn't the oom-killer be kicking in?
Kicks in for me (albeit the world comes to a screeching halt for a few moments before it kicks in). I'm using this patch below from Andrew. Does it help you?
mm/vmscan.c renamed from mm/vmscan.c~vmscan-handle-empty-zones +9 -0
@@ -851,6 +851,9 @@ shrink_caches(struct zone **zones, strucfor (i =0 ; zones[i] !=NULL ; i++) {struct zone *zone = zones[i]; +if (zone->present_pages ==0 ) +continue ; + zone->temp_priority = sc->priority;if (zone->prev_priority > sc->priority) zone->prev_priority = sc->priority;
@@ -999,6 +1002,9 @@ static int balance_pgdat(pg_data_t *pgdafor (i = pgdat->nr_zones -1 ; i >=0 ; i--) {struct zone *zone = pgdat->node_zones + i; +if (zone->present_pages ==0 ) +continue ; +if (zone->all_unreclaimable && priority !=DEF_PRIORITY )continue ;
@@ -1033,6 +1039,9 @@ static int balance_pgdat(pg_data_t *pgdafor (i =0 ; i <= end_zone; i++) {struct zone *zone = pgdat->node_zones + i; +if (zone->present_pages ==0 ) +continue ; +if (zone->all_unreclaimable && priority !=DEF_PRIORITY )continue ;
_
The OOM killer is a heuristic. Switch the machine to strict accounting and it'll kill or block memory access correctly.
Quoted 1 line · Alan Cox
The OOM killer is a heuristic.
Sure, but presumably it's a bad thing for a user with no priorities to be able to lock up a machine by running two tasks? I'm not complaining that its killing the wrong thing, I'm complaining that the machine locked up.
Quoted 2 lines
Switch the machine to strict accounting
and it'll kill or block memory access correctly.
I must be able to run an app that uses over 90% of system memory, and calls fork(). I was under the impression this made strict accounting unfeasable?
Chris
Quoted 1 line · Alan Cox
Quoted 1 line
The OOM killer is a heuristic.
Quoted 3 lines · Chris Friesen
Sure, but presumably it's a bad thing for a user with no priorities to be able to lock up a machine by running two tasks? I'm not complaining that its killing the wrong thing, I'm complaining that the machine locked up.
Quoted 2 lines · Alan Cox
Quoted 2 lines
Switch the machine to strict accounting
and it'll kill or block memory access correctly.
Quoted 2 lines · Chris Friesen
I must be able to run an app that uses over 90% of system memory, and calls
fork(). I was under the impression this made strict accounting unfeasable?
Not so. Just add enough swapspace to act as the backing store for the aggregate anonymous virtualspace.
-- wli
Quoted 7 lines · William Lee Irwin III
Quoted 1 line · Chris Friesen
I must be able to run an app that uses over 90% of system memory, and calls
fork(). I was under the impression this made strict accounting unfeasable?Not so. Just add enough swapspace to act as the backing store for the aggregate anonymous virtualspace.
In my first message I mentioned that I had no swap. It's embedded, so I do not have the ability to add swap.
Chris
Quoted 4 lines · Chris Friesen
Quoted 3 lines
Quoted 3 lines
I must be able to run an app that uses over 90% of system memory, and calls
fork(). I was under the impression this made strict accounting unfeasable?
Quoted 2 lines · William Lee Irwin III
Quoted 2 lines
Not so. Just add enough swapspace to act as the backing store for the aggregate anonymous virtualspace.
Quoted 2 lines · Chris Friesen
In my first message I mentioned that I had no swap. It's embedded, so I do not have the ability to add swap.
Then the closest thing to a good idea may be to enable overcommitment. echo 1 > /proc/sys/vm/overcommit_memory to do that.
-- wli
Quoted 7 lines · Chris Friesen
Quoted 1 line · Alan Cox
The OOM killer is a heuristic.
Sure, but presumably it's a bad thing for a user with no priorities to be able to lock up a machine by running two tasks? I'm not complaining that its killing the wrong thing, I'm complaining that the machine locked up.
Quoted 7 lines
Quoted 2 lines
Switch the machine to strict accounting
and it'll kill or block memory access correctly.I must be able to run an app that uses over 90% of system memory, and calls
fork(). I was under the impression this made strict accounting unfeasable?Chris
No.
The default allows a job to take ten times what is available, and bad things happen later.
With overcommit mode 2 there is an upper bound, but you can twiddle the bound as desired. From proc(5):
/proc/sys/vm/overcommit_memory
This file contains the kernel virtual memory
accounting mode. Values are:
0: heuristic overcommit (this is the default) 1: always overcommit, never check
2: always check, never overcommit
In mode 0, calls of mmap(2) with MAP_NORESERVE set are not checked, and the default check is very weak, leading to the risk of getting a process
"OOM-killed". Under Linux 2.4 any nonzero value
implies mode 1. In mode 2 (available since Linux 2.6), the total virtual address space on the system is limited to (SS + RAM*(r/100)), where SS is the size of the swap space, and RAM is the size of the physical memory, and r is the contents of the file /proc/sys/vm/overcommit_ratio.
/proc/sys/vm/overcommit_ratio
See the description of /proc/sys/vm/overcommit_mem
So, what you have is a bound on virtual memory, and that bound can very easily be larger than twice physical memory.
Andries
Quoted 4 lines · Andries Brouwer
The default allows a job to take ten times what is available, and bad things happen later.
With overcommit mode 2 there is an upper bound, but you can twiddle the bound as desired. From proc(5):
Okay, that may be a possibility. I'll look into that.
However, isn't it a bad thing that a vanilla 2.6.9-rc3 can be totally locked up by an unpriviledged user by running two tasks?
It seems to me that the OOM-killer not waking up is a bug. I should not be able to lock up the system by running it out of memory--it should wake up and start killing things rather than hang the system.
Chris
Quoted 2 lines · * Chris Friesen ([email protected]) wrote:
However, isn't it a bad thing that a vanilla 2.6.9-rc3 can be totally locked up by an unpriviledged user by running two tasks?
Chris, did you try the patch I sent you (it's in mainline now, so if you re-test on 2.6.9-rc4 you'd pick it up)? With that patch, with 2G of memory and no swap, my machine did not lock up, and the conditions that the patch protect against were triggered. And, with the patch backed out, kswapd spins out of control. I believe this is fixed.
thanks,
-chris
Signature · 1 line
Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net
Re: [BUG] oom killer not triggering in 2.6.9-rc3 -- FIXED in -rc4
Quoted 2 lines · Chris Wright
Chris, did you try the patch I sent you (it's in mainline now, so if you re-test on 2.6.9-rc4 you'd pick it up)?
Oops. No I didn't--meant to, but then forgot. I should have, it seems to be fixed in -rc4.
Quoted 4 lines
With that patch, with 2G of
memory and no swap, my machine did not lock up, and the conditions that the patch protect against were triggered. And, with the patch backed out, kswapd spins out of control. I believe this is fixed.
2.6.9-rc4 seems sane again. Start up two memory hogs, one gets killed immediately.
Sweet.
Thanks for your help,
Chris
Re: [BUG] oom killer not triggering in 2.6.9-rc3 -- FIXED in -rc4
Quoted 1 line · * Chris Friesen ([email protected]) wrote:
2.6.9-rc4 seems sane again. Start up two memory hogs, one gets killed immediately.
Good, thanks for checking.
-chris
Signature · 1 line
Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net
Quoted 5 lines · Chris Friesen
Quoted 2 lines
Switch the machine to strict accounting
and it'll kill or block memory access correctly.I must be able to run an app that uses over 90% of system memory, and calls
fork(). I was under the impression this made strict accounting unfeasable?
Its rather smarter than that, you'll want swap probably. The strict accountant is a virtual address accountant not a memory accountant. It knows shared r/o segments don't need charging all the time etc
Quoted 7 lines · Alan Cox
Quoted 1 line · Chris Friesen
I must be able to run an app that uses over 90% of system memory, and calls
fork(). I was under the impression this made strict accounting unfeasable?Its rather smarter than that, you'll want swap probably. The strict accountant is a virtual address accountant not a memory accountant. It knows shared r/o segments don't need charging all the time etc
As I said in the first message, I've got no swap.
In any case, moving to -rc4 seems to have cleared up the issue, the patch Chris Wright suggested seems to have worked. Oom killer now wakes up immediately and kills one of the memory hogs, and the system continues on.
Chris