Skip to contentSkip to conversation

oom killer not triggering in 2.6.9-rc3

BUG Chris Friesen 5 participants 14 messages Oct 9, 2004 – Oct 12, 2004

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

I have an Xserve running 2.6.9-rc3 and patched to run the ppc kernel rather than the ppc64 kernel. It's configured with 2GB of memory, no swap.

If I run one instance of the following program, it allocates all but about 3MB of memory, and the memory hog spins with 100% of the cpu.

If I run two instances of the program, the machine locks up, doesn't respond to pings, and is basically dead to the world.

Shouldn't the oom-killer be kicking in?

Chris

#include <stdlib.h>
#include <unistd.h>

#define PAGES 1000
#define BLOCKSIZE (pgsz * PAGES)

int main()

{
	int pgsz = sysconf(_SC_PAGESIZE);

	while(1) {
		char *p = (char *)malloc(BLOCKSIZE);
		if (p)
			for (int i=0;i<PAGES;i++)
				*(p+(i*pgsz)) = 1;
	}
	return 0;
}
#1 lore raw
Quoted 11 lines · * Chris Friesen ([email protected]) wrote:

I have an Xserve running 2.6.9-rc3 and patched to run the ppc kernel rather than the ppc64 kernel. It's configured with 2GB of memory, no swap.

If I run one instance of the following program, it allocates all but about 3MB of memory, and the memory hog spins with 100% of the cpu.

If I run two instances of the program, the machine locks up, doesn't respond to pings, and is basically dead to the world.

Shouldn't the oom-killer be kicking in?

Kicks in for me (albeit the world comes to a screeching halt for a few moments before it kicks in). I'm using this patch below from Andrew. Does it help you?

mm/vmscan.c renamed from mm/vmscan.c~vmscan-handle-empty-zones +9 -0
diff -puN mm/vmscan.c~vmscan-handle-empty-zones mm/vmscan.c
@@ -851,6 +851,9 @@ shrink_caches(struct zone **zones, struc
 	for (i = 0; zones[i] != NULL; i++) {
 		struct zone *zone = zones[i];
 
+		if (zone->present_pages == 0)
+			continue;
+
 		zone->temp_priority = sc->priority;
 		if (zone->prev_priority > sc->priority)
 			zone->prev_priority = sc->priority;
@@ -999,6 +1002,9 @@ static int balance_pgdat(pg_data_t *pgda
 			for (i = pgdat->nr_zones - 1; i >= 0; i--) {
 				struct zone *zone = pgdat->node_zones + i;
 
+				if (zone->present_pages == 0)
+					continue;
+
 				if (zone->all_unreclaimable &&
 						priority != DEF_PRIORITY)
 					continue;
@@ -1033,6 +1039,9 @@ static int balance_pgdat(pg_data_t *pgda
 		for (i = 0; i <= end_zone; i++) {
 			struct zone *zone = pgdat->node_zones + i;
 
+			if (zone->present_pages == 0)
+				continue;
+
 			if (zone->all_unreclaimable && priority != DEF_PRIORITY)
 				continue;
 

_

#3 lore raw
Quoted 1 line · Alan Cox

The OOM killer is a heuristic.

Sure, but presumably it's a bad thing for a user with no priorities to be able to lock up a machine by running two tasks? I'm not complaining that its killing the wrong thing, I'm complaining that the machine locked up.

Quoted 2 lines

Switch the machine to strict accounting
and it'll kill or block memory access correctly.

I must be able to run an app that uses over 90% of system memory, and calls fork(). I was under the impression this made strict accounting unfeasable?

Chris

#4 lore raw
Quoted 1 line · Alan Cox
Quoted 1 line

The OOM killer is a heuristic.

Quoted 3 lines · Chris Friesen

Sure, but presumably it's a bad thing for a user with no priorities to be able to lock up a machine by running two tasks? I'm not complaining that its killing the wrong thing, I'm complaining that the machine locked up.

Quoted 2 lines · Alan Cox
Quoted 2 lines

Switch the machine to strict accounting
and it'll kill or block memory access correctly.

Quoted 2 lines · Chris Friesen

I must be able to run an app that uses over 90% of system memory, and calls fork(). I was under the impression this made strict accounting unfeasable?

Not so. Just add enough swapspace to act as the backing store for the aggregate anonymous virtualspace.

-- wli

#5 lore raw
Quoted 7 lines · William Lee Irwin III
Quoted 1 line · Chris Friesen

I must be able to run an app that uses over 90% of system memory, and calls fork(). I was under the impression this made strict accounting unfeasable?

Not so. Just add enough swapspace to act as the backing store for the aggregate anonymous virtualspace.

In my first message I mentioned that I had no swap. It's embedded, so I do not have the ability to add swap.

Chris

#6 lore raw
Quoted 4 lines · Chris Friesen
Quoted 3 lines
Quoted 3 lines

I must be able to run an app that uses over 90% of system memory, and calls fork(). I was under the impression this made strict accounting unfeasable?

Quoted 2 lines · William Lee Irwin III
Quoted 2 lines

Not so. Just add enough swapspace to act as the backing store for the aggregate anonymous virtualspace.

Quoted 2 lines · Chris Friesen

In my first message I mentioned that I had no swap. It's embedded, so I do not have the ability to add swap.

Then the closest thing to a good idea may be to enable overcommitment. echo 1 > /proc/sys/vm/overcommit_memory to do that.

-- wli

#7 lore raw
In reply to Chris Friesen
Quoted 7 lines · Chris Friesen
Quoted 1 line · Alan Cox

The OOM killer is a heuristic.

Sure, but presumably it's a bad thing for a user with no priorities to be able to lock up a machine by running two tasks? I'm not complaining that its killing the wrong thing, I'm complaining that the machine locked up.

Quoted 7 lines
Quoted 2 lines

Switch the machine to strict accounting
and it'll kill or block memory access correctly.

I must be able to run an app that uses over 90% of system memory, and calls fork(). I was under the impression this made strict accounting unfeasable?

Chris

No.

The default allows a job to take ten times what is available, and bad things happen later.
With overcommit mode 2 there is an upper bound, but you can twiddle the bound as desired. From proc(5):

/proc/sys/vm/overcommit_memory
This file contains the kernel virtual memory
accounting mode. Values are:
0: heuristic overcommit (this is the default) 1: always overcommit, never check
2: always check, never overcommit
In mode 0, calls of mmap(2) with MAP_NORESERVE set are not checked, and the default check is very weak, leading to the risk of getting a process
"OOM-killed". Under Linux 2.4 any nonzero value
implies mode 1. In mode 2 (available since Linux 2.6), the total virtual address space on the system is limited to (SS + RAM*(r/100)), where SS is the size of the swap space, and RAM is the size of the physical memory, and r is the contents of the file /proc/sys/vm/overcommit_ratio.

/proc/sys/vm/overcommit_ratio
See the description of /proc/sys/vm/overcommit_mem

So, what you have is a bound on virtual memory, and that bound can very easily be larger than twice physical memory.

Andries

#8 lore raw
Quoted 4 lines · Andries Brouwer

The default allows a job to take ten times what is available, and bad things happen later.
With overcommit mode 2 there is an upper bound, but you can twiddle the bound as desired. From proc(5):

Okay, that may be a possibility. I'll look into that.

However, isn't it a bad thing that a vanilla 2.6.9-rc3 can be totally locked up by an unpriviledged user by running two tasks?

It seems to me that the OOM-killer not waking up is a bug. I should not be able to lock up the system by running it out of memory--it should wake up and start killing things rather than hang the system.

Chris

#9 lore raw
Quoted 2 lines · * Chris Friesen ([email protected]) wrote:

However, isn't it a bad thing that a vanilla 2.6.9-rc3 can be totally locked up by an unpriviledged user by running two tasks?

Chris, did you try the patch I sent you (it's in mainline now, so if you re-test on 2.6.9-rc4 you'd pick it up)? With that patch, with 2G of memory and no swap, my machine did not lock up, and the conditions that the patch protect against were triggered. And, with the patch backed out, kswapd spins out of control. I believe this is fixed.

thanks,
-chris

Signature · 1 line
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net
#10 lore raw

Re: [BUG] oom killer not triggering in 2.6.9-rc3 -- FIXED in -rc4

Quoted 2 lines · Chris Wright

Chris, did you try the patch I sent you (it's in mainline now, so if you re-test on 2.6.9-rc4 you'd pick it up)?

Oops. No I didn't--meant to, but then forgot. I should have, it seems to be fixed in -rc4.

Quoted 4 lines

With that patch, with 2G of
memory and no swap, my machine did not lock up, and the conditions that the patch protect against were triggered. And, with the patch backed out, kswapd spins out of control. I believe this is fixed.

2.6.9-rc4 seems sane again. Start up two memory hogs, one gets killed immediately.

Sweet.

Thanks for your help,

Chris

#11 lore raw

Re: [BUG] oom killer not triggering in 2.6.9-rc3 -- FIXED in -rc4

Quoted 1 line · * Chris Friesen ([email protected]) wrote:

2.6.9-rc4 seems sane again. Start up two memory hogs, one gets killed immediately.

Good, thanks for checking.
-chris

Signature · 1 line
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net
#12 lore raw
In reply to Chris Friesen
Quoted 5 lines · Chris Friesen
Quoted 2 lines

Switch the machine to strict accounting
and it'll kill or block memory access correctly.

I must be able to run an app that uses over 90% of system memory, and calls fork(). I was under the impression this made strict accounting unfeasable?

Its rather smarter than that, you'll want swap probably. The strict accountant is a virtual address accountant not a memory accountant. It knows shared r/o segments don't need charging all the time etc

#13 lore raw
Quoted 7 lines · Alan Cox
Quoted 1 line · Chris Friesen

I must be able to run an app that uses over 90% of system memory, and calls fork(). I was under the impression this made strict accounting unfeasable?

Its rather smarter than that, you'll want swap probably. The strict accountant is a virtual address accountant not a memory accountant. It knows shared r/o segments don't need charging all the time etc

As I said in the first message, I've got no swap.

In any case, moving to -rc4 seems to have cleared up the issue, the patch Chris Wright suggested seems to have worked. Oom killer now wakes up immediately and kills one of the memory hogs, and the system continues on.

Chris