Skip to contentSkip to conversation

oom kill oddness.

DISCUSSION Dave Jones 4 participants 7 messages Sep 27, 2006 – Sep 29, 2006

lore
Share & tools
lore

Dave Jones

author
#0 lore raw

So I have two boxes that are very similar.
Both have 2GB of RAM & 1GB of swap space.
One has a 2.8GHz CPU, the other a 2.93GHz CPU, both dualcore.

The slower box survives a 'make -j bzImage' of a 2.6.18 kernel tree without incident. (Although it takes ~4 minutes longer than a -j2)

The faster box goes absolutely nuts, oomkilling everything in sight, until eventually after about 10 minutes, the box locks up dead, and won't even respond to pings.

Oh, the only other difference - the slower box has 1 disk, whereas the
faster box has two in RAID0. I'm not surprised that stuff is getting
oom-killed given the pathological scenario, but the fact that the box never recovered at all is a little odd. Does md lack some means of dealing with low memory scenarios ?

Dave

#1 lore raw
Quoted 11 lines · On Wed, 27 Sep 2006 16:54:35 -0400 Dave Jones <[email protected]> wrote:

So I have two boxes that are very similar.
Both have 2GB of RAM & 1GB of swap space.
One has a 2.8GHz CPU, the other a 2.93GHz CPU, both dualcore.

The slower box survives a 'make -j bzImage' of a 2.6.18 kernel tree without incident. (Although it takes ~4 minutes longer than a -j2)

The faster box goes absolutely nuts, oomkilling everything in sight, until eventually after about 10 minutes, the box locks up dead, and won't even respond to pings.

Quoted 5 lines

Oh, the only other difference - the slower box has 1 disk, whereas the
faster box has two in RAID0. I'm not surprised that stuff is getting
oom-killed given the pathological scenario, but the fact that the box never recovered at all is a little odd. Does md lack some means of dealing with low memory scenarios ?

Are you sure it isn't a memory leak?

Suggest you kill things just before it locks up, have a look at /proc/meminfo, /proc/slabinfo, sysrq-M, echo 3>/proc/sys/vm/drop_caches, etc.

#2 lore raw
In reply to Dave Jones

Hi,

Quoted 11 lines · Dave Jones

So I have two boxes that are very similar.
Both have 2GB of RAM & 1GB of swap space.
One has a 2.8GHz CPU, the other a 2.93GHz CPU, both dualcore.

The slower box survives a 'make -j bzImage' of a 2.6.18 kernel tree without incident. (Although it takes ~4 minutes longer than a -j2)

The faster box goes absolutely nuts, oomkilling everything in sight, until eventually after about 10 minutes, the box locks up dead, and won't even respond to pings.

Quoted 5 lines

Oh, the only other difference - the slower box has 1 disk, whereas the
faster box has two in RAID0. I'm not surprised that stuff is getting
oom-killed given the pathological scenario, but the fact that the box never recovered at all is a little odd. Does md lack some means of dealing with low memory scenarios ?

I think I see the same thing on the other end on slow machines, here it only takes a single compile job, which doesn't quite fit into memory and another task (like top) which occasionally wakes up and tries to allocate memory and then kills the compile job - that's very annoying.

AFAICT the basic problem is that "did_some_progress" in __alloc_pages() is rather local information, other processes can still make progress and keep this process from making progress, which gets grumpy and starts killing. What's happing here is that most memory is either mapped or in the swap cache, so we have a race between processes trying to free memory from the cache and processes mapping memory back into their address space.

If someone wants to play with the problem, the example program below triggers the problem relatively easily (booting with only little ram helps), it starts a number of readers, which should touch a bit more memory than is available and a few writers, which occasionally allocate memory.

bye, Roman

#include <stdlib.h>
#include <string.h>
#include <time.h>

#define MEM_SIZE (24 << 20)

int main(int ac, char **av)

{
	volatile char *mem;
	int i, memsize;

	memsize = MEM_SIZE;
	if (ac > 1)
		memsize = atoi(av[1]) << 20;
	mem = malloc(memsize);

	memset(mem, 0, memsize);
	for (i = 0; i < 32; i++) {
		if (!fork()) {
			while (1) {
				*(mem + random() % memsize);
			}
		}
	}
	for (i = 0; i < 5; i++) {
		if (!fork()) {
			while (1) {
				volatile char *p;
				struct timespec ts;
				int t = random() % 5000;
				ts.tv_sec = t / 1000;
				ts.tv_nsec = (t % 1000) * 1000000;
				nanosleep(&ts, NULL);
				p = malloc(1 << 16);
				memset(p, 0, 1 << 16);
				free(p);
			}
		}
	}
	while (1)
		pause();
}
#3 lore raw
Quoted 26 lines · On Fri, 29 Sep 2006 01:03:16 +0200 (CEST) Roman Zippel <[email protected]>…

Hi,

Quoted 11 lines · Dave Jones

So I have two boxes that are very similar.
Both have 2GB of RAM & 1GB of swap space.
One has a 2.8GHz CPU, the other a 2.93GHz CPU, both dualcore.

The slower box survives a 'make -j bzImage' of a 2.6.18 kernel tree without incident. (Although it takes ~4 minutes longer than a -j2)

The faster box goes absolutely nuts, oomkilling everything in sight, until eventually after about 10 minutes, the box locks up dead, and won't even respond to pings.

Quoted 5 lines

Oh, the only other difference - the slower box has 1 disk, whereas the
faster box has two in RAID0. I'm not surprised that stuff is getting
oom-killed given the pathological scenario, but the fact that the box never recovered at all is a little odd. Does md lack some means of dealing with low memory scenarios ?

I think I see the same thing on the other end on slow machines, here it only takes a single compile job, which doesn't quite fit into memory and another task (like top) which occasionally wakes up and tries to allocate memory and then kills the compile job - that's very annoying.

Quoted 6 lines

AFAICT the basic problem is that "did_some_progress" in __alloc_pages() is rather local information, other processes can still make progress and keep this process from making progress, which gets grumpy and starts killing. What's happing here is that most memory is either mapped or in the swap cache, so we have a race between processes trying to free memory from the cache and processes mapping memory back into their address space.

Kernel versions please, guys. There have been a lot of oom-killer changes post-2.6.18.

Quoted 6 lines

If someone wants to play with the problem, the example program below triggers the problem relatively easily (booting with only little ram helps), it starts a number of readers, which should touch a bit more memory than is available and a few writers, which occasionally allocate memory.

How much ram, how much swap?

author
#4 lore raw
Quoted 36 lines · Andrew Morton
Quoted 26 lines · On Fri, 29 Sep 2006 01:03:16 +0200 (CEST) Roman Zippel <[email protected]>…

Hi,

Quoted 11 lines · Dave Jones

So I have two boxes that are very similar.
Both have 2GB of RAM & 1GB of swap space.
One has a 2.8GHz CPU, the other a 2.93GHz CPU, both dualcore.

The slower box survives a 'make -j bzImage' of a 2.6.18 kernel tree without incident. (Although it takes ~4 minutes longer than a -j2)

The faster box goes absolutely nuts, oomkilling everything in sight, until eventually after about 10 minutes, the box locks up dead, and won't even respond to pings.

Quoted 5 lines

Oh, the only other difference - the slower box has 1 disk, whereas the
faster box has two in RAID0. I'm not surprised that stuff is getting
oom-killed given the pathological scenario, but the fact that the box never recovered at all is a little odd. Does md lack some means of dealing with low memory scenarios ?

I think I see the same thing on the other end on slow machines, here it only takes a single compile job, which doesn't quite fit into memory and another task (like top) which occasionally wakes up and tries to allocate memory and then kills the compile job - that's very annoying.

Quoted 6 lines

AFAICT the basic problem is that "did_some_progress" in __alloc_pages() is rather local information, other processes can still make progress and keep this process from making progress, which gets grumpy and starts killing. What's happing here is that most memory is either mapped or in the swap cache, so we have a race between processes trying to free memory from the cache and processes mapping memory back into their address space.

Quoted 2 lines

Kernel versions please, guys. There have been a lot of oom-killer changes post-2.6.18.

Sorry, I've been stuck on 2.6.18 as that's what we're shipping in FC6 soon.

Dave

#5 lore raw
In reply to Andrew Morton

Hi,

Quoted 2 lines · Andrew Morton

Kernel versions please, guys. There have been a lot of oom-killer changes post-2.6.18.

Last I tested this was with 2.6.18.
The latest changes to vmscan.c should help...

Quoted 8 lines
Quoted 6 lines

If someone wants to play with the problem, the example program below triggers the problem relatively easily (booting with only little ram helps), it starts a number of readers, which should touch a bit more memory than is available and a few writers, which occasionally allocate memory.

How much ram, how much swap?

I tested it with 32MB and 64MB and plenty of swap.

bye, Roman

#6 lore raw
Quoted 5 lines · Roman Zippel

Hi,

On Thu, 28 Sep 2006, Andrew Morton wrote:

Quoted 7 lines
Quoted 3 lines

Kernel versions please, guys. There have been a lot of oom-killer changes post-2.6.18.

Last I tested this was with 2.6.18.
The latest changes to vmscan.c should help...

It would be good if you could confirm that. I basically got the kernel to the point where it used up all swap before going OOM on the workload I was looking at (MySQL running in virtual machines).

--

Send instant messages to your online friends http://au.messenger.yahoo.com