oom kill oddness.
So I have two boxes that are very similar.
Both have 2GB of RAM & 1GB of swap space.
One has a 2.8GHz CPU, the other a 2.93GHz CPU, both dualcore.
The slower box survives a 'make -j bzImage' of a 2.6.18 kernel tree without incident. (Although it takes ~4 minutes longer than a -j2)
The faster box goes absolutely nuts, oomkilling everything in sight, until eventually after about 10 minutes, the box locks up dead, and won't even respond to pings.
Oh, the only other difference - the slower box has 1 disk, whereas the
faster box has two in RAID0. I'm not surprised that stuff is getting
oom-killed given the pathological scenario, but the fact that the box never recovered at all is a little odd. Does md lack some means of dealing with low memory scenarios ?
Dave
Quoted 11 lines · On Wed, 27 Sep 2006 16:54:35 -0400 Dave Jones <[email protected]> wrote:
So I have two boxes that are very similar.
Both have 2GB of RAM & 1GB of swap space.
One has a 2.8GHz CPU, the other a 2.93GHz CPU, both dualcore.The slower box survives a 'make -j bzImage' of a 2.6.18 kernel tree without incident. (Although it takes ~4 minutes longer than a -j2)
The faster box goes absolutely nuts, oomkilling everything in sight, until eventually after about 10 minutes, the box locks up dead, and won't even respond to pings.
Quoted 5 lines
Oh, the only other difference - the slower box has 1 disk, whereas the
faster box has two in RAID0. I'm not surprised that stuff is getting
oom-killed given the pathological scenario, but the fact that the box never recovered at all is a little odd. Does md lack some means of dealing with low memory scenarios ?
Are you sure it isn't a memory leak?
Suggest you kill things just before it locks up, have a look at /proc/meminfo, /proc/slabinfo, sysrq-M, echo 3>/proc/sys/vm/drop_caches, etc.
Hi,
Quoted 11 lines · Dave Jones
So I have two boxes that are very similar.
Both have 2GB of RAM & 1GB of swap space.
One has a 2.8GHz CPU, the other a 2.93GHz CPU, both dualcore.The slower box survives a 'make -j bzImage' of a 2.6.18 kernel tree without incident. (Although it takes ~4 minutes longer than a -j2)
The faster box goes absolutely nuts, oomkilling everything in sight, until eventually after about 10 minutes, the box locks up dead, and won't even respond to pings.
Quoted 5 lines
Oh, the only other difference - the slower box has 1 disk, whereas the
faster box has two in RAID0. I'm not surprised that stuff is getting
oom-killed given the pathological scenario, but the fact that the box never recovered at all is a little odd. Does md lack some means of dealing with low memory scenarios ?
I think I see the same thing on the other end on slow machines, here it only takes a single compile job, which doesn't quite fit into memory and another task (like top) which occasionally wakes up and tries to allocate memory and then kills the compile job - that's very annoying.
AFAICT the basic problem is that "did_some_progress" in __alloc_pages() is rather local information, other processes can still make progress and keep this process from making progress, which gets grumpy and starts killing. What's happing here is that most memory is either mapped or in the swap cache, so we have a race between processes trying to free memory from the cache and processes mapping memory back into their address space.
If someone wants to play with the problem, the example program below triggers the problem relatively easily (booting with only little ram helps), it starts a number of readers, which should touch a bit more memory than is available and a few writers, which occasionally allocate memory.
bye, Roman
#include <stdlib.h>
#include <string.h>
#include <time.h>
#define MEM_SIZE (24 << 20)
int main(int ac, char **av)
{
volatile char *mem;
int i, memsize;
memsize = MEM_SIZE;
if (ac > 1)
memsize = atoi(av[1]) << 20;
mem = malloc(memsize);
memset(mem, 0, memsize);
for (i = 0; i < 32; i++) {
if (!fork()) {
while (1) {
*(mem + random() % memsize);
}
}
}
for (i = 0; i < 5; i++) {
if (!fork()) {
while (1) {
volatile char *p;
struct timespec ts;
int t = random() % 5000;
ts.tv_sec = t / 1000;
ts.tv_nsec = (t % 1000) * 1000000;
nanosleep(&ts, NULL);
p = malloc(1 << 16);
memset(p, 0, 1 << 16);
free(p);
}
}
}
while (1)
pause();
}Quoted 26 lines · On Fri, 29 Sep 2006 01:03:16 +0200 (CEST) Roman Zippel <[email protected]>…
Hi,
Quoted 11 lines · Dave Jones
So I have two boxes that are very similar.
Both have 2GB of RAM & 1GB of swap space.
One has a 2.8GHz CPU, the other a 2.93GHz CPU, both dualcore.The slower box survives a 'make -j bzImage' of a 2.6.18 kernel tree without incident. (Although it takes ~4 minutes longer than a -j2)
The faster box goes absolutely nuts, oomkilling everything in sight, until eventually after about 10 minutes, the box locks up dead, and won't even respond to pings.
Quoted 5 lines
Oh, the only other difference - the slower box has 1 disk, whereas the
faster box has two in RAID0. I'm not surprised that stuff is getting
oom-killed given the pathological scenario, but the fact that the box never recovered at all is a little odd. Does md lack some means of dealing with low memory scenarios ?I think I see the same thing on the other end on slow machines, here it only takes a single compile job, which doesn't quite fit into memory and another task (like top) which occasionally wakes up and tries to allocate memory and then kills the compile job - that's very annoying.
Quoted 6 lines
AFAICT the basic problem is that "
did_some_progress" in__alloc_pages()is rather local information, other processes can still make progress and keep this process from making progress, which gets grumpy and starts killing. What's happing here is that most memory is either mapped or in the swap cache, so we have a race between processes trying to free memory from the cache and processes mapping memory back into their address space.
Kernel versions please, guys. There have been a lot of oom-killer changes post-2.6.18.
Quoted 6 lines
If someone wants to play with the problem, the example program below triggers the problem relatively easily (booting with only little ram helps), it starts a number of readers, which should touch a bit more memory than is available and a few writers, which occasionally allocate memory.
How much ram, how much swap?
Quoted 36 lines · Andrew Morton
Quoted 26 lines · On Fri, 29 Sep 2006 01:03:16 +0200 (CEST) Roman Zippel <[email protected]>…
Hi,
Quoted 11 lines · Dave Jones
So I have two boxes that are very similar.
Both have 2GB of RAM & 1GB of swap space.
One has a 2.8GHz CPU, the other a 2.93GHz CPU, both dualcore.The slower box survives a 'make -j bzImage' of a 2.6.18 kernel tree without incident. (Although it takes ~4 minutes longer than a -j2)
The faster box goes absolutely nuts, oomkilling everything in sight, until eventually after about 10 minutes, the box locks up dead, and won't even respond to pings.
Quoted 5 lines
Oh, the only other difference - the slower box has 1 disk, whereas the
faster box has two in RAID0. I'm not surprised that stuff is getting
oom-killed given the pathological scenario, but the fact that the box never recovered at all is a little odd. Does md lack some means of dealing with low memory scenarios ?I think I see the same thing on the other end on slow machines, here it only takes a single compile job, which doesn't quite fit into memory and another task (like top) which occasionally wakes up and tries to allocate memory and then kills the compile job - that's very annoying.
Quoted 6 lines
AFAICT the basic problem is that "
did_some_progress" in__alloc_pages()is rather local information, other processes can still make progress and keep this process from making progress, which gets grumpy and starts killing. What's happing here is that most memory is either mapped or in the swap cache, so we have a race between processes trying to free memory from the cache and processes mapping memory back into their address space.
Quoted 2 lines
Kernel versions please, guys. There have been a lot of oom-killer changes post-2.6.18.
Sorry, I've been stuck on 2.6.18 as that's what we're shipping in FC6 soon.
Dave
Hi,
Quoted 2 lines · Andrew Morton
Kernel versions please, guys. There have been a lot of oom-killer changes post-2.6.18.
Last I tested this was with 2.6.18.
The latest changes to vmscan.c should help...
Quoted 8 lines
Quoted 6 lines
If someone wants to play with the problem, the example program below triggers the problem relatively easily (booting with only little ram helps), it starts a number of readers, which should touch a bit more memory than is available and a few writers, which occasionally allocate memory.
How much ram, how much swap?
I tested it with 32MB and 64MB and plenty of swap.
bye, Roman
Quoted 5 lines · Roman Zippel
Hi,
On Thu, 28 Sep 2006, Andrew Morton wrote:
Quoted 7 lines
Quoted 3 lines
Kernel versions please, guys. There have been a lot of oom-killer changes post-2.6.18.
Last I tested this was with 2.6.18.
The latest changes tovmscan.cshould help...
It would be good if you could confirm that. I basically got the kernel to the point where it used up all swap before going OOM on the workload I was looking at (MySQL running in virtual machines).
--
Send instant messages to your online friends http://au.messenger.yahoo.com