Skip to contentSkip to conversation

2.6.21 frozen for a few minutes, swapping to disk

DISCUSSION Miguel Figueiredo 3 participants 5 messages Apr 29, 2007 – May 1, 2007

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi all,

today, with 2.6.21, my laptop had a really odd behaviour. It started writing to disk for a few minutes with no interactivity at all (no redraw on screen, only hdd led on). It's the first time i noticed OOM-killer started do kill programs.

It was totally unresponsive for minutes, after back to life it had a load of ~19.0, and 300+ MB on swap (first time i saw this).

It's an HP pavillon core duo 2.0 GHz, 1 GB RAM

kern.log details: http://www.debianpt.org/~elmig/pool/kernel/20070429/kern.log
.config: http://www.debianpt.org/~elmig/pool/kernel/20070429/2.6.21.config
dmesg: http://www.debianpt.org/~elmig/pool/kernel/20070429/dmesg

As this is the first time it happened and it felt odd i am reporting.

If aditional info is needed please CC me as i am not on the list.

Signature · 4 lines
Com os melhores cumprimentos/Best regards,

Miguel Figueiredo
http://www.DebianPT.org
maintainer
#1 lore raw
Quoted 12 lines · On Sun, 29 Apr 2007 11:28:05 +0100 Miguel Figueiredo <[email protected]> wrote:

Hi all,

today, with 2.6.21, my laptop had a really odd behaviour. It started writing to disk for a few minutes with no interactivity at all (no redraw on screen, only hdd led on). It's the first time i noticed OOM-killer started do kill programs.

It was totally unresponsive for minutes, after back to life it had a load of ~19.0, and 300+ MB on swap (first time i saw this).

It's an HP pavillon core duo 2.0 GHz, 1 GB RAM

Quoted 9 lines

kern.log details:
http://www.debianpt.org/~elmig/pool/kernel/20070429/kern.log
.config: http://www.debianpt.org/~elmig/pool/kernel/20070429/2.6.21.config
dmesg: http://www.debianpt.org/~elmig/pool/kernel/20070429/dmesg

As this is the first time it happened and it felt odd i am reporting.

If aditional info is needed please CC me as i am not on the list.

hm, a genuine oom on an all-ext3 data=ordered i386 system, just like a million other people. How very weird.

I assume all those pages on the LRU are pagecache pages which for some reason we're unable to reclaim.

If some privileged application went berzerk mlock()ing everything then that might explain it. It sounds improbable, but then, something improbable has happened.

We cleverly managed to not display the pagecache totals in the oom-killer output. Could you please take a copy of /proc/meminfo after an oom-killing, send that? And /proc/vmstat, I guess.

If you're keen, we could eliminate the mlock possibility by adding this:

mm/mlock.c renamed from mm/mlock.c~a +4 -0
@@ -127,6 +127,8 @@ asmlinkage long sys_mlock(unsigned long 
 	unsigned long lock_limit;
 	int error = -ENOMEM;
 
+	return 0;
+
 	if (!can_do_mlock())
 		return -EPERM;
 
@@ -151,6 +153,8 @@ asmlinkage long sys_munlock(unsigned lon
 {
 	int ret;
 
+	return 0;
+
 	down_write(&current->mm->mmap_sem);
 	len = PAGE_ALIGN(len + (start & ~PAGE_MASK));
 	start &= PAGE_MASK;

_

#2 lore raw
Quoted 21 lines · Andrew Morton
Quoted 8 lines · On Sun, 29 Apr 2007 11:28:05 +0100 Miguel Figueiredo <[email protected]> wrote:

Hi all,

today, with 2.6.21, my laptop had a really odd behaviour. It started writing to disk for a few minutes with no interactivity at all (no redraw on screen, only hdd led on). It's the first time i noticed OOM-killer started do kill programs.

It was totally unresponsive for minutes, after back to life it had a load of ~19.0, and 300+ MB on swap (first time i saw this).

It's an HP pavillon core duo 2.0 GHz, 1 GB RAM

Quoted 8 lines

kern.log details: http://www.debianpt.org/~elmig/pool/kernel/20070429/kern.log
.config: http://www.debianpt.org/~elmig/pool/kernel/20070429/2.6.21.config
dmesg: http://www.debianpt.org/~elmig/pool/kernel/20070429/dmesg

As this is the first time it happened and it felt odd i am reporting.

If aditional info is needed please CC me as i am not on the list.

Quoted 5 lines

hm, a genuine oom on an all-ext3 data=ordered i386 system, just like a million other people. How very weird.

I assume all those pages on the LRU are pagecache pages which for some reason we're unable to reclaim.

It looks like it used up all swap? I'd guess a memory leak in some application, or maybe a page refcount leak somewhere.

Quoted 33 lines

If some privileged application went berzerk mlock()ing everything then that might explain it. It sounds improbable, but then, something improbable has happened.

We cleverly managed to not display the pagecache totals in the oom-killer output. Could you please take a copy of /proc/meminfo after an oom-killing, send that? And /proc/vmstat, I guess.

If you're keen, we could eliminate the mlock possibility by adding this:

mm/mlock.c renamed from mm/mlock.c~a +4 -0
@@ -127,6 +127,8 @@ asmlinkage long sys_mlock(unsigned long 
 	unsigned long lock_limit;
 	int error = -ENOMEM;
 
+	return 0;
+
 	if (!can_do_mlock())
 		return -EPERM;
 
@@ -151,6 +153,8 @@ asmlinkage long sys_munlock(unsigned lon
 {
 	int ret;
 
+	return 0;
+
 	down_write(&current->mm->mmap_sem);
 	len = PAGE_ALIGN(len + (start & ~PAGE_MASK));
 	start &= PAGE_MASK;

_

Signature · 1 line
SUSE Labs, Novell Inc.
maintainer
#3 lore raw
Quoted 8 lines · On Tue, 01 May 2007 15:42:30 +1000 Nick Piggin <[email protected]> wrote:
Quoted 5 lines

hm, a genuine oom on an all-ext3 data=ordered i386 system, just like a million other people. How very weird.

I assume all those pages on the LRU are pagecache pages which for some reason we're unable to reclaim.

It looks like it used up all swap? I'd guess a memory leak in some application, or maybe a page refcount leak somewhere.

yes, I missed that. The number of mapped pages is tiny so the thing has
been trying to swap out like.

The question is: how much memory is free after the oom-killing storm? If it's "lots" then it's probably an application problem. If it's "not much" then perhaps there's a kernel leak.

#4 lore raw
Quoted 9 lines · Andrew Morton
Quoted 6 lines · On Tue, 01 May 2007 15:42:30 +1000 Nick Piggin <[email protected]> wrote:
Quoted 5 lines

hm, a genuine oom on an all-ext3 data=ordered i386 system, just like a million other people. How very weird.

I assume all those pages on the LRU are pagecache pages which for some reason we're unable to reclaim.

Quoted 6 lines
Quoted 2 lines

It looks like it used up all swap? I'd guess a memory leak in some application, or maybe a page refcount leak somewhere.

yes, I missed that. The number of mapped pages is tiny so the thing has
been trying to swap out like.

I didn't quite parse this :)

If the memory is leaking slowly, it could be eventually pushing everything out to swap without having a large amount of mapped pages.

Or if something is slowly writing stuff to tmpfs, that may not show up in mapped pages either.

Quoted 3 lines

The question is: how much memory is free after the oom-killing storm? If it's "lots" then it's probably an application problem. If it's "not much" then perhaps there's a kernel leak.

Yeah, or a tmpfs filesystem being filled up (what does df say?).

Signature · 1 line
SUSE Labs, Novell Inc.