2.4.18-14 kernel stuck during ext3 umount with ping still responding

From: Andrew Morton <[email protected]>
To: yuval yeret <[email protected]>
CC: [email protected], [email protected]
Subject: Re: 2.4.18-14 kernel stuck during ext3 umount with ping still responding
Date: Thu, 09 Jan 2003 02:06:55 -0800

Quoted 4 lines · yuval yeret

Hi,

I'm running a 2.4.18-14 kernel with a heavy IO profile using ext3 over

RAID

Quoted 3 lines

0+1 volumes.

Quoted 1 line

From time to time I get a black screen stuck machine while trying to

umount

Quoted 1 line

a volume during an IO workload (as part of a failback solution - but

after

Quoted 1 line

killing all IO processes ), with ping still responding, but everything

else

Quoted 3 lines

mostly dead.

I tried using the forcedumount patch to solve this problem - to no

avail.

Quoted 1 line

Also tried upgrading the qlogic drivers to the latest drivers from

Qlogic.

Quoted 9 lines

After one of the occurences I managed to get some output using the sysrq keys.

This seems similar to what is described in
http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=77508 but with a different call trace

What I have here is what I managed to copy down (for some reason

pgup/pgdown

Quoted 1 line

didn't work so not all information is full...) together with a manual

lookup

Quoted 16 lines

of the call trace
from /proc/ksyms :

process umount
EIP c01190b8 (set_running_and_schedule)
call trace:

c01144c9        f25f9ec0        IO_APIC_get_PCI_irq_vector
c010a8b0        f25f9ed0        enable_irq
c014200c        f25f9ef0        fsync_buffers_list
c0155595        f25f9efc        clear_inode
c015553d        f25f9f2c        invalidate_inodes
c01461d8        f25f9f78        get_super
c014a629        f25f9f94        path_release
c0157c58        f25f9fc0        sys_umount
c0108cab                        sys_sigaltstack

Quoted 2 lines

Any idea what can cause this ?

If you have a large amount of data against two or more filesystems, and you try to unmount one of them the kernel can seize up for a very long time in the fsync_dev()->sync_buffers() function. Under these circumstances that function has O(n*n) search complexity and n is quite large.

However your backtrace shows neither of those functions.

Still, as an experiment it would be interesting to see if the below patch fixes it up. It converts O(n*n) to O(m), where m > n.

 fs/buffer.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

fs/buffer.c renamed from fs/buffer.c~a +3 -3

@@ -307,11 +307,11 @@ int sync_buffers(kdev_t dev, int wait)
 	 * 2) write out all dirty, unlocked buffers;
 	 * 2) wait for completion by waiting for all buffers to unlock.
 	 */
-	write_unlocked_buffers(dev);
+	write_unlocked_buffers(NODEV);
 	if (wait) {
-		err = wait_for_locked_buffers(dev, BUF_DIRTY, 0);
+		err = wait_for_locked_buffers(NODEV, BUF_DIRTY, 0);
 		write_unlocked_buffers(dev);
-		err |= wait_for_locked_buffers(dev, BUF_LOCKED, 1);
+		err |= wait_for_locked_buffers(NODEV, BUF_LOCKED, 1);
 	}
 	return err;
 }

_