Crash when unmounting NFS/TCP with -f
Subsystems (3)
Hi Trond,
I'm using NFS (v2) over TCP (in a SSH tunnel).
Each time the SSH dies before a umount NFS, I have to umount -f and I get a crash (only sysrq works).
Actually, the crash occurs a few seconds after umount -f.
It seems that killing SSH by hand does _not_ lead to crash. But a long network failure does.
I remember seeing this bug several times with all stable releases from 2.6.7 to 2.6.11. I didn't try with earlier versions.
I didn't see anything in the logs (after reboot). But I can't be sure there was nothing in dmesg since I didn't get a chance to chvt 1 and see console messages before rebooting (with sysrq).
Do you have any idea how to debug this ?
Thanks,
Brice
fr den 22.04.2005 Klokka 14:32 (+0200) skreiv Brice Goglin:
Quoted 7 lines
Hi Trond,
I'm using NFS (v2) over TCP (in a SSH tunnel).
Each time the SSH dies before a umount NFS, I have to umount -f and I get a crash (only sysrq works).
Actually, the crash occurs a few seconds after umount -f.
Quoted 8 lines
It seems that killing SSH by hand does _not_ lead to crash. But a long network failure does.
I remember seeing this bug several times with all stable releases from 2.6.7 to 2.6.11. I didn't try with earlier versions.I didn't see anything in the logs (after reboot). But I can't be sure there was nothing in dmesg since I didn't get a chance to chvt 1 and see console messages before rebooting (with sysrq).
I'll try to reproduce. There has just been a discussion about "umount -f" on the NFS mailing list ([email protected]), where Peter Cendio said he was seeing the following Oops:
http://www.cendio.se/~peter/fc3-umount-crash.png
I am unable to reproduce Peter's crash, but I didn't try the scenario that you describe above.
Cheers,
Trond
Signature · 1 line
Trond Myklebust <[email protected]>
Quoted 12 lines · Brice Goglin
Hi Trond,
I'm using NFS (v2) over TCP (in a SSH tunnel).
Each time the SSH dies before a umount NFS, I have to umount -f and I get a crash (only sysrq works).
Actually, the crash occurs a few seconds after umount -f.It seems that killing SSH by hand does _not_ lead to crash. But a long network failure does.
I remember seeing this bug several times with all stable releases from 2.6.7 to 2.6.11. I didn't try with earlier versions.
Quoted 5 lines
I didn't see anything in the logs (after reboot). But I can't be sure there was nothing in dmesg since I didn't get a chance to chvt 1 and see console messages before rebooting (with sysrq).
Do you have any idea how to debug this ?
No clue, but a question: is this a hard or soft mount? Could you post your ssh and mount commands, munged as needed for security? That might give someone a clue.
I did this "back when" but I don't recall having a problem with it.
Signature · 3 lines
-bill davidsen ([email protected]) "The secret to procrastination is to put things off until the last possible moment - but no longer" -me
Quoted 14 lines · Bill Davidsen
Quoted 12 lines · Brice Goglin
Hi Trond,
I'm using NFS (v2) over TCP (in a SSH tunnel).
Each time the SSH dies before a umount NFS, I have to umount -f and I get a crash (only sysrq works).
Actually, the crash occurs a few seconds after umount -f.It seems that killing SSH by hand does _not_ lead to crash. But a long network failure does.
I remember seeing this bug several times with all stable releases from 2.6.7 to 2.6.11. I didn't try with earlier versions.
Quoted 8 lines
Quoted 5 lines
I didn't see anything in the logs (after reboot). But I can't be sure there was nothing in dmesg since I didn't get a chance to chvt 1 and see console messages before rebooting (with sysrq).
Do you have any idea how to debug this ?
No clue, but a question: is this a hard or soft mount? Could you post your ssh and mount commands, munged as needed for security? That might give someone a clue.
The ssh command is just
$ ssh kwad -L 2249:localhost:2049 -L 2248:localhost:870 -N -f (port is forwarded to 2249 while mountport if forwarded to 2248)
Options is /proc/mounts are rw,v2,rsize=8192,wsize=8192,hard,tcp,nolock,addr=localhost
I just had another network failure. I ran umount -f from vt1 to see kernel message. I waited for about 1 minute but didn't get any crash. So I switched back to X... and got the crash then.
Looks like this crash doesn't want me to see any message...
Brice
Quoted 11 lines · Brice Goglin
The ssh command is just
$ ssh kwad -L 2249:localhost:2049 -L 2248:localhost:870 -N -f (port is forwarded to 2249 while mountport if forwarded to 2248)Options is
/proc/mountsare rw,v2,rsize=8192,wsize=8192,hard,tcp,nolock,addr=localhostI just had another network failure. I ran umount -f from vt1 to see kernel message. I waited for about 1 minute but didn't get any crash. So I switched back to X... and got the crash then.
Looks like this crash doesn't want me to see any message...
I just got it through netconsole.
Unfortunatelly, the call trace doesn't appear.
Maybe the netconsole didn't have time send it before crashing. Hope this helps.
Brice
RPC: error 5 connecting to server localhost
RPC: error 5 connecting to server localhost
RPC: error 5 connecting to server localhost
RPC: error 5 connecting to server localhost
RPC: error 5 connecting to server localhost
Unable to handle kernel paging request at virtual address ffffff98 printing eip: e0aaa07a *pde = 00002067 *pte = 00000000 Oops: 0002 [#1]
PREEMPT
Modules linked in: netconsole sd_mod usb_storage vfat fat loop isofs
zlib_inflate nls_cp850 nls_iso8859_15 smbfs nfs lockd sunrpc i915 tun ipt_MASQUERADE iptable_nat ipt_state ip_conntrack iptable_filter ip_tables floppy uhci_hcd ehci_hcd dm_mod snd_intel8x0 snd_ac97_codec
CPU: 0 EIP: 0060:[<e0aaa07a>] Not tainted VLI EFLAGS: 00010297 (2.6.11=Macvin)
EIP is at rpc_wake_up_status+0x6a/0x80 [sunrpc]
eax: ffffff84 ebx: d0065888 ecx: 00000001 edx: c146e000
esi: fffffffb edi: d0065888 ebp: d0065800 esp: c146ef14
ds: 007b es: 007b ss: 0068
Process events/0 (pid: 3, threadinfo=c146e000 task=c1473020)
Stack: c146ef44 d0065800 00000283 fffffffb e0aa710e d0065888 fffffffb
00120dcb c1473184 00000000 d0065904
to den 05.05.2005 Klokka 12:17 (+0200) skreiv Brice Goglin:
Quoted 16 lines
Unable to handle kernel paging request at virtual address ffffff98 printing eip: e0aaa07a *pde = 00002067 *pte = 00000000 Oops: 0002 [#1]PREEMPT
Modules linked in: netconsole sd_mod usb_storage vfat fat loop isofs
zlib_inflatenls_cp850nls_iso8859_15smbfs nfs lockd sunrpc i915 tunipt_MASQUERADEiptable_natipt_stateip_conntrackiptable_filterip_tablesfloppyuhci_hcdehci_hcddm_modsnd_intel8x0snd_ac97_codecCPU: 0 EIP: 0060:[<e0aaa07a>] Not tainted VLI EFLAGS: 00010297 (2.6.11=Macvin)
Quoted 7 lines
EIP is at
rpc_wake_up_status+0x6a/0x80[sunrpc]
eax:ffffff84ebx:d0065888ecx: 00000001 edx:c146e000
esi: fffffffb edi:d0065888ebp:d0065800esp:c146ef14
ds: 007b es: 007b ss: 0068
Process events/0 (pid: 3, threadinfo=c146e000 task=c1473020)Stack: c146ef44 d0065800 00000283 fffffffb e0aa710e d0065888 fffffffb
00120dcbc147318400000000d0065904
Have you tried the attached patch? Andrew has already included it in the -mm series.
Cheers,
Trond
Make the socket transport kick the event queue to start socket connects immediately. This should improve responsiveness of applications that are sensitive to slow mount operations (like automounters).
We are now also careful to cancel the connect worker before destroying the xprt. This eliminates a race where xprt_destroy can finish before the connect worker is even allowed to run.
Test-plan:
Destructive testing (unplugging the network temporarily). Connectathon with UDP and TCP. Hard-code impossibly small connect timeout.
Version: Fri, 29 Apr 2005 15:32:01 -0400
---
net/sunrpc/xprt.c | 9 ++++++++- 1 files changed, 8 insertions(+), 1 deletion(-)
net/sunrpc/xprt.c +8 -1
@@ -569,8 +569,11 @@ void xprt_connect(struct rpc_task *task)if (xprt->sock !=NULL )schedule_delayed_work (&xprt->sock_connect,RPC_REESTABLISH_TIMEOUT ); -else +else {schedule_work (&xprt->sock_connect); +if (!RPC_IS_ASYNC (task)) +flush_scheduled_work (); + } }return ; out_write:
@@ -1666,6 +1669,10 @@ xprt_shutdown(struct rpc_xprt *xprt)rpc_wake_up (&xprt->backlog);wake_up (&xprt->cong_wait);del_timer_sync (&xprt->timer); + +/* synchronously wait for connect worker to finish */ +cancel_delayed_work (&xprt->sock_connect); +flush_scheduled_work (); }/*