more nfsv4 lockups w/ nexenta
Subsystems (2)
I have a Debian client running 2.6.27.5 connecting over nfsv4 to a Nexenta nfs server running b85 (and even with b103) that is experiencing nfs lockups. The symptoms are basically that nfs stops working (usually first noticed as Firefox locking up, but trying to log in as a user w/ an nfs mounted home dir hangs as well, as does trying to list said nfs mounted home directory). Trying a umount -f doesn't usually resolve the issue. Unfortunately there are no logs indicating what the issue is, so I've done some preliminary dumps. Here is the relevant portion of the process list:
3295 ? S< 24:55 \_ [rpciod/0] 3296 ? S< 46:14 \_ [rpciod/1] 3297 ? S< 55:17 \_ [rpciod/2] 3298 ? S< 19:01 \_ [rpciod/3] 3315 ? R< 185:15 \_ [nfsiod] 4464 ? D< 0:00 \_ [nfsv4-svc] 25203 ? S 0:00 \_ [pdflush] 26618 ? S 0:00 \_ [pdflush] 27343 ? R< 252:35 \_ [192.168.0.10-re] 27344 ? D 0:00 \_ [nfsv4-delegretu]
I've also put up a /var/log/messages with rpcdebug -m rpc -s all and rpcdebug -m nfs -s all turned on:
http://s120158928.onlinehome.us/messages.2.gz
And a tcpdump -s 0 -w nfs_dump_argento5 -x -i eth0:
http://s120158928.onlinehome.us/nfs_dump_argento5
Looking forward to debugging and resolving,
Thomas
A message that is not in the archive (sent off-list or not yet received): 9ef864c50812110940o7ab17a2dvc83ac7653ad6dd21-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 21 lines · Thomas Garner
I have a Debian client running 2.6.27.5 connecting over nfsv4 to a Nexenta nfs server running b85 (and even with b103) that is experiencing nfs lockups. The symptoms are basically that nfs stops working (usually first noticed as Firefox locking up, but trying to log in as a user w/ an nfs mounted home dir hangs as well, as does trying to list said nfs mounted home directory). Trying a
umount -fdoesn't usually resolve the issue. Unfortunately there are no logs indicating what the issue is, so I've done some preliminary dumps. Here is the relevant portion of the process list:3295 ? S< 24:55 \_ [rpciod/0] 3296 ? S< 46:14 \_ [rpciod/1] 3297 ? S< 55:17 \_ [rpciod/2] 3298 ? S< 19:01 \_ [rpciod/3] 3315 ? R< 185:15 \_ [nfsiod] 4464 ? D< 0:00 \_ [nfsv4-svc] 25203 ? S 0:00 \_ [pdflush] 26618 ? S 0:00 \_ [pdflush] 27343 ? R< 252:35 \_ [192.168.0.10-re] 27344 ? D 0:00 \_ [nfsv4-delegretu]
Quoted 8 lines
I've also put up a
/var/log/messageswithrpcdebug -m rpc -s allandrpcdebug -m nfs -s allturned on:http://s120158928.onlinehome.us/messages.2.gz
And a
tcpdump -s 0 -w nfs_dump_argento5 -x -i eth0:From the logs, it looks like it is recovery related. Did the server
reboot just before the hang?
Cheers
Trond
A message that is not in the archive (sent off-list or not yet received): 1229018528.6463.23.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
The logs in particular are from Nov, but a lockup today prompted the email. Today's event does not appear to have been due to a reboot or server restart (though I'm not as familiar with the intricacies of Sun's daemon management), as both seem to have been up since Dec 2:
[root@filer0 ~]# uptime
4:25pm up 9 day(s), 12:23, 2 users, load average: 0.26, 0.26, 0.27
[root@filer0 ~]# svcs nfs/server
STATE STIME FMRI online Dec_02 svc:/network/nfs/server:default
I can provide specific info from today (though I'll need to gather it). Just let me know.
Thomas
Quoted 36 lines · Trond Myklebust
Quoted 21 lines · Thomas Garner
I have a Debian client running 2.6.27.5 connecting over nfsv4 to a Nexenta nfs server running b85 (and even with b103) that is experiencing nfs lockups. The symptoms are basically that nfs stops working (usually first noticed as Firefox locking up, but trying to log in as a user w/ an nfs mounted home dir hangs as well, as does trying to list said nfs mounted home directory). Trying a
umount -fdoesn't usually resolve the issue. Unfortunately there are no logs indicating what the issue is, so I've done some preliminary dumps. Here is the relevant portion of the process list:3295 ? S< 24:55 \_ [rpciod/0] 3296 ? S< 46:14 \_ [rpciod/1] 3297 ? S< 55:17 \_ [rpciod/2] 3298 ? S< 19:01 \_ [rpciod/3] 3315 ? R< 185:15 \_ [nfsiod] 4464 ? D< 0:00 \_ [nfsv4-svc] 25203 ? S 0:00 \_ [pdflush] 26618 ? S 0:00 \_ [pdflush] 27343 ? R< 252:35 \_ [192.168.0.10-re] 27344 ? D 0:00 \_ [nfsv4-delegretu]Quoted 8 lines
I've also put up a
/var/log/messageswithrpcdebug -m rpc -s allandrpcdebug -m nfs -s allturned on:http://s120158928.onlinehome.us/messages.2.gz
And a
tcpdump -s 0 -w nfs_dump_argento5 -x -i eth0:From the logs, it looks like it is recovery related. Did the server
reboot just before the hang?
Cheers
Trond
A message that is not in the archive (sent off-list or not yet received): 9ef864c50812111330v1c9dc200s49dacb5504bbfeca-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
This has happened twice since I wrote last. More debugging from the last occurrence:
http://s120158928.onlinehome.us/messages.7.gz
http://s120158928.onlinehome.us/nfs_dump_argento7
Thomas
Quoted 55 lines · Thomas Garner
The logs in particular are from Nov, but a lockup today prompted the email. Today's event does not appear to have been due to a reboot or server restart (though I'm not as familiar with the intricacies of Sun's daemon management), as both seem to have been up since Dec 2:
[root@filer0 ~]# uptime
4:25pm up 9 day(s), 12:23, 2 users, load average: 0.26, 0.26, 0.27
[root@filer0 ~]# svcs nfs/serverSTATE STIME FMRI online Dec_02 svc:/network/nfs/server:defaultI can provide specific info from today (though I'll need to gather it). Just let me know.
Thomas
Quoted 36 lines · Trond Myklebust
Quoted 21 lines · Thomas Garner
I have a Debian client running 2.6.27.5 connecting over nfsv4 to a Nexenta nfs server running b85 (and even with b103) that is experiencing nfs lockups. The symptoms are basically that nfs stops working (usually first noticed as Firefox locking up, but trying to log in as a user w/ an nfs mounted home dir hangs as well, as does trying to list said nfs mounted home directory). Trying a
umount -fdoesn't usually resolve the issue. Unfortunately there are no logs indicating what the issue is, so I've done some preliminary dumps. Here is the relevant portion of the process list:3295 ? S< 24:55 \_ [rpciod/0] 3296 ? S< 46:14 \_ [rpciod/1] 3297 ? S< 55:17 \_ [rpciod/2] 3298 ? S< 19:01 \_ [rpciod/3] 3315 ? R< 185:15 \_ [nfsiod] 4464 ? D< 0:00 \_ [nfsv4-svc] 25203 ? S 0:00 \_ [pdflush] 26618 ? S 0:00 \_ [pdflush] 27343 ? R< 252:35 \_ [192.168.0.10-re] 27344 ? D 0:00 \_ [nfsv4-delegretu]Quoted 8 lines
I've also put up a
/var/log/messageswithrpcdebug -m rpc -s allandrpcdebug -m nfs -s allturned on:http://s120158928.onlinehome.us/messages.2.gz
And a
tcpdump -s 0 -w nfs_dump_argento5 -x -i eth0:From the logs, it looks like it is recovery related. Did the server
reboot just before the hang?
Cheers
Trond
A message that is not in the archive (sent off-list or not yet received): 9ef864c50812190834p7a63d572s3bd99f964254cc-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 6 lines · Thomas Garner
This has happened twice since I wrote last. More debugging from the last occurrence:
http://s120158928.onlinehome.us/messages.7.gz
http://s120158928.onlinehome.us/nfs_dump_argento7
It's the same thing. The WRITE requests are waiting for NFSv4 state recovery to finish.
As far as I can see from your tcpdump, there is no NFS network activity at all, so the recovery thread is obviously hung. What does 'echo t > /proc/sysrq-trigger' tell you that the thread is doing?
Quoted 58 lines
Thomas
Quoted 55 lines · Thomas Garner
The logs in particular are from Nov, but a lockup today prompted the email. Today's event does not appear to have been due to a reboot or server restart (though I'm not as familiar with the intricacies of Sun's daemon management), as both seem to have been up since Dec 2:
[root@filer0 ~]# uptime
4:25pm up 9 day(s), 12:23, 2 users, load average: 0.26, 0.26, 0.27
[root@filer0 ~]# svcs nfs/serverSTATE STIME FMRI online Dec_02 svc:/network/nfs/server:defaultI can provide specific info from today (though I'll need to gather it). Just let me know.
Thomas
Quoted 36 lines · Trond Myklebust
Quoted 21 lines · Thomas Garner
I have a Debian client running 2.6.27.5 connecting over nfsv4 to a Nexenta nfs server running b85 (and even with b103) that is experiencing nfs lockups. The symptoms are basically that nfs stops working (usually first noticed as Firefox locking up, but trying to log in as a user w/ an nfs mounted home dir hangs as well, as does trying to list said nfs mounted home directory). Trying a
umount -fdoesn't usually resolve the issue. Unfortunately there are no logs indicating what the issue is, so I've done some preliminary dumps. Here is the relevant portion of the process list:3295 ? S< 24:55 \_ [rpciod/0] 3296 ? S< 46:14 \_ [rpciod/1] 3297 ? S< 55:17 \_ [rpciod/2] 3298 ? S< 19:01 \_ [rpciod/3] 3315 ? R< 185:15 \_ [nfsiod] 4464 ? D< 0:00 \_ [nfsv4-svc] 25203 ? S 0:00 \_ [pdflush] 26618 ? S 0:00 \_ [pdflush] 27343 ? R< 252:35 \_ [192.168.0.10-re] 27344 ? D 0:00 \_ [nfsv4-delegretu]Quoted 8 lines
I've also put up a
/var/log/messageswithrpcdebug -m rpc -s allandrpcdebug -m nfs -s allturned on:http://s120158928.onlinehome.us/messages.2.gz
And a
tcpdump -s 0 -w nfs_dump_argento5 -x -i eth0:From the logs, it looks like it is recovery related. Did the server
reboot just before the hang?
Cheers
Trond
A message that is not in the archive (sent off-list or not yet received): 1229710786.7789.138.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 3 lines
As far as I can see from your tcpdump, there is no NFS network activity at all, so the recovery thread is obviously hung. What does 'echo t >
/proc/sysrq-trigger' tell you that the thread is doing?
Here you go:
A message that is not in the archive (sent off-list or not yet received): 9ef864c50812191036s35f30e5bia33b8024fec4d42e-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 7 lines · Thomas Garner
Quoted 3 lines
As far as I can see from your tcpdump, there is no NFS network activity at all, so the recovery thread is obviously hung. What does 'echo t >
/proc/sysrq-trigger' tell you that the thread is doing?Here you go:
OK. It looks as if that thread is hanging waiting for some other thread to release the sequence lock. However, I couldn't see anything other than a firefox session actually in the NFS code in your sysrq trace. That thread couldn't holding the sequence lock, since it was actually doing those WRITE calls that figure in the above messages.
The syslog trace seems incomplete, though. Can you see more information if you use
dmesg -s 90000
Trond
A message that is not in the archive (sent off-list or not yet received): 1229713906.7789.159.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
I'm not super familiar with how to get dmesg to print out absolutely everything if it's missing something. I tried dmesg -s 900000 > messages_dmesg.7 after running another sysrq, but this still looks to me to still be cut off:
http://s120158928.onlinehome.us/messages_dmesg.7.gz
Quoted 6 lines
The syslog trace seems incomplete, though. Can you see more information if you use
dmesg -s 90000Trond