2.6.27.14 breakage when grace period expires
Subsystems (2)
I'm sorry to inform you but... it seems that there is a similar problem in the NLM subsystem as reported previously but this time it is triggered when the grace time expires after a reboot.
Client and server run 2.6.27.14 + previous fix, NFSv3.
On the client there are three shells running:
while :; do lck -w /mnt/foo 2; done
The "lck" program is the same as posted before and it obtains an exclusive write lock then waits 2 seconds in above invocation (there's probably an "fcntl" command equivalent). After an orderly server reboot + grace time expiration one of above command loops reports:
lck: fcntl: No locks available
and all three get stuck. After ^C-ing all "lck" loops the server still shows an entry in /proc/locks which causes the file to be locked indefinately. Maybe two loops are sufficient to reproduce the issue or maybe you need more, I don't know.
Interestingly, during the grace time at least one of the "lck" processes should have re-obtained the lock but it didn't show up in /proc/locks on the server.
Interestingly (#2), after removing the file on the server (i.e. no sillyrename) the now free inode is still locked according to /proc/locks. Even stopping/starting /etc/init.d/nfs-kernel-server plus "echo 3 >/proc/sys/vm/drop_caches" did not remove the lock (it did re-enter grace).
Signature · 1 line
Frank
Quoted 6 lines · Frank van Maarseveen
I'm sorry to inform you but... it seems that there is a similar problem in the NLM subsystem as reported previously but this time it is triggered when the grace time expires after a reboot.
Client and server run 2.6.27.14 + previous fix, NFSv3.
Quoted 7 lines
On the client there are three shells running:
while :; do lck -w
/mnt/foo2; doneThe "lck" program is the same as posted before and it obtains an exclusive write lock then waits 2 seconds in above invocation (there's probably an "fcntl" command equivalent). After an orderly server reboot + grace time
How are you rebooting the server?
--b.
Quoted 25 lines
expiration one of above command loops reports:
lck: fcntl: No locks available
and all three get stuck. After ^C-ing all "lck" loops the server still shows an entry in
/proc/lockswhich causes the file to be locked indefinately. Maybe two loops are sufficient to reproduce the issue or maybe you need more, I don't know.Interestingly, during the grace time at least one of the "lck" processes should have re-obtained the lock but it didn't show up in
/proc/lockson the server.Interestingly (#2), after removing the file on the server (i.e. no sillyrename) the now free inode is still locked according to
/proc/locks. Even stopping/starting/etc/init.d/nfs-kernel-serverplus "echo 3 >/proc/sys/vm/drop_caches" did not remove the lock (it did re-enter grace).Signature · 5 lines
Frank -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · J. Bruce Fields
Quoted 8 lines · Frank van Maarseveen
I'm sorry to inform you but... it seems that there is a similar problem in the NLM subsystem as reported previously but this time it is triggered when the grace time expires after a reboot.
Client and server run 2.6.27.14 + previous fix, NFSv3.
On the client there are three shells running:
Quoted 7 lines
Quoted 5 lines
while :; do lck -w
/mnt/foo2; doneThe "lck" program is the same as posted before and it obtains an exclusive write lock then waits 2 seconds in above invocation (there's probably an "fcntl" command equivalent). After an orderly server reboot + grace time
How are you rebooting the server?
"reboot"
Quoted 28 lines
--b.
Quoted 25 lines
expiration one of above command loops reports:
lck: fcntl: No locks available
and all three get stuck. After ^C-ing all "lck" loops the server still shows an entry in
/proc/lockswhich causes the file to be locked indefinately. Maybe two loops are sufficient to reproduce the issue or maybe you need more, I don't know.Interestingly, during the grace time at least one of the "lck" processes should have re-obtained the lock but it didn't show up in
/proc/lockson the server.Interestingly (#2), after removing the file on the server (i.e. no sillyrename) the now free inode is still locked according to
/proc/locks. Even stopping/starting/etc/init.d/nfs-kernel-serverplus "echo 3 >/proc/sys/vm/drop_caches" did not remove the lock (it did re-enter grace).Signature · 5 lines
Frank -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 1 line
Frank
Quoted 12 lines · Frank van Maarseveen
Quoted 11 lines · J. Bruce Fields
Quoted 10 lines · Frank van Maarseveen
I'm sorry to inform you but... it seems that there is a similar problem in the NLM subsystem as reported previously but this time it is triggered when the grace time expires after a reboot.
Client and server run 2.6.27.14 + previous fix, NFSv3.
On the client there are three shells running:
while :; do lck -w
/mnt/foo2; done
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
The "lck" program is the same as posted before and it obtains an exclusive write lock then waits 2 seconds in above invocation (there's probably an "fcntl" command equivalent). After an orderly server reboot + grace time
How are you rebooting the server?
"reboot"
Could you watch the nfs/nlm/nsm traffic on reboot and make sure that the server is actually sending the reboot notification to the client, and that the client is trying to reclaim? (Wireshark should make this all fairly clear. But capture the traffic with tcpdump -s0 -wtmp.pcap and send it to me if you're having trouble interpreting it.)
--b.
Quoted 32 lines
Quoted 29 lines
--b.
Quoted 25 lines
expiration one of above command loops reports:
lck: fcntl: No locks available
and all three get stuck. After ^C-ing all "lck" loops the server still shows an entry in
/proc/lockswhich causes the file to be locked indefinately. Maybe two loops are sufficient to reproduce the issue or maybe you need more, I don't know.Interestingly, during the grace time at least one of the "lck" processes should have re-obtained the lock but it didn't show up in
/proc/lockson the server.Interestingly (#2), after removing the file on the server (i.e. no sillyrename) the now free inode is still locked according to
/proc/locks. Even stopping/starting/etc/init.d/nfs-kernel-serverplus "echo 3 >/proc/sys/vm/drop_caches" did not remove the lock (it did re-enter grace).Signature · 5 lines
Frank -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlSignature · 1 line
Frank
Quoted 19 lines · J. Bruce Fields
Quoted 18 lines · Frank van Maarseveen
Quoted 16 lines · J. Bruce Fields
Quoted 6 lines · Frank van Maarseveen
I'm sorry to inform you but... it seems that there is a similar problem in the NLM subsystem as reported previously but this time it is triggered when the grace time expires after a reboot.
Client and server run 2.6.27.14 + previous fix, NFSv3.
Quoted 7 lines
On the client there are three shells running:
while :; do lck -w
/mnt/foo2; doneThe "lck" program is the same as posted before and it obtains an exclusive write lock then waits 2 seconds in above invocation (there's probably an "fcntl" command equivalent). After an orderly server reboot + grace time
How are you rebooting the server?
Quoted 7 lines
Quoted 1 line
"reboot"
Could you watch the nfs/nlm/nsm traffic on reboot and make sure that the server is actually sending the reboot notification to the client, and that the client is trying to reclaim? (Wireshark should make this all fairly clear. But capture the traffic with tcpdump -s0 -wtmp.pcap and send it to me if you're having trouble interpreting it.)
Can't try it right now but tomorrow I can. However, I'm pretty sure at least the reboot notification is there because:
1) The issue happens too in a totally different NFS server setup which
by definition invokes sm-notify in a script. This is the real use case.
2) If not, then I would expect different behavior anyway compared to
what I saw. A lost reboot notification is always possible but in that case the client(s) might end up holding more locks than the server, not the other way around as it is right now.
I'll make a capture.
Signature · 1 line
Frank
Quoted 19 lines · J. Bruce Fields
Quoted 18 lines · Frank van Maarseveen
Quoted 16 lines · J. Bruce Fields
Quoted 6 lines · Frank van Maarseveen
I'm sorry to inform you but... it seems that there is a similar problem in the NLM subsystem as reported previously but this time it is triggered when the grace time expires after a reboot.
Client and server run 2.6.27.14 + previous fix, NFSv3.
Quoted 7 lines
On the client there are three shells running:
while :; do lck -w
/mnt/foo2; doneThe "lck" program is the same as posted before and it obtains an exclusive write lock then waits 2 seconds in above invocation (there's probably an "fcntl" command equivalent). After an orderly server reboot + grace time
How are you rebooting the server?
Quoted 7 lines
Quoted 1 line
"reboot"
Could you watch the nfs/nlm/nsm traffic on reboot and make sure that the server is actually sending the reboot notification to the client, and that the client is trying to reclaim? (Wireshark should make this all fairly clear. But capture the traffic with tcpdump -s0 -wtmp.pcap and send it to me if you're having trouble interpreting it.)
I have a capture with comment below. It raised so many questions that I decided to do some more testing, trying to figure out how it looks when the locking works. This issue now appears to predate the fuse changes and is also present when both client and server run 2.6.24.4. I decided to stick with the traffic capture for 2.7.27.14 + previous fix as discussed earlier. The full capture is available at
http://www.frankvm.com/tmp/2.6.27.14-nlm-grace.pcap. It's about 33k and was started on the server as part of initscripts, right after the reboot and filtered on client IP address.
Exported by wireshark (filter: nfs or stat or nlm) and condensed:
# time src prot 1 0.000000 client: NFS V3 GETATTR Call (Reply In 42), FH:0x0308030a 2 0.000018 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 5 0.000583 server: ICMP Destination unreachable (Port unreachable) 6 0.000589 server: ICMP Destination unreachable (Port unreachable) 7 1.891277 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 8 1.891320 server: ICMP Destination unreachable (Port unreachable) 9 5.827053 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 10 5.827119 server: ICMP Destination unreachable (Port unreachable) 11 14.626501 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 12 14.626587 server: ICMP Destination unreachable (Port unreachable) 15 15.726426 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 16 15.726505 server: ICMP Destination unreachable (Port unreachable) 17 17.926284 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 18 17.926368 server: ICMP Destination unreachable (Port unreachable) 25 22.326006 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 26 22.326090 server: ICMP Destination unreachable (Port unreachable) 35 30.022271 client: NLM V4 UNLOCK Call (Reply In 36) FH:0xcafa61cc svid:114 pos:0-0 36 30.029511 server: NLM V4 UNLOCK Reply (Call In 35) NLM_DENIED_GRACE_PERIOD 37 30.029660 client: NLM V4 LOCK Call (Reply In 39) FH:0xcafa61cc svid:116 pos:0-0 38 30.029691 client: NLM V4 LOCK Call (Reply In 40) FH:0xcafa61cc svid:115 pos:0-0 39 30.029884 server: NLM V4 LOCK Reply (Call In 37) NLM_DENIED_GRACE_PERIOD 40 30.029914 server: NLM V4 LOCK Reply (Call In 38) NLM_DENIED_GRACE_PERIOD 41 31.125403 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 42 31.127499 server: NFS V3 GETATTR Reply (Call In 1) Directory mode:0755 uid:0 gid:0 43 31.127942 client: NFS V3 GETATTR Call (Reply In 45), FH:0x0308030a 45 31.129378 server: NFS V3 GETATTR Reply (Call In 43) Directory mode:0755 uid:0 gid:0 47 31.129958 server: STAT V1 NOTIFY Call (Reply In 48) 48 31.130301 client: STAT V1 NOTIFY Reply (Call In 47)
Reboot notification ok.
51 35.029968 client: NLM V4 UNLOCK Call (Reply In 54) FH:0xcafa61cc svid:114 pos:0-0 52 35.030003 client: NLM V4 LOCK Call (Reply In 55) FH:0xcafa61cc svid:116 pos:0-0 53 35.030016 client: NLM V4 LOCK Call (Reply In 56) FH:0xcafa61cc svid:115 pos:0-0 54 35.030085 server: NLM V4 UNLOCK Reply (Call In 51) NLM_DENIED_GRACE_PERIOD 55 35.030126 server: NLM V4 LOCK Reply (Call In 52) NLM_DENIED_GRACE_PERIOD 56 35.030153 server: NLM V4 LOCK Reply (Call In 53) NLM_DENIED_GRACE_PERIOD
The three contending client processes. I don't see a lock registration for svid:114, only UNLOCK calls which fail with NLM_DENIED_GRACE_PERIOD. The above goes on for a while. Neither the server or client shows any lock in /proc/locks at this point.
166 115.028376 client: NLM V4 LOCK Call (Reply In 168) FH:0xcafa61cc svid:115 pos:0-0 167 115.028394 client: NLM V4 LOCK Call (Reply In 169) FH:0xcafa61cc svid:116 pos:0-0 168 115.028440 server: NLM V4 LOCK Reply (Call In 166) NLM_DENIED_GRACE_PERIOD 169 115.028465 server: NLM V4 LOCK Reply (Call In 167) NLM_DENIED_GRACE_PERIOD 170 120.027233 client: NLM V4 UNLOCK Call (Reply In 171) FH:0xcafa61cc svid:114 pos:0-0 171 120.027337 server: NLM V4 UNLOCK Reply (Call In 170) NLM_DENIED_GRACE_PERIOD 172 120.028234 client: NLM V4 LOCK Call (Reply In 175) FH:0xcafa61cc svid:116 pos:0-0 173 120.028258 client: NLM V4 LOCK Call (Reply In 174) FH:0xcafa61cc svid:115 pos:0-0 174 120.030601 server: NLM V4 LOCK Reply (Call In 173) 175 120.030656 server: NLM V4 LOCK Reply (Call In 172) NLM_BLOCKED
This doesn't add up. There hasn't been a successful unlock for svid:114 (see #213 for that) but still one of the locks is granted.
176 120.030781 client: NLM V4 LOCK Call (Reply In 177) FH:0xcafa61cc svid:115 pos:0-0 177 120.030849 server: NLM V4 LOCK Reply (Call In 176)
Strange: an identical lock request but with a different rpc xid (i.e. no packet duplication).
178 120.031078 client: NFS V3 GETATTR Call (Reply In 179), FH:0xcafa61cc 179 120.031154 server: NFS V3 GETATTR Reply (Call In 178) Regular File mode:0644 uid:363 gid:1500 180 120.033973 client: NFS V3 ACCESS Call (Reply In 181), FH:0x0308030a 181 120.034030 server: NFS V3 ACCESS Reply (Call In 180) 182 120.034223 client: NFS V3 LOOKUP Call (Reply In 183), DH:0x0308030a/loc 183 120.034285 server: NFS V3 LOOKUP Reply (Call In 182), FH:0x81685ca0 184 120.034472 client: NFS V3 ACCESS Call (Reply In 185), FH:0x0308030c 185 120.034526 server: NFS V3 ACCESS Reply (Call In 184) 186 120.034722 client: NFS V3 ACCESS Call (Reply In 187), FH:0x0308030c 187 120.034776 server: NFS V3 ACCESS Reply (Call In 186) 188 120.034922 client: NFS V3 LOOKUP Call (Reply In 189), DH:0x0308030c/locktest 189 120.034993 server: NFS V3 LOOKUP Reply (Call In 188), FH:0xcafa61cc 190 120.035172 client: NFS V3 ACCESS Call (Reply In 191), FH:0xcafa61cc 191 120.035230 server: NFS V3 ACCESS Reply (Call In 190) 193 122.032218 client: NLM V4 UNLOCK Call (Reply In 195) FH:0xcafa61cc svid:115 pos:0-0 194 122.032253 client: NLM V4 LOCK Call (Reply In 197) FH:0xcafa61cc svid:119 pos:0-0 195 122.032343 server: NLM V4 UNLOCK Reply (Call In 193) 197 122.032794 server: NLM V4 LOCK Reply (Call In 194) NLM_BLOCKED 201 122.033767 server: NLM V4 GRANTED_MSG Call (Reply In 202) FH:0xcafa61cc svid:116 pos:0-0 202 122.034066 client: NLM V4 GRANTED_MSG Reply (Call In 201) 205 122.034665 client: NLM V4 GRANTED_RES Call (Reply In 206) NLM_DENIED 206 122.034753 server: NLM V4 GRANTED_RES Reply (Call In 205) 207 122.036312 client: NFS V3 GETATTR Call (Reply In 208), FH:0xcafa61cc 208 122.036394 server: NFS V3 GETATTR Reply (Call In 207) Regular File mode:0644 uid:363 gid:1500 209 122.036611 client: NLM V4 LOCK Call (Reply In 210) FH:0xcafa61cc svid:120 pos:0-0 210 122.036674 server: NLM V4 LOCK Reply (Call In 209) NLM_BLOCKED 213 125.027091 client: NLM V4 UNLOCK Call (Reply In 214) FH:0xcafa61cc svid:114 pos:0-0 214 125.027194 server: NLM V4 UNLOCK Reply (Call In 213) 215 125.029487 client: NFS V3 GETATTR Call (Reply In 216), FH:0xcafa61cc 216 125.029570 server: NFS V3 GETATTR Reply (Call In 215) Regular File mode:0644 uid:363 gid:1500 217 125.029836 client: NLM V4 LOCK Call (Reply In 218) FH:0xcafa61cc svid:121 pos:0-0 218 125.029895 server: NLM V4 LOCK Reply (Call In 217) NLM_BLOCKED 224 152.032157 client: NLM V4 LOCK Call (Reply In 225) FH:0xcafa61cc svid:119 pos:0-0 225 152.032283 server: NLM V4 LOCK Reply (Call In 224) NLM_BLOCKED 226 152.035103 client: NLM V4 LOCK Call (Reply In 227) FH:0xcafa61cc svid:120 pos:0-0 227 152.035157 server: NLM V4 LOCK Reply (Call In 226) NLM_BLOCKED 230 155.029676 client: NLM V4 LOCK Call (Reply In 231) FH:0xcafa61cc svid:121 pos:0-0 231 155.029761 server: NLM V4 LOCK Reply (Call In 230) NLM_BLOCKED
To recap the problem: one of the fcntl calls to obtain a write lock returns
lck: fcntl: No locks available
shortly after the grace period expires. After that everything gets stuck, server holding a write lock with no corresponding client side lock.
IMO looks like the client is to blame, even if/when the server should/could have accepted UNLOCK during grace (I don't know, I'm not an expert on that one).
Signature · 1 line
Frank
Quoted 95 lines · Frank van Maarseveen
Quoted 19 lines · J. Bruce Fields
Quoted 18 lines · Frank van Maarseveen
Quoted 16 lines · J. Bruce Fields
Quoted 6 lines · Frank van Maarseveen
I'm sorry to inform you but... it seems that there is a similar problem in the NLM subsystem as reported previously but this time it is triggered when the grace time expires after a reboot.
Client and server run 2.6.27.14 + previous fix, NFSv3.
Quoted 7 lines
On the client there are three shells running:
while :; do lck -w
/mnt/foo2; doneThe "lck" program is the same as posted before and it obtains an exclusive write lock then waits 2 seconds in above invocation (there's probably an "fcntl" command equivalent). After an orderly server reboot + grace time
How are you rebooting the server?
Quoted 7 lines
Quoted 1 line
"reboot"
Could you watch the nfs/nlm/nsm traffic on reboot and make sure that the server is actually sending the reboot notification to the client, and that the client is trying to reclaim? (Wireshark should make this all fairly clear. But capture the traffic with tcpdump -s0 -wtmp.pcap and send it to me if you're having trouble interpreting it.)
I have a capture with comment below. It raised so many questions that I decided to do some more testing, trying to figure out how it looks when the locking works. This issue now appears to predate the fuse changes and is also present when both client and server run 2.6.24.4. I decided to stick with the traffic capture for 2.7.27.14 + previous fix as discussed earlier. The full capture is available at
http://www.frankvm.com/tmp/2.6.27.14-nlm-grace.pcap. It's about 33k and was started on the server as part of initscripts, right after the reboot and filtered on client IP address.Exported by wireshark (filter: nfs or stat or nlm) and condensed:
# time src prot 1 0.000000 client: NFS V3 GETATTR Call (Reply In 42), FH:0x0308030a 2 0.000018 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 5 0.000583 server: ICMP Destination unreachable (Port unreachable) 6 0.000589 server: ICMP Destination unreachable (Port unreachable) 7 1.891277 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 8 1.891320 server: ICMP Destination unreachable (Port unreachable) 9 5.827053 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 10 5.827119 server: ICMP Destination unreachable (Port unreachable) 11 14.626501 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 12 14.626587 server: ICMP Destination unreachable (Port unreachable) 15 15.726426 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 16 15.726505 server: ICMP Destination unreachable (Port unreachable) 17 17.926284 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 18 17.926368 server: ICMP Destination unreachable (Port unreachable) 25 22.326006 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 26 22.326090 server: ICMP Destination unreachable (Port unreachable) 35 30.022271 client: NLM V4 UNLOCK Call (Reply In 36) FH:0xcafa61cc svid:114 pos:0-0 36 30.029511 server: NLM V4 UNLOCK Reply (Call In 35) NLM_DENIED_GRACE_PERIOD 37 30.029660 client: NLM V4 LOCK Call (Reply In 39) FH:0xcafa61cc svid:116 pos:0-0 38 30.029691 client: NLM V4 LOCK Call (Reply In 40) FH:0xcafa61cc svid:115 pos:0-0 39 30.029884 server: NLM V4 LOCK Reply (Call In 37) NLM_DENIED_GRACE_PERIOD 40 30.029914 server: NLM V4 LOCK Reply (Call In 38) NLM_DENIED_GRACE_PERIOD 41 31.125403 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 42 31.127499 server: NFS V3 GETATTR Reply (Call In 1) Directory mode:0755 uid:0 gid:0 43 31.127942 client: NFS V3 GETATTR Call (Reply In 45), FH:0x0308030a 45 31.129378 server: NFS V3 GETATTR Reply (Call In 43) Directory mode:0755 uid:0 gid:0 47 31.129958 server: STAT V1 NOTIFY Call (Reply In 48) 48 31.130301 client: STAT V1 NOTIFY Reply (Call In 47)Reboot notification ok.
51 35.029968 client: NLM V4 UNLOCK Call (Reply In 54) FH:0xcafa61cc svid:114 pos:0-0 52 35.030003 client: NLM V4 LOCK Call (Reply In 55) FH:0xcafa61cc svid:116 pos:0-0 53 35.030016 client: NLM V4 LOCK Call (Reply In 56) FH:0xcafa61cc svid:115 pos:0-0 54 35.030085 server: NLM V4 UNLOCK Reply (Call In 51) NLM_DENIED_GRACE_PERIOD 55 35.030126 server: NLM V4 LOCK Reply (Call In 52) NLM_DENIED_GRACE_PERIOD 56 35.030153 server: NLM V4 LOCK Reply (Call In 53) NLM_DENIED_GRACE_PERIODThe three contending client processes. I don't see a lock registration for svid:114, only UNLOCK calls which fail with
NLM_DENIED_GRACE_PERIOD. The above goes on for a while. Neither the server or client shows any lock in/proc/locksat this point.166 115.028376 client: NLM V4 LOCK Call (Reply In 168) FH:0xcafa61cc svid:115 pos:0-0 167 115.028394 client: NLM V4 LOCK Call (Reply In 169) FH:0xcafa61cc svid:116 pos:0-0 168 115.028440 server: NLM V4 LOCK Reply (Call In 166) NLM_DENIED_GRACE_PERIOD 169 115.028465 server: NLM V4 LOCK Reply (Call In 167) NLM_DENIED_GRACE_PERIOD 170 120.027233 client: NLM V4 UNLOCK Call (Reply In 171) FH:0xcafa61cc svid:114 pos:0-0 171 120.027337 server: NLM V4 UNLOCK Reply (Call In 170) NLM_DENIED_GRACE_PERIOD 172 120.028234 client: NLM V4 LOCK Call (Reply In 175) FH:0xcafa61cc svid:116 pos:0-0 173 120.028258 client: NLM V4 LOCK Call (Reply In 174) FH:0xcafa61cc svid:115 pos:0-0 174 120.030601 server: NLM V4 LOCK Reply (Call In 173) 175 120.030656 server: NLM V4 LOCK Reply (Call In 172) NLM_BLOCKED
Quoted 2 lines
This doesn't add up. There hasn't been a successful unlock for svid:114 (see #213 for that) but still one of the locks is granted.
Has the lock for svid:114 been attempted recovered by the client? If not, then the server has no knowledge of that lock.
Quoted 5 lines
176 120.030781 client: NLM V4 LOCK Call (Reply In 177) FH:0xcafa61cc svid:115 pos:0-0 177 120.030849 server: NLM V4 LOCK Reply (Call In 176)Strange: an identical lock request but with a different rpc xid (i.e. no packet duplication).
No. That would be the non-blocking lock that is intended as a 'ping' to see if the server is still alive. It duplicates the blocking lock in all details except that the 'block' flag is not set.
Quoted 14 lines
178 120.031078 client: NFS V3 GETATTR Call (Reply In 179), FH:0xcafa61cc 179 120.031154 server: NFS V3 GETATTR Reply (Call In 178) Regular File mode:0644 uid:363 gid:1500 180 120.033973 client: NFS V3 ACCESS Call (Reply In 181), FH:0x0308030a 181 120.034030 server: NFS V3 ACCESS Reply (Call In 180) 182 120.034223 client: NFS V3 LOOKUP Call (Reply In 183), DH:0x0308030a/loc 183 120.034285 server: NFS V3 LOOKUP Reply (Call In 182), FH:0x81685ca0 184 120.034472 client: NFS V3 ACCESS Call (Reply In 185), FH:0x0308030c 185 120.034526 server: NFS V3 ACCESS Reply (Call In 184) 186 120.034722 client: NFS V3 ACCESS Call (Reply In 187), FH:0x0308030c 187 120.034776 server: NFS V3 ACCESS Reply (Call In 186) 188 120.034922 client: NFS V3 LOOKUP Call (Reply In 189), DH:0x0308030c/locktest 189 120.034993 server: NFS V3 LOOKUP Reply (Call In 188), FH:0xcafa61cc 190 120.035172 client: NFS V3 ACCESS Call (Reply In 191), FH:0xcafa61cc 191 120.035230 server: NFS V3 ACCESS Reply (Call In 190)
Quoted 8 lines
193 122.032218 client: NLM V4 UNLOCK Call (Reply In 195) FH:0xcafa61cc svid:115 pos:0-0 194 122.032253 client: NLM V4 LOCK Call (Reply In 197) FH:0xcafa61cc svid:119 pos:0-0 195 122.032343 server: NLM V4 UNLOCK Reply (Call In 193) 197 122.032794 server: NLM V4 LOCK Reply (Call In 194) NLM_BLOCKED 201 122.033767 server: NLM V4 GRANTED_MSG Call (Reply In 202) FH:0xcafa61cc svid:116 pos:0-0 202 122.034066 client: NLM V4 GRANTED_MSG Reply (Call In 201) 205 122.034665 client: NLM V4 GRANTED_RES Call (Reply In 206) NLM_DENIED 206 122.034753 server: NLM V4 GRANTED_RES Reply (Call In 205)
What happened here? Why did the client refuse the lock for svid 116?
Did the task get signalled? If so, where is the CANCEL request?
Quoted 22 lines
207 122.036312 client: NFS V3 GETATTR Call (Reply In 208), FH:0xcafa61cc 208 122.036394 server: NFS V3 GETATTR Reply (Call In 207) Regular File mode:0644 uid:363 gid:1500 209 122.036611 client: NLM V4 LOCK Call (Reply In 210) FH:0xcafa61cc svid:120 pos:0-0 210 122.036674 server: NLM V4 LOCK Reply (Call In 209) NLM_BLOCKED 213 125.027091 client: NLM V4 UNLOCK Call (Reply In 214) FH:0xcafa61cc svid:114 pos:0-0 214 125.027194 server: NLM V4 UNLOCK Reply (Call In 213) 215 125.029487 client: NFS V3 GETATTR Call (Reply In 216), FH:0xcafa61cc 216 125.029570 server: NFS V3 GETATTR Reply (Call In 215) Regular File mode:0644 uid:363 gid:1500 217 125.029836 client: NLM V4 LOCK Call (Reply In 218) FH:0xcafa61cc svid:121 pos:0-0 218 125.029895 server: NLM V4 LOCK Reply (Call In 217) NLM_BLOCKED 224 152.032157 client: NLM V4 LOCK Call (Reply In 225) FH:0xcafa61cc svid:119 pos:0-0 225 152.032283 server: NLM V4 LOCK Reply (Call In 224) NLM_BLOCKED 226 152.035103 client: NLM V4 LOCK Call (Reply In 227) FH:0xcafa61cc svid:120 pos:0-0 227 152.035157 server: NLM V4 LOCK Reply (Call In 226) NLM_BLOCKED 230 155.029676 client: NLM V4 LOCK Call (Reply In 231) FH:0xcafa61cc svid:121 pos:0-0 231 155.029761 server: NLM V4 LOCK Reply (Call In 230) NLM_BLOCKEDTo recap the problem: one of the fcntl calls to obtain a write lock returns
lck: fcntl: No locks available
Quoted 7 lines
shortly after the grace period expires. After that everything gets stuck, server holding a write lock with no corresponding client side lock.
IMO looks like the client is to blame, even if/when the server should/could have accepted UNLOCK during grace (I don't know, I'm not an expert on that one).
Possibly... It depends entirely on what happened to cause it to deny the GRANTED callback...
Trond
Quoted 96 lines · Trond Myklebust
Quoted 95 lines · Frank van Maarseveen
Quoted 19 lines · J. Bruce Fields
Quoted 18 lines · Frank van Maarseveen
Quoted 16 lines · J. Bruce Fields
Quoted 6 lines · Frank van Maarseveen
I'm sorry to inform you but... it seems that there is a similar problem in the NLM subsystem as reported previously but this time it is triggered when the grace time expires after a reboot.
Client and server run 2.6.27.14 + previous fix, NFSv3.
Quoted 7 lines
On the client there are three shells running:
while :; do lck -w
/mnt/foo2; doneThe "lck" program is the same as posted before and it obtains an exclusive write lock then waits 2 seconds in above invocation (there's probably an "fcntl" command equivalent). After an orderly server reboot + grace time
How are you rebooting the server?
Quoted 7 lines
Quoted 1 line
"reboot"
Could you watch the nfs/nlm/nsm traffic on reboot and make sure that the server is actually sending the reboot notification to the client, and that the client is trying to reclaim? (Wireshark should make this all fairly clear. But capture the traffic with tcpdump -s0 -wtmp.pcap and send it to me if you're having trouble interpreting it.)
I have a capture with comment below. It raised so many questions that I decided to do some more testing, trying to figure out how it looks when the locking works. This issue now appears to predate the fuse changes and is also present when both client and server run 2.6.24.4. I decided to stick with the traffic capture for 2.7.27.14 + previous fix as discussed earlier. The full capture is available at
http://www.frankvm.com/tmp/2.6.27.14-nlm-grace.pcap. It's about 33k and was started on the server as part of initscripts, right after the reboot and filtered on client IP address.Exported by wireshark (filter: nfs or stat or nlm) and condensed:
# time src prot 1 0.000000 client: NFS V3 GETATTR Call (Reply In 42), FH:0x0308030a 2 0.000018 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 5 0.000583 server: ICMP Destination unreachable (Port unreachable) 6 0.000589 server: ICMP Destination unreachable (Port unreachable) 7 1.891277 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 8 1.891320 server: ICMP Destination unreachable (Port unreachable) 9 5.827053 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 10 5.827119 server: ICMP Destination unreachable (Port unreachable) 11 14.626501 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 12 14.626587 server: ICMP Destination unreachable (Port unreachable) 15 15.726426 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 16 15.726505 server: ICMP Destination unreachable (Port unreachable) 17 17.926284 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 18 17.926368 server: ICMP Destination unreachable (Port unreachable) 25 22.326006 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 26 22.326090 server: ICMP Destination unreachable (Port unreachable) 35 30.022271 client: NLM V4 UNLOCK Call (Reply In 36) FH:0xcafa61cc svid:114 pos:0-0 36 30.029511 server: NLM V4 UNLOCK Reply (Call In 35) NLM_DENIED_GRACE_PERIOD 37 30.029660 client: NLM V4 LOCK Call (Reply In 39) FH:0xcafa61cc svid:116 pos:0-0 38 30.029691 client: NLM V4 LOCK Call (Reply In 40) FH:0xcafa61cc svid:115 pos:0-0 39 30.029884 server: NLM V4 LOCK Reply (Call In 37) NLM_DENIED_GRACE_PERIOD 40 30.029914 server: NLM V4 LOCK Reply (Call In 38) NLM_DENIED_GRACE_PERIOD 41 31.125403 client: NFS [RPC retransmission of #1]V3 GETATTR Call (Reply In 42), FH:0x0308030a 42 31.127499 server: NFS V3 GETATTR Reply (Call In 1) Directory mode:0755 uid:0 gid:0 43 31.127942 client: NFS V3 GETATTR Call (Reply In 45), FH:0x0308030a 45 31.129378 server: NFS V3 GETATTR Reply (Call In 43) Directory mode:0755 uid:0 gid:0 47 31.129958 server: STAT V1 NOTIFY Call (Reply In 48) 48 31.130301 client: STAT V1 NOTIFY Reply (Call In 47)Reboot notification ok.
51 35.029968 client: NLM V4 UNLOCK Call (Reply In 54) FH:0xcafa61cc svid:114 pos:0-0 52 35.030003 client: NLM V4 LOCK Call (Reply In 55) FH:0xcafa61cc svid:116 pos:0-0 53 35.030016 client: NLM V4 LOCK Call (Reply In 56) FH:0xcafa61cc svid:115 pos:0-0 54 35.030085 server: NLM V4 UNLOCK Reply (Call In 51) NLM_DENIED_GRACE_PERIOD 55 35.030126 server: NLM V4 LOCK Reply (Call In 52) NLM_DENIED_GRACE_PERIOD 56 35.030153 server: NLM V4 LOCK Reply (Call In 53) NLM_DENIED_GRACE_PERIODThe three contending client processes. I don't see a lock registration for svid:114, only UNLOCK calls which fail with
NLM_DENIED_GRACE_PERIOD. The above goes on for a while. Neither the server or client shows any lock in/proc/locksat this point.166 115.028376 client: NLM V4 LOCK Call (Reply In 168) FH:0xcafa61cc svid:115 pos:0-0 167 115.028394 client: NLM V4 LOCK Call (Reply In 169) FH:0xcafa61cc svid:116 pos:0-0 168 115.028440 server: NLM V4 LOCK Reply (Call In 166) NLM_DENIED_GRACE_PERIOD 169 115.028465 server: NLM V4 LOCK Reply (Call In 167) NLM_DENIED_GRACE_PERIOD 170 120.027233 client: NLM V4 UNLOCK Call (Reply In 171) FH:0xcafa61cc svid:114 pos:0-0 171 120.027337 server: NLM V4 UNLOCK Reply (Call In 170) NLM_DENIED_GRACE_PERIOD 172 120.028234 client: NLM V4 LOCK Call (Reply In 175) FH:0xcafa61cc svid:116 pos:0-0 173 120.028258 client: NLM V4 LOCK Call (Reply In 174) FH:0xcafa61cc svid:115 pos:0-0 174 120.030601 server: NLM V4 LOCK Reply (Call In 173) 175 120.030656 server: NLM V4 LOCK Reply (Call In 172) NLM_BLOCKED
Quoted 5 lines
Quoted 2 lines
This doesn't add up. There hasn't been a successful unlock for svid:114 (see #213 for that) but still one of the locks is granted.
Has the lock for svid:114 been attempted recovered by the client? If not, then the server has no knowledge of that lock.
exactly. Apparently the client tries to unlock an unrecovered lock.
Quoted 34 lines
Quoted 6 lines
176 120.030781 client: NLM V4 LOCK Call (Reply In 177) FH:0xcafa61cc svid:115 pos:0-0 177 120.030849 server: NLM V4 LOCK Reply (Call In 176)Strange: an identical lock request but with a different rpc xid (i.e. no packet duplication).
No. That would be the non-blocking lock that is intended as a 'ping' to see if the server is still alive. It duplicates the blocking lock in all details except that the 'block' flag is not set.
Quoted 14 lines
178 120.031078 client: NFS V3 GETATTR Call (Reply In 179), FH:0xcafa61cc 179 120.031154 server: NFS V3 GETATTR Reply (Call In 178) Regular File mode:0644 uid:363 gid:1500 180 120.033973 client: NFS V3 ACCESS Call (Reply In 181), FH:0x0308030a 181 120.034030 server: NFS V3 ACCESS Reply (Call In 180) 182 120.034223 client: NFS V3 LOOKUP Call (Reply In 183), DH:0x0308030a/loc 183 120.034285 server: NFS V3 LOOKUP Reply (Call In 182), FH:0x81685ca0 184 120.034472 client: NFS V3 ACCESS Call (Reply In 185), FH:0x0308030c 185 120.034526 server: NFS V3 ACCESS Reply (Call In 184) 186 120.034722 client: NFS V3 ACCESS Call (Reply In 187), FH:0x0308030c 187 120.034776 server: NFS V3 ACCESS Reply (Call In 186) 188 120.034922 client: NFS V3 LOOKUP Call (Reply In 189), DH:0x0308030c/locktest 189 120.034993 server: NFS V3 LOOKUP Reply (Call In 188), FH:0xcafa61cc 190 120.035172 client: NFS V3 ACCESS Call (Reply In 191), FH:0xcafa61cc 191 120.035230 server: NFS V3 ACCESS Reply (Call In 190)Quoted 8 lines
193 122.032218 client: NLM V4 UNLOCK Call (Reply In 195) FH:0xcafa61cc svid:115 pos:0-0 194 122.032253 client: NLM V4 LOCK Call (Reply In 197) FH:0xcafa61cc svid:119 pos:0-0 195 122.032343 server: NLM V4 UNLOCK Reply (Call In 193) 197 122.032794 server: NLM V4 LOCK Reply (Call In 194) NLM_BLOCKED 201 122.033767 server: NLM V4 GRANTED_MSG Call (Reply In 202) FH:0xcafa61cc svid:116 pos:0-0 202 122.034066 client: NLM V4 GRANTED_MSG Reply (Call In 201) 205 122.034665 client: NLM V4 GRANTED_RES Call (Reply In 206) NLM_DENIED 206 122.034753 server: NLM V4 GRANTED_RES Reply (Call In 205)
Quoted 3 lines
What happened here? Why did the client refuse the lock for svid 116?
Did the task get signalled? If so, where is the CANCEL request?
The task did not get signaled, there is no CANCEL.
Quoted 27 lines
Quoted 27 lines
207 122.036312 client: NFS V3 GETATTR Call (Reply In 208), FH:0xcafa61cc 208 122.036394 server: NFS V3 GETATTR Reply (Call In 207) Regular File mode:0644 uid:363 gid:1500 209 122.036611 client: NLM V4 LOCK Call (Reply In 210) FH:0xcafa61cc svid:120 pos:0-0 210 122.036674 server: NLM V4 LOCK Reply (Call In 209) NLM_BLOCKED 213 125.027091 client: NLM V4 UNLOCK Call (Reply In 214) FH:0xcafa61cc svid:114 pos:0-0 214 125.027194 server: NLM V4 UNLOCK Reply (Call In 213) 215 125.029487 client: NFS V3 GETATTR Call (Reply In 216), FH:0xcafa61cc 216 125.029570 server: NFS V3 GETATTR Reply (Call In 215) Regular File mode:0644 uid:363 gid:1500 217 125.029836 client: NLM V4 LOCK Call (Reply In 218) FH:0xcafa61cc svid:121 pos:0-0 218 125.029895 server: NLM V4 LOCK Reply (Call In 217) NLM_BLOCKED 224 152.032157 client: NLM V4 LOCK Call (Reply In 225) FH:0xcafa61cc svid:119 pos:0-0 225 152.032283 server: NLM V4 LOCK Reply (Call In 224) NLM_BLOCKED 226 152.035103 client: NLM V4 LOCK Call (Reply In 227) FH:0xcafa61cc svid:120 pos:0-0 227 152.035157 server: NLM V4 LOCK Reply (Call In 226) NLM_BLOCKED 230 155.029676 client: NLM V4 LOCK Call (Reply In 231) FH:0xcafa61cc svid:121 pos:0-0 231 155.029761 server: NLM V4 LOCK Reply (Call In 230) NLM_BLOCKEDTo recap the problem: one of the fcntl calls to obtain a write lock returns
lck: fcntl: No locks available
shortly after the grace period expires. After that everything gets stuck, server holding a write lock with no corresponding client side lock.
Quoted 6 lines
Quoted 3 lines
IMO looks like the client is to blame, even if/when the server should/could have accepted UNLOCK during grace (I don't know, I'm not an expert on that one).
Possibly... It depends entirely on what happened to cause it to deny the GRANTED callback...
A little theorizing:
If the unlock of a yet unrecovered lock has failed up to that point then the client sure must remember the lock somehow. That might explain the secondary error when a conflicting lock is granted by the server.
Signature · 1 line
Frank
Quoted 4 lines · Frank van Maarseveen
A little theorizing:
If the unlock of a yet unrecovered lock has failed up to that point then the client sure must remember the lock somehow. That might explain the secondary error when a conflicting lock is granted by the server.
Sorry, but that doesn't hold water. The client will release the VFS 'mirror' of the lock before it attempts to unlock. Otherwise, you could have some nasty races between the unlock thread and the recovery thread...
Besides, the granted callback handler on the client only checks the list of blocked locks for a match.
Oh, bugger, I know what this is... It's the same thing that happened to the NFSv4 callback server. If you compile with CONFIG_IPV6 or CONFIG_IPV6_MODULE enabled, and also set CONFIG_SUNRPC_REGISTER_V4, then the NLM server will listen on an IPv6 socket, and so the RPC request come in with their IPv4 address mapped into the IPv6 namespace. The client, on the other hand, is using an IPv4 socket, 'cos you specified an IPv4 address to the mount command.
The result is that the call to nlm_cmp_addr() in nlmclnt_grant() always fails...
Basically, we need to replace nlm_cmp_addr() with something akin to nfs_sockaddr_match_ipaddr(), which will compare v4 mapped addresses.
The workaround should be simply to turn off CONFIG_SUNRPC_REGISTER_V4 if you're not planning on ever using NFS-over-IPv6...
Cheers
Trond
Quoted 6 lines · Trond Myklebust
Quoted 4 lines · Frank van Maarseveen
A little theorizing:
If the unlock of a yet unrecovered lock has failed up to that point then the client sure must remember the lock somehow. That might explain the secondary error when a conflicting lock is granted by the server.
Quoted 6 lines
Sorry, but that doesn't hold water. The client will release the VFS 'mirror' of the lock before it attempts to unlock. Otherwise, you could have some nasty races between the unlock thread and the recovery thread...
Besides, the granted callback handler on the client only checks the list of blocked locks for a match.
ok, then we have more than one NLM bug to resolve.
Quoted 6 lines
Oh, bugger, I know what this is... It's the same thing that happened to the NFSv4 callback server. If you compile with
CONFIG_IPV6orCONFIG_IPV6_MODULEenabled, and also setCONFIG_SUNRPC_REGISTER_V4, then the NLM server will listen on an IPv6 socket, and so the RPC request come in with their IPv4 address mapped into the IPv6 namespace.
Nope:
$ zgrep IPV6 /proc/config.gz # CONFIG_IPV6 is not set $ zgrep SUNRPC /proc/config.gz CONFIG_SUNRPC=y CONFIG_SUNRPC_GSS=y # CONFIG_SUNRPC_BIND34 is not set
And remember this is not a recent regression.
Signature · 1 line
Frank
Quoted 23 lines · Frank van Maarseveen
Quoted 6 lines · Trond Myklebust
Quoted 4 lines · Frank van Maarseveen
A little theorizing:
If the unlock of a yet unrecovered lock has failed up to that point then the client sure must remember the lock somehow. That might explain the secondary error when a conflicting lock is granted by the server.Quoted 6 lines
Sorry, but that doesn't hold water. The client will release the VFS 'mirror' of the lock before it attempts to unlock. Otherwise, you could have some nasty races between the unlock thread and the recovery thread...
Besides, the granted callback handler on the client only checks the list of blocked locks for a match.ok, then we have more than one NLM bug to resolve.
Quoted 6 lines
Oh, bugger, I know what this is... It's the same thing that happened to the NFSv4 callback server. If you compile with
CONFIG_IPV6orCONFIG_IPV6_MODULEenabled, and also setCONFIG_SUNRPC_REGISTER_V4, then the NLM server will listen on an IPv6 socket, and so the RPC request come in with their IPv4 address mapped into the IPv6 namespace.
Quoted 8 lines
Nope:
$ zgrep IPV6 /proc/config.gz # CONFIG_IPV6 is not set $ zgrep SUNRPC /proc/config.gz CONFIG_SUNRPC=y CONFIG_SUNRPC_GSS=y # CONFIG_SUNRPC_BIND34 is not set
Sorry, yes... 2.6.27.x should be OK. The lockd v4mapped addresses bug is specific to 2.6.29. Chuck, are you planning on fixing this before 2.6.29-final comes out?
Quoted 1 line
And remember this is not a recent regression.
It would help if you sent us the full binary tcpdump, instead of just the summary. That should enable us to figure out which of the tests is failing in nlmclnt_grant().
Trond
Quoted 37 lines · Trond Myklebust
Quoted 23 lines · Frank van Maarseveen
Quoted 6 lines · Trond Myklebust
Quoted 4 lines · Frank van Maarseveen
A little theorizing:
If the unlock of a yet unrecovered lock has failed up to that point then the client sure must remember the lock somehow. That might explain the secondary error when a conflicting lock is granted by the server.Quoted 6 lines
Sorry, but that doesn't hold water. The client will release the VFS 'mirror' of the lock before it attempts to unlock. Otherwise, you could have some nasty races between the unlock thread and the recovery thread...
Besides, the granted callback handler on the client only checks the list of blocked locks for a match.ok, then we have more than one NLM bug to resolve.
Quoted 6 lines
Oh, bugger, I know what this is... It's the same thing that happened to the NFSv4 callback server. If you compile with
CONFIG_IPV6orCONFIG_IPV6_MODULEenabled, and also setCONFIG_SUNRPC_REGISTER_V4, then the NLM server will listen on an IPv6 socket, and so the RPC request come in with their IPv4 address mapped into the IPv6 namespace.Quoted 8 lines
Nope:
$ zgrep IPV6 /proc/config.gz # CONFIG_IPV6 is not set $ zgrep SUNRPC /proc/config.gz CONFIG_SUNRPC=y CONFIG_SUNRPC_GSS=y # CONFIG_SUNRPC_BIND34 is not setSorry, yes... 2.6.27.x should be OK. The lockd v4mapped addresses bug is specific to 2.6.29. Chuck, are you planning on fixing this before 2.6.29-final comes out?
Quoted 5 lines
Quoted 1 line
And remember this is not a recent regression.
It would help if you sent us the full binary tcpdump, instead of just the summary. That should enable us to figure out which of the tests is failing in
nlmclnt_grant().
I posted the link already. Anyway, see attachment.
Signature · 1 line
Frank
Quoted 33 lines · Trond Myklebust
Quoted 23 lines · Frank van Maarseveen
Quoted 6 lines · Trond Myklebust
Quoted 4 lines · Frank van Maarseveen
A little theorizing:
If the unlock of a yet unrecovered lock has failed up to that point then the client sure must remember the lock somehow. That might explain the secondary error when a conflicting lock is granted by the server.Quoted 6 lines
Sorry, but that doesn't hold water. The client will release the VFS 'mirror' of the lock before it attempts to unlock. Otherwise, you could have some nasty races between the unlock thread and the recovery thread...
Besides, the granted callback handler on the client only checks the list of blocked locks for a match.ok, then we have more than one NLM bug to resolve.
Quoted 6 lines
Oh, bugger, I know what this is... It's the same thing that happened to the NFSv4 callback server. If you compile with
CONFIG_IPV6orCONFIG_IPV6_MODULEenabled, and also setCONFIG_SUNRPC_REGISTER_V4, then the NLM server will listen on an IPv6 socket, and so the RPC request come in with their IPv4 address mapped into the IPv6 namespace.Quoted 8 lines
Nope:
$ zgrep IPV6 /proc/config.gz # CONFIG_IPV6 is not set $ zgrep SUNRPC /proc/config.gz CONFIG_SUNRPC=y CONFIG_SUNRPC_GSS=y # CONFIG_SUNRPC_BIND34 is not set
Quoted 3 lines
Sorry, yes... 2.6.27.x should be OK. The lockd v4mapped addresses bug is specific to 2.6.29. Chuck, are you planning on fixing this before 2.6.29-final comes out?
I wasn't sure exactly where the compared addresses came from. I had assumed that they all came through the listener, so we wouldn't need this kind of translation. It shouldn't be difficult to map addresses passed in via nlmclnt_init() to AF_INET6.
But this is the kind of thing that makes "falling back" to an AF_INET listener a little challenging. We will have to record what flavor the listener is and do a translation depending on what listener family was actually created.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 9 lines · Chuck Lever
I wasn't sure exactly where the compared addresses came from. I had assumed that they all came through the listener, so we wouldn't need this kind of translation. It shouldn't be difficult to map addresses passed in via
nlmclnt_init()toAF_INET6.But this is the kind of thing that makes "falling back" to an
AF_INETlistener a little challenging. We will have to record what flavor the listener is and do a translation depending on what listener family was actually created.
Why? Should we care whether we're receiving IPv4 addresses or IPv6 v4-mapped addresses? They're the same thing...
We're already doing the mapping for the NFSv4 callback channel. See nfs_sockaddr_match_ipaddr() in fs/nfs/client.c
Trond
Quoted 6 lines · Trond Myklebust
Quoted 5 lines · Chuck Lever
I wasn't sure exactly where the compared addresses came from. I had assumed that they all came through the listener, so we wouldn't need this kind of translation. It shouldn't be difficult to map addresses passed in via
nlmclnt_init()toAF_INET6.
Quoted 7 lines
Quoted 4 lines
But this is the kind of thing that makes "falling back" to an
AF_INETlistener a little challenging. We will have to record what flavor the listener is and do a translation depending on what listener family was actually created.Why? Should we care whether we're receiving IPv4 addresses or IPv6 v4-mapped addresses? They're the same thing...
The problem is the listener family is now decided at run-time. If an AF_INET6 listener can't be created, an AF_INET listener is created instead, even if CONFIG_IPV6 || CONFIG_IPV6_MODULE is enabled. If an AF_INET listener is created, we get only IPv4 addresses in svc_rqst->rq_addr.
So we can do it either way. Taking lockd as an example:
1. Have nlmclnt_init() map AF_INET mount addresses to AF_INET6 iff the lockd listener is AF_INET6, so nlm_cmp_addr() is always dealing with AF_INET6 in this case, or
2. If CONFIG_IPV6 || CONFIG_IPV6_MODULE, unconditionally map AF_INET addresses in nlmclnt_init and for incoming NLM requests (when lockd happens to have fallen back to an AF_INET listener)
Personally I think solution 1. will be less confusing operationally and less invasive code-wise. I suppose IPv6 purists would prefer keeping the whole stack in AF_INET6, so they would like solution 2.
Eventually we could map incoming addresses on AF_INET listeners in the RPC server code, but I prefer to wait until all kernel RPC services have IPv6 support.
Since 2.6.29 has the CONFIG_SUNRPC_REGISTER_V4=N workaround, do we need to fix 2.6.29, or can this wait until 2.6.30?
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 38 lines · Frank van Maarseveen
Quoted 37 lines · Trond Myklebust
Quoted 23 lines · Frank van Maarseveen
Quoted 6 lines · Trond Myklebust
Quoted 4 lines · Frank van Maarseveen
A little theorizing:
If the unlock of a yet unrecovered lock has failed up to that point then the client sure must remember the lock somehow. That might explain the secondary error when a conflicting lock is granted by the server.Quoted 6 lines
Sorry, but that doesn't hold water. The client will release the VFS 'mirror' of the lock before it attempts to unlock. Otherwise, you could have some nasty races between the unlock thread and the recovery thread...
Besides, the granted callback handler on the client only checks the list of blocked locks for a match.ok, then we have more than one NLM bug to resolve.
Quoted 6 lines
Oh, bugger, I know what this is... It's the same thing that happened to the NFSv4 callback server. If you compile with
CONFIG_IPV6orCONFIG_IPV6_MODULEenabled, and also setCONFIG_SUNRPC_REGISTER_V4, then the NLM server will listen on an IPv6 socket, and so the RPC request come in with their IPv4 address mapped into the IPv6 namespace.Quoted 8 lines
Nope:
$ zgrep IPV6 /proc/config.gz # CONFIG_IPV6 is not set $ zgrep SUNRPC /proc/config.gz CONFIG_SUNRPC=y CONFIG_SUNRPC_GSS=y # CONFIG_SUNRPC_BIND34 is not setSorry, yes... 2.6.27.x should be OK. The lockd v4mapped addresses bug is specific to 2.6.29. Chuck, are you planning on fixing this before 2.6.29-final comes out?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
And remember this is not a recent regression.
It would help if you sent us the full binary tcpdump, instead of just the summary. That should enable us to figure out which of the tests is failing in
nlmclnt_grant().I posted the link already. Anyway, see attachment.
Yeah... It looks alright. The one thing that looks a bit odd is the GRANTED lock has a 'caller_name' field that is set to the name of the server. I pretty sure we don't care about that, though...
Hmm... I wonder if the problem isn't just that we're failing to cancel the lock request when the process is signalled. Can you try the following patch?
--------------------------------------------------------------------
From: Trond Myklebust <[email protected]>
NLM/lockd: Always cancel blocked locks when exiting early from nlmclnt_lock
---
fs/lockd/clntproc.c | 9 +++++++-- 1 files changed, 7 insertions(+), 2 deletions(-)
fs/lockd/clntproc.c +7 -2
@@ -542,9 +542,14 @@ again: status =nlmclnt_call (cred, req,NLMPROC_LOCK );if (status <0 )break ; -/* Did a reclaimer thread notify us of a server reboot? */ -if (resp->status == nlm_lck_denied_grace_period) +/* Is the server in a grace period state? +* If so, we need to reset the resp->status, and +* retry... +*/ +if (resp->status == nlm_lck_denied_grace_period) { + resp->status = nlm_lck_blocked;continue ; + }if (resp->status != nlm_lck_blocked)break ;/* Wait on an NLM blocking lock */
Quoted 14 lines · Chuck Lever
Quoted 13 lines · Trond Myklebust
Quoted 11 lines · Chuck Lever
I wasn't sure exactly where the compared addresses came from. I had assumed that they all came through the listener, so we wouldn't need this kind of translation. It shouldn't be difficult to map addresses passed in via
nlmclnt_init()toAF_INET6.But this is the kind of thing that makes "falling back" to an
AF_INETlistener a little challenging. We will have to record what flavor the
listener is and do a translation depending on what listener family was
actually created.
Quoted 8 lines
Quoted 2 lines
Why? Should we care whether we're receiving IPv4 addresses or IPv6 v4-mapped addresses? They're the same thing...
The problem is the listener family is now decided at run-time. If an
AF_INET6listener can't be created, anAF_INETlistener is created instead, even ifCONFIG_IPV6||CONFIG_IPV6_MODULEis enabled. If anAF_INETlistener is created, we get only IPv4 addresses in svc_rqst-Quoted 1 line
rq_addr.
You're missing my point. Why should we care if it's one or the other? In the NFSv4 case, we v4map all IPv4 addresses _unconditionally_ if it turns out that CONFIG_IPV6 is enabled.
IOW: we always compare IPv6 addresses.
Trond
Quoted 22 lines · Trond Myklebust
Quoted 21 lines · Chuck Lever
Quoted 13 lines · Trond Myklebust
Quoted 11 lines · Chuck Lever
I wasn't sure exactly where the compared addresses came from. I had assumed that they all came through the listener, so we wouldn't need this kind of translation. It shouldn't be difficult to map addresses passed in via
nlmclnt_init()toAF_INET6.But this is the kind of thing that makes "falling back" to an
AF_INETlistener a little challenging. We will have to record what flavor the
listener is and do a translation depending on what listener family was
actually created.Quoted 2 lines
Why? Should we care whether we're receiving IPv4 addresses or IPv6 v4-mapped addresses? They're the same thing...
The problem is the listener family is now decided at run-time. If an
AF_INET6listener can't be created, anAF_INETlistener is created instead, even ifCONFIG_IPV6||CONFIG_IPV6_MODULEis enabled. If anAF_INETlistener is created, we get only IPv4 addresses in svc_rqst-
Quoted 7 lines
Quoted 1 line
Quoted 1 line
rq_addr.You're missing my point. Why should we care if it's one or the other? In the NFSv4 case, we v4map all IPv4 addresses _unconditionally_ if it turns out that
CONFIG_IPV6is enabled.IOW: we always compare IPv6 addresses.
The reason we might care in this case is nlm_cmp_addr() is executed more frequently than nfs_sockaddr_match_ipaddr().
Mapping the server address in nlmclnt_init() means we translate the server address once and are done with it. We never have to map incoming AF_INET addresses in NLM requests, and we don't have the extra conditionals every time we go through nlm_cmp_addr().
This keeps nlm_cmp_addr() as simple as it can be: it compares only two AF_INET addresses or two AF_INET6 addresses.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 39 lines · Chuck Lever
Quoted 26 lines · Trond Myklebust
Quoted 25 lines · Chuck Lever
Quoted 17 lines · Trond Myklebust
Quoted 8 lines · Chuck Lever
I wasn't sure exactly where the compared addresses came from. I had
assumed that they all came through the listener, so we wouldn't need
this kind of translation. It shouldn't be difficult to map addresses
passed in vianlmclnt_init()toAF_INET6.Quoted 7 lines
But this is the kind of thing that makes "falling back" to an
AF_INET
listener a little challenging. We will have to record what flavor the
listener is and do a translation depending on what listener family was
actually created.Quoted 2 lines
Why? Should we care whether we're receiving IPv4 addresses or IPv6 v4-mapped addresses? They're the same thing...
The problem is the listener family is now decided at run-time. If an
AF_INET6listener can't be created, anAF_INETlistener is created instead, even ifCONFIG_IPV6||CONFIG_IPV6_MODULEis enabled. If anAF_INETlistener is created, we get only IPv4 addresses in svc_rqst-Quoted 8 lines
Quoted 1 line
Quoted 1 line
rq_addr.You're missing my point. Why should we care if it's one or the other? In
the NFSv4 case, we v4map all IPv4 addresses _unconditionally_ if it turns out thatCONFIG_IPV6is enabled.IOW: we always compare IPv6 addresses.
The reason we might care in this case is
nlm_cmp_addr()is executed more frequently thannfs_sockaddr_match_ipaddr().
Quoted 7 lines
Mapping the server address in
nlmclnt_init()means we translate the server address once and are done with it. We never have to map incomingAF_INETaddresses in NLM requests, and we don't have the extra conditionals every time we go throughnlm_cmp_addr().This keeps
nlm_cmp_addr()as simple as it can be: it compares only twoAF_INETaddresses or twoAF_INET6addresses.
I don't see how that changes the general principle. All it means is that you should be caching v4 mapped addresses instead of ipv4 addresses. That would allow you to simplify nlm_cmp_addr() even further...
Trond
Quoted 45 lines · Trond Myklebust
Quoted 44 lines · Chuck Lever
Quoted 26 lines · Trond Myklebust
Quoted 25 lines · Chuck Lever
Quoted 17 lines · Trond Myklebust
Quoted 8 lines · Chuck Lever
I wasn't sure exactly where the compared addresses came from. I had
assumed that they all came through the listener, so we wouldn't need
this kind of translation. It shouldn't be difficult to map addresses
passed in vianlmclnt_init()toAF_INET6.Quoted 7 lines
But this is the kind of thing that makes "falling back" to an
AF_INET
listener a little challenging. We will have to record what flavor the
listener is and do a translation depending on what listener family was
actually created.Quoted 2 lines
Why? Should we care whether we're receiving IPv4 addresses or IPv6 v4-mapped addresses? They're the same thing...
The problem is the listener family is now decided at run-time. If an
AF_INET6listener can't be created, anAF_INETlistener is created instead, even ifCONFIG_IPV6||CONFIG_IPV6_MODULEis enabled. If anAF_INETlistener is created, we get only IPv4 addresses in svc_rqst-Quoted 8 lines
Quoted 1 line
Quoted 1 line
rq_addr.You're missing my point. Why should we care if it's one or the other? In
the NFSv4 case, we v4map all IPv4 addresses _unconditionally_ if it turns out thatCONFIG_IPV6is enabled.IOW: we always compare IPv6 addresses.
The reason we might care in this case is
nlm_cmp_addr()is executed more frequently thannfs_sockaddr_match_ipaddr().Mapping the server address in
nlmclnt_init()means we translate the server address once and are done with it. We never have to map incomingAF_INETaddresses in NLM requests, and we don't have the extra conditionals every time we go throughnlm_cmp_addr().
Quoted 6 lines
Quoted 2 lines
This keeps
nlm_cmp_addr()as simple as it can be: it compares only twoAF_INETaddresses or twoAF_INET6addresses.I don't see how that changes the general principle. All it means is that you should be caching v4 mapped addresses instead of ipv4 addresses. That would allow you to simplify
nlm_cmp_addr()even further...
Operationally we have to support both AF_INET and AF_INET6 addresses in the cache, because we don't know what kind of lockd listener can be created until runtime. So, I can't see how we can eliminate the AF_INET arm in nlm_cmp_addr() unless we unconditionally convert all incoming AF_INET addresses from putative PF_INET listeners _and_ convert incoming IPv4 server addresses in NFS mount requests to AF_INET6.
Doesn't that add computational overhead to a fairly common case?
This goes away if we ensure that the address family of the server address passed to nlmclnt_lookup_host() always matches the protocol family of lockd's listener sockets. Then address mapping overhead is entirely removed from the common cases involving PF_INET listeners.
For PF_INET6 listeners, incoming IPv4 addresses are already mapped by the underlying network layer. Nothing can be done about that. But we can make sure the address family of the server address passed to nlmclnt_lookup_host() matches the incoming mapped addresses to eliminate the need for nlm_cmp_addr() to do the mapping every time it wants to compare an address.
It should be fairly simple to record the listener's protocol family, check it against incoming server addresses in nlmclnt_init(), then map the address as needed.
Having nlm_cmp_addr() do the mapping solves some problems, but at the cost of extra CPU time every time it is called; each loop iteration in nlm_lookup_host() for example. All I'm doing is removing a loop invariant, essentially.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 2 lines · Chuck Lever
The reason we might care in this case is
nlm_cmp_addr()is executed more frequently thannfs_sockaddr_match_ipaddr().
Actually, I'm not sure this assertion is correct. The only users of nlm_cmp_addr() are nlmclnt_grant(), nlm_lookup_host() and nlmsvc_unlock_all_by_ip().
AFAICS, the only one that needs to be v4 mapped should be nlmclnt_grant, which is not in a performance critical path...
Quoted 4 lines · Chuck Lever
Having
nlm_cmp_addr()do the mapping solves some problems, but at the cost of extra CPU time every time it is called; each loop iteration innlm_lookup_host()for example. All I'm doing is removing a loop invariant, essentially.
nlm_lookup_host() shouldn't need to compare v4 mapped addresses and IPv4 addresses afaics.
Quoted 10 lines · Trond Myklebust
Quoted 2 lines · Chuck Lever
The reason we might care in this case is
nlm_cmp_addr()is executed more frequently thannfs_sockaddr_match_ipaddr().Actually, I'm not sure this assertion is correct. The only users of
nlm_cmp_addr()arenlmclnt_grant(),nlm_lookup_host()andnlmsvc_unlock_all_by_ip().AFAICS, the only one that needs to be v4 mapped should be
nlmclnt_grant, which is not in a performance critical path...
So then your proposal is to ensure the two arguments of the nlm_cmp_addr() callsite in nlmclnt_grant() are both AF_INET6?
That doesn't sound so bad.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 9 lines · Chuck Lever
Quoted 8 lines · Trond Myklebust
Quoted 2 lines · Chuck Lever
The reason we might care in this case is
nlm_cmp_addr()is executed more frequently thannfs_sockaddr_match_ipaddr().Actually, I'm not sure this assertion is correct. The only users of
nlm_cmp_addr()arenlmclnt_grant(),nlm_lookup_host()andnlmsvc_unlock_all_by_ip().
Quoted 6 lines
Quoted 3 lines
AFAICS, the only one that needs to be v4 mapped should be
nlmclnt_grant,
which is not in a performance critical path...So then your proposal is to ensure the two arguments of the
nlm_cmp_addr()callsite innlmclnt_grant()are bothAF_INET6?
Yup... I can't see that the other two callsites need anything like that.
Quoted 27 lines · Trond Myklebust
Hmm... I wonder if the problem isn't just that we're failing to cancel the lock request when the process is signalled. Can you try the following patch?
--------------------------------------------------------------------
From: Trond Myklebust <[email protected]>
NLM/lockd: Always cancel blocked locks when exiting early fromnlmclnt_lockSigned-off-by Trond Myklebust <[email protected]>---
fs/lockd/clntproc.c | 9 +++++++-- 1 files changed, 7 insertions(+), 2 deletions(-)fs/lockd/clntproc.c +1
-2@@ -542,9 +542,14 @@ again: status =nlmclnt_call (cred, req,NLMPROC_LOCK );if (status <0 )break ; -/* Did a reclaimer thread notify us of a server reboot? */ -if (resp->status == nlm_lck_denied_grace_period) +/* Is the server in a grace period state?
fs/lockd/clntproc.c again: · line 555
Quoted 10 lines
fs/lockd/clntproc.c +6
-0+* If so, we need to reset the resp->status, and +* retry... +*/ +if (resp->status == nlm_lck_denied_grace_period) { + resp->status = nlm_lck_blocked;continue ; + }if (resp->status != nlm_lck_blocked)break ;/* Wait on an NLM blocking lock */
Patch tried but didn't make any difference. Note that there isn't any ^C or any other signal involved. The client runs three loops in the shell
while :; do lck -w /mnt/locktest 2; done
and every "lck" opens the file, obtains an exclusive write lock (waits if necessary), calls sleep(2), closes the fd (releasing the lock) and goes exit.
The "lck" which ends up unlocking during grace terminates normally but one of the others gets a "fcntl: No locks available" when trying to obtain the lock.
Question: shouldn't the server drop the lock after a sequence like:
201 122.033767 server: NLM V4 GRANTED_MSG Call (Reply In 202) FH:0xcafa61cc svid:116 pos:0-0 202 122.034066 client: NLM V4 GRANTED_MSG Reply (Call In 201) 205 122.034665 client: NLM V4 GRANTED_RES Call (Reply In 206) NLM_DENIED 206 122.034753 server: NLM V4 GRANTED_RES Reply (Call In 205)
?
Signature · 1 line
Frank