2.6.5-pre TCP connect problems
Subsystems (2)
Hi,
I'm currently debugging a problem with TCP reconnects in 2.6.5-pre where the TCP reconnect code got rewritten to use worker queues. What happens is that the NFS server drops the connection immediately and that state change isn't propagated to the transport. The typical sequence of events goes like this
Mar 29 15:44:00 e36 kernel: RPC: 28611 call_transmit (status 0) Mar 29 15:44:00 e36 kernel: RPC: 28611 xprt_prepare_transmit Mar 29 15:44:00 e36 kernel: RPC: 28611 call_status (status -107) Mar 29 15:44:00 e36 kernel: RPC: 28611 call_bind xprt d42c4200 is not connected Mar 29 15:44:00 e36 kernel: RPC: 28611 call_connect status 0 Mar 29 15:44:00 e36 kernel: RPC: 28611 xprt_connect xprt d42c4200 is not connected Mar 29 15:44:00 e36 kernel: RPC: disconnected transport d42c4200 Mar 29 15:44:00 e36 kernel: RPC: xprt_create_socket(tcp 6) Mar 29 15:44:00 e36 kernel: RPC: d42c4200 connect status 115 connected 0 sock state 2 Mar 29 15:44:00 e36 kernel: RPC: tcp_state_change client d42c4200... Mar 29 15:44:00 e36 kernel: RPC: state 1 conn 0 dead 0 zapped 0 Mar 29 15:44:00 e36 kernel: RPC: tcp_state_change client d42c4200... Mar 29 15:44:00 e36 kernel: RPC: state 8 conn 1 dead 0 zapped 0 Mar 29 15:44:00 e36 kernel: RPC: disconnected transport d42c4200 Mar 29 15:44:00 e36 kernel: RPC: tcp_data_ready... Mar 29 15:44:15 e36 kernel: RPC: 28611 call_transmit (status 0) Mar 29 15:44:15 e36 kernel: RPC: 28611 xprt_prepare_transmit Mar 29 15:44:15 e36 kernel: RPC: 28611 call_status (status -107)
Does this sound familiar to anyone?
Olaf
Signature · 13 lines
Olaf Kirch | The Hardware Gods hate me. [email protected] | ---------------+ ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 4 lines · Olaf Kirch
I'm currently debugging a problem with TCP reconnects in 2.6.5-pre where the TCP reconnect code got rewritten to use worker queues. What happens is that the NFS server drops the connection immediately and that state change isn't propagated to the transport.
I debugged this a little more. The problem I'be been seeing was caused by too many TCP connections. The NFS server was dropping connections randomly. Randomly means that the newest connection will be dropped with a probability of 50%, so that the connection dies before the client has sent the first packet. This causes the client to back off for 60 seconds.
I'm not sure why this effect wasn't visible with 2.6.4, but it seems it used a lower timeout (REESTABLISH_TIMEOUT = 15sec) when the connection was refused or dropped instantly, and may have been less noticeable therefore.
I'm not sure if it's a good idea to be more aggressive about reconnecting, but I think the client should at least log a message to syslog that a connection attempt failed. Likewise, the server should probably log a message when it finds it's dropping too many TCP connections.
Finally, I think the way nfsd drops connections is bad. Dropping the most recent connection doesn't prevent DoS, and as this example demonstrates, it does unexpected things to your clients.
Olaf
Signature · 13 lines
Olaf Kirch | The Hardware Gods hate me. [email protected] | ---------------+ ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs