new timeout behavior for RPC requests on TCP sockets
Quoted 11 lines · Chuck
make RPC timeout behavior over TCP sockets behave more like reference client implementations. reference behavior is to transmit the same request three times at 60 second intervals; if there is no response, close and reestablish the socket connection. we modify the Linux RPC client as follows:
(quoted diff) +2
-0+ after a minor retransmit timeout, use the same timeout value when retrying on a TCP socket rather than doubling the value + after a major retransmit timeout, close the socket and attempt to reestablish a fresh TCP connection
Quoted 3 lines
note that today mount uses a 6 second timeout with 5 retries for NFS over TCP by default; proper default behavior is 2 retries each with 60 second timeouts. a separate patch for mount is pending.
Chuck, can you briefly explain why RPC does any minor retransmits at all over TCP?
Shouldn't TCP's natural retransmit take care of that?
- Dan
Quoted 12 lines · Dan Kegel
Quoted 11 lines · Chuck
make RPC timeout behavior over TCP sockets behave more like reference client implementations. reference behavior is to transmit the same request three times at 60 second intervals; if there is no response, close and reestablish the socket connection. we modify the Linux RPC client as follows:
(quoted diff) +2
-0+ after a minor retransmit timeout, use the same timeout value when retrying on a TCP socket rather than doubling the value + after a major retransmit timeout, close the socket and attempt to reestablish a fresh TCP connection
Quoted 7 lines
Quoted 3 lines
note that today mount uses a 6 second timeout with 5 retries for NFS over TCP by default; proper default behavior is 2 retries each with 60 second timeouts. a separate patch for mount is pending.
Chuck, can you briefly explain why RPC does any minor retransmits at all over TCP?
Shouldn't TCP's natural retransmit take care of that?
the socket layer guarantees delivery only to the RPC server application... if the application itself chooses to drop the request, an RPC retransmit is still required.
- Chuck Lever
--
corporate: <cel at netapp dot com>
personal: <chucklever at bigfoot dot com>
Quoted 22 lines · Chuck Lever
Quoted 12 lines · Dan Kegel
Quoted 11 lines · Chuck
make RPC timeout behavior over TCP sockets behave more like reference client implementations. reference behavior is to transmit the same request three times at 60 second intervals; if there is no response, close and reestablish the socket connection. we modify the Linux RPC client as follows:
(quoted diff) +2
-0+ after a minor retransmit timeout, use the same timeout value when retrying on a TCP socket rather than doubling the value + after a major retransmit timeout, close the socket and attempt to reestablish a fresh TCP connectionQuoted 7 lines
Quoted 3 lines
note that today mount uses a 6 second timeout with 5 retries for NFS over TCP by default; proper default behavior is 2 retries each with 60 second timeouts. a separate patch for mount is pending.
Chuck, can you briefly explain why RPC does any minor retransmits at all over TCP?
Shouldn't TCP's natural retransmit take care of that?
Quoted 6 lines
the socket layer guarantees delivery only to the RPC server application... if the application itself chooses to drop the request, an RPC retransmit is still required.
- Chuck Lever
--
If the application "chooses to drop the request", the kernel is not required to fix that application. The RPC cannot retransmit if it has been shut-down or disconnected, which is about the only way the application could "choose to drop the request". So something doesn't smell right here.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Bush : The Fourth Reich of America
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Richard B Johnson <[email protected]> writes:
> If the application "chooses to drop the request", the kernel is > not required to fix that application. The RPC cannot retransmit > if it has been shut-down or disconnected, which is about the > only way the application could "choose to drop the request". So > something doesn't smell right here.
An NFS server is perfectly free to drop an RPC request if it doesn't have the necessary free resources to service it (i.e. if it is out of memory). If the client doesn't time out + retry, you lose data. Not a good idea...
Cheers,
Trond
Quoted 8 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Richard B Johnson <[email protected]> writes:
> If the application "chooses to drop the request", the kernel is > not required to fix that application. The RPC cannot retransmit > if it has been shut-down or disconnected, which is about the > only way the application could "choose to drop the request". So > something doesn't smell right here.
Quoted 7 lines
An NFS server is perfectly free to drop an RPC request if it doesn't have the necessary free resources to service it (i.e. if it is out of memory). If the client doesn't time out + retry, you lose data. Not a good idea...
Cheers,
Trond
The Client is the guy that just retries, as you say from a time-out. This shouldn't affect any internal TCP/IP code. The time-out is at the application (client) level. It sent a request, the data was sent or promised to be sent because the write() or send() didn't block, now it expects to get the data it asked for. It waits, nothing happens. It times-out and sends the exact same request again.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Bush : The Fourth Reich of America
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Richard B Johnson <[email protected]> writes:
> The Client is the guy that just retries, as you say from a > time-out. This shouldn't affect any internal TCP/IP code. The > time-out is at the application (client) level. It sent a > request, the data was sent or promised to be sent because the > write() or send() didn't block, now it expects to get the data > it asked for. It waits, nothing happens. It times-out and sends > the exact same request again.
Huh??? There's no 'application level' involved here at all, nor any 'internal TCP/IP code'.
Chuck's patch touches the way the kernel Sun RPC client code (as used exclusively by the kernel NFS client and the kernel NLM client) handles the generic case of message timeout + resend. Why would we want to even consider pushing that sort of thing down into the NFS code itself?
Cheers,
Trond
Quoted 13 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Richard B Johnson <[email protected]> writes:
> The Client is the guy that just retries, as you say from a > time-out. This shouldn't affect any internal TCP/IP code. The > time-out is at the application (client) level. It sent a > request, the data was sent or promised to be sent because the >
write()orsend()didn't block, now it expects to get the data > it asked for. It waits, nothing happens. It times-out and sends > the exact same request again.Huh??? There's no 'application level' involved here at all, nor any 'internal TCP/IP code'.
Quoted 9 lines
Chuck's patch touches the way the kernel Sun RPC client code (as used exclusively by the kernel NFS client and the kernel NLM client) handles the generic case of message timeout + resend. Why would we want to even consider pushing that sort of thing down into the NFS code itself?
Cheers,
Trond
Because all of the RPC stuff was, initially, user-mode code. For performance reasons or otherwise, it was moved into the kernel. Okay, so far? Now, when something goes wrong with that code, should that code be fixed, or should the unrelated TCP/IP code be modified to accommodate? I think the time-outs should be put at the correct places and not added to generic network code.
Once the client side gets a buffer of data from the TCP/IP stack, the TCP/IP should not care (ever) what it does with it. Putting the timeout(s) in the TCP/IP stack, makes it care, and adds code to accommodate.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Bush : The Fourth Reich of America
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Richard B Johnson <[email protected]> writes:
> Because all of the RPC stuff was, initially, user-mode > code. For performance reasons or otherwise, it was moved into > the kernel. Okay, so far? Now, when something goes wrong with > that code, should that code be fixed, or should the unrelated > TCP/IP code be modified to accommodate? I think the time-outs > should be put at the correct places and not added to generic > network code.
No. The kernel RPC code has never been user mode code, nor has it ever been exported to userland. It exists purely for the benefit of NFS and friends. It is located in a subdirectory of the network code, but it is certainly not 'generic network code'.
Cheers,
Trond
Quoted 1 line · Richard B. Johnson
Because all of the RPC stuff was, initially, user-mode code.
if you mean ti-rpc, that stuff comes from sun. the linux kernel ONC/RPC implementation is not based on the ti-rpc code because, being Transport Independent, ti-rpc is less than optimally efficient. also, it is covered by a restrictive license agreement, so that code base can't be included in the linux kernel.
Quoted 3 lines
Now, when something goes wrong with that code, should that code be fixed, or should the unrelated TCP/IP code be modified to accommodate?
obviously the RPC client should be fixed....
Quoted 2 lines
I think the time-outs should be put at the correct places and not added to generic network code.
...which is exactly what i did.
the new RPC retransmission logic is in net/sunrpc/clnt.c:call_timeout, which is strictly a part of the RPC client's finite state machine. underlying TCP retransmit behavior is not changed by this patch. the changes apply to the RPC client only, which resides above the socket layer.
let me go over the changes again. the RPC client sets a timeout after sending each request. if it doesn't receive a valid reply for a request within the timeout interval, a "minor" timeout occurs. after each timeout, the RPC client doubles the timeout interval until it reaches a maximum value.
for RPC over UDP, short timeouts and retransmission back-off make sense. for TCP, retransmission is built into the underlying protocol, so it makes more sense to use a constant long retransmit timeout.
a "major" timeout occurs after several "minor" timeouts. this is an ad-hoc mechanism for detecting that a server is actually down, rather than just a few requests have been lost. a "server not responding" message in the kernel log appears when a major timeout occurs.
for UDP, there is no way a client can tell the server has gone away except by noticing that the server is not sending any replies. TCP sockets require a bit more cleanup when one end dies, however, since both ends maintain some connection state.
i've changed the RPC client's timeout behavior when it uses a TCP socket rather than a UDP socket to connect to a server:
+ after a minor RPC retransmit timeout on a TCP socket, the RPC client uses the same retransmit timeout value when retransmitting the request rather than doubling it, as it would on a UDP socket.
+ after a major RPC retransmit timeout on a TCP socket, close the socket. the RPC finite state machine will notice the socket is no longer connected, and attempt to reestablish a connection when it retries the request again.
this means that after a few retransmissions, the RPC client closes the transport socket. if a server hasn't responded after several retransmissions, the client now assumes that it has crashed and has lost all connection state, so it will reestablish a fresh connection with the server.
this behavior is recommended for NFSv2 and v3 over TCP, and is required for NFSv4 over TCP (RFC3010).
- Chuck Lever
--
corporate: <cel at netapp dot com>
personal: <chucklever at bigfoot dot com>
Quoted 64 lines · Chuck Lever
Quoted 1 line · Richard B. Johnson
Because all of the RPC stuff was, initially, user-mode code.
if you mean ti-rpc, that stuff comes from sun. the linux kernel ONC/RPC implementation is not based on the ti-rpc code because, being Transport Independent, ti-rpc is less than optimally efficient. also, it is covered by a restrictive license agreement, so that code base can't be included in the linux kernel.
Quoted 3 lines
Now, when something goes wrong with that code, should that code be fixed, or should the unrelated TCP/IP code be modified to accommodate?
obviously the RPC client should be fixed....
Quoted 2 lines
I think the time-outs should be put at the correct places and not added to generic network code.
...which is exactly what i did.
the new RPC retransmission logic is in net/sunrpc/clnt.c:call_timeout, which is strictly a part of the RPC client's finite state machine. underlying TCP retransmit behavior is not changed by this patch. the changes apply to the RPC client only, which resides above the socket layer.
let me go over the changes again. the RPC client sets a timeout after sending each request. if it doesn't receive a valid reply for a request within the timeout interval, a "minor" timeout occurs. after each timeout, the RPC client doubles the timeout interval until it reaches a maximum value.
for RPC over UDP, short timeouts and retransmission back-off make sense. for TCP, retransmission is built into the underlying protocol, so it makes more sense to use a constant long retransmit timeout.
a "major" timeout occurs after several "minor" timeouts. this is an ad-hoc mechanism for detecting that a server is actually down, rather than just a few requests have been lost. a "server not responding" message in the kernel log appears when a major timeout occurs.
for UDP, there is no way a client can tell the server has gone away except by noticing that the server is not sending any replies. TCP sockets require a bit more cleanup when one end dies, however, since both ends maintain some connection state.
i've changed the RPC client's timeout behavior when it uses a TCP socket rather than a UDP socket to connect to a server:
(quoted diff) +2
-0+ after a minor RPC retransmit timeout on a TCP socket, the RPC client uses the same retransmit timeout value when retransmitting the request rather than doubling it, as it would on a UDP socket. + after a major RPC retransmit timeout on a TCP socket, close the socket. the RPC finite state machine will notice the socket is no longer connected, and attempt to reestablish a connection when it retries the request again.this means that after a few retransmissions, the RPC client closes the transport socket. if a server hasn't responded after several retransmissions, the client now assumes that it has crashed and has lost all connection state, so it will reestablish a fresh connection with the server.
Quoted 9 lines
this behavior is recommended for NFSv2 and v3 over TCP, and is required for NFSv4 over TCP (RFC3010).
- Chuck Lever
--
corporate: <cel at netapp dot com>
personal: <chucklever at bigfoot dot com>
Okay. Thanks a lot for the complete explaination. The early information about the patch, and the patch itself that I tried to follow, seemed to show that new retransmit timer bahavior was applied at the TCP/IP level (actually socket level). This is what I was bitching about.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Bush : The Fourth Reich of America
Quoted 1 line · Richard B. Johnson
Because all of the RPC stuff was, initially, user-mode code.
if you mean ti-rpc, that stuff comes from sun. the linux kernel ONC/RPC implementation is not based on the ti-rpc code because, being Transport Independent, ti-rpc is less than optimally efficient. also, it is covered by a restrictive license agreement, so that code base can't be included in the linux kernel.
Quoted 3 lines
Now, when something goes wrong with that code, should that code be fixed, or should the unrelated TCP/IP code be modified to accommodate?
obviously the RPC client should be fixed....
Quoted 2 lines
I think the time-outs should be put at the correct places and not added to generic network code.
...which is exactly what i did.
the new RPC retransmission logic is in net/sunrpc/clnt.c:call_timeout, which is strictly a part of the RPC client's finite state machine. underlying TCP retransmit behavior is not changed by this patch. the changes apply to the RPC client only, which resides above the socket layer.
let me go over the changes again. the RPC client sets a timeout after sending each request. if it doesn't receive a valid reply for a request within the timeout interval, a "minor" timeout occurs. after each timeout, the RPC client doubles the timeout interval until it reaches a maximum value.
for RPC over UDP, short timeouts and retransmission back-off make sense. for TCP, retransmission is built into the underlying protocol, so it makes more sense to use a constant long retransmit timeout.
a "major" timeout occurs after several "minor" timeouts. this is an ad-hoc mechanism for detecting that a server is actually down, rather than just a few requests have been lost. a "server not responding" message in the kernel log appears when a major timeout occurs.
for UDP, there is no way a client can tell the server has gone away except by noticing that the server is not sending any replies. TCP sockets require a bit more cleanup when one end dies, however, since both ends maintain some connection state.
i've changed the RPC client's timeout behavior when it uses a TCP socket rather than a UDP socket to connect to a server:
+ after a minor RPC retransmit timeout on a TCP socket, the RPC client uses the same retransmit timeout value when retransmitting the request rather than doubling it, as it would on a UDP socket.
+ after a major RPC retransmit timeout on a TCP socket, close the socket. the RPC finite state machine will notice the socket is no longer connected, and attempt to reestablish a connection when it retries the request again.
this means that after a few retransmissions, the RPC client closes the transport socket. if a server hasn't responded after several retransmissions, the client now assumes that it has crashed and has lost all connection state, so it will reestablish a fresh connection with the server.
this behavior is recommended for NFSv2 and v3 over TCP, and is required for NFSv4 over TCP (RFC3010).
- Chuck Lever
--
corporate: <cel at netapp dot com>
personal: <chucklever at bigfoot dot com>
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 5 lines · Richard B. Johnson
If the application "chooses to drop the request", the kernel is not required to fix that application. The RPC cannot retransmit if it has been shut-down or disconnected, which is about the only way the application could "choose to drop the request". So something doesn't smell right here.
Check your socks...
As far as RPC goes the RPC server can choose to drop a request whenever it pleases by simply throwing it away (eg reading it from the socket and binning it) depending on its workload. There are actually reasons for that in some situations (eg if the top requests are all for a volume that is down its better to throw them away so you can get requests for a volume that is functional)
Alan
Quoted 9 lines · Alan Cox
Quoted 5 lines · Richard B. Johnson
If the application "chooses to drop the request", the kernel is not required to fix that application. The RPC cannot retransmit if it has been shut-down or disconnected, which is about the only way the application could "choose to drop the request". So something doesn't smell right here.
Check your socks...
Quoted 8 lines
As far as RPC goes the RPC server can choose to drop a request whenever it pleases by simply throwing it away (eg reading it from the socket and binning it) depending on its workload. There are actually reasons for that in some situations (eg if the top requests are all for a volume that is down its better to throw them away so you can get requests for a volume that is functional)
Alan
Yes! But, the Client it is perfectly free to request it again and should (must) to keep any mounted volumes intact. This doesn't affect the internal TCP/IP stack (or it shouldn't). Since the whole NFS thing is "stateless", the client just issues another request.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Bush : The Fourth Reich of America