SVC sockets don't disable Nagle
Subsystems (3)
PATCH 1/1 SVC sockets don't disable Nagle
I noticed that the Nagle algorithm was disabled just recently on the client side, while it still seems to be enabled on the server side. The previous patch came from Chuck Lever.
Below patch disables it on the server side as well. For latency reasons, this should be the desired behaviour NFS at both client and server.
Diff is against 2.4.21-rc1.
---------------------------
net/sunrpc/svcsock.c renamed from net/sunrpc/svcsock.c.orig +3 -0
@@ -932,6 +932,7 @@ static int svc_tcp_init(struct svc_sock *svsk) {struct sock *sk = svsk->sk_sk; +struct tcp_opt *tp = &(sk->tp_pinfo.af_tcp); svsk->sk_recvfrom = svc_tcp_recvfrom; svsk->sk_sendto = svc_tcp_sendto;
@@ -948,6 +949,8 @@ svc_tcp_init(struct svc_sock *svsk) svsk->sk_reclen =0 ; svsk->sk_tcplen =0 ; + tp->nonagle =1 ;/* disable Nagle's algorithm */ +/* initialise setting must have enough space to * receive and respond to one request. * svc_tcp_recvfrom will re-adjust if necessary
---------------------------
Thanks,
Olof
Signature · 14 lines
Olof Johansson Office: 4E002/905 pSeries Linux Development IBM Systems Group Email: [email protected] Phone: 512-838-9858 All opinions are my own and not those of IBM. ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 2 lines · Olof Johansson
Below patch disables it on the server side as well. For latency reasons, this should be the desired behaviour NFS at both client and server.
I disagree with this general statement. Nagle is there in the TCP protocol for a specific reason and enabling or disabling it gives the opportunity to better tune the connection.
It seems like disabling Nagle is the bolierplate answer for anyone who encounters GigEth or better networking simply because the TCP protocol was not designed for such fast networks. But what happens to others that do not have GigEth ? What is the impact of this change for a slower/congested network ?
If anything, I would like to see this as a per-mount-point and per-export-entry option...
--
Bogdan Costescu
IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [email protected]
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 6 lines · Bogdan Costescu
Quoted 1 line · Olof Johansson
Below patch disables it on the server side as well. For latency reasons, this should be the desired behaviour NFS at both client and server.
I disagree with this general statement. Nagle is there in the TCP protocol for a specific reason and enabling or disabling it gives the opportunity to better tune the connection.
Yes, Nagle is there to reduce overhead by aggregating data into segments and piggybacking them with ACKs. Most significant differences will be when dealing with a steady but slow pace of characters, say text being typed over a 9600bps serial line.
Quoted 1 line
It seems like disabling Nagle is the bolierplate answer for anyone who encounters GigEth or better networking simply because the TCP protocol was not designed for such fast networks. But what happens to others that do not have GigEth ? What is the impact of this change for a slower/congested network ?
There is a chance that small requests and replies (~100 bytes) will not be aggregated into the same segments but instead sent out separately. In essence, this will result in some additional network overhead due to headers, but the response times will be perceived as faster even for slower networks.
Since NFS traffic is bursty, it is unlikely that any individual request or reply will be split up between two segments when it didn't need to.
Quoted 1 line
If anything, I would like to see this as a per-mount-point and per-export-entry option...
I think it would just clutter the documentation and list of options, since I don't forsee any practical scenarios in which anyone would have a positive performance impact from having Nagle enabled.
On a side note: As far as I know, most (all?) other NFS implementations out there already disable Nagle.
-Olof
Signature · 14 lines
Olof Johansson Office: 4E002/905 pSeries Linux Development IBM Systems Group Email: [email protected] Phone: 512-838-9858 ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 2 lines · Olof Johansson
There is a chance that small requests and replies (~100 bytes) will not be aggregated into the same segments but instead sent out separately.
Although I haven't looked at the code recently, I think that what you said is true when this is the only request from the client. But what happens when there are several requests or when GETATTR calls happen at the same time with some data transfer (which is normally the case for a multiuser NFS client) ?
Quoted 3 lines
In essence, this will result in some additional network overhead due to headers, but the response times will be perceived as faster even for slower networks.
True, until the network becomes too slow, then you see "server not responding" messages...
Quoted 1 line
I think it would just clutter the documentation and list of options,
Yes, but Samba has this option and many others and still lots of people are using it :-)
Quoted 2 lines
since I don't forsee any practical scenarios in which anyone would have a positive performance impact from having Nagle enabled.
Well, my previous message included this question: "What is the impact of this change for a slower/congested network ?" So I'll rephrase it: do you have some data to show that disabling Nagle does not have a negative impact ?
Quoted 2 lines
On a side note: As far as I know, most (all?) other NFS implementations out there already disable Nagle.
I didn't say that the other NFS implementations are better than the Linux implementation :-)
Signature · 17 lines
Bogdan Costescu IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868 E-mail: [email protected] ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Bogdan Costescu <[email protected]> writes:
In essence, this will result in some additional network overhead due to headers, but the response times will be perceived as faster even for slower networks.
> True, until the network becomes too slow, then you see "server > not responding" messages...
Anybody trying to run multi-user NFS over a 9.6Kbaud link needs their head examined.
For *real use* situations, the total number of on-the-wire NFS requests is currently limited to 16. In addition, TCP timeouts are supposed to be > 1 minute. The above scenario simply will not happen.
For any reasonable setup over a slow network, the aggregation will happen naturally anyway since the NFS layer both can and will fill the buffers as soon as they start to empty (provided we have pending datagrams).
For fast networks, we should at least ensure that single datagrams are aggregated properly. The way to do that is to use MSG_MORE...
Cheers,
Trond
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · Bogdan Costescu
Quoted 2 lines · Olof Johansson
There is a chance that small requests and replies (~100 bytes) will not be aggregated into the same segments but instead sent out separately.
Although I haven't looked at the code recently, I think that what you said is true when this is the only request from the client. But what happens when there are several requests or when GETATTR calls happen at the same time with some data transfer (which is normally the case for a multiuser NFS client) ?
This is exactly what I meant. There is a chance that the GETATTR (or other small request and/or reply) will be sent out in an individual segment instead of being aggregated. This will give some additional network overhead, but give latency improvement. Actually, for a busy NFS connection the small requests/replies will still be aggregated without Nagle. At least that's what I've been seeing here.
Quoted 5 lines
Quoted 3 lines
In essence, this will result in some additional network overhead due to headers, but the response times will be perceived as faster even for slower networks.
True, until the network becomes too slow, then you see "server not responding" messages...
I can't see how disabling Nagle would lead to RPC timeouts:
Lower network overhead during congestion is taken care of automatically: As the network throughput decreases, TCP segments will be sent out at a lower pace and/or smaller size (due to window sizes going down). There will be more time to wait for more data to fill the segment with, even without having Nagle enabled.
Quoted 4 lines
Quoted 1 line
I think it would just clutter the documentation and list of options,
Yes, but Samba has this option and many others and still lots of people are using it :-)
I didn't say that Samba is better than NFS :-)
Seriously: Samba has a configuration file, while NFS needs to stuff all the options into a line or two of export and/or mount options. Option bloat is not as big a problem when you have a large configuration file.
Quoted 4 lines
Quoted 2 lines
since I don't forsee any practical scenarios in which anyone would have a positive performance impact from having Nagle enabled.
Well, my previous message included this question: "What is the impact of this change for a slower/congested network ?" So I'll rephrase it: do you have some data to show that disabling Nagle does not have a negative impact ?
No, this would be to prove a negative which is always hard. But the same reasoning as above regarding congested networks apply. Actually, it applies to any scenario where the machine can process requests faster than the network can send the replies.
Quoted 3 lines
Quoted 1 line
On a side note: As far as I know, most (all?) other NFS implementations out there already disable Nagle.
I didn't say that the other NFS implementations are better than the Linux implementation :-)
I'm of the humble opinion that there is still a bit of room for improvement in the Linux implementation, but it's continuosly getting better. :-)
-Olof
Signature · 14 lines
Olof Johansson Office: 4E002/905 pSeries Linux Development IBM Systems Group Email: [email protected] Phone: 512-838-9858 All opinions are my own and not those of IBM ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs