why set send and receive buffers in svc_sock_setbufsize()?

I've been reviewing a patch for CIFS, and out of the discussions over that patch, there's been the recommendation to not set sk_sndbuf and sk_rcvbuf. Apparently, setting these to a value will place a hard limit on the buffer sizes for the socket, so there's not much need to set these unless we want to limit memory usage to this value for some reason.

One question is that the sunrpc code sets these values in svc_sock_setbufsize(). What is the reasoning behind this?

If we're doing it to limit memory consumption then that's certainly valid, but if we're doing this to attempt to make sure that we have "enough" buffer space for the expected load then this may be counter-productive.

Jeff Layton <[email protected]>

Yep! See

	http://marc.info/?l=linux-nfs&m=122461390600382&w=2

and followups; Olga and Jim have found this makes a huge difference to performance on fast high-latency networks.

--b.

Jeff Layton wrote:

I've been reviewing a patch for CIFS, and out of the discussions over that patch, there's been the recommendation to not set sk_sndbuf and sk_rcvbuf. Apparently, setting these to a value will place a hard limit on the buffer sizes for the socket, so there's not much need to set these unless we want to limit memory usage to this value for some reason.

There are two issues to be aware of. One is that the socket buffers have to be big enough for the tcp congestion window. In the old days, the application would have to know ahead of time how big this is, and call setsockopt(), which sets these numbers.

Now however, the tcp stack "autotunes" the buffer sizes to the correct amount. If you call setsockopt() to set a buffer size, or set sk_*buf, or set the SOCK_*BUF_LOCK bits in sk_userlocks, you defeat this autotuning. This is almost always a bad idea.

The other issue is that at least for NFS, the receive buffer must be big enough to hold the biggest possible rpc. If not, a partial rpc will get stuck in the buffer and no progress will be made.

Issue one is easy to deal with, just don't muck with the socket internal data structure. The second one is harder. What's really needed is a new api into the tcp layer that will reserve a minimum amount of memory in the socket buffer so that receives won't stall. For now, our patch sets the sk_*buf values without setting the lock flags.