Skip to contentSkip to conversation

PATCH Ben Myers 1 participant 1 message Jan 25, 2013

Subsystems (3)
lore
Share & tools
lore

Ben Myers

author
#0 lore raw

Bcc:
Subject: Re: sunrpc: socket buffer size tuneable
Reply-To:
In-Reply-To: <[email protected]>

Hey,

Quoted 23 lines · Ben Myers

Hey Bruce & Jim & Olga,

Quoted 8 lines · Jim Rees

J. Bruce Fields wrote:

Quoted 5 lines · Ben Myers

At 1020 threads the send buffer size wraps and becomes negative causing the nfs server to grind to a halt. Rather than setting bufsize based upon the number of nfsd threads, make the buffer sizes tuneable via module parameters.

Quoted 6 lines
Quoted 2 lines

Set the buffer sizes in terms of the number of rpcs you want to fit into the buffer.

From private communication, my understanding is that the original problem here was due to memory pressure forcing the tcp send buffer size below the size required to hold a single rpc.

Years ago I did see wrapping of the buffer size when tcp was used with many threads. Today's problem is timeouts on a cluster with a heavy read workload... and I seem to remember seeing that the send buffer size was too small.

Quoted 6 lines
Quoted 2 lines

In which case the important variable here is lock_bufsize, as that's what prevents the buffer size from going too low.

I tested removing the lock of bufsize and did hit the timeouts, so the overflow is starting to look less relevant. I will test your minimal overflow fix to see if this is the case.

The minimal overflow fix did not resolve the timeouts. I will test with this to see if it resolves the timeouts:

---
 net/sunrpc/svcsock.c |    7 +++++++
 1 file changed, 7 insertions(+)
net/sunrpc/svcsock.c +7 -0
Index: b/net/sunrpc/svcsock.c
===================================================================
@@ -435,6 +435,13 @@ static void svc_sock_setbufsize(struct s
 	lock_sock(sock->sk);
 	sock->sk->sk_sndbuf = snd * 2;
 	sock->sk->sk_rcvbuf = rcv * 2;
+
+	/*
+	 * The socket buffer can be resized by the networking code
+	 * unless you specify that this is not to be done.
+	 */
+	sock->sk->sk_userlocks |= SOCK_SNDBUF_LOCK|SOCK_RCVBUF_LOCK;
+
 	sock->sk->sk_write_space(sock->sk);
 	release_sock(sock->sk);
 #endif

Thanks,
Ben