NFSD: fix use of setsockopt

The following patch fixes NFS server's use of setsockopt. For this function to take an effect it first needs be called after socket creation but before sock binding.

This patch also changes the size of the receive sock buffer to be same as the send sock buffer. Both buffers are now a multiple of maxpayload and number of nfsd threads.

This patch fixes the problem that receive window never opens beyond the default TCP receive window size set by the 2nd parameter of the net.ipv4.tcp_rmem sysctl.

Signed-off-by Olga Kornievskaia <[email protected]>

Hi Olga,

This makes sense, if NFSD is going to ignore global Linux TCP settings and 'go it alone', then it shouldn't be constrained by them.

At least now we can increase the rcv buffer size by increasing the number of NFSDs. I would still like to pursue my sysctl patch for the rcv and snd buffer though since we have seen situations where too many NFSDs can increase the randomness of requests to the underlying file system, reducing the effectiveness readahead/write gathering.

Dean

Quoted 8 lines · Olga Kornievskaia

The following patch fixes NFS server's use of setsockopt. For this function to take an effect it first needs be called after socket creation but before sock binding.

This patch also changes the size of the receive sock buffer to be same as the send sock buffer. Both buffers are now a multiple of maxpayload and number of nfsd threads.

Quoted 5 lines

This patch fixes the problem that receive window never opens beyond the default TCP receive window size set by the 2nd parameter of the net.ipv4.tcp_rmem sysctl.

Signed-off-by Olga Kornievskaia <[email protected]>

Signed-off-by Olga Kornievskaia <[email protected]>

------------------------------------------------------------------------

net/sunrpc/svcsock.c +1 -6

index c75bffe..178b397 100644
@@ -1191,7 +1191,7 @@ svc_tcp_recvfrom(struct svc_rqst *rqstp)
 		 */
 		svc_sock_setbufsize(svsk->sk_sock,
 				    (serv->sv_nrthreads+3) * serv->sv_max_mesg,
-				    3 * serv->sv_max_mesg);
+				    (serv->sv_nrthreads+3) * serv->sv_max_mesg);
 
 	clear_bit(SK_DATA, &svsk->sk_flags);
 

@@ -1372,11 +1372,6 @@ svc_tcp_init(struct svc_sock *svsk)
 		 * receive and respond to one request.
 		 * svc_tcp_recvfrom will re-adjust if necessary
 		 */
-		svc_sock_setbufsize(svsk->sk_sock,
-				    3 * svsk->sk_server->sv_max_mesg,
-				    3 * svsk->sk_server->sv_max_mesg);
-
-		set_bit(SK_CHNGBUF, &svsk->sk_flags);
 
 		set_bit(SK_DATA, &svsk->sk_flags);
 		if (sk->sk_state != TCP_ESTABLISHED)

			set_bit(SK_CLOSE, &svsk->sk_flags);

net/sunrpc/svcsock.c +2 -1

@@ -1761,6 +1756,8 @@ static int svc_create_socket(struct svc_serv *serv, int protocol,
 
 	if (type == SOCK_STREAM)
 		sock->sk->sk_reuse = 1;		/* allow address reuse */
+	svc_sock_setbufsize(sock, (serv->sv_nrthreads+3) * serv->sv_max_mesg,
+				(serv->sv_nrthreads+3) * serv->sv_max_mesg);
 	error = kernel_bind(sock, sin, len);
 	if (error < 0)
 		goto bummer;
 
--

To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

The following patch fixes NFS server's use of setsockopt. For this function to take an effect it first needs be called after socket creation but before sock binding.

This patch also changes the size of the receive sock buffer to be same as the send sock buffer. Both buffers are now a multiple of maxpayload and number of nfsd threads.

This patch fixes the problem that receive window never opens beyond the default TCP receive window size set by the 2nd parameter of the net.ipv4.tcp_rmem sysctl.

Signed-off-by Olga Kornievskaia <[email protected]>

net/sunrpc/svcsock.c +3 -6

index c75bffe..178b397 100644
@@ -1191,7 +1191,7 @@ svc_tcp_recvfrom(struct svc_rqst *rqstp)
 		 */
 		svc_sock_setbufsize(svsk->sk_sock,
 				    (serv->sv_nrthreads+3) * serv->sv_max_mesg,
-				    3 * serv->sv_max_mesg);
+				    (serv->sv_nrthreads+3) * serv->sv_max_mesg);
 
 	clear_bit(SK_DATA, &svsk->sk_flags);
 

@@ -1372,11 +1372,6 @@ svc_tcp_init(struct svc_sock *svsk)
 		 * receive and respond to one request.
 		 * svc_tcp_recvfrom will re-adjust if necessary
 		 */
-		svc_sock_setbufsize(svsk->sk_sock,
-				    3 * svsk->sk_server->sv_max_mesg,
-				    3 * svsk->sk_server->sv_max_mesg);
-
-		set_bit(SK_CHNGBUF, &svsk->sk_flags);
 		set_bit(SK_DATA, &svsk->sk_flags);
 		if (sk->sk_state != TCP_ESTABLISHED)
 			set_bit(SK_CLOSE, &svsk->sk_flags);

@@ -1761,6 +1756,8 @@ static int svc_create_socket(struct svc_serv *serv, int protocol,
 
 	if (type == SOCK_STREAM)
 		sock->sk->sk_reuse = 1;		/* allow address reuse */
+	svc_sock_setbufsize(sock, (serv->sv_nrthreads+3) * serv->sv_max_mesg,
+				(serv->sv_nrthreads+3) * serv->sv_max_mesg);
 	error = kernel_bind(sock, sin, len);
 	if (error < 0)
 		goto bummer;

Quoted 2 lines · Olga Kornievskaia

The following patch fixes NFS server's use of setsockopt. For this function to take an effect it first needs be called after socket creation but before sock binding.

The tcp(7) man page actually claims that it's listen() and connect() that matter (so the setsockopt is effective on (and only on) unconnected sockets), so probably this could go after the bind and before the listen? Not that it matters.

Quoted 2 lines

This patch also changes the size of the receive sock buffer to be same as the send sock buffer. Both buffers are now a multiple of maxpayload and number of nfsd threads.

It would be nice if we could get some review from someone who remembers what the justification for the smaller receive buffer size was (Neil?).

Quoted 2 lines

This patch fixes the problem that receive window never opens beyond the default TCP receive window size set by the 2nd parameter of the net.ipv4.tcp_rmem sysctl.

Do you know what it does in the udp case?

--b.

Quoted 39 lines

Signed-off-by Olga Kornievskaia <[email protected]>

net/sunrpc/svcsock.c +3 -6

index c75bffe..178b397 100644
@@ -1191,7 +1191,7 @@ svc_tcp_recvfrom(struct svc_rqst *rqstp)
 		 */
 		svc_sock_setbufsize(svsk->sk_sock,
 				    (serv->sv_nrthreads+3) * serv->sv_max_mesg,
-				    3 * serv->sv_max_mesg);
+				    (serv->sv_nrthreads+3) * serv->sv_max_mesg);
 
 	clear_bit(SK_DATA, &svsk->sk_flags);
 

@@ -1372,11 +1372,6 @@ svc_tcp_init(struct svc_sock *svsk)
 		 * receive and respond to one request.
 		 * svc_tcp_recvfrom will re-adjust if necessary
 		 */
-		svc_sock_setbufsize(svsk->sk_sock,
-				    3 * svsk->sk_server->sv_max_mesg,
-				    3 * svsk->sk_server->sv_max_mesg);
-
-		set_bit(SK_CHNGBUF, &svsk->sk_flags);
 		set_bit(SK_DATA, &svsk->sk_flags);
 		if (sk->sk_state != TCP_ESTABLISHED)
 			set_bit(SK_CLOSE, &svsk->sk_flags);

@@ -1761,6 +1756,8 @@ static int svc_create_socket(struct svc_serv *serv, int protocol,
 
 	if (type == SOCK_STREAM)
 		sock->sk->sk_reuse = 1;		/* allow address reuse */
+	svc_sock_setbufsize(sock, (serv->sv_nrthreads+3) * serv->sv_max_mesg,
+				(serv->sv_nrthreads+3) * serv->sv_max_mesg);
 	error = kernel_bind(sock, sin, len);
 	if (error < 0)
 		goto bummer;

Looking at the kernel code, when setsockopt() is called on a UDP socket to set send/receive buffer for UPD the code will not do anything:
udp_setsockopt() and udp_lib_setsockopt() will return -ENOPROTOOPT. However, we bypass the call to setsockopt() and instead set the buffer sizes directly. From what I understand sk_sndbuf/sk_rcvbuf are not used by the UDP code. We are setting the fields that are never used.

Then perhaps we can remove calls to svc_sock_setbufsize() from svc_udp_init() and svc_udp_recvfrom()?

Hi,

Does anybody know how sm-notify exactly works on a suse sles 9 system?

Hi,

Does anybody know how sm-notify exactly works on a suse sles 9 system=

Hi,
maybe I can specify Dirks question.

We are trying to set up a NFS Cluster and want to use sm-notify to inform the Clients if the Server has failed over. We have found a Howto that says: sm-notify uses files under /var/lib/nfs/sm to determine the Clients it has to inform. Our Problem is, that ther are no files in this place even if we connet with a client. Is there anything we have forgotten?

-----Original Message-----
From: Neil Brown [mailto:[email protected]]
Sent: Saturday, June 28, 2008 1:44 AM
To: Oeltze, Benjamin
Cc: Laurenz, Dirk; [email protected]
Subject: RE: Sm-notify

Quoted 2 lines · On Friday June 27, Benjamin.Oeltze-/ixSogHR0HOS/[email protected]…

Hi,
maybe I can specify Dirks question.

Thanks.

Quoted 2 lines

We are trying to set up a NFS Cluster and want to use sm-notify to inform the

Clients if the Server has failed over.

Quoted 1 line

We have found a Howto that says: sm-notify uses files under /var/lib/nfs/sm to

determine the Clients it has to inform.

Quoted 1 line

Our Problem is, that ther are no files in this place even if we connet with a client.

Is there anything we have forgotten?

Files do not get created in /var/lib/nfs/sm until the client tried to get a lock on some file (or part of a file). Then the file is created.
Some minutes after the last lock is dropped, the file should be removed (though I recall there were some issues related to that, it might depend on exactly which kernel you are running).

If no file locking is involved, there is no need to inform the client of failover. They will transparently start accessing the new server through the old IP address, possible after a short timeout.

NeilBrown

i, the time out is from node a to b seconds, but from node b to a 15 minutes. what I saw was, that the handle changed in /var/lib/nfs/rmtab from 0x000002 to 0x000003 although the fsid is the same on both nodes.
how can the 15minute timeout occur?