Skip to contentSkip to conversation

kernel not recovering from statd port change

PATCH J. Bruce Fields 3 participants 3 messages Aug 21, 2014 – Sep 4, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

While testing server restart somebody noticed that knfsd can't recover from statd restarting with a new port.

Quoted 1 line

From only a very quick skim of the code it looked like creating the nsm

client with RPC_CLNT_CREATE_AUTOBIND should cause us to call rpcbind again on connection failures, but that doesn't seem to be working.

Any ideas? I'll keep looking....

--b.

commit 2c9fb5570fe2

Author: J. Bruce Fields <[email protected]>
Date:   Wed Aug 20 17:21:32 2014 -0400

lockd: allow rebinding to statd

During normal operation statd isn't restarted, but it may be if, for example, the server is shut down and restarted to simulate a shutdown or perform some kind of failover. In that case the kernel may need to query rpcbind again to get statd's new port number.

Symptoms were locking failures after a manual server restart (without rebooting the machine), and loopback network traces showing the new kernel nfsd attempting to contact statd at its old port number.

This was probably introduced by cb7323fffa85, which first allowed reusing the statd rpc client, but it looks like a reference count may typically have prevented any symptoms until e498daa81295 "LOCKD: Clear ln->nsm_clnt only when ln->nsm_users is zero".

Fixes cb7323fffa85 "lockd: create and use per-net NSM RPC clients on MON/UNMON requests"
fs/lockd/mon.c +2 -1
index 1812f026960c..3bce1d318435 100644
@@ -80,7 +80,8 @@ static struct rpc_clnt *nsm_create(struct net *net)
 		.program		= &nsm_program,
 		.version		= NSM_VERSION,
 		.authflavor		= RPC_AUTH_NULL,
-		.flags			= RPC_CLNT_CREATE_NOPING,
+		.flags			= RPC_CLNT_CREATE_NOPING|
+			                  RPC_CLNT_CREATE_AUTOBIND,
 	};
 
 	return rpc_create(&args);
#1 lore raw

I've been debugging the same thing on an Ubuntu 12.04 server running 3.8.0-44, and ended up in the same place you are. Did you find out anything more? I have carefully inserted rpc_force_rebind() near nlm_client_get, but I don't think it is a good fix for others.

In production servers, I am starting rpc.statd with "--port #####", which does seem to solve the problem. NLM (vs NSM) apparently doesn't suffer from it.

One thing that puzzles me is that of several hundred NFS clients only a handful have a problem getting a lock. The problem clients are running 3.2 and 2.6.26. The not-problem clients are 3.8 mostly. NFSv3.

Quoted 53 lines · J. Bruce Fields

While testing server restart somebody noticed that knfsd can't recover from statd restarting with a new port.

From only a very quick skim of the code it looked like creating the nsm
client with RPC_CLNT_CREATE_AUTOBIND should cause us to call rpcbind again on connection failures, but that doesn't seem to be working.

Any ideas? I'll keep looking....

--b.

commit 2c9fb5570fe2

Author: J. Bruce Fields <[email protected]>
Date:   Wed Aug 20 17:21:32 2014 -0400

lockd: allow rebinding to statd

During normal operation statd isn't restarted, but it may be if, for example, the server is shut down and restarted to simulate a shutdown or perform some kind of failover. In that case the kernel may need to query rpcbind again to get statd's new port number.

Symptoms were locking failures after a manual server restart (without rebooting the machine), and loopback network traces showing the new kernel nfsd attempting to contact statd at its old port number.

This was probably introduced by cb7323fffa85, which first allowed reusing the statd rpc client, but it looks like a reference count may typically have prevented any symptoms until e498daa81295 "LOCKD: Clear ln->nsm_clnt only when ln->nsm_users is zero".

Fixes cb7323fffa85 "lockd: create and use per-net NSM RPC clients on MON/UNMON requests"
fs/lockd/mon.c +2 -2
index 1812f026960c..3bce1d318435 100644
@@ -80,7 +80,8 @@ static struct rpc_clnt *nsm_create(struct net *net)
  		.program		= &nsm_program,
  		.version		= NSM_VERSION,
  		.authflavor		= RPC_AUTH_NULL,
-		.flags			= RPC_CLNT_CREATE_NOPING,
+		.flags			= RPC_CLNT_CREATE_NOPING|
+			                  RPC_CLNT_CREATE_AUTOBIND,
  	};
 
  	return rpc_create(&args);
--

To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
In reply to J. Bruce Fields
Quoted 39 lines · J. Bruce Fields

While testing server restart somebody noticed that knfsd can't recover from statd restarting with a new port.

From only a very quick skim of the code it looked like creating the nsm client with RPC_CLNT_CREATE_AUTOBIND should cause us to call rpcbind again on connection failures, but that doesn't seem to be working.

Any ideas? I'll keep looking....

--b.

commit 2c9fb5570fe2

Author: J. Bruce Fields <[email protected]>
Date:   Wed Aug 20 17:21:32 2014 -0400

lockd: allow rebinding to statd

During normal operation statd isn't restarted, but it may be if, for example, the server is shut down and restarted to simulate a shutdown or perform some kind of failover. In that case the kernel may need to query rpcbind again to get statd's new port number.

Symptoms were locking failures after a manual server restart (without rebooting the machine), and loopback network traces showing the new kernel nfsd attempting to contact statd at its old port number.

This was probably introduced by cb7323fffa85, which first allowed reusing the statd rpc client, but it looks like a reference count may typically have prevented any symptoms until e498daa81295 "LOCKD: Clear ln->nsm_clnt only when ln->nsm_users is zero".

Fixes cb7323fffa85 "lockd: create and use per-net NSM RPC clients on MON/UNMON requests"
fs/lockd/mon.c +0 -0
index 1812f026960c..3bce1d318435 100644

fs/lockd/mon.c static struct rpc_clnt *nsm_create(struct net *net) · line 83

Quoted 5 lines
fs/lockd/mon.c +1 -1
@@ -80,7 +80,8 @@ static struct rpc_clnt *nsm_create(struct net *net)
                .program                = &nsm_program,
                .version                = NSM_VERSION,
                .authflavor             = RPC_AUTH_NULL,
-               .flags                  = RPC_CLNT_CREATE_NOPING,
+               .flags                  = RPC_CLNT_CREATE_NOPING|

RPC_CLNT_CREATE_HARDRTRY |

Quoted 4 lines
fs/lockd/mon.c +1 -0
+                                         RPC_CLNT_CREATE_AUTOBIND,
        };
 
        return rpc_create(&args);
Signature · 5 lines
Trond Myklebust

Linux NFS client maintainer, PrimaryData

[email protected]