Skip to contentSkip to conversation

NFSv4: Infinite loop in lease recovery when rpc.gssd is not running.

PATCH In review Steve Dickson 2 participants 5 messages Feb 10, 2014 – Feb 11, 2014

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 NFSv4: Infinite loop in lease recovery when rpc.gssd is not running.

[ Resent with Trond's correct email address ]

Commit 0ea9de0e introduce a regression in the lease recovery code.

An infinite loop is caused when nfs4_establish_lease() fails with -EACCES. This causes nfs4_handle_reclaim_lease_error() to sleep a bit and resets the NFS4CLNT_LEASE_EXPIRED bit. This in turn causes nfs4_state_manager() to try and reestablished the lease, again, again, again...

The problem is a valid RPCSEC_GSS client is being created when rpc.gssd is not running. This is causing the RPC code to fail with the -EACCES sending the lease reestablished off the deep end.

Moving the gssd_running() check back into nfs4_init_client(), stopping the RPCSEC_GSS client from being create, stops the looping

---
 fs/nfs/nfs4client.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)
fs/nfs/nfs4client.c +4 -1
index 860ad26..a60269f 100644
@@ -372,7 +372,10 @@ struct nfs_client *nfs4_init_client(struct nfs_client *clp,
 	__set_bit(NFS_CS_DISCRTRY, &clp->cl_flags);
 	__set_bit(NFS_CS_NO_RETRANS_TIMEOUT, &clp->cl_flags);
 
-	error = nfs_create_rpc_client(clp, timeparms, RPC_AUTH_GSS_KRB5I);
+	error =  -EINVAL;
+	if (gssd_running(clp->cl_net))
+		error = nfs_create_rpc_client(clp, timeparms,
+				RPC_AUTH_GSS_KRB5I);
 	if (error == -EINVAL)
 		error = nfs_create_rpc_client(clp, timeparms, RPC_AUTH_UNIX);
 	if (error < 0)
Signature · 1 line
1.7.1
#1 lore raw
Quoted 43 lines · Steve Dickson

[ Resent with Trond's correct email address ]

Commit 0ea9de0e introduce a regression in the lease recovery code.

An infinite loop is caused when nfs4_establish_lease() fails with -EACCES. This causes nfs4_handle_reclaim_lease_error() to sleep a bit and resets the NFS4CLNT_LEASE_EXPIRED bit. This in turn causes nfs4_state_manager() to try and reestablished the lease, again, again, again...

The problem is a valid RPCSEC_GSS client is being created when rpc.gssd is not running. This is causing the RPC code to fail with the -EACCES sending the lease reestablished off the deep end.

Moving the gssd_running() check back into nfs4_init_client(), stopping the RPCSEC_GSS client from being create, stops the looping

---
fs/nfs/nfs4client.c | 5 ++++-

1 files changed, 4 insertions(+), 1 deletions(-)
fs/nfs/nfs4client.c +4 -1
index 860ad26..a60269f 100644
@@ -372,7 +372,10 @@ struct nfs_client *nfs4_init_client(struct nfs_client *clp,
 	__set_bit(NFS_CS_DISCRTRY, &clp->cl_flags);
 	__set_bit(NFS_CS_NO_RETRANS_TIMEOUT, &clp->cl_flags);
 
-	error = nfs_create_rpc_client(clp, timeparms, RPC_AUTH_GSS_KRB5I);
+	error =  -EINVAL;
+	if (gssd_running(clp->cl_net))
+		error = nfs_create_rpc_client(clp, timeparms,
+				RPC_AUTH_GSS_KRB5I);
 	if (error == -EINVAL)
 		error = nfs_create_rpc_client(clp, timeparms, RPC_AUTH_UNIX);
 	if (error < 0)
Signature · 1 line
1.7.1

NACK. gssd_running() is not an acceptable solution outside of the RPC layer.

_________________________________
Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]

#2 lore raw

PATCH 1/1 SUNRPC: Don't create a gss auth cache unless rpc.gssd is running

An infinite loop is caused when nfs4_establish_lease() fails with -EACCES. This causes nfs4_handle_reclaim_lease_error() to sleep a bit and resets the NFS4CLNT_LEASE_EXPIRED bit. This in turn causes nfs4_state_manager() to try and reestablished the lease, again, again, again...

The problem is a valid RPCSEC_GSS client is being created when rpc.gssd is not running.

Fixes 0ea9de0ea6a4 (sunrpc: turn warn_gssd() log message into a dprintk())
---
 net/sunrpc/auth_gss/auth_gss.c | 2 ++
 1 file changed, 2 insertions(+)
net/sunrpc/auth_gss/auth_gss.c +2 -0
index 6c0513a7f992..44a61e8fda6f 100644
@@ -991,6 +991,8 @@ gss_create_new(struct rpc_auth_create_args *args, struct rpc_clnt *clnt)
 	gss_auth->service = gss_pseudoflavor_to_service(gss_auth->mech, flavor);
 	if (gss_auth->service == 0)
 		goto err_put_mech;
+	if (!gssd_running(gss_auth->net))
+		goto err_put_mech;
 	auth = &gss_auth->rpc_auth;
 	auth->au_cslack = GSS_CRED_SLACK >> 2;
 	auth->au_rslack = GSS_VERF_SLACK >> 2;
Signature · 1 line
1.8.5.3
#3 lore raw

Re: [PATCH] SUNRPC: Don't create a gss auth cache unless rpc.gssd is running

Quoted 23 lines · Trond Myklebust

An infinite loop is caused when nfs4_establish_lease() fails with -EACCES. This causes nfs4_handle_reclaim_lease_error() to sleep a bit and resets the NFS4CLNT_LEASE_EXPIRED bit. This in turn causes nfs4_state_manager() to try and reestablished the lease, again, again, again...

The problem is a valid RPCSEC_GSS client is being created when rpc.gssd is not running.

Fixes 0ea9de0ea6a4 (sunrpc: turn warn_gssd() log message into a dprintk())
---
 net/sunrpc/auth_gss/auth_gss.c | 2 ++
 1 file changed, 2 insertions(+)
net/sunrpc/auth_gss/auth_gss.c +0 -0
index 6c0513a7f992..44a61e8fda6f 100644

net/sunrpc/auth_gss/auth_gss.c gss_create_new(struct rpc_auth_create_args *args, struct rpc_clnt *clnt) · line 998

Quoted 8 lines
net/sunrpc/auth_gss/auth_gss.c +2 -0
@@ -991,6 +991,8 @@ gss_create_new(struct rpc_auth_create_args *args, struct rpc_clnt *clnt)
 	gss_auth->service = gss_pseudoflavor_to_service(gss_auth->mech, flavor);
 	if (gss_auth->service == 0)
 		goto err_put_mech;
+	if (!gssd_running(gss_auth->net))
+		goto err_put_mech;
 	auth = &gss_auth->rpc_auth;
 	auth->au_cslack = GSS_CRED_SLACK >> 2;
 	auth->au_rslack = GSS_VERF_SLACK >> 2;

Unfortunately I'm seeing the same loop but this time its with _nfs4_proc_exchange_id

Here is the trace point output:
192.168.62.8-ma-20371 [000] .... 955443.604229: nfs4_exchange_id: error=-13 (EACCES) dstaddr=192.168.62.8

and here is the rpcdebug output:

[ 2782.341981] NFS call  exchange_id auth=RPCSEC_GSS, 'Linux NFSv4.1 <client>'
[ 2782.360540] NFS reply exchange_id: -13

All three mounts (v4.0, v4.1, v4.2) are hung...

Looking into it...

steved.

author Tested-by
#4 lore raw

Re: [PATCH] SUNRPC: Don't create a gss auth cache unless rpc.gssd is running

Quoted 39 lines · Steve Dickson
Quoted 23 lines · Trond Myklebust

An infinite loop is caused when nfs4_establish_lease() fails with -EACCES. This causes nfs4_handle_reclaim_lease_error() to sleep a bit and resets the NFS4CLNT_LEASE_EXPIRED bit. This in turn causes nfs4_state_manager() to try and reestablished the lease, again, again, again...

The problem is a valid RPCSEC_GSS client is being created when rpc.gssd is not running.

Fixes 0ea9de0ea6a4 (sunrpc: turn warn_gssd() log message into a dprintk())
---
 net/sunrpc/auth_gss/auth_gss.c | 2 ++
 1 file changed, 2 insertions(+)
net/sunrpc/auth_gss/auth_gss.c +0 -0
index 6c0513a7f992..44a61e8fda6f 100644

net/sunrpc/auth_gss/auth_gss.c gss_create_new(struct rpc_auth_create_args *args, struct rpc_clnt *clnt) · line 998

Quoted 8 lines
net/sunrpc/auth_gss/auth_gss.c +2 -0
@@ -991,6 +991,8 @@ gss_create_new(struct rpc_auth_create_args *args, struct rpc_clnt *clnt)
 	gss_auth->service = gss_pseudoflavor_to_service(gss_auth->mech, flavor);
 	if (gss_auth->service == 0)
 		goto err_put_mech;
+	if (!gssd_running(gss_auth->net))
+		goto err_put_mech;
 	auth = &gss_auth->rpc_auth;
 	auth->au_cslack = GSS_CRED_SLACK >> 2;
 	auth->au_rslack = GSS_VERF_SLACK >> 2;

Unfortunately I'm seeing the same loop but this time its with _nfs4_proc_exchange_id

Here is the trace point output:
192.168.62.8-ma-20371 [000] .... 955443.604229: nfs4_exchange_id: error=-13 (EACCES) dstaddr=192.168.62.8

Quoted 7 lines

and here is the rpcdebug output:

[ 2782.341981] NFS call  exchange_id auth=RPCSEC_GSS, 'Linux NFSv4.1 <client>'
[ 2782.360540] NFS reply exchange_id: -13

All three mounts (v4.0, v4.1, v4.2) are hung...

Looking into it...

Pilot error on my part... I only reloaded sunrpc.ko not auth_rpcgss.ko What a good night sleep can do for you... :-)

Question, should we be checking that gssd still running when gss_auth pointer is found in the hash table? I'm thinking of the case where gssd was started and then stopped.

steved.