Skip to contentSkip to conversation

NFSv4 wait on recovery for async session errors

PATCH In review [email protected] 4 participants 7 messages Nov 15, 2013 – Nov 21, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 NFSv4 wait on recovery for async session errors

From: Andy Adamson <[email protected]>

When the state manager is processing the NFS4CLNT_DELEGRETURN flag, session draining is off, but DELEGRETURN can still get a session error. The async handler calls nfs4_schedule_session_recovery returns -EAGAIN, and the DELEGRETURN done then restarts the RPC task in the prepare state. With the state manager still processing the NFS4CLNT_DELEGRETURN flag with session draining off, these DELEGRETURNs will cycle with errors filling up the session slots.

This prevents OPEN reclaims (from nfs_delegation_claim_opens) required by the NFS4CLNT_DELEGRETURN state manager processing from completing, hanging the state manager in the __rpc_wait_for_completion_task in nfs4_run_open_task as seen in this kernel thread dump:

kernel: 4.12.32.53-ma D 0000000000000000 0 3393 2 0x00000000
kernel: ffff88013995fb60 0000000000000046 ffff880138cc5400 ffff88013a9df140
kernel: ffff8800000265c0 ffffffff8116eef0 ffff88013fc10080 0000000300000001
kernel: ffff88013a4ad058 ffff88013995ffd8 000000000000fbc8 ffff88013a4ad058
kernel: Call Trace:
kernel: [<ffffffff8116eef0>] ? cache_alloc_refill+0x1c0/0x240
kernel: [<ffffffffa0358110>] ? rpc_wait_bit_killable+0x0/0xa0 [sunrpc]
kernel: [<ffffffffa0358152>] rpc_wait_bit_killable+0x42/0xa0 [sunrpc]
kernel: [<ffffffff8152914f>] __wait_on_bit+0x5f/0x90
kernel: [<ffffffffa0358110>] ? rpc_wait_bit_killable+0x0/0xa0 [sunrpc]
kernel: [<ffffffff815291f8>] out_of_line_wait_on_bit+0x78/0x90
kernel: [<ffffffff8109b520>] ? wake_bit_function+0x0/0x50
kernel: [<ffffffffa035810d>] __rpc_wait_for_completion_task+0x2d/0x30 [sunrpc]
kernel: [<ffffffffa040d44c>] nfs4_run_open_task+0x11c/0x160 [nfs]
kernel: [<ffffffffa04114e7>] nfs4_open_recover_helper+0x87/0x120 [nfs]
kernel: [<ffffffffa0411646>] nfs4_open_recover+0xc6/0x150 [nfs]
kernel: [<ffffffffa040cc6f>] ? nfs4_open_recoverdata_alloc+0x2f/0x60 [nfs]
kernel: [<ffffffffa0414e1a>] nfs4_open_delegation_recall+0x6a/0xa0 [nfs]
kernel: [<ffffffffa0424020>] nfs_end_delegation_return+0x120/0x2e0 [nfs]
kernel: [<ffffffff8109580f>] ? queue_work+0x1f/0x30
kernel: [<ffffffffa0424347>] nfs_client_return_marked_delegations+0xd7/0x110 [nfs]
kernel: [<ffffffffa04225d8>] nfs4_run_state_manager+0x548/0x620 [nfs]
kernel: [<ffffffffa0422090>] ? nfs4_run_state_manager+0x0/0x620 [nfs]
kernel: [<ffffffff8109b0f6>] kthread+0x96/0xa0
kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
kernel: [<ffffffff8109b060>] ? kthread+0x0/0xa0
kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20

The state manager can not therefore process the DELEGRETURN session errors. Change the async handler to wait for recovery on session errors.

---
 fs/nfs/nfs4proc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfs/nfs4proc.c +1 -1
index 5ab33c0..1f4edfb 100644
@@ -4803,7 +4803,7 @@ nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server,
 			dprintk("%s ERROR %d, Reset session\n", __func__,
 				task->tk_status);
 			nfs4_schedule_session_recovery(clp->cl_session, task->tk_status);
-			goto restart_call;
+			goto wait_on_recovery;
 #endif /* CONFIG_NFS_V4_1 */
 		case -NFS4ERR_DELAY:
 			nfs_inc_server_stats(server, NFSIOS_DELAY);
Signature · 1 line
1.8.3.1
#1 lore raw
Quoted 10 lines · On Fri, 2013-11-15 at 16:36 -0500, [email protected] wrote:

From: Andy Adamson <[email protected]>

When the state manager is processing the NFS4CLNT_DELEGRETURN flag, session draining is off, but DELEGRETURN can still get a session error. The async handler calls nfs4_schedule_session_recovery returns -EAGAIN, and the DELEGRETURN done then restarts the RPC task in the prepare state. With the state manager still processing the NFS4CLNT_DELEGRETURN flag with session draining off, these DELEGRETURNs will cycle with errors filling up the session slots.

Quoted 5 lines

This prevents OPEN reclaims (from nfs_delegation_claim_opens) required by the NFS4CLNT_DELEGRETURN state manager processing from completing, hanging the state manager in the __rpc_wait_for_completion_task in nfs4_run_open_task as seen in this kernel thread dump:

Hi Andy,

There is a second patch that goes with this problem. Please see the following attachment.

Cheers
Trond

Signature · 6 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com

From 20a4067243f81c1417bf62ecea7697b79901926f Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Tue, 19 Nov 2013 16:34:14 -0500
Subject: [PATCH] NFSv4: Update list of irrecoverable errors on DELEGRETURN

If the DELEGRETURN errors out with something like NFS4ERR_BAD_STATEID then there is no recovery possible.

Also, the client must not assume that the NFSv4 lease has been renewed when it sees an error on DELEGRETURN.

---
 fs/nfs/nfs4proc.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
fs/nfs/nfs4proc.c +6 -2
index 1f4edfbb4a70..aa16a22ad349 100644
@@ -4988,10 +4988,14 @@ static void nfs4_delegreturn_done(struct rpc_task *task, void *calldata)
 
 	trace_nfs4_delegreturn_exit(&data->args, &data->res, task->tk_status);
 	switch (task->tk_status) {
-	case -NFS4ERR_STALE_STATEID:
-	case -NFS4ERR_EXPIRED:
 	case 0:
 		renew_lease(data->res.server, data->timestamp);
+	case -NFS4ERR_ADMIN_REVOKED:
+	case -NFS4ERR_DELEG_REVOKED:
+	case -NFS4ERR_BAD_STATEID:
+	case -NFS4ERR_OLD_STATEID:
+	case -NFS4ERR_STALE_STATEID:
+	case -NFS4ERR_EXPIRED:
 		break;
 	default:
 		if (nfs4_async_handle_error(task, data->res.server, NULL) ==
Signature · 1 line
1.8.3.1
#2 lore raw
Quoted 2 lines · Trond Myklebust

There is a second patch that goes with this problem. Please see the following attachment.

V2: Don't return an error when we know that the stateid is no longer valid.

Signature · 6 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com

From ba64b3649ce323004cc2bd75e5c9f785e6386779 Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Tue, 19 Nov 2013 16:34:14 -0500
Subject: [PATCH v2] NFSv4: Update list of irrecoverable errors on DELEGRETURN

If the DELEGRETURN errors out with something like NFS4ERR_BAD_STATEID then there is no recovery possible. Just quit without returning an error.

Also, note that the client must not assume that the NFSv4 lease has been renewed when it sees an error on DELEGRETURN.

---
 fs/nfs/nfs4proc.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
fs/nfs/nfs4proc.c +8 -2
index 1f4edfbb4a70..ca36d0d50b7d 100644
@@ -4988,11 +4988,17 @@ static void nfs4_delegreturn_done(struct rpc_task *task, void *calldata)
 
 	trace_nfs4_delegreturn_exit(&data->args, &data->res, task->tk_status);
 	switch (task->tk_status) {
-	case -NFS4ERR_STALE_STATEID:
-	case -NFS4ERR_EXPIRED:
 	case 0:
 		renew_lease(data->res.server, data->timestamp);
 		break;
+	case -NFS4ERR_ADMIN_REVOKED:
+	case -NFS4ERR_DELEG_REVOKED:
+	case -NFS4ERR_BAD_STATEID:
+	case -NFS4ERR_OLD_STATEID:
+	case -NFS4ERR_STALE_STATEID:
+	case -NFS4ERR_EXPIRED:
+		task->tk_status = 0;
+		break;
 	default:
 		if (nfs4_async_handle_error(task, data->res.server, NULL) ==
 				-EAGAIN) {
Signature · 1 line
1.8.3.1
#3 lore raw

Why not recover the lease? Shouldn't NFS4ERR_EXPIRED be left for the async handler?

-->Andy

Quoted 15 lines · Trond
Quoted 2 lines · Trond Myklebust

There is a second patch that goes with this problem. Please see the following attachment.

V2: Don't return an error when we know that the stateid is no longer valid.

Signature · 7 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com
<0001-NFSv4-Update-list-of-irrecoverable-errors-on-DELEGRE.patch>
#4 lore raw
Quoted 1 line · Andy

Why not recover the lease? Shouldn't NFS4ERR_EXPIRED be left for the async handler?

No, because then it will retry the operation. Since it is impossible to recover the delegation at this point, then that will result in another failure.

Quoted 21 lines

-->Andy

Quoted 15 lines · Trond
Quoted 2 lines · Trond Myklebust

There is a second patch that goes with this problem. Please see the following attachment.

V2: Don't return an error when we know that the stateid is no longer valid.

Signature · 7 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com
<0001-NFSv4-Update-list-of-irrecoverable-errors-on-DELEGRE.patch>
Signature · 6 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com
#5 lore raw
In reply to Myklebust, Trond

Hi Trond,

Quoted 2 lines · Trond

There is a second patch that goes with this problem. Please see the following attachment.

ba64b36 NFSv4: Update list of irrecoverable errors on DELEGRETURN is cc'ed for stable

If these two patches are connected, why the first one:
d3b173a NFSv4 wait on recovery for async session errors is not cc'ed for stable as well?

For example in stable v3.10.x we currently have:

        case -NFS4ERR_SEQ_MISORDERED:
            dprintk("%s ERROR %d, Reset session\n", __func__,
                task->tk_status);
            nfs4_schedule_session_recovery(clp->cl_session, task->tk_status);
            task->tk_status = 0;
            return -EAGAIN;

but with the last pacthes we now have in mainline:

        case -NFS4ERR_SEQ_MISORDERED:
            dprintk("%s ERROR: %d Reset session\n", __func__,
                errorcode);
            nfs4_schedule_session_recovery(clp->cl_session, errorcode);
            goto wait_on_recovery;

and this was orignally change because of commit f1478c1 NFS: Re-use exit code in nfs4_async_handle_error()

The result in stable v3.10.x doesn't really make sense to me. Did I missed something?

Regards,

Signature · 1 line
William
#6 lore raw
Quoted 6 lines · William Dauchy

ba64b36 NFSv4: Update list of irrecoverable errors on DELEGRETURN is cc'ed for stable

If these two patches are connected, why the first one:
d3b173a NFSv4 wait on recovery for async session errors is not cc'ed for stable as well?

you last push
4a82fd7 NFSv4 wait on recovery for async session errors is now cc'ed for stable

Thanks,

Signature · 1 line
William