Skip to contentSkip to conversation

NFSD: fix hang in nfsd4_shutdown_callback

PATCH Superseded Dai Ngo 4 participants 6 messages Jan 29, 2025

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Dai Ngo

author reviewer
#0 lore raw

PATCH 1/1 NFSD: fix hang in nfsd4_shutdown_callback

If nfs4_client is in COURTESY state then there is no point to retry the callback. This causes nfsd4_shutdown_callback to hang since cl_cb_inflight is not 0. This hang lasts about 15 minutes until TCP notifies NFSD that the connection was closed.

This patch modifies nfsd4_cb_sequence_done to skip the restart the RPC if nfs4_client is in COURTESY state.

---
 fs/nfsd/nfs4callback.c | 5 +++++
 1 file changed, 5 insertions(+)
fs/nfsd/nfs4callback.c +5 -0
index 50e468bdb8d4..c90f94898cc5 100644
@@ -1372,6 +1372,11 @@ static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback
 		ret = false;
 		break;
 	case 1:
+		if (clp->cl_state == NFSD4_COURTESY) {
+			nfsd4_mark_cb_fault(cb->cb_clp);
+			ret = false;
+			break;
+		}
 		/*
 		 * cb_seq_status remains 1 if an RPC Reply was never
 		 * received. NFSD can't know if the client processed
Signature · 1 line
2.43.5
#1 lore raw
Quoted 18 lines · Dai Ngo

If nfs4_client is in COURTESY state then there is no point to retry the callback. This causes nfsd4_shutdown_callback to hang since cl_cb_inflight is not 0. This hang lasts about 15 minutes until TCP notifies NFSD that the connection was closed.

This patch modifies nfsd4_cb_sequence_done to skip the restart the RPC if nfs4_client is in COURTESY state.

---
 fs/nfsd/nfs4callback.c | 5 +++++
 1 file changed, 5 insertions(+)
fs/nfsd/nfs4callback.c +0 -0
index 50e468bdb8d4..c90f94898cc5 100644

fs/nfsd/nfs4callback.c static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback · line 1379

Quoted 8 lines
fs/nfsd/nfs4callback.c +5 -0
@@ -1372,6 +1372,11 @@ static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback
 		ret = false;
 		break;
 	case 1:
+		if (clp->cl_state == NFSD4_COURTESY) {
+			nfsd4_mark_cb_fault(cb->cb_clp);
+			ret = false;
+			break;
+		}

We could do toss this operation here or at the top of nfsd4_run_cb_work().

Quoted 3 lines
fs/nfsd/nfs4callback.c +0 -0
 		/*
 		 * cb_seq_status remains 1 if an RPC Reply was never
 		 * received. NFSD can't know if the client processed
Signature · 1 line
Chuck Lever
maintainer
#2 lore raw
Quoted 14 lines · Chuck Lever
Quoted 13 lines · Dai Ngo

If nfs4_client is in COURTESY state then there is no point to retry the callback. This causes nfsd4_shutdown_callback to hang since cl_cb_inflight is not 0. This hang lasts about 15 minutes until TCP notifies NFSD that the connection was closed.

This patch modifies nfsd4_cb_sequence_done to skip the restart the RPC if nfs4_client is in COURTESY state.

---
 fs/nfsd/nfs4callback.c | 5 +++++
 1 file changed, 5 insertions(+)
Quoted 17 lines

fs/nfsd/nfs4callback.c static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback · line 1379

Quoted 13 lines
fs/nfsd/nfs4callback.c +5 -0
index 50e468bdb8d4..c90f94898cc5 100644
@@ -1372,6 +1372,11 @@ static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback
 		ret = false;
 		break;
 	case 1:
+		if (clp->cl_state == NFSD4_COURTESY) {
+			nfsd4_mark_cb_fault(cb->cb_clp);
+			ret = false;
+			break;
+		}

We could do toss this operation here or at the top of nfsd4_run_cb_work().

Like Olga, I'm wondering if this is the culprit on the recently reported rpc_shutdown_client hangs.

I'm assuming that with a courtesy client that we don't want to do any callbacks? If that's the case, then I think doing it early in nfsd4_run_cb_work() would be better. That would prevent new cb's from being queued until the cl_state changes too.

Quoted 6 lines

fs/nfsd/nfs4callback.c static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback · line 1382

Quoted 4 lines
fs/nfsd/nfs4callback.c +0 -0
 		/*
 		 * cb_seq_status remains 1 if an RPC Reply was never
 		 * received. NFSD can't know if the client processed

Nice catch though!

Signature · 1 line
Jeff Layton <[email protected]>
author reviewer
#3 lore raw
Quoted 31 lines · Jeff Layton
Quoted 14 lines · Chuck Lever
Quoted 13 lines · Dai Ngo

If nfs4_client is in COURTESY state then there is no point to retry the callback. This causes nfsd4_shutdown_callback to hang since cl_cb_inflight is not 0. This hang lasts about 15 minutes until TCP notifies NFSD that the connection was closed.

This patch modifies nfsd4_cb_sequence_done to skip the restart the RPC if nfs4_client is in COURTESY state.

---
  fs/nfsd/nfs4callback.c | 5 +++++
  1 file changed, 5 insertions(+)
Quoted 16 lines

fs/nfsd/nfs4callback.c static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback · line 1379

Quoted 13 lines
fs/nfsd/nfs4callback.c +5 -0
index 50e468bdb8d4..c90f94898cc5 100644
@@ -1372,6 +1372,11 @@ static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback
  		ret = false;
  		break;
  	case 1:
+		if (clp->cl_state == NFSD4_COURTESY) {
+			nfsd4_mark_cb_fault(cb->cb_clp);
+			ret = false;
+			break;
+		}

We could do toss this operation here or at the top of nfsd4_run_cb_work().

Quoted 7 lines

Like Olga, I'm wondering if this is the culprit on the recently reported rpc_shutdown_client hangs.

I'm assuming that with a courtesy client that we don't want to do any callbacks? If that's the case, then I think doing it early in nfsd4_run_cb_work() would be better. That would prevent new cb's from being queued until the cl_state changes too.

I'll move this check into nfsd4_run_cb_work.

Thanks Chuck and Jeff.
-Dai

Quoted 7 lines
Quoted 6 lines

fs/nfsd/nfs4callback.c static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback · line 1382

Quoted 5 lines
fs/nfsd/nfs4callback.c +0 -0
  		/*
  		 * cb_seq_status remains 1 if an RPC Reply was never
  		 * received. NFSD can't know if the client processed

Nice catch though!

#4 lore raw
In reply to Dai Ngo
Quoted 9 lines · Dai Ngo

If nfs4_client is in COURTESY state then there is no point to retry the callback. This causes nfsd4_shutdown_callback to hang since cl_cb_inflight is not 0. This hang lasts about 15 minutes until TCP notifies NFSD that the connection was closed.

This patch modifies nfsd4_cb_sequence_done to skip the restart the RPC if nfs4_client is in COURTESY state.

Curious, does this patch address the problem seen/discussed in the thread "NFSD threads hang when destroying a session or client ID" or that is something else?

Quoted 25 lines
---
 fs/nfsd/nfs4callback.c | 5 +++++
 1 file changed, 5 insertions(+)
fs/nfsd/nfs4callback.c +5 -1
index 50e468bdb8d4..c90f94898cc5 100644
@@ -1372,6 +1372,11 @@ static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback
                ret = false;
                break;
        case 1:
+               if (clp->cl_state == NFSD4_COURTESY) {
+                       nfsd4_mark_cb_fault(cb->cb_clp);
+                       ret = false;
+                       break;
+               }
                /*
                 * cb_seq_status remains 1 if an RPC Reply was never
                 * received. NFSD can't know if the client processed
--

2.43.5

author reviewer
#5 lore raw
Quoted 6 lines · Olga Kornievskaia
Quoted 5 lines · Dai Ngo

If nfs4_client is in COURTESY state then there is no point to retry the callback. This causes nfsd4_shutdown_callback to hang since cl_cb_inflight is not 0. This hang lasts about 15 minutes until TCP notifies NFSD that the connection was closed.

Quoted 6 lines
Quoted 3 lines

This patch modifies nfsd4_cb_sequence_done to skip the restart the RPC if nfs4_client is in COURTESY state.

Curious, does this patch address the problem seen/discussed in the thread "NFSD threads hang when destroying a session or client ID" or that is something else?

I'm not sure about the symptom in 6.1.y kernel.

The problem that I reproduced here has the same symptom described in the thread for newer kernel; NFSv4 callback shutdown hang while waiting for cl_cb_inflight to drop to 0.

-Dai

Quoted 26 lines
Quoted 26 lines
---
  fs/nfsd/nfs4callback.c | 5 +++++
  1 file changed, 5 insertions(+)
fs/nfsd/nfs4callback.c +5 -1
index 50e468bdb8d4..c90f94898cc5 100644
@@ -1372,6 +1372,11 @@ static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback
                 ret = false;
                 break;
         case 1:
+               if (clp->cl_state == NFSD4_COURTESY) {
+                       nfsd4_mark_cb_fault(cb->cb_clp);
+                       ret = false;
+                       break;
+               }
                 /*
                  * cb_seq_status remains 1 if an RPC Reply was never
                  * received. NFSD can't know if the client processed
--

2.43.5