NFSD: fix hang in nfsd4_shutdown_callback
Subsystems (2)
Dai Ngo
reviewerPATCH 1/1 NFSD: fix hang in nfsd4_shutdown_callback
If nfs4_client is in COURTESY state then there is no point to retry the callback. This causes nfsd4_shutdown_callback to hang since cl_cb_inflight is not 0. This hang lasts about 15 minutes until TCP notifies NFSD that the connection was closed.
This patch modifies nfsd4_cb_sequence_done to skip the restart the RPC if nfs4_client is in COURTESY state.
--- fs/nfsd/nfs4callback.c | 5 +++++ 1 file changed, 5 insertions(+)
fs/nfsd/nfs4callback.c +5 -0
@@ -1372,6 +1372,11 @@ static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback ret = false;break ;case 1 : +if (clp->cl_state ==NFSD4_COURTESY ) { +nfsd4_mark_cb_fault (cb->cb_clp); + ret = false; +break ; + }/* * cb_seq_status remains 1 if an RPC Reply was never * received. NFSD can't know if the client processed
Signature · 1 line
2.43.5
Quoted 18 lines · Dai Ngo
If
nfs4_clientis in COURTESY state then there is no point to retry the callback. This causesnfsd4_shutdown_callbackto hang sincecl_cb_inflightis not 0. This hang lasts about 15 minutes until TCP notifies NFSD that the connection was closed.This patch modifies
nfsd4_cb_sequence_doneto skip the restart the RPC ifnfs4_clientis in COURTESY state.Signed-off-by Dai Ngo <[email protected]>--- fs/nfsd/nfs4callback.c | 5 +++++ 1 file changed, 5 insertions(+)fs/nfsd/nfs4callback.c +0
-0
fs/nfsd/nfs4callback.c static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback · line 1379
Quoted 8 lines
fs/nfsd/nfs4callback.c +5
-0@@ -1372,6 +1372,11 @@ static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback ret = false;break ;case 1 : +if (clp->cl_state ==NFSD4_COURTESY ) { +nfsd4_mark_cb_fault (cb->cb_clp); + ret = false; +break ; + }
We could do toss this operation here or at the top of nfsd4_run_cb_work().
Quoted 3 lines
fs/nfsd/nfs4callback.c +0
-0/* * cb_seq_status remains 1 if an RPC Reply was never * received. NFSD can't know if the client processed
Signature · 1 line
Chuck Lever
Jeff Layton
maintainerQuoted 14 lines · Chuck Lever
Quoted 13 lines · Dai Ngo
If
nfs4_clientis in COURTESY state then there is no point to retry the callback. This causesnfsd4_shutdown_callbackto hang sincecl_cb_inflightis not 0. This hang lasts about 15 minutes until TCP notifies NFSD that the connection was closed.This patch modifies
nfsd4_cb_sequence_doneto skip the restart the RPC ifnfs4_clientis in COURTESY state.Signed-off-by Dai Ngo <[email protected]>--- fs/nfsd/nfs4callback.c | 5 +++++ 1 file changed, 5 insertions(+)
Quoted 17 lines
fs/nfsd/nfs4callback.c static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback · line 1379
Quoted 13 lines
fs/nfsd/nfs4callback.c +5
-0@@ -1372,6 +1372,11 @@ static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback ret = false;break ;case 1 : +if (clp->cl_state ==NFSD4_COURTESY ) { +nfsd4_mark_cb_fault (cb->cb_clp); + ret = false; +break ; + }We could do toss this operation here or at the top of
nfsd4_run_cb_work().
Like Olga, I'm wondering if this is the culprit on the recently reported rpc_shutdown_client hangs.
I'm assuming that with a courtesy client that we don't want to do any callbacks? If that's the case, then I think doing it early in nfsd4_run_cb_work() would be better. That would prevent new cb's from being queued until the cl_state changes too.
Quoted 6 lines
fs/nfsd/nfs4callback.c static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback · line 1382
Quoted 4 lines
fs/nfsd/nfs4callback.c +0
-0/* * cb_seq_status remains 1 if an RPC Reply was never * received. NFSD can't know if the client processed
Nice catch though!
Signature · 1 line
Jeff Layton <[email protected]>
Dai Ngo
reviewerQuoted 31 lines · Jeff Layton
Quoted 14 lines · Chuck Lever
Quoted 13 lines · Dai Ngo
If
nfs4_clientis in COURTESY state then there is no point to retry the callback. This causesnfsd4_shutdown_callbackto hang sincecl_cb_inflightis not 0. This hang lasts about 15 minutes until TCP notifies NFSD that the connection was closed.This patch modifies
nfsd4_cb_sequence_doneto skip the restart the RPC ifnfs4_clientis in COURTESY state.Signed-off-by Dai Ngo <[email protected]>--- fs/nfsd/nfs4callback.c | 5 +++++ 1 file changed, 5 insertions(+)Quoted 16 lines
fs/nfsd/nfs4callback.c static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback · line 1379
Quoted 13 lines
fs/nfsd/nfs4callback.c +5
-0@@ -1372,6 +1372,11 @@ static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback ret = false;break ;case 1 : +if (clp->cl_state ==NFSD4_COURTESY ) { +nfsd4_mark_cb_fault (cb->cb_clp); + ret = false; +break ; + }We could do toss this operation here or at the top of
nfsd4_run_cb_work().
Quoted 7 lines
Like Olga, I'm wondering if this is the culprit on the recently reported
rpc_shutdown_clienthangs.I'm assuming that with a courtesy client that we don't want to do any callbacks? If that's the case, then I think doing it early in
nfsd4_run_cb_work()would be better. That would prevent new cb's from being queued until thecl_statechanges too.
I'll move this check into nfsd4_run_cb_work.
Thanks Chuck and Jeff.
-Dai
Quoted 7 lines
Quoted 6 lines
fs/nfsd/nfs4callback.c static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback · line 1382
Quoted 5 lines
fs/nfsd/nfs4callback.c +0
-0/* * cb_seq_status remains 1 if an RPC Reply was never * received. NFSD can't know if the client processed Nice catch though!
Quoted 9 lines · Dai Ngo
If
nfs4_clientis in COURTESY state then there is no point to retry the callback. This causesnfsd4_shutdown_callbackto hang sincecl_cb_inflightis not 0. This hang lasts about 15 minutes until TCP notifies NFSD that the connection was closed.This patch modifies
nfsd4_cb_sequence_doneto skip the restart the RPC ifnfs4_clientis in COURTESY state.
Curious, does this patch address the problem seen/discussed in the thread "NFSD threads hang when destroying a session or client ID" or that is something else?
Quoted 25 lines
Signed-off-by Dai Ngo <[email protected]>--- fs/nfsd/nfs4callback.c | 5 +++++ 1 file changed, 5 insertions(+)fs/nfsd/nfs4callback.c +5
-1@@ -1372,6 +1372,11 @@ static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback ret = false;break ;case 1 : +if (clp->cl_state ==NFSD4_COURTESY ) { +nfsd4_mark_cb_fault (cb->cb_clp); + ret = false; +break ; + }/* * cb_seq_status remains 1 if an RPC Reply was never * received. NFSD can't know if the client processed -- 2.43.5
Dai Ngo
reviewerQuoted 6 lines · Olga Kornievskaia
Quoted 5 lines · Dai Ngo
If
nfs4_clientis in COURTESY state then there is no point to retry the callback. This causesnfsd4_shutdown_callbackto hang sincecl_cb_inflightis not 0. This hang lasts about 15 minutes until TCP notifies NFSD that the connection was closed.
Quoted 6 lines
Quoted 3 lines
This patch modifies
nfsd4_cb_sequence_doneto skip the restart the RPC ifnfs4_clientis in COURTESY state.Curious, does this patch address the problem seen/discussed in the thread "NFSD threads hang when destroying a session or client ID" or that is something else?
I'm not sure about the symptom in 6.1.y kernel.
The problem that I reproduced here has the same symptom described in the thread for newer kernel; NFSv4 callback shutdown hang while waiting for cl_cb_inflight to drop to 0.
-Dai
Quoted 26 lines
Quoted 26 lines
Signed-off-by Dai Ngo <[email protected]>--- fs/nfsd/nfs4callback.c | 5 +++++ 1 file changed, 5 insertions(+)fs/nfsd/nfs4callback.c +5
-1@@ -1372,6 +1372,11 @@ static bool nfsd4_cb_sequence_done(struct rpc_task *task, struct nfsd4_callback ret = false;break ;case 1 : +if (clp->cl_state ==NFSD4_COURTESY ) { +nfsd4_mark_cb_fault (cb->cb_clp); + ret = false; +break ; + }/* * cb_seq_status remains 1 if an RPC Reply was never * received. NFSD can't know if the client processed -- 2.43.5