nfsd: Reset cb_status in nfsd4_cb_prepare() at retrying
Subsystems (2)
PATCH 1/2 nfsd: Reset cb_status in nfsd4_cb_prepare() at retrying
nfsd enters a infinite loop and print message per 10 seconds,
May 31 18:33:52 test-server kernel: Error sending entire callback! May 31 18:34:01 test-server kernel: Error sending entire callback!
It is caused by a cb_layoutreturn got error -10008 (NFS4ERR_DELAY), and then, the client crash, nfsd enter the infinite loop.
bc_sendto --> call_timeout --> nfsd4_cb_done --> nfsd4_cb_layout_done
with error -10008 --> rpc_delay(task, HZ/100) --> bc_sendto ...
--- fs/nfsd/nfs4callback.c | 1 + 1 file changed, 1 insertion(+)
fs/nfsd/nfs4callback.c +1 -0
@@ -875,6 +875,7 @@ static void nfsd4_cb_prepare(struct rpc_task *task, void *calldata)u32 minorversion = clp->cl_minorversion; cb->cb_minorversion = minorversion; + cb->cb_status =0 ;if (minorversion) {if (!nfsd41_cb_get_slot (clp, task))return ;
Signature · 1 line
2.4.2
Quoted 10 lines · Kinglong Mee
nfsd enters a infinite loop and print message per 10 seconds,
May 31 18:33:52 test-server kernel: Error sending entire callback! May 31 18:34:01 test-server kernel: Error sending entire callback!It is caused by a
cb_layoutreturngot error -10008 (NFS4ERR_DELAY), and then, the client crash, nfsd enter the infinite loop.
bc_sendto-->call_timeout-->nfsd4_cb_done-->nfsd4_cb_layout_done
with error -10008 -->rpc_delay(task, HZ/100)-->bc_sendto...
How are you reproducing this?
--b.
Quoted 20 lines
Signed-off-by Kinglong Mee <[email protected]>--- fs/nfsd/nfs4callback.c | 1 + 1 file changed, 1 insertion(+)fs/nfsd/nfs4callback.c +1
-0@@ -875,6 +875,7 @@ static void nfsd4_cb_prepare(struct rpc_task *task, void *calldata)u32 minorversion = clp->cl_minorversion; cb->cb_minorversion = minorversion; + cb->cb_status =0 ;if (minorversion) {if (!nfsd41_cb_get_slot (clp, task))return ;Signature · 1 line
2.4.2
Quoted 6 lines · J. Bruce Fields
Quoted 5 lines · Kinglong Mee
nfsd enters a infinite loop and print message per 10 seconds,
May 31 18:33:52 test-server kernel: Error sending entire callback! May 31 18:34:01 test-server kernel: Error sending entire callback!
Quoted 7 lines
Quoted 5 lines
It is caused by a
cb_layoutreturngot error -10008 (NFS4ERR_DELAY), and then, the client crash, nfsd enter the infinite loop.
bc_sendto-->call_timeout-->nfsd4_cb_done-->nfsd4_cb_layout_done
with error -10008 -->rpc_delay(task, HZ/100)-->bc_sendto...How are you reproducing this?
Yes,
I test it by xfstests 074 with nfs client's kdump is on, set CONFIG_DEFAULT_HUNG_TASK_TIMEOUT, and client's blkmapd is down.
1. nfs client's write operation will get the layout of file,
and then the getdeviceinfo,
2. but layout segment is not record by client for blkmapd is down,
3. client write data by sending WRITE to server,
4. nfs server will recall the layout of the file before WRITE,
5. network error cause the client reset the session and return NFS4ERR_DELAY,
6. so client's WRITE operation is waiting the reply,
if the task hang 120s, client will crash.
7. so that, the next bc_sendto will fail with TIMEOUT,
and cb_status is NFS4ERR_DELAY.
thanks,
Kinglong Mee
Quoted 24 lines
--b.
Quoted 20 lines
Signed-off-by Kinglong Mee <[email protected]>--- fs/nfsd/nfs4callback.c | 1 + 1 file changed, 1 insertion(+)fs/nfsd/nfs4callback.c +1
-0@@ -875,6 +875,7 @@ static void nfsd4_cb_prepare(struct rpc_task *task, void *calldata)u32 minorversion = clp->cl_minorversion; cb->cb_minorversion = minorversion; + cb->cb_status =0 ;if (minorversion) {if (!nfsd41_cb_get_slot (clp, task))return ;Signature · 1 line
2.4.2
Quoted 20 lines · Kinglong Mee
Quoted 6 lines · J. Bruce Fields
Quoted 5 lines · Kinglong Mee
nfsd enters a infinite loop and print message per 10 seconds,
May 31 18:33:52 test-server kernel: Error sending entire callback! May 31 18:34:01 test-server kernel: Error sending entire callback!Quoted 7 lines
Quoted 5 lines
It is caused by a
cb_layoutreturngot error -10008 (NFS4ERR_DELAY), and then, the client crash, nfsd enter the infinite loop.
bc_sendto-->call_timeout-->nfsd4_cb_done-->nfsd4_cb_layout_done
with error -10008 -->rpc_delay(task, HZ/100)-->bc_sendto...How are you reproducing this?
Yes,
I test it by xfstests 074 with nfs client's kdump is on, set
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT, and client's blkmapd is down.
Quoted 10 lines
1. nfs client's write operation will get the layout of file,
and then the getdeviceinfo,
2. but layout segment is not record by client for blkmapd is down,
3. client write data by sending WRITE to server,
4. nfs server will recall the layout of the file before WRITE,
5. network error cause the client reset the session and returnNFS4ERR_DELAY,
6. so client's WRITE operation is waiting the reply,
if the task hang 120s, client will crash.
7. so that, the nextbc_sendtowill fail with TIMEOUT,
andcb_statusisNFS4ERR_DELAY.
OK, that's complicated. Sounds like you're giving this code a workout--thanks. I'll add the reproducer to the changelog....
--b.
Quoted 28 lines
thanks,
Kinglong MeeQuoted 24 lines
--b.
Quoted 20 lines
Signed-off-by Kinglong Mee <[email protected]>--- fs/nfsd/nfs4callback.c | 1 + 1 file changed, 1 insertion(+)fs/nfsd/nfs4callback.c +1
-0@@ -875,6 +875,7 @@ static void nfsd4_cb_prepare(struct rpc_task *task, void *calldata)u32 minorversion = clp->cl_minorversion; cb->cb_minorversion = minorversion; + cb->cb_status =0 ;if (minorversion) {if (!nfsd41_cb_get_slot (clp, task))return ;Signature · 1 line
2.4.2