SUNRPC: Ensure that call_connect times out correctly
Subsystems (3)
PATCH 1/2 SUNRPC: Ensure that call_connect times out correctly
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
--- net/sunrpc/clnt.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
net/sunrpc/clnt.c +3 -5
@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :
@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;
Signature · 1 line
1.8.5.3
PATCH 2/2 SUNRPC: Ensure that call_bind times out correctly
If the rpcbind server is unavailable, we still want the RPC client to respect the timeout.
--- net/sunrpc/clnt.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
net/sunrpc/clnt.c +2 -3
@@ -1728,9 +1728,7 @@ call_bind_status(struct rpc_task *task)case -EPROTONOSUPPORT :dprintk ("RPC: %5u remote rpcbind version unavailable, retrying\n" , task->tk_pid); - task->tk_status =0 ; - task->tk_action = call_bind; -return ; + goto retry_timeout;case -ECONNREFUSED :/* connection problems */ case -ECONNRESET :case -ECONNABORTED :
@@ -1756,6 +1754,7 @@ call_bind_status(struct rpc_task *task)return ; retry_timeout: + task->tk_status =0 ; task->tk_action = call_timeout; }
Signature · 1 line
1.8.5.3
Re: [PATCH 2/2] SUNRPC: Ensure that call_bind times out correctly
Quoted 4 lines · Trond Myklebust
If the rpcbind server is unavailable, we still want the RPC client to respect the timeout.
Signed-off-by Trond Myklebust <[email protected]>
steved.
Quoted 28 lines
--- net/sunrpc/clnt.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)net/sunrpc/clnt.c +2
-3@@ -1728,9 +1728,7 @@ call_bind_status(struct rpc_task *task)case -EPROTONOSUPPORT :dprintk ("RPC: %5u remote rpcbind version unavailable, retrying\n" , task->tk_pid); - task->tk_status =0 ; - task->tk_action = call_bind; -return ; + goto retry_timeout;case -ECONNREFUSED :/* connection problems */ case -ECONNRESET :case -ECONNABORTED :@@ -1756,6 +1754,7 @@ call_bind_status(struct rpc_task *task)return ; retry_timeout: + task->tk_status =0 ; task->tk_action = call_timeout; }
Hey,
Quoted 26 lines · Trond Myklebust
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
Fixes 561ec1603171 (SUNRPC: call_connect_status should recheck bind..)Signed-off-by Trond Myklebust <[email protected]>--- net/sunrpc/clnt.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)net/sunrpc/clnt.c +0
-4@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :
net/sunrpc/clnt.c call_connect_status(struct rpc_task *task) · line 1816
Quoted 10 lines
net/sunrpc/clnt.c +3
-1@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;
How is this support to work if the trunking code still ignores timeouts?
[ 2076.045176] NFS: nfs4_discover_server_trunking after status -110, retrying
steved.
Quoted 41 lines · Steve Dickson
Hey,
Quoted 26 lines · Trond Myklebust
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
Fixes 561ec1603171 (SUNRPC: call_connect_status should recheck bind..)Signed-off-by Trond Myklebust <[email protected]>---
net/sunrpc/clnt.c| 8 +++-----1 file changed, 3 insertions(+), 5 deletions(-)net/sunrpc/clnt.c +0
-4@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :net/sunrpc/clnt.c call_connect_status(struct rpc_task *task) · line 1816
Quoted 10 lines
net/sunrpc/clnt.c +3
-1@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;How is this support to work if the trunking code still ignores timeouts?
Quoted 1 line
[ 2076.045176] NFS: nfs4_discover_server_trunking after status -110, retrying
The above patch fixes the regression that Neil tracked down in Linux 3.12, and that affects the generic RPC handling of soft timeouts.
The trunking code's handling of ETIMEDOUT has been there since Linux 3.7 and hasn’t changed, so I really don’t see how it can have worked at one time before 3.12.
_________________________________
Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]
Quoted 44 lines · Trond Myklebust
Quoted 41 lines · Steve Dickson
Hey,
Quoted 26 lines · Trond Myklebust
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
Fixes 561ec1603171 (SUNRPC: call_connect_status should recheck bind..)Signed-off-by Trond Myklebust <[email protected]>---
net/sunrpc/clnt.c| 8 +++-----1 file changed, 3 insertions(+), 5 deletions(-)net/sunrpc/clnt.c +0
-4@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :net/sunrpc/clnt.c call_connect_status(struct rpc_task *task) · line 1816
Quoted 10 lines
net/sunrpc/clnt.c +3
-1@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;How is this support to work if the trunking code still ignores timeouts?
Quoted 7 lines
Quoted 1 line
[ 2076.045176] NFS: nfs4_discover_server_trunking after status -110, retryingThe above patch fixes the regression that Neil tracked down in Linux 3.12, and that affects the generic RPC handling of soft timeouts.
The trunking code's handling of ETIMEDOUT has been there since Linux 3.7 and hasn’t changed, so I really don’t see how it can have worked at one time before 3.12.
Maybe it been broken that long.... :-)
But here is the obvious loop that stop that hangs a mount forever:
#8 [ffff88007a22b7e8] rpc_call_sync at ffffffffa0220210 [sunrpc]
#9 [ffff88007a22b840] nfs4_proc_setclientid at ffffffffa0505c49 [nfsv4]
#10 [ffff88007a22b988] nfs40_discover_server_trunking at ffffffffa0514489 [nfsv4] #11 [ffff88007a22b9d0] nfs4_discover_server_trunking at ffffffffa0516f2d [nfsv4] #12 [ffff88007a22ba28] nfs4_init_client at ffffffffa051e9a4 [nfsv4] #13 [ffff88007a22bb20] nfs_get_client at ffffffffa04bd6ba [nfs]
#14 [ffff88007a22bb80] nfs4_set_client at ffffffffa051dfb0 [nfsv4] #15 [ffff88007a22bc00] nfs4_create_server at ffffffffa051f4ce [nfsv4] #16 [ffff88007a22bc88] nfs4_remote_mount at ffffffffa051790e [nfsv4] #17 [ffff88007a22bcb0] mount_fs at ffffffff811b3dd9
The SETCLIENT times out
NFS call setclientid auth=UNIX, 'Linux NFSv4.0 10.19.60.77/10.19.60.33 tcp' NFS reply setclientid: -110
The nfs4_discover_server_trunking() retries
NFS: nfs4_discover_server_trunking after status -110, retrying
The happens when there server is down and so the connections fail with ECONNREFUSED:
RPC: 2 call_connect_status (status -111)
The mount system call never times out in which it did in the past.
So who do I get the system call to time out again?
steved.
Quoted 72 lines · Steve Dickson
Quoted 44 lines · Trond Myklebust
Quoted 41 lines · Steve Dickson
Hey,
Quoted 26 lines · Trond Myklebust
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
Fixes 561ec1603171 (SUNRPC: call_connect_status should recheck bind..)Signed-off-by Trond Myklebust <[email protected]>---
net/sunrpc/clnt.c| 8 +++-----1 file changed, 3 insertions(+), 5 deletions(-)net/sunrpc/clnt.c +0
-4@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :net/sunrpc/clnt.c call_connect_status(struct rpc_task *task) · line 1816
Quoted 10 lines
net/sunrpc/clnt.c +3
-1@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;How is this support to work if the trunking code still ignores timeouts?
Quoted 7 lines
Quoted 1 line
[ 2076.045176] NFS: nfs4_discover_server_trunking after status -110, retryingThe above patch fixes the regression that Neil tracked down in Linux 3.12, and that affects the generic RPC handling of soft timeouts.
The trunking code's handling of ETIMEDOUT has been there since Linux 3.7 and hasn’t changed, so I really don’t see how it can have worked at one time before 3.12.
Maybe it been broken that long.... :-)
But here is the obvious loop that stop that hangs a mount forever:
#8 [
ffff88007a22b7e8]rpc_call_syncatffffffffa0220210[sunrpc]
#9 [ffff88007a22b840]nfs4_proc_setclientidatffffffffa0505c49[nfsv4]
#10 [ffff88007a22b988]nfs40_discover_server_trunkingatffffffffa0514489[nfsv4] #11 [ffff88007a22b9d0]nfs4_discover_server_trunkingatffffffffa0516f2d[nfsv4] #12 [ffff88007a22ba28]nfs4_init_clientatffffffffa051e9a4[nfsv4] #13 [ffff88007a22bb20]nfs_get_clientatffffffffa04bd6ba[nfs]
#14 [ffff88007a22bb80]nfs4_set_clientatffffffffa051dfb0[nfsv4] #15 [ffff88007a22bc00]nfs4_create_serveratffffffffa051f4ce[nfsv4] #16 [ffff88007a22bc88]nfs4_remote_mountatffffffffa051790e[nfsv4] #17 [ffff88007a22bcb0]mount_fsatffffffff811b3dd9The SETCLIENT times out
NFS call setclientid auth=UNIX, 'Linux NFSv4.0 10.19.60.77/10.19.60.33 tcp' NFS reply setclientid: -110
Quoted 8 lines
The
nfs4_discover_server_trunking()retries
NFS:nfs4_discover_server_trunkingafter status -110, retryingThe happens when there server is down and so the connections fail with ECONNREFUSED:
RPC: 2call_connect_status(status -111)The mount system call never times out in which it did in the past.
Why should a mount system call time out other than perhaps in the case of a soft mount?
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, PrimaryData [email protected]
Quoted 76 lines · Trond Myklebust
Quoted 75 lines · Steve Dickson
Quoted 44 lines · Trond Myklebust
Quoted 41 lines · Steve Dickson
Hey,
Quoted 26 lines · Trond Myklebust
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
Fixes 561ec1603171 (SUNRPC: call_connect_status should recheck bind..)Signed-off-by Trond Myklebust <[email protected]>---
net/sunrpc/clnt.c| 8 +++-----1 file changed, 3 insertions(+), 5 deletions(-)net/sunrpc/clnt.c +0
-4@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :net/sunrpc/clnt.c call_connect_status(struct rpc_task *task) · line 1816
Quoted 10 lines
net/sunrpc/clnt.c +3
-1@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;How is this support to work if the trunking code still ignores timeouts?
Quoted 7 lines
Quoted 1 line
[ 2076.045176] NFS: nfs4_discover_server_trunking after status -110, retryingThe above patch fixes the regression that Neil tracked down in Linux 3.12, and that affects the generic RPC handling of soft timeouts.
The trunking code's handling of ETIMEDOUT has been there since Linux 3.7 and hasn’t changed, so I really don’t see how it can have worked at one time before 3.12.
Maybe it been broken that long.... :-)
But here is the obvious loop that stop that hangs a mount forever:
#8 [
ffff88007a22b7e8]rpc_call_syncatffffffffa0220210[sunrpc]
#9 [ffff88007a22b840]nfs4_proc_setclientidatffffffffa0505c49[nfsv4]
#10 [ffff88007a22b988]nfs40_discover_server_trunkingatffffffffa0514489[nfsv4] #11 [ffff88007a22b9d0]nfs4_discover_server_trunkingatffffffffa0516f2d[nfsv4] #12 [ffff88007a22ba28]nfs4_init_clientatffffffffa051e9a4[nfsv4] #13 [ffff88007a22bb20]nfs_get_clientatffffffffa04bd6ba[nfs]
#14 [ffff88007a22bb80]nfs4_set_clientatffffffffa051dfb0[nfsv4] #15 [ffff88007a22bc00]nfs4_create_serveratffffffffa051f4ce[nfsv4] #16 [ffff88007a22bc88]nfs4_remote_mountatffffffffa051790e[nfsv4] #17 [ffff88007a22bcb0]mount_fsatffffffff811b3dd9The SETCLIENT times out
NFS call setclientid auth=UNIX, 'Linux NFSv4.0 10.19.60.77/10.19.60.33 tcp' NFS reply setclientid: -110The
nfs4_discover_server_trunking()retries
NFS:nfs4_discover_server_trunkingafter status -110, retrying
Quoted 9 lines
Quoted 5 lines
The happens when there server is down and so the connections fail with ECONNREFUSED:
RPC: 2call_connect_status(status -111)The mount system call never times out in which it did in the past.
Why should a mount system call time out other than perhaps in the case of a soft mount?
So the mount can go in background. As you know the -o bg is used in the /etc/fstab so the boot does not get hung up with downed servers. That's how it always worked...
steved.
Quoted 83 lines · Steve Dickson
Quoted 80 lines · Trond Myklebust
Quoted 79 lines · Steve Dickson
Quoted 44 lines · Trond Myklebust
Quoted 41 lines · Steve Dickson
Hey,
Quoted 26 lines · Trond Myklebust
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
Fixes 561ec1603171 (SUNRPC: call_connect_status should recheck bind..)Signed-off-by Trond Myklebust <[email protected]>---
net/sunrpc/clnt.c| 8 +++-----1 file changed, 3 insertions(+), 5 deletions(-)net/sunrpc/clnt.c +0
-4@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :net/sunrpc/clnt.c call_connect_status(struct rpc_task *task) · line 1816
Quoted 10 lines
net/sunrpc/clnt.c +3
-1@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;How is this support to work if the trunking code still ignores timeouts?
Quoted 7 lines
Quoted 1 line
[ 2076.045176] NFS: nfs4_discover_server_trunking after status -110, retryingThe above patch fixes the regression that Neil tracked down in Linux 3.12, and that affects the generic RPC handling of soft timeouts.
The trunking code's handling of ETIMEDOUT has been there since Linux 3.7 and hasn’t changed, so I really don’t see how it can have worked at one time before 3.12.
Maybe it been broken that long.... :-)
But here is the obvious loop that stop that hangs a mount forever:
#8 [
ffff88007a22b7e8]rpc_call_syncatffffffffa0220210[sunrpc]
#9 [ffff88007a22b840]nfs4_proc_setclientidatffffffffa0505c49[nfsv4] #10 [ffff88007a22b988]nfs40_discover_server_trunkingatffffffffa0514489[nfsv4] #11 [ffff88007a22b9d0]nfs4_discover_server_trunkingatffffffffa0516f2d[nfsv4] #12 [ffff88007a22ba28]nfs4_init_clientatffffffffa051e9a4[nfsv4] #13 [ffff88007a22bb20]nfs_get_clientatffffffffa04bd6ba[nfs]
#14 [ffff88007a22bb80]nfs4_set_clientatffffffffa051dfb0[nfsv4] #15 [ffff88007a22bc00]nfs4_create_serveratffffffffa051f4ce[nfsv4] #16 [ffff88007a22bc88]nfs4_remote_mountatffffffffa051790e[nfsv4] #17 [ffff88007a22bcb0]mount_fsatffffffff811b3dd9The SETCLIENT times out
NFS call setclientid auth=UNIX, 'Linux NFSv4.0 10.19.60.77/10.19.60.33 tcp' NFS reply setclientid: -110The
nfs4_discover_server_trunking()retries
NFS:nfs4_discover_server_trunkingafter status -110, retryingThe happens when there server is down and so the connections fail with ECONNREFUSED:
RPC: 2call_connect_status(status -111)
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
The mount system call never times out in which it did in the past.
Why should a mount system call time out other than perhaps in the case of a soft mount?
So the mount can go in background. As you know the -o bg is used in the
/etc/fstabso the boot does not get hung up with downed servers. That's how it always worked...
No. As I’ve told you already, this has never worked correctly for NFSv4, and is not expected to work even if we do change the trunking discovery because path walks etc will still hang. Please do this in userland as previously suggested.
_________________________________
Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]
Quoted 91 lines · Trond Myklebust
Quoted 88 lines · Steve Dickson
Quoted 85 lines · Trond Myklebust
Quoted 72 lines · Steve Dickson
Quoted 44 lines · Trond Myklebust
Quoted 41 lines · Steve Dickson
Hey,
Quoted 26 lines · Trond Myklebust
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
Fixes 561ec1603171 (SUNRPC: call_connect_status should recheck bind..)Signed-off-by Trond Myklebust <[email protected]>---
net/sunrpc/clnt.c| 8 +++-----1 file changed, 3 insertions(+), 5 deletions(-)net/sunrpc/clnt.c +0
-4@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :net/sunrpc/clnt.c call_connect_status(struct rpc_task *task) · line 1816
Quoted 10 lines
net/sunrpc/clnt.c +3
-1@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;How is this support to work if the trunking code still ignores timeouts?
Quoted 7 lines
Quoted 1 line
[ 2076.045176] NFS: nfs4_discover_server_trunking after status -110, retryingThe above patch fixes the regression that Neil tracked down in Linux 3.12, and that affects the generic RPC handling of soft timeouts.
The trunking code's handling of ETIMEDOUT has been there since Linux 3.7 and hasn’t changed, so I really don’t see how it can have worked at one time before 3.12.
Maybe it been broken that long.... :-)
But here is the obvious loop that stop that hangs a mount forever:
#8 [
ffff88007a22b7e8]rpc_call_syncatffffffffa0220210[sunrpc]
#9 [ffff88007a22b840]nfs4_proc_setclientidatffffffffa0505c49[nfsv4] #10 [ffff88007a22b988]nfs40_discover_server_trunkingatffffffffa0514489[nfsv4] #11 [ffff88007a22b9d0]nfs4_discover_server_trunkingatffffffffa0516f2d[nfsv4] #12 [ffff88007a22ba28]nfs4_init_clientatffffffffa051e9a4[nfsv4] #13 [ffff88007a22bb20]nfs_get_clientatffffffffa04bd6ba[nfs]
#14 [ffff88007a22bb80]nfs4_set_clientatffffffffa051dfb0[nfsv4] #15 [ffff88007a22bc00]nfs4_create_serveratffffffffa051f4ce[nfsv4] #16 [ffff88007a22bc88]nfs4_remote_mountatffffffffa051790e[nfsv4] #17 [ffff88007a22bcb0]mount_fsatffffffff811b3dd9The SETCLIENT times out
NFS call setclientid auth=UNIX, 'Linux NFSv4.0 10.19.60.77/10.19.60.33 tcp' NFS reply setclientid: -110Quoted 8 lines
The
nfs4_discover_server_trunking()retries
NFS:nfs4_discover_server_trunkingafter status -110, retryingThe happens when there server is down and so the connections fail with ECONNREFUSED:
RPC: 2call_connect_status(status -111)The mount system call never times out in which it did in the past.
Why should a mount system call time out other than perhaps in the case of a soft mount?
Quoted 7 lines
Quoted 3 lines
So the mount can go in background. As you know the -o bg is used in the
/etc/fstabso the boot does not get hung up with downed servers. That's how it always worked...No. As I’ve told you already, this has never worked correctly for NFSv4, and is not expected to work even if we do change the trunking discovery because path walks etc will still hang.
Just to be clear... Are you saying that v4 mounts can/will hang in the kernel forever regardless of what the timeout and retry mount options are?
Quoted 1 line
Please do this in userland as previously suggested.
If the above is indeed the case, then I'll have to use signals to interrupt the foreground mount...
But, if setting mount options like
http://www.spinics.net/lists/linux-nfs/msg41993.html
will guarantee will indeed timeout in a timely manner, than I would rather using the mount options instead of introducing signals into the mix...
steved.
Quoted 98 lines · Steve Dickson
Quoted 95 lines · Trond Myklebust
Quoted 91 lines · Steve Dickson
Quoted 76 lines · Trond Myklebust
Quoted 75 lines · Steve Dickson
Quoted 44 lines · Trond Myklebust
Quoted 41 lines · Steve Dickson
Hey,
Quoted 26 lines · Trond Myklebust
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
Fixes 561ec1603171 (SUNRPC: call_connect_status should recheck bind..)Signed-off-by Trond Myklebust <[email protected]>---
net/sunrpc/clnt.c| 8 +++-----1 file changed, 3 insertions(+), 5 deletions(-)net/sunrpc/clnt.c +0
-4@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :net/sunrpc/clnt.c call_connect_status(struct rpc_task *task) · line 1816
Quoted 10 lines
net/sunrpc/clnt.c +3
-1@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;How is this support to work if the trunking code still ignores timeouts?
Quoted 7 lines
Quoted 1 line
[ 2076.045176] NFS: nfs4_discover_server_trunking after status -110, retryingThe above patch fixes the regression that Neil tracked down in Linux 3.12, and that affects the generic RPC handling of soft timeouts.
The trunking code's handling of ETIMEDOUT has been there since Linux 3.7 and hasn’t changed, so I really don’t see how it can have worked at one time before 3.12.
Maybe it been broken that long.... :-)
But here is the obvious loop that stop that hangs a mount forever:
#8 [
ffff88007a22b7e8]rpc_call_syncatffffffffa0220210[sunrpc]
#9 [ffff88007a22b840]nfs4_proc_setclientidatffffffffa0505c49[nfsv4] #10 [ffff88007a22b988]nfs40_discover_server_trunkingatffffffffa0514489[nfsv4] #11 [ffff88007a22b9d0]nfs4_discover_server_trunkingatffffffffa0516f2d[nfsv4] #12 [ffff88007a22ba28]nfs4_init_clientatffffffffa051e9a4[nfsv4] #13 [ffff88007a22bb20]nfs_get_clientatffffffffa04bd6ba[nfs]
#14 [ffff88007a22bb80]nfs4_set_clientatffffffffa051dfb0[nfsv4] #15 [ffff88007a22bc00]nfs4_create_serveratffffffffa051f4ce[nfsv4] #16 [ffff88007a22bc88]nfs4_remote_mountatffffffffa051790e[nfsv4] #17 [ffff88007a22bcb0]mount_fsatffffffff811b3dd9The SETCLIENT times out
NFS call setclientid auth=UNIX, 'Linux NFSv4.0 10.19.60.77/10.19.60.33 tcp' NFS reply setclientid: -110The
nfs4_discover_server_trunking()retries
NFS:nfs4_discover_server_trunkingafter status -110, retryingQuoted 9 lines
Quoted 5 lines
The happens when there server is down and so the connections fail with ECONNREFUSED:
RPC: 2call_connect_status(status -111)The mount system call never times out in which it did in the past.
Why should a mount system call time out other than perhaps in the case of a soft mount?
So the mount can go in background. As you know the -o bg is used in the
/etc/fstabso the boot does not get hung up with downed servers. That's how it always worked...
Quoted 6 lines
Quoted 3 lines
No. As I’ve told you already, this has never worked correctly for NFSv4, and is not expected to work even if we do change the trunking discovery because path walks etc will still hang.
Just to be clear... Are you saying that v4 mounts can/will hang in the kernel forever regardless of what the timeout and retry mount options are?
I’m saying that we should respect the ‘soft’ mount option, but we shouldn’t need to add any new special features for the mount call.
Quoted 10 lines
Quoted 1 line
Please do this in userland as previously suggested.
If the above is indeed the case, then I'll have to use signals to interrupt the foreground mount...
But, if setting mount options like
http://www.spinics.net/lists/linux-nfs/msg41993.htmlwill guarantee will indeed timeout in a timely manner, than I would rather using the mount options instead of introducing signals into the mix…
For the case of NFSv4:
1) ‘retry' is pretty much obsolete, since the protocol tells us we must not resend unless the connection breaks. All it does today is to act as a multiplier for ‘timeo’.
2) ‘timeo’ itself only has a user-visible effect when you request a soft mount or a soft RPC call. Otherwise, its main effect is to tell the socket when to ping the server with a TCP ‘keepalive’.
_________________________________
Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]
Quoted 121 lines · Trond Myklebust
Quoted 98 lines · Steve Dickson
Quoted 95 lines · Trond Myklebust
Quoted 91 lines · Steve Dickson
Quoted 76 lines · Trond Myklebust
Quoted 75 lines · Steve Dickson
Quoted 44 lines · Trond Myklebust
Quoted 41 lines · Steve Dickson
Hey,
Quoted 26 lines · Trond Myklebust
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
Fixes 561ec1603171 (SUNRPC: call_connect_status should recheck bind..)Signed-off-by Trond Myklebust <[email protected]>---
net/sunrpc/clnt.c| 8 +++-----1 file changed, 3 insertions(+), 5 deletions(-)net/sunrpc/clnt.c +0
-4@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :net/sunrpc/clnt.c call_connect_status(struct rpc_task *task) · line 1816
Quoted 10 lines
net/sunrpc/clnt.c +3
-1@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;How is this support to work if the trunking code still ignores timeouts?
Quoted 7 lines
Quoted 1 line
[ 2076.045176] NFS: nfs4_discover_server_trunking after status -110, retryingThe above patch fixes the regression that Neil tracked down in Linux 3.12, and that affects the generic RPC handling of soft timeouts.
The trunking code's handling of ETIMEDOUT has been there since Linux 3.7 and hasn’t changed, so I really don’t see how it can have worked at one time before 3.12.
Maybe it been broken that long.... :-)
But here is the obvious loop that stop that hangs a mount forever:
#8 [
ffff88007a22b7e8]rpc_call_syncatffffffffa0220210[sunrpc]
#9 [ffff88007a22b840]nfs4_proc_setclientidatffffffffa0505c49[nfsv4] #10 [ffff88007a22b988]nfs40_discover_server_trunkingatffffffffa0514489[nfsv4] #11 [ffff88007a22b9d0]nfs4_discover_server_trunkingatffffffffa0516f2d[nfsv4] #12 [ffff88007a22ba28]nfs4_init_clientatffffffffa051e9a4[nfsv4] #13 [ffff88007a22bb20]nfs_get_clientatffffffffa04bd6ba[nfs]
#14 [ffff88007a22bb80]nfs4_set_clientatffffffffa051dfb0[nfsv4] #15 [ffff88007a22bc00]nfs4_create_serveratffffffffa051f4ce[nfsv4] #16 [ffff88007a22bc88]nfs4_remote_mountatffffffffa051790e[nfsv4] #17 [ffff88007a22bcb0]mount_fsatffffffff811b3dd9The SETCLIENT times out
NFS call setclientid auth=UNIX, 'Linux NFSv4.0 10.19.60.77/10.19.60.33 tcp' NFS reply setclientid: -110The
nfs4_discover_server_trunking()retries
NFS:nfs4_discover_server_trunkingafter status -110, retryingQuoted 9 lines
Quoted 5 lines
The happens when there server is down and so the connections fail with ECONNREFUSED:
RPC: 2call_connect_status(status -111)The mount system call never times out in which it did in the past.
Why should a mount system call time out other than perhaps in the case of a soft mount?
So the mount can go in background. As you know the -o bg is used in the
/etc/fstabso the boot does not get hung up with downed servers. That's how it always worked...Quoted 6 lines
Quoted 3 lines
No. As I’ve told you already, this has never worked correctly for NFSv4, and is not expected to work even if we do change the trunking discovery because path walks etc will still hang.
Just to be clear... Are you saying that v4 mounts can/will hang in the kernel forever regardless of what the timeout and retry mount options are?
I’m saying that we should respect the ‘soft’ mount option, but we shouldn’t need to add any new special features for the mount call.
Quoted 10 lines
Quoted 1 line
Please do this in userland as previously suggested.
If the above is indeed the case, then I'll have to use signals to interrupt the foreground mount...
But, if setting mount options like
http://www.spinics.net/lists/linux-nfs/msg41993.htmlwill guarantee will indeed timeout in a timely manner, than I would rather using the mount options instead of introducing signals into the mix…
Quoted 5 lines
For the case of NFSv4:
1) ‘retry' is pretty much obsolete, since the protocol tells us we must not resend unless the connection breaks. All it does today is to act as a multiplier for ‘timeo’.
I saw that.... I was thinking '1' was a better multiplier that the default... but it appears it does not matter...
Quoted 3 lines
2) ‘timeo’ itself only has a user-visible effect when you request a soft mount or a soft RPC call. Otherwise, its main effect is to tell the socket when to ping the server with a TCP ‘keepalive’.
Setting 'timeo=100' really cuts down the foreground timeout. Without it takes over three minutes for the fg to timeout. Setting it to 100, brings the timeout to 30 seconds, consistently
Question. You mention some about "path walks etc will still hang". I'm assuming we would get to those hangs because we were able to connect the server. My question is would make sense to interrupt out of those "path walks"? Maybe its just a slow server??
steved.
Quoted 132 lines · Steve Dickson
Quoted 121 lines · Trond Myklebust
Quoted 98 lines · Steve Dickson
Quoted 95 lines · Trond Myklebust
Quoted 91 lines · Steve Dickson
Quoted 76 lines · Trond Myklebust
Quoted 75 lines · Steve Dickson
Quoted 44 lines · Trond Myklebust
Quoted 41 lines · Steve Dickson
Hey,
Quoted 26 lines · Trond Myklebust
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
Fixes 561ec1603171 (SUNRPC: call_connect_status should recheck bind..)Signed-off-by Trond Myklebust <[email protected]>---
net/sunrpc/clnt.c| 8 +++-----1 file changed, 3 insertions(+), 5 deletions(-)net/sunrpc/clnt.c +0
-4@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :net/sunrpc/clnt.c call_connect_status(struct rpc_task *task) · line 1816
Quoted 10 lines
net/sunrpc/clnt.c +3
-1@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;How is this support to work if the trunking code still ignores timeouts?
Quoted 7 lines
Quoted 1 line
[ 2076.045176] NFS: nfs4_discover_server_trunking after status -110, retryingThe above patch fixes the regression that Neil tracked down in Linux 3.12, and that affects the generic RPC handling of soft timeouts.
The trunking code's handling of ETIMEDOUT has been there since Linux 3.7 and hasn’t changed, so I really don’t see how it can have worked at one time before 3.12.
Maybe it been broken that long.... :-)
But here is the obvious loop that stop that hangs a mount forever:
#8 [
ffff88007a22b7e8]rpc_call_syncatffffffffa0220210[sunrpc]
#9 [ffff88007a22b840]nfs4_proc_setclientidatffffffffa0505c49[nfsv4] #10 [ffff88007a22b988]nfs40_discover_server_trunkingatffffffffa0514489[nfsv4] #11 [ffff88007a22b9d0]nfs4_discover_server_trunkingatffffffffa0516f2d[nfsv4] #12 [ffff88007a22ba28]nfs4_init_clientatffffffffa051e9a4[nfsv4] #13 [ffff88007a22bb20]nfs_get_clientatffffffffa04bd6ba[nfs]
#14 [ffff88007a22bb80]nfs4_set_clientatffffffffa051dfb0[nfsv4] #15 [ffff88007a22bc00]nfs4_create_serveratffffffffa051f4ce[nfsv4] #16 [ffff88007a22bc88]nfs4_remote_mountatffffffffa051790e[nfsv4] #17 [ffff88007a22bcb0]mount_fsatffffffff811b3dd9The SETCLIENT times out
NFS call setclientid auth=UNIX, 'Linux NFSv4.0 10.19.60.77/10.19.60.33 tcp' NFS reply setclientid: -110The
nfs4_discover_server_trunking()retries
NFS:nfs4_discover_server_trunkingafter status -110, retryingQuoted 9 lines
Quoted 5 lines
The happens when there server is down and so the connections fail with ECONNREFUSED:
RPC: 2call_connect_status(status -111)The mount system call never times out in which it did in the past.
Why should a mount system call time out other than perhaps in the case of a soft mount?
So the mount can go in background. As you know the -o bg is used in the
/etc/fstabso the boot does not get hung up with downed servers. That's how it always worked...Quoted 6 lines
Quoted 3 lines
No. As I’ve told you already, this has never worked correctly for NFSv4, and is not expected to work even if we do change the trunking discovery because path walks etc will still hang.
Just to be clear... Are you saying that v4 mounts can/will hang in the kernel forever regardless of what the timeout and retry mount options are?
I’m saying that we should respect the ‘soft’ mount option, but we shouldn’t need to add any new special features for the mount call.
Quoted 10 lines
Quoted 1 line
Please do this in userland as previously suggested.
If the above is indeed the case, then I'll have to use signals to interrupt the foreground mount...
But, if setting mount options like
http://www.spinics.net/lists/linux-nfs/msg41993.htmlwill guarantee will indeed timeout in a timely manner, than I would rather using the mount options instead of introducing signals into the mix…
Quoted 5 lines
For the case of NFSv4:
1) ‘retry' is pretty much obsolete, since the protocol tells us we must not resend unless the connection breaks. All it does today is to act as a multiplier for ‘timeo’.
I saw that.... I was thinking '1' was a better multiplier that the default... but it appears it does not matter...
Quoted 6 lines
Quoted 3 lines
2) ‘timeo’ itself only has a user-visible effect when you request a soft mount or a soft RPC call. Otherwise, its main effect is to tell the socket when to ping the server with a TCP ‘keepalive’.
Setting 'timeo=100' really cuts down the foreground timeout. Without it takes over three minutes for the fg to timeout. Setting it to 100, brings the timeout to 30 seconds, consistently
Yes, but a 10 second soft timeout can screw you over badly if your server crashes or reboots at some point.
Tuning the mount ‘timeo’ and ‘retrans’ values purely to make the mount system call fast, would be like choosing sport shoes based solely on their ability to slip onto your feet faster: you may end up playing your game of football in a pair of clogs...
Quoted 5 lines
Question. You mention some about "path walks etc will still hang". I'm assuming we would get to those hangs because we were able to connect the server. My question is would make sense to interrupt out of those "path walks"? Maybe its just a slow server??
The kernel will continue to honour ‘soft’ in those path walks, so yes, if the server is slow, and it causes a timeout then that timeout is passed back to the user. That is fully in accordance with the documentation for ‘soft’.
If, however, the server crashes while we’re looking up the path as part of the mount call, and that mount call does not specify ‘soft’, then the client will hang until that server becomes available again. It will not return any errors. This behaviour has not changed since we first introduced NFSv4 in Linux 2.6.0 AFAIK, and there are no plans to change it. Another consideration is that if the target directory on top of which you are mounting the new filesystem is itself NFS mounted, then the mount system call may hang if/when that server goes down. Again, the timeout behaviour will vary depending on the mount options that apply to that target directory.
IOW: there is no way to make mount.nfs honour the ‘retry’ and/or ‘bg' mount options in any consistent fashion by solely relying on kernel timeouts.
_________________________________
Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]
Quoted 2 lines · Trond Myklebust
IOW: there is no way to make mount.nfs honour the ‘retry’ and/or ‘bg' mount options in any consistent fashion by solely relying on kernel timeouts.
I went back and took a look at how bg mounts worked in a number of older kernels f19(3.12) all the way back to RHEL6 kernel (2.6).
I turns out you are right. The bg mounts were not depending on timeouts they were depended on the mount to fail with ECONNREFUSED The very first one, which is the reason the bg mount happen so fast...
Its seems these days ECONNREFUSED are no longer return as an error codes. They basically are turned into a timeout... Just curious as to why ECONNREFUSED are no longer returned?
Again, thanks for the cycles!
steved.
Quoted 12 lines · Steve Dickson
Quoted 2 lines · Trond Myklebust
IOW: there is no way to make mount.nfs honour the ‘retry’ and/or ‘bg' mount options in any consistent fashion by solely relying on kernel timeouts.
I went back and took a look at how bg mounts worked in a number of older kernels f19(3.12) all the way back to RHEL6 kernel (2.6).
I turns out you are right. The bg mounts were not depending on timeouts they were depended on the mount to fail with ECONNREFUSED The very first one, which is the reason the bg mount happen so fast...
Quoted 6 lines
Its seems these days ECONNREFUSED are no longer return as an error codes. They basically are turned into a timeout... Just curious as to why ECONNREFUSED are no longer returned?
Again, thanks for the cycles!
If the server is down during the initial rpc client creation, then I’d still expect that to fail with ECONNREFUSED due to the rpc_ping() call.
Does the following patch help?
8<------------------------------------------------------------
Quoted 1 line
From
dad628cc357a06cff8ce04300ba5c19bd92e73ebMon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Wed, 19 Mar 2014 13:25:43 -0400
Subject: [PATCH] SUNRPC: Ensure call_status() deals correctly with SOFTCONN
tasks
--- net/sunrpc/clnt.c | 4 ++++ 1 file changed, 4 insertions(+)
net/sunrpc/clnt.c +4 -0
@@ -2004,6 +2004,10 @@ call_status(struct rpc_task *task)case -EHOSTDOWN :case -EHOSTUNREACH :case -ENETUNREACH : +if (RPC_IS_SOFTCONN (task)) { +rpc_exit (task, status); +break ; + }/* * Delay any retries for 3 seconds, then handle as if it * were a timeout.
Signature · 7 lines
1.8.5.3 -- Trond Myklebust Linux NFS client maintainer, PrimaryData [email protected]
Quoted 44 lines · Trond Myklebust
Quoted 12 lines · Steve Dickson
Quoted 2 lines · Trond Myklebust
IOW: there is no way to make mount.nfs honour the ‘retry’ and/or ‘bg' mount options in any consistent fashion by solely relying on kernel timeouts.
I went back and took a look at how bg mounts worked in a number of older kernels f19(3.12) all the way back to RHEL6 kernel (2.6).
I turns out you are right. The bg mounts were not depending on timeouts they were depended on the mount to fail with ECONNREFUSED The very first one, which is the reason the bg mount happen so fast...
Quoted 6 lines
Its seems these days ECONNREFUSED are no longer return as an error codes. They basically are turned into a timeout... Just curious as to why ECONNREFUSED are no longer returned?
Again, thanks for the cycles!
If the server is down during the initial rpc client creation, then I’d still expect that to fail with ECONNREFUSED due to the
rpc_ping()call.Does the following patch help?
8<------------------------------------------------------------ From
dad628cc357a06cff8ce04300ba5c19bd92e73ebMon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Wed, 19 Mar 2014 13:25:43 -0400
Subject: [PATCH] SUNRPC: Ensurecall_status()deals correctly with SOFTCONN
tasksSigned-off-by Trond Myklebust <[email protected]>--- net/sunrpc/clnt.c | 4 ++++ 1 file changed, 4 insertions(+)net/sunrpc/clnt.c +0
-0
net/sunrpc/clnt.c call_status(struct rpc_task *task) · line 2013
Quoted 10 lines
net/sunrpc/clnt.c +4
-0@@ -2004,6 +2004,10 @@ call_status(struct rpc_task *task)case -EHOSTDOWN :case -EHOSTUNREACH :case -ENETUNREACH : +if (RPC_IS_SOFTCONN (task)) { +rpc_exit (task, status); +break ; + }/* * Delay any retries for 3 seconds, then handle as if it * were a timeout.
No... but I do thing that patch make sense...
What's going on is-ECONNREFUSED is being seen in call_connect_status() and the task is not a soft connection. So call_timeout() is call which eventual times out the mount...
So just for fun I make the SETCLIENTID rpc soft, but for some reason that didn't work either... I thought for sure it would...
steved.
Quoted 4 lines · Steve Dickson
What's going on is-ECONNREFUSED is being seen in
call_connect_status()and the task is not a soft connection. Socall_timeout()is call which eventual times out the mount…
This is what is confusing me. I’d expect that the rpc_ping() would be the first thing to be sent on the wire by rpc_create(), and that ping should normally have the RPC_SOFTCONN flag set.
Quoted 4 lines
So just for fun I make the SETCLIENTID rpc soft, but for some reason that didn't work either... I thought for sure it would...
steved.
_________________________________
Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]
Quoted 13 lines · Trond Myklebust
Quoted 4 lines · Steve Dickson
What's going on is-ECONNREFUSED is being seen in
call_connect_status()and the task is not a soft connection. Socall_timeout()is call which eventual times out the mount…This is what is confusing me. I’d expect that the
rpc_ping()would be the first thing to be sent on the wire byrpc_create(), and that ping should normally have theRPC_SOFTCONNflag set.Quoted 2 lines
So just for fun I make the SETCLIENTID rpc soft, but for some reason that didn't work either... I thought for sure it would...
I see you point... Using quick systemtap scripts it turns out rpc_ping is not failing (returns 0x0) and it should because the server is definitely down!
steved.
Quoted 14 lines · Steve Dickson
Quoted 11 lines · Trond Myklebust
Quoted 4 lines · Steve Dickson
What's going on is-ECONNREFUSED is being seen in
call_connect_status()and the task is not a soft connection. Socall_timeout()is call which eventual times out the mount…This is what is confusing me. I’d expect that the
rpc_ping()would be the first thing to be sent on the wire byrpc_create(), and that ping should normally have theRPC_SOFTCONNflag set.
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
So just for fun I make the SETCLIENTID rpc soft, but for some reason that didn't work either... I thought for sure it would...
I see you point... Using quick systemtap scripts it turns out
rpc_pingis not failing (returns0x0) and it should because the server is definitely down!
Found it... The rpc_delay call in call_connect_status() is causing a callback to be scheduled and run before rpc_exit_task. Unfortunately that call back (__rpc_atrun) clears the status...
I'll posting my version of the fix shortly..
steved.
Quoted 24 lines · Steve Dickson
Quoted 2 lines
The trunking code's handling of ETIMEDOUT has been there since Linux 3.7
Quoted 1 line
and hasn’t changed, so I really don’t see how it can have worked at one time before 3.12.
Maybe it been broken that long.... :-)
But here is the obvious loop that stop that hangs a mount forever:
#8 [
ffff88007a22b7e8]rpc_call_syncatffffffffa0220210[sunrpc]
#9 [ffff88007a22b840]nfs4_proc_setclientidatffffffffa0505c49[nfsv4]
#10 [ffff88007a22b988]nfs40_discover_server_trunkingatffffffffa0514489[nfsv4] #11 [ffff88007a22b9d0]nfs4_discover_server_trunkingatffffffffa0516f2d[nfsv4] #12 [ffff88007a22ba28]nfs4_init_clientatffffffffa051e9a4[nfsv4] #13 [ffff88007a22bb20]nfs_get_clientatffffffffa04bd6ba[nfs]
#14 [ffff88007a22bb80]nfs4_set_clientatffffffffa051dfb0[nfsv4] #15 [ffff88007a22bc00]nfs4_create_serveratffffffffa051f4ce[nfsv4] #16 [ffff88007a22bc88]nfs4_remote_mountatffffffffa051790e[nfsv4] #17 [ffff88007a22bcb0]mount_fsatffffffff811b3dd9The SETCLIENT times out
NFS call setclientid auth=UNIX, 'Linux NFSv4.0 10.19.60.77/10.19.60.33 tcp' NFS reply setclientid: -110The
nfs4_discover_server_trunking()retries
NFS:nfs4_discover_server_trunkingafter status -110, retrying
Quoted 7 lines
The happens when there server is down and so the connections fail with ECONNREFUSED:
RPC: 2call_connect_status(status -111)The mount system call never times out in which it did in the past.
So who do I get the system call to time out again?
I'm thinking the problem here is the ECONNREFUSED is never being passed up to the trunking code. The major timeout in call_timeout() turns the ECONNREFUSED into a ETIMEDOUT So the trunking code never know the server is refusing the connection...
Why are ECONNREFUSED masked into ETIMEDOUTs? Should the ECONNREFUSED passed up?
steved.
Quoted 2 lines
steved.
Quoted 5 lines · Trond Myklebust
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
Fixes 561ec1603171 (SUNRPC: call_connect_status should recheck bind..)Signed-off-by Trond Myklebust <[email protected]>
steved.
Quoted 31 lines
--- net/sunrpc/clnt.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)net/sunrpc/clnt.c +3
-5@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;
Quoted 25 lines · On Mon, 17 Mar 2014 14:40:44 -0400 Trond Myklebust…
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
Fixes 561ec1603171 (SUNRPC: call_connect_status should recheck bind..)Signed-off-by Trond Myklebust <[email protected]>--- net/sunrpc/clnt.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)net/sunrpc/clnt.c +0
-4@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :
net/sunrpc/clnt.c call_connect_status(struct rpc_task *task) · line 1816
Quoted 10 lines
net/sunrpc/clnt.c +3
-1@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;
I believe this patch may have broken the v4.0 callback channel establishment code in nfsd. I think what's happening is this:
nfsd tries to create a RPC_TASK_SOFTCONN call to probe the cb channel with a CB_NULL. It queues the connect_worker to the workqueue. That establishes the socket and then gets a callback from the socket layer into xs_tcp_state_change for TCP_ESTABLISHED.
That code does:
xprt_wake_pending_tasks(xprt, -EAGAIN);
...that wakes the task up, and sets the tk_status to -EAGAIN, and it then moves on to call_timeout due to this patch. That code then does this:
if (RPC_IS_SOFTCONN(task)) {
rpc_exit(task, -ETIMEDOUT);
return;
}...and the callback ping then fails with an error. Reverting this patch seems to fix it. I see several ways that we could fix this, but I'm not clear on the right way. Maybe we shouldn't be waking up the tasks with -EAGAIN in the TCP_ESTABLISHED case?
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 60 lines · Jeff Layton
Quoted 25 lines · On Mon, 17 Mar 2014 14:40:44 -0400 Trond Myklebust…
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
Fixes 561ec1603171 (SUNRPC: call_connect_status should recheck bind..)Signed-off-by Trond Myklebust <[email protected]>---
net/sunrpc/clnt.c| 8 +++-----1 file changed, 3 insertions(+), 5 deletions(-)net/sunrpc/clnt.c +0
-4@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :net/sunrpc/clnt.c call_connect_status(struct rpc_task *task) · line 1816
Quoted 10 lines
net/sunrpc/clnt.c +3
-1@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;I believe this patch may have broken the v4.0 callback channel establishment code in nfsd. I think what's happening is this:
nfsd tries to create a
RPC_TASK_SOFTCONNcall to probe the cb channel with aCB_NULL. It queues theconnect_workerto the workqueue. That establishes the socket and then gets a callback from the socket layer intoxs_tcp_state_changeforTCP_ESTABLISHED.That code does:
xprt_wake_pending_tasks(xprt, -EAGAIN);...that wakes the task up, and sets the
tk_statusto-EAGAIN, and it then moves on tocall_timeoutdue to this patch. That code then does this:if (RPC_IS_SOFTCONN(task)) { rpc_exit(task, -ETIMEDOUT); return; }
Quoted 4 lines
...and the callback ping then fails with an error. Reverting this patch seems to fix it. I see several ways that we could fix this, but I'm not clear on the right way. Maybe we shouldn't be waking up the tasks with
-EAGAINin theTCP_ESTABLISHEDcase?
...or, possibly setup_callback_client should be setting the timeparms.to_maxval to a non-zero value so that xprt_adjust_timeout() and xprt_reset_majortimeo() behave as expected.
_________________________________
Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]
Quoted 68 lines · On Mon, 14 Apr 2014 12:57:58 -0400 Trond Myklebust…
Quoted 60 lines · Jeff Layton
Quoted 25 lines · On Mon, 17 Mar 2014 14:40:44 -0400 Trond Myklebust…
When the server is unavailable due to a networking error, etc, we want the RPC client to respect the timeout delays when attempting to reconnect.
Fixes 561ec1603171 (SUNRPC: call_connect_status should recheck bind..)Signed-off-by Trond Myklebust <[email protected]>---
net/sunrpc/clnt.c| 8 +++-----1 file changed, 3 insertions(+), 5 deletions(-)net/sunrpc/clnt.c +0
-4@@ -1798,10 +1798,6 @@ call_connect_status(struct rpc_task *task)trace_rpc_connect_status (task, status); task->tk_status =0 ;switch (status) { -/* if soft mounted, test if we've timed out */ -case -ETIMEDOUT : - task->tk_action = call_timeout; -return ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :net/sunrpc/clnt.c call_connect_status(struct rpc_task *task) · line 1816
Quoted 10 lines
net/sunrpc/clnt.c +3
-1@@ -1812,7 +1808,9 @@ call_connect_status(struct rpc_task *task)if (RPC_IS_SOFTCONN (task))break ;case -EAGAIN : - task->tk_action = call_bind; +case -ETIMEDOUT : +/* Check if we've timed out before looping back to call_bind */ + task->tk_action = call_timeout;return ;case 0 : clnt->cl_stats->netreconn++;I believe this patch may have broken the v4.0 callback channel establishment code in nfsd. I think what's happening is this:
nfsd tries to create a
RPC_TASK_SOFTCONNcall to probe the cb channel with aCB_NULL. It queues theconnect_workerto the workqueue. That establishes the socket and then gets a callback from the socket layer intoxs_tcp_state_changeforTCP_ESTABLISHED.That code does:
xprt_wake_pending_tasks(xprt, -EAGAIN);...that wakes the task up, and sets the
tk_statusto-EAGAIN, and it then moves on tocall_timeoutdue to this patch. That code then does this:if (RPC_IS_SOFTCONN(task)) { rpc_exit(task, -ETIMEDOUT); return; }Quoted 4 lines
...and the callback ping then fails with an error. Reverting this patch seems to fix it. I see several ways that we could fix this, but I'm not clear on the right way. Maybe we shouldn't be waking up the tasks with
-EAGAINin theTCP_ESTABLISHEDcase?
Quoted 7 lines
...or, possibly
setup_callback_clientshould be setting the timeparms.to_maxval to a non-zero value so thatxprt_adjust_timeout()andxprt_reset_majortimeo()behave as expected._________________________________
Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]
Well spotted. That does indeed fix it. I'll spin up a patch and send it to Bruce.
Thanks!
Signature · 1 line
Jeff Layton <[email protected]>