NFS: add a sysctl for disable the reconnect delay
Subsystems (3)
PATCH 1/1 NFS: add a sysctl for disable the reconnect delay
If network partition or some other reason cause a reconnect, it cannot succeed immediately when environment recover, but client want to connect timely sometimes.
This patch can provide a proc file(/proc/sys/fs/nfs/nfs_disable_reconnect_delay) to allow client disable the reconnect delay(reestablish_timeout) when using NFS.
It's only useful for NFS.
--- fs/nfs/client.c | 3 +++ fs/nfs/sysctl.c | 8 ++++++++ include/linux/nfs_fs.h | 6 ++++++ include/linux/sunrpc/clnt.h | 1 + include/linux/sunrpc/xprt.h | 3 ++- net/sunrpc/clnt.c | 2 ++ net/sunrpc/xprtsock.c | 2 +- 7 files changed, 23 insertions(+), 2 deletions(-)
fs/nfs/client.c +3 -0
@@ -55,6 +55,8 @@ static LIST_HEAD(nfs_client_list);static LIST_HEAD (nfs_volume_list);static DECLARE_WAIT_QUEUE_HEAD (nfs_client_active_wq); +int nfs_disable_reconnect_delay =0 ; +/* * RPC cruft for NFS */
@@ -607,6 +609,7 @@ static int nfs_create_rpc_client(struct nfs_client *clp, .program = &nfs_program, .version = clp->rpc_ops->version, .authflavor = flavor, + .no_recon_delay = nfs_disable_reconnect_delay, };if (discrtry)
fs/nfs/sysctl.c +8 -0
@@ -58,6 +58,14 @@ static ctl_table nfs_cb_sysctls[] = { .mode = 0644, .proc_handler = &proc_dointvec, }, + { + .ctl_name = CTL_UNNUMBERED, + .procname = "nfs_disable_reconnect_delay", + .data = &nfs_disable_reconnect_delay, + .maxlen = sizeof(nfs_disable_reconnect_delay), + .mode = 0644, + .proc_handler = &proc_dointvec, + }, { .ctl_name = 0 } };
include/linux/nfs_fs.h +6 -0
@@ -390,6 +390,12 @@ static inline struct rpc_cred *nfs_file_cred(struct file *file) }/* +* linux/fs/nfs/client.c +*/ + +extern int nfs_disable_reconnect_delay; + +/* * linux/fs/nfs/xattr.c */ #ifdef CONFIG_NFS_V3_ACL
include/linux/sunrpc/clnt.h +1 -0
@@ -115,6 +115,7 @@ struct rpc_create_args {rpc_authflavor_t authflavor;unsigned long flags;char *client_name; +int no_recon_delay;/* no delay when reconnect */ };/* Values for "flags" field */
include/linux/sunrpc/xprt.h +2 -1
@@ -153,7 +153,8 @@ struct rpc_xprt {unsigned int max_reqs;/* total slots */ unsigned long state;/* transport state */ unsigned char shutdown : 1, /* being shut down */ - resvport : 1; /* use a reserved port */ + resvport : 1, /* use a reserved port */ + no_recon_delay: 1; /* no delay when reconnect */unsigned int bind_index;/* bind function index */ /*
net/sunrpc/clnt.c +2 -0
@@ -316,6 +316,8 @@ struct rpc_clnt *rpc_create(struct rpc_create_args *args)if (args->flags &RPC_CLNT_CREATE_NONPRIVPORT ) xprt->resvport =0 ; + xprt->no_recon_delay = !!args->no_recon_delay; + clnt =rpc_new_client (args, xprt);if (IS_ERR (clnt))return clnt;
net/sunrpc/xprtsock.c +1 -1
@@ -2089,7 +2089,7 @@ static void xs_connect(struct rpc_task *task)if (xprt_test_and_set_connecting (xprt))return ; -if (transport->sock !=NULL ) { +if (!xprt->no_recon_delay && transport->sock !=NULL ) {dprintk ("RPC: xs_connect delayed xprt %p for %lu " "seconds\n" , xprt, xprt->reestablish_timeout /HZ );
Signature · 1 line
1.6.4
Hi Mi-
Quoted 8 lines · Mi Jinlong
If network partition or some other reason cause a reconnect, it cannot succeed immediately when environment recover, but client want to connect timely sometimes.
This patch can provide a proc
file(/proc/sys/fs/nfs/nfs_disable_reconnect_delay)to allow client disable the reconnectdelay(reestablish_timeout)when using NFS.It's only useful for NFS.
There's a good reason for the connection re-establishment delay, and only very few instances where you'd want to disable it. A sysctl is the wrong place for this, as it would disable the reconnect delay across the board, instead of for just those occasions when it is actually necessary to connect immediately.
I assume that because the grace period has a time limit, you would want the client to reconnect at all costs? I think that this is actually when a client should take care not to spuriously reconnect: during a server reboot, a server may be sluggish or not completely ready to accept client requests. It's not a time when a client should be showering a server with connection attempts.
The reconnect delay is an exponential backoff that starts at 3 seconds, so if the server is really ready to accept connections, the actual connection delay ought to be quick.
We're already considering shortening the maximum amount of time the client can wait before trying a reconnect. And, it might possibly be that the network layer itself is interfering with the backoff logic that is already built into the RPC client. (If true, that would be the real bug in this case). I'm not interested in a workaround when we really should fix any underlying issues to make this work correctly.
Perhaps the RPC client needs to distinguish between connection refusal (where a lengthening exponential backoff between connection attempts makes sense) and no server response (where we want the client's network layer to keep sending SYN requests so that it can reconnect as soon as possible).
The second scenario might disable the reconnect timer so that only one ->connect() call would be outstanding until the network layer tells us it's given up on SYN retries.
Quoted 120 lines
Signed-off-by Mi Jinlong<[email protected]>--- fs/nfs/client.c | 3 +++ fs/nfs/sysctl.c | 8 ++++++++ include/linux/nfs_fs.h | 6 ++++++ include/linux/sunrpc/clnt.h | 1 + include/linux/sunrpc/xprt.h | 3 ++- net/sunrpc/clnt.c | 2 ++ net/sunrpc/xprtsock.c | 2 +- 7 files changed, 23 insertions(+), 2 deletions(-)fs/nfs/client.c +3
-0@@ -55,6 +55,8 @@ static LIST_HEAD(nfs_client_list);static LIST_HEAD (nfs_volume_list);static DECLARE_WAIT_QUEUE_HEAD (nfs_client_active_wq); +int nfs_disable_reconnect_delay =0 ; +/* * RPC cruft for NFS */ @@ -607,6 +609,7 @@ static int nfs_create_rpc_client(struct nfs_client *clp, .program =&nfs_program, .version = clp->rpc_ops->version, .authflavor = flavor, + .no_recon_delay = nfs_disable_reconnect_delay, };if (discrtry)fs/nfs/sysctl.c +8
-0@@ -58,6 +58,14 @@ static ctl_table nfs_cb_sysctls[] = { .mode = 0644, .proc_handler =&proc_dointvec, }, + { + .ctl_name = CTL_UNNUMBERED, + .procname = "nfs_disable_reconnect_delay", + .data =&nfs_disable_reconnect_delay, + .maxlen = sizeof(nfs_disable_reconnect_delay), + .mode = 0644, + .proc_handler =&proc_dointvec, + }, { .ctl_name = 0 } };include/linux/nfs_fs.h +6
-0@@ -390,6 +390,12 @@ static inline struct rpc_cred *nfs_file_cred(struct file *file) }/* +* linux/fs/nfs/client.c +*/ + +extern int nfs_disable_reconnect_delay; + +/* * linux/fs/nfs/xattr.c */ #ifdef CONFIG_NFS_V3_ACL include/linux/sunrpc/clnt.h +1
-0@@ -115,6 +115,7 @@ struct rpc_create_args {rpc_authflavor_t authflavor;unsigned long flags;char *client_name; +int no_recon_delay;/* no delay when reconnect */ };/* Values for "flags" field */ include/linux/sunrpc/xprt.h +2
-1@@ -153,7 +153,8 @@ struct rpc_xprt {unsigned int max_reqs;/* total slots */ unsigned long state;/* transport state */ unsigned char shutdown : 1, /* being shut down */ - resvport : 1; /* use a reserved port */ + resvport : 1, /* use a reserved port */ + no_recon_delay: 1; /* no delay when reconnect */unsigned int bind_index;/* bind function index */ /* net/sunrpc/clnt.c +2
-0@@ -316,6 +316,8 @@ struct rpc_clnt *rpc_create(struct rpc_create_args *args)if (args->flags&RPC_CLNT_CREATE_NONPRIVPORT ) xprt->resvport =0 ; + xprt->no_recon_delay = !!args->no_recon_delay; + clnt =rpc_new_client (args, xprt);if (IS_ERR (clnt))return clnt;net/sunrpc/xprtsock.c +1
-1@@ -2089,7 +2089,7 @@ static void xs_connect(struct rpc_task *task)if (xprt_test_and_set_connecting (xprt))return ; -if (transport->sock !=NULL ) { +if (!xprt->no_recon_delay&& transport->sock !=NULL ) {dprintk ("RPC: xs_connect delayed xprt %p for %lu " "seconds\n" , xprt, xprt->reestablish_timeout /HZ );
Signature · 1 line
chuck[dot]lever[at]oracle[dot]com
Hi Chuck,
Sorry for replying your message so later.
Chuck Lever =E5=86=99=E9=81=93:
Quoted 4 lines
Hi Mi-
=20Quoted 1 line · Mi Jinlong
If network partition or some other reason cause a reconnect, it cann=
ot
Quoted 1 line
Quoted 1 line
succeed immediately when environment recover, but client want to con=
nect
Quoted 5 lines
Quoted 5 lines
timely sometimes.
This patch can provide a proc
file(/proc/sys/fs/nfs/nfs_disable_reconnect_delay)
to allow client disable the reconnectdelay(reestablish_timeout)whe=
n
Quoted 6 lines
Quoted 3 lines
using NFS.
It's only useful for NFS.
=20
There's a good reason for the connection re-establishment delay, and only very few instances where you'd want to disable it. A sysctl is =
the
Quoted 1 line
wrong place for this, as it would disable the reconnect delay across =
the
Quoted 1 line
board, instead of for just those occasions when it is actually necess=
ary
Quoted 1 line
to connect immediately.
Yes, I agree with you.
Quoted 2 lines
=20
I assume that because the grace period has a time limit, you would wa=
nt
Quoted 7 lines
the client to reconnect at all costs? I think that this is actually when a client should take care not to spuriously reconnect: during a server reboot, a server may be sluggish or not completely ready to accept client requests. It's not a time when a client should be showering a server with connection attempts.
=20
The reconnect delay is an exponential backoff that starts at 3 second=
s,
Quoted 6 lines
so if the server is really ready to accept connections, the actual connection delay ought to be quick.
=20
We're already considering shortening the maximum amount of time the client can wait before trying a reconnect. And, it might possibly be that the network layer itself is interfering with the backoff logic t=
hat
Quoted 1 line
is already built into the RPC client. (If true, that would be the re=
al
Quoted 4 lines
bug in this case). I'm not interested in a workaround when we really should fix any underlying issues to make this work correctly. =20
Perhaps the RPC client needs to distinguish between connection refusa=
l
Quoted 2 lines
(where a lengthening exponential backoff between connection attempts makes sense) and no server response (where we want the client's netwo=
rk
Quoted 1 line
layer to keep sending SYN requests so that it can reconnect as soon a=
s
Quoted 1 line
possible).
When reading the kernel's code and testing, I find there are three ca=
se:
A. network partition:
Becasue the client can't communicate with server's rpcbind,=20 so there is no influence.
B. server's nfs service stop:
The client call xprt_connect to conncet, but get err(111: Connecti=
on refused).
C. server's nfs service sotp, and ifdown the NIC after about 60s:
At first, when the NIC is up, xprt_connect get err(111: Connection=
refused) as 2.
After NIC is down, xprt_connect get err(113: No route to host).
When connecting fail, the sunrpc level only get a ETIMEDOUT or EAGAIN =
err, it will also
call xprt_connect to reconnect.
If we make the network layer to keep sending SYN requests, but there w=
ill be more request=20
be delayed at the request queue, and the reestablish_timeout also be i=
ncreased.
Can we distinguish those refusal at sunrpc level, but not at xprt leve=
l ?
If we can do that, the problem will solved easily.
=20
[NOTE]
the testing process:
client server
1. mount nfs (OK)=20
2. df (OK)
3. nfs stop
4. df (hang)
I get message through rpcdebug.
Quoted 2 lines
=20
The second scenario might disable the reconnect timer so that only on=
e
Quoted 1 line
->connect() call would be outstanding until the network layer tells u=
s
Quoted 1 line
it's given up on SYN retries.
=20
I think that's a good idea, but implementation may be a great work.
=20
thanks,
Mi Jinlong
Quoted 9 lines · Mi Jinlong
Hi Chuck,
Sorry for replying your message so later.
Chuck Lever =E5=86=99=E9=81=93:
Quoted 4 lines
Hi Mi-
Quoted 1 line · Mi Jinlong
If network partition or some other reason cause a reconnect, it can=
not
Quoted 1 line
Quoted 1 line
Quoted 1 line
succeed immediately when environment recover, but client want to co=
nnect
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
timely sometimes.
This patch can provide a proc
file(/proc/sys/fs/nfs/nfs_disable_reconnect_delay)
to allow client disable the reconnectdelay(reestablish_timeout)wh=
en
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
using NFS.
It's only useful for NFS.
There's a good reason for the connection re-establishment delay, and only very few instances where you'd want to disable it. A sysctl is=
the
Quoted 1 line
Quoted 1 line
wrong place for this, as it would disable the reconnect delay across=
the
Quoted 1 line
Quoted 1 line
board, instead of for just those occasions when it is actually neces=
sary
Quoted 6 lines
Quoted 1 line
to connect immediately.
Yes, I agree with you.
Quoted 2 lines
I assume that because the grace period has a time limit, you would w=
ant
Quoted 7 lines
Quoted 7 lines
the client to reconnect at all costs? I think that this is actually when a client should take care not to spuriously reconnect: during a server reboot, a server may be sluggish or not completely ready to accept client requests. It's not a time when a client should be showering a server with connection attempts.
The reconnect delay is an exponential backoff that starts at 3 secon=
ds,
Quoted 5 lines
Quoted 5 lines
so if the server is really ready to accept connections, the actual connection delay ought to be quick.
We're already considering shortening the maximum amount of time the client can wait before trying a reconnect. And, it might possibly b=
e
Quoted 1 line
Quoted 1 line
that the network layer itself is interfering with the backoff logic =
that
Quoted 1 line
Quoted 1 line
is already built into the RPC client. (If true, that would be the r=
eal
Quoted 1 line
Quoted 1 line
bug in this case). I'm not interested in a workaround when we reall=
y
Quoted 3 lines
Quoted 3 lines
should fix any underlying issues to make this work correctly.
Perhaps the RPC client needs to distinguish between connection refus=
al
Quoted 2 lines
Quoted 2 lines
(where a lengthening exponential backoff between connection attempts makes sense) and no server response (where we want the client's netw=
ork
Quoted 1 line
Quoted 1 line
layer to keep sending SYN requests so that it can reconnect as soon =
as
Quoted 3 lines
Quoted 1 line
possible).
When reading the kernel's code and testing, I find there are three=
case:
Quoted 7 lines
A. network partition:
Becasue the client can't communicate with server's rpcbind, so there is no influence.B. server's nfs service stop:
The client callxprt_connectto conncet, but get err(111: Conne=
ction refused).
Quoted 3 lines
C. server's nfs service sotp, and ifdown the NIC after about 60s:
At first, when the NIC is up,xprt_connectget err(111: Connect=
ion refused) as 2.
Quoted 4 lines
After NIC is down,
xprt_connectget err(113: No route to host).When connecting fail, the sunrpc level only get a ETIMEDOUT or EAGA=
IN err, it will also
Quoted 2 lines
call
xprt_connectto reconnect.
If we make the network layer to keep sending SYN requests, but ther=
e will be more request
Quoted 1 line
be delayed at the request queue, and the
reestablish_timeoutalso b=
e increased.
Quoted 2 lines
Can we distinguish those refusal at sunrpc level, but not at xprt l=
evel ?
Quoted 11 lines
If we can do that, the problem will solved easily.
[NOTE]
the testing process:
client server
1. mount nfs (OK)
2. df (OK)
3. nfs stop
4. df (hang)I get message through rpcdebug.
We have a matrix of cases. "soft" v. "hard" RPCs, ECONNREFUSED v. no=20 response, connection previously closed by server disconnect v. client=20 idle timeout.
I've found at least one major bug in this logic, and that is that the 6= 0=20
second transport connect timer is clobbered in the ECONNREFUSED case, s= o=20
soft RPCs never time out if the server refuses a connection, for=20 example. I handed all of this off to Trond.
Quoted 1 line
Quoted 1 line
The second scenario might disable the reconnect timer so that only o=
ne
Quoted 1 line
Quoted 1 line
->connect() call would be outstanding until the network layer tells =
us
Quoted 3 lines
Quoted 1 line
it's given up on SYN retries.
I think that's a good idea, but implementation may be a great work=
=2E
Quoted 4 lines
thanks,
Mi Jinlong
--=20
chuck[dot]lever[at]oracle[dot]com
Chuck Lever =E5=86=99=E9=81=93:
Quoted 10 lines · Mi Jinlong
Quoted 9 lines
Hi Chuck,
Sorry for replying your message so later.
Chuck Lever =E5=86=99=E9=81=93:
Quoted 4 lines
Hi Mi-
Quoted 1 line · Mi Jinlong
If network partition or some other reason cause a reconnect, it ca=
nnot
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
succeed immediately when environment recover, but client want to connect
timely sometimes.This patch can provide a proc
file(/proc/sys/fs/nfs/nfs_disable_reconnect_delay)
to allow client disable the reconnectdelay(reestablish_timeout)w=
hen
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
using NFS.
It's only useful for NFS.
There's a good reason for the connection re-establishment delay, an=
d
Quoted 1 line
Quoted 1 line
Quoted 1 line
only very few instances where you'd want to disable it. A sysctl i=
s the
Quoted 1 line
Quoted 1 line
Quoted 1 line
wrong place for this, as it would disable the reconnect delay acros=
s the
Quoted 1 line
Quoted 1 line
Quoted 1 line
board, instead of for just those occasions when it is actually nece=
ssary
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
to connect immediately.
Yes, I agree with you.
Quoted 2 lines
I assume that because the grace period has a time limit, you would =
want
Quoted 1 line
Quoted 1 line
Quoted 1 line
the client to reconnect at all costs? I think that this is actuall=
y
Quoted 1 line
Quoted 1 line
Quoted 1 line
when a client should take care not to spuriously reconnect: during =
a
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
server reboot, a server may be sluggish or not completely ready to accept client requests. It's not a time when a client should be showering a server with connection attempts.
The reconnect delay is an exponential backoff that starts at 3 seco=
nds,
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
so if the server is really ready to accept connections, the actual connection delay ought to be quick.
We're already considering shortening the maximum amount of time the client can wait before trying a reconnect. And, it might possibly =
be
Quoted 1 line
Quoted 1 line
Quoted 1 line
that the network layer itself is interfering with the backoff logic=
that
Quoted 1 line
Quoted 1 line
Quoted 1 line
is already built into the RPC client. (If true, that would be the =
real
Quoted 1 line
Quoted 1 line
Quoted 1 line
bug in this case). I'm not interested in a workaround when we real=
ly
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
should fix any underlying issues to make this work correctly.
Perhaps the RPC client needs to distinguish between connection refu=
sal
Quoted 1 line
Quoted 1 line
Quoted 1 line
(where a lengthening exponential backoff between connection attempt=
s
Quoted 1 line
Quoted 1 line
Quoted 1 line
makes sense) and no server response (where we want the client's net=
work
Quoted 1 line
Quoted 1 line
Quoted 1 line
layer to keep sending SYN requests so that it can reconnect as soon=
as
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
possible).
When reading the kernel's code and testing, I find there are thre=
e
Quoted 10 lines
Quoted 10 lines
case:
A. network partition:
Becasue the client can't communicate with server's rpcbind, so there is no influence.B. server's nfs service stop:
The client callxprt_connectto conncet, but get err(111:
Connection refused).
Quoted 5 lines
Quoted 5 lines
C. server's nfs service sotp, and ifdown the NIC after about 60s:
At first, when the NIC is up,xprt_connectget err(111:
Connection refused) as 2.After NIC is down,
xprt_connectget err(113: No route to host)=
=2E
Quoted 5 lines
Quoted 5 lines
When connecting fail, the sunrpc level only get a ETIMEDOUT or
EAGAIN err, it will also
callxprt_connectto reconnect.
If we make the network layer to keep sending SYN requests, but the=
re
Quoted 2 lines
Quoted 2 lines
will be more request
be delayed at the request queue, and thereestablish_timeoutalso =
be
Quoted 4 lines
Quoted 4 lines
increased.
Can we distinguish those refusal at sunrpc level, but not at xprt
level ?
What do you think that I show yesterday?
Quoted 10 lines
Quoted 10 lines
If we can do that, the problem will solved easily.
[NOTE]
the testing process:
client server
1. mount nfs (OK)
2. df (OK)
3. nfs stop
4. df (hang)
Quoted 5 lines
Quoted 1 line
I get message through rpcdebug.
=20
We have a matrix of cases. "soft" v. "hard" RPCs, ECONNREFUSED v. no response, connection previously closed by server disconnect v. client idle timeout.
connection previously closed by server disconnect v. client idle time=
out?
Can you explain to me in some sort? Maybe it's useful for me. Thanks.
Quoted 2 lines
=20
I've found at least one major bug in this logic, and that is that the=
60
Quoted 1 line
second transport connect timer is clobbered in the ECONNREFUSED case,=
so
Quoted 2 lines
soft RPCs never time out if the server refuses a connection, for example. I handed all of this off to Trond.
Really?=20
I mount the nfs file through soft(-o soft), and then I using "df" com=
mand
to see the mount information after server's nfs stop.
The "df" will return with error -5(Input/output error), maybe it's RP=
Cs=20
timeout cause the df return?
thanks,
Mi Jinlong
Quoted 11 lines · Mi Jinlong
Chuck Lever =E5=86=99=E9=81=93:
Quoted 10 lines · Mi Jinlong
Quoted 9 lines
Hi Chuck,
=20
Sorry for replying your message so later.
=20
Chuck Lever =E5=86=99=E9=81=93:Quoted 4 lines
Hi Mi-
=20Quoted 1 line · Mi Jinlong
If network partition or some other reason cause a reconnect, it c=
annot
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
succeed immediately when environment recover, but client want to connect
timely sometimes.
=20
This patch can provide a procfile(/proc/sys/fs/nfs/nfs_disable_reconnect_delay)
to allow client disable the reconnectdelay(reestablish_timeout)=
when
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
using NFS.
=20
It's only useful for NFS.=20
There's a good reason for the connection re-establishment delay, a=
nd
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
only very few instances where you'd want to disable it. A sysctl =
is the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wrong place for this, as it would disable the reconnect delay acro=
ss the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
board, instead of for just those occasions when it is actually nec=
essary
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
to connect immediately.
=20
Yes, I agree with you.
=20Quoted 2 lines
=20
I assume that because the grace period has a time limit, you would=
want
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the client to reconnect at all costs? I think that this is actual=
ly
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
when a client should take care not to spuriously reconnect: during=
a
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
server reboot, a server may be sluggish or not completely ready to accept client requests. It's not a time when a client should be showering a server with connection attempts.
=20
The reconnect delay is an exponential backoff that starts at 3 sec=
onds,
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
so if the server is really ready to accept connections, the actual connection delay ought to be quick.
=20
We're already considering shortening the maximum amount of time th=
e
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
client can wait before trying a reconnect. And, it might possibly=
be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
that the network layer itself is interfering with the backoff logi=
c that
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
is already built into the RPC client. (If true, that would be the=
real
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
bug in this case). I'm not interested in a workaround when we rea=
lly
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
should fix any underlying issues to make this work correctly. =20
Perhaps the RPC client needs to distinguish between connection ref=
usal
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(where a lengthening exponential backoff between connection attemp=
ts
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
makes sense) and no server response (where we want the client's ne=
twork
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
layer to keep sending SYN requests so that it can reconnect as soo=
n as
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
possible).
=20
When reading the kernel's code and testing, I find there are thre=
e
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
case:
=20
A. network partition:
Becasue the client can't communicate with server's rpcbind, so there is no influence.
=20
B. server's nfs service stop:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The client call
xprt_connectto conncet, but get err(111:
Connection refused).
=20
C. server's nfs service sotp, and ifdown the NIC after about 60s:
At first, when the NIC is up,xprt_connectget err(111:
Connection refused) as 2.
=20
After NIC is down,xprt_connectget err(113: No route to host)=
=2E
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
=20
When connecting fail, the sunrpc level only get a ETIMEDOUT or
EAGAIN err, it will also
callxprt_connectto reconnect.
If we make the network layer to keep sending SYN requests, but the=
re
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
will be more request
be delayed at the request queue, and thereestablish_timeoutalso =
be
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
increased.
=20
Can we distinguish those refusal at sunrpc level, but not at xprt
level ?=20
What do you think that I show yesterday?
In xprtsock.c, these reconnection errors are distinguished. In the gen= eric sunrpc client (xprt.c) they are not -- when an RPC transmission is=
sent, xprtsock.c returns ENOTCONN for any connection error. Trond mad=
e this change after 2.6.18. The differences matter in how the client r= e-establishes the connection, and that logic is all in xprtsock.c. So,=
the RPC client already makes this distinction, but the logic may have =
bugs.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
If we can do that, the problem will solved easily.
=20
[NOTE]
the testing process:
client server
1. mount nfs (OK)
2. df (OK)
3. nfs stop
4. df (hang)
=20
I get message through rpcdebug.
Quoted 2 lines
Quoted 2 lines
=20
We have a matrix of cases. "soft" v. "hard" RPCs, ECONNREFUSED v. n=
o
Quoted 1 line
Quoted 1 line
response, connection previously closed by server disconnect v. clien=
t
Quoted 3 lines
Quoted 1 line
idle timeout.
=20
connection previously closed by server disconnect v. client idle tim=
eout?
Quoted 1 line
Can you explain to me in some sort? Maybe it's useful for me. Thanks=
=2E
If the server closed the connection, the client should use the re-estab= lish timeout to delay the reconnection in order to prevent a hard loop = of client connection retries. If the client idled the connection out, = then the client should reconnect immediately.
Quoted 1 line
Quoted 1 line
I've found at least one major bug in this logic, and that is that th=
e 60
Quoted 1 line
Quoted 1 line
second transport connect timer is clobbered in the ECONNREFUSED case=
, so
Quoted 5 lines
Quoted 2 lines
soft RPCs never time out if the server refuses a connection, for example. I handed all of this off to Trond.
=20
Really?=20
I mount the nfs file through soft(-o soft), and then I using "df" co=
mmand
Quoted 2 lines
to see the mount information after server's nfs stop.
The "df" will return with error -5(Input/output error), maybe it's R=
PCs=20
Quoted 1 line
timeout cause the df return?
RPC timeouts generally cause an EIO. However, if the server continues = to refuse a connection, the timeout never occurs.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com