Skip to contentSkip to conversation

NFS: add a sysctl for disable the reconnect delay

PATCH In review Mi Jinlong 2 participants 6 messages Mar 18, 2010 – Apr 14, 2010

Subsystems (3)
lore
Share & tools
lore

Mi Jinlong

author
#0 lore raw

PATCH 1/1 NFS: add a sysctl for disable the reconnect delay

If network partition or some other reason cause a reconnect, it cannot succeed immediately when environment recover, but client want to connect timely sometimes.

This patch can provide a proc file(/proc/sys/fs/nfs/nfs_disable_reconnect_delay) to allow client disable the reconnect delay(reestablish_timeout) when using NFS.

It's only useful for NFS.

---
 fs/nfs/client.c             |    3 +++
 fs/nfs/sysctl.c             |    8 ++++++++
 include/linux/nfs_fs.h      |    6 ++++++
 include/linux/sunrpc/clnt.h |    1 +
 include/linux/sunrpc/xprt.h |    3 ++-
 net/sunrpc/clnt.c           |    2 ++
 net/sunrpc/xprtsock.c       |    2 +-
 7 files changed, 23 insertions(+), 2 deletions(-)
fs/nfs/client.c +3 -0
index 8d25ccb..e878724 100644
@@ -55,6 +55,8 @@ static LIST_HEAD(nfs_client_list);
 static LIST_HEAD(nfs_volume_list);
 static DECLARE_WAIT_QUEUE_HEAD(nfs_client_active_wq);
 
+int nfs_disable_reconnect_delay = 0;
+
 /*
  * RPC cruft for NFS
  */
@@ -607,6 +609,7 @@ static int nfs_create_rpc_client(struct nfs_client *clp,
 		.program	= &nfs_program,
 		.version	= clp->rpc_ops->version,
 		.authflavor	= flavor,
+		.no_recon_delay	= nfs_disable_reconnect_delay,
 	};
 
 	if (discrtry)
fs/nfs/sysctl.c +8 -0
index b62481d..6c04479 100644
@@ -58,6 +58,14 @@ static ctl_table nfs_cb_sysctls[] = {
 		.mode		= 0644,
 		.proc_handler	= &proc_dointvec,
 	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname	= "nfs_disable_reconnect_delay",
+		.data		= &nfs_disable_reconnect_delay,
+		.maxlen		= sizeof(nfs_disable_reconnect_delay),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+	},
 	{ .ctl_name = 0 }
 };
 
include/linux/nfs_fs.h +6 -0
index f6b9024..e031496 100644
@@ -390,6 +390,12 @@ static inline struct rpc_cred *nfs_file_cred(struct file *file)
 }
 
 /*
+ * linux/fs/nfs/client.c
+ */
+
+extern int nfs_disable_reconnect_delay;
+
+/*
  * linux/fs/nfs/xattr.c
  */
 #ifdef CONFIG_NFS_V3_ACL
include/linux/sunrpc/clnt.h +1 -0
index 5bd17f6..f73eae1 100644
@@ -115,6 +115,7 @@ struct rpc_create_args {
 	rpc_authflavor_t	authflavor;
 	unsigned long		flags;
 	char			*client_name;
+	int			no_recon_delay;  /* no delay when reconnect */
 };
 
 /* Values for "flags" field */
include/linux/sunrpc/xprt.h +2 -1
index 1175d58..a177348 100644
@@ -153,7 +153,8 @@ struct rpc_xprt {
 	unsigned int		max_reqs;	/* total slots */
 	unsigned long		state;		/* transport state */
 	unsigned char		shutdown   : 1,	/* being shut down */
-				resvport   : 1; /* use a reserved port */
+				resvport   : 1, /* use a reserved port */
+				no_recon_delay: 1; /* no delay when reconnect */
 	unsigned int		bind_index;	/* bind function index */
 
 	/*
net/sunrpc/clnt.c +2 -0
index df1039f..7a90d1a 100644
@@ -316,6 +316,8 @@ struct rpc_clnt *rpc_create(struct rpc_create_args *args)
 	if (args->flags & RPC_CLNT_CREATE_NONPRIVPORT)
 		xprt->resvport = 0;
 
+	xprt->no_recon_delay = !!args->no_recon_delay;
+
 	clnt = rpc_new_client(args, xprt);
 	if (IS_ERR(clnt))
 		return clnt;
net/sunrpc/xprtsock.c +1 -1
index 24c9605..52f2367 100644
@@ -2089,7 +2089,7 @@ static void xs_connect(struct rpc_task *task)
 	if (xprt_test_and_set_connecting(xprt))
 		return;
 
-	if (transport->sock != NULL) {
+	if (!xprt->no_recon_delay && transport->sock != NULL) {
 		dprintk("RPC:       xs_connect delayed xprt %p for %lu "
 				"seconds\n",
 				xprt, xprt->reestablish_timeout / HZ);
Signature · 1 line
1.6.4
#1 lore raw

Hi Mi-

Quoted 8 lines · Mi Jinlong

If network partition or some other reason cause a reconnect, it cannot succeed immediately when environment recover, but client want to connect timely sometimes.

This patch can provide a proc file(/proc/sys/fs/nfs/nfs_disable_reconnect_delay) to allow client disable the reconnect delay(reestablish_timeout) when using NFS.

It's only useful for NFS.

There's a good reason for the connection re-establishment delay, and only very few instances where you'd want to disable it. A sysctl is the wrong place for this, as it would disable the reconnect delay across the board, instead of for just those occasions when it is actually necessary to connect immediately.

I assume that because the grace period has a time limit, you would want the client to reconnect at all costs? I think that this is actually when a client should take care not to spuriously reconnect: during a server reboot, a server may be sluggish or not completely ready to accept client requests. It's not a time when a client should be showering a server with connection attempts.

The reconnect delay is an exponential backoff that starts at 3 seconds, so if the server is really ready to accept connections, the actual connection delay ought to be quick.

We're already considering shortening the maximum amount of time the client can wait before trying a reconnect. And, it might possibly be that the network layer itself is interfering with the backoff logic that is already built into the RPC client. (If true, that would be the real bug in this case). I'm not interested in a workaround when we really should fix any underlying issues to make this work correctly.

Perhaps the RPC client needs to distinguish between connection refusal (where a lengthening exponential backoff between connection attempts makes sense) and no server response (where we want the client's network layer to keep sending SYN requests so that it can reconnect as soon as possible).

The second scenario might disable the reconnect timer so that only one ->connect() call would be outstanding until the network layer tells us it's given up on SYN retries.

Quoted 120 lines
---
  fs/nfs/client.c             |    3 +++
  fs/nfs/sysctl.c             |    8 ++++++++
  include/linux/nfs_fs.h      |    6 ++++++
  include/linux/sunrpc/clnt.h |    1 +
  include/linux/sunrpc/xprt.h |    3 ++-
  net/sunrpc/clnt.c           |    2 ++
  net/sunrpc/xprtsock.c       |    2 +-
  7 files changed, 23 insertions(+), 2 deletions(-)
fs/nfs/client.c +3 -0
index 8d25ccb..e878724 100644
@@ -55,6 +55,8 @@ static LIST_HEAD(nfs_client_list);
  static LIST_HEAD(nfs_volume_list);
  static DECLARE_WAIT_QUEUE_HEAD(nfs_client_active_wq);
 
+int nfs_disable_reconnect_delay = 0;
+
  /*
   * RPC cruft for NFS
   */
@@ -607,6 +609,7 @@ static int nfs_create_rpc_client(struct nfs_client *clp,
  		.program	=&nfs_program,
  		.version	= clp->rpc_ops->version,
  		.authflavor	= flavor,
+		.no_recon_delay	= nfs_disable_reconnect_delay,
  	};
 
  	if (discrtry)
fs/nfs/sysctl.c +8 -0
index b62481d..6c04479 100644
@@ -58,6 +58,14 @@ static ctl_table nfs_cb_sysctls[] = {
  		.mode		= 0644,
  		.proc_handler	=&proc_dointvec,
  	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname	= "nfs_disable_reconnect_delay",
+		.data		=&nfs_disable_reconnect_delay,
+		.maxlen		= sizeof(nfs_disable_reconnect_delay),
+		.mode		= 0644,
+		.proc_handler	=&proc_dointvec,
+	},
  	{ .ctl_name = 0 }
  };
 
include/linux/nfs_fs.h +6 -0
index f6b9024..e031496 100644
@@ -390,6 +390,12 @@ static inline struct rpc_cred *nfs_file_cred(struct file *file)
  }
 
  /*
+ * linux/fs/nfs/client.c
+ */
+
+extern int nfs_disable_reconnect_delay;
+
+/*
   * linux/fs/nfs/xattr.c
   */
  #ifdef CONFIG_NFS_V3_ACL
include/linux/sunrpc/clnt.h +1 -0
index 5bd17f6..f73eae1 100644
@@ -115,6 +115,7 @@ struct rpc_create_args {
  	rpc_authflavor_t	authflavor;
  	unsigned long		flags;
  	char			*client_name;
+	int			no_recon_delay;  /* no delay when reconnect */
  };
 
  /* Values for "flags" field */
include/linux/sunrpc/xprt.h +2 -1
index 1175d58..a177348 100644
@@ -153,7 +153,8 @@ struct rpc_xprt {
  	unsigned int		max_reqs;	/* total slots */
  	unsigned long		state;		/* transport state */
  	unsigned char		shutdown   : 1,	/* being shut down */
-				resvport   : 1; /* use a reserved port */
+				resvport   : 1, /* use a reserved port */
+				no_recon_delay: 1; /* no delay when reconnect */
  	unsigned int		bind_index;	/* bind function index */
 
  	/*
net/sunrpc/clnt.c +2 -0
index df1039f..7a90d1a 100644
@@ -316,6 +316,8 @@ struct rpc_clnt *rpc_create(struct rpc_create_args *args)
  	if (args->flags&  RPC_CLNT_CREATE_NONPRIVPORT)
  		xprt->resvport = 0;
 
+	xprt->no_recon_delay = !!args->no_recon_delay;
+
  	clnt = rpc_new_client(args, xprt);
  	if (IS_ERR(clnt))
  		return clnt;
net/sunrpc/xprtsock.c +1 -1
index 24c9605..52f2367 100644
@@ -2089,7 +2089,7 @@ static void xs_connect(struct rpc_task *task)
  	if (xprt_test_and_set_connecting(xprt))
  		return;
 
-	if (transport->sock != NULL) {
+	if (!xprt->no_recon_delay&&  transport->sock != NULL) {
  		dprintk("RPC:       xs_connect delayed xprt %p for %lu "
  				"seconds\n",
  				xprt, xprt->reestablish_timeout / HZ);
Signature · 1 line
chuck[dot]lever[at]oracle[dot]com
author
#2 lore raw

Hi Chuck,

Sorry for replying your message so later.

Chuck Lever =E5=86=99=E9=81=93:

Quoted 4 lines

Hi Mi-
=20

Quoted 1 line · Mi Jinlong

If network partition or some other reason cause a reconnect, it cann=

ot

Quoted 1 line
Quoted 1 line

succeed immediately when environment recover, but client want to con=

nect

Quoted 5 lines
Quoted 5 lines

timely sometimes.

This patch can provide a proc file(/proc/sys/fs/nfs/nfs_disable_reconnect_delay)
to allow client disable the reconnect delay(reestablish_timeout) whe=

n

Quoted 6 lines
Quoted 3 lines

using NFS.

It's only useful for NFS.

=20
There's a good reason for the connection re-establishment delay, and only very few instances where you'd want to disable it. A sysctl is =

the

Quoted 1 line

wrong place for this, as it would disable the reconnect delay across =

the

Quoted 1 line

board, instead of for just those occasions when it is actually necess=

ary

Quoted 1 line

to connect immediately.

Yes, I agree with you.

Quoted 2 lines

=20
I assume that because the grace period has a time limit, you would wa=

nt

Quoted 7 lines

the client to reconnect at all costs? I think that this is actually when a client should take care not to spuriously reconnect: during a server reboot, a server may be sluggish or not completely ready to accept client requests. It's not a time when a client should be showering a server with connection attempts.
=20
The reconnect delay is an exponential backoff that starts at 3 second=

s,

Quoted 6 lines

so if the server is really ready to accept connections, the actual connection delay ought to be quick.
=20
We're already considering shortening the maximum amount of time the client can wait before trying a reconnect. And, it might possibly be that the network layer itself is interfering with the backoff logic t=

hat

Quoted 1 line

is already built into the RPC client. (If true, that would be the re=

al

Quoted 4 lines

bug in this case). I'm not interested in a workaround when we really should fix any underlying issues to make this work correctly. =20
Perhaps the RPC client needs to distinguish between connection refusa=

l

Quoted 2 lines

(where a lengthening exponential backoff between connection attempts makes sense) and no server response (where we want the client's netwo=

rk

Quoted 1 line

layer to keep sending SYN requests so that it can reconnect as soon a=

s

Quoted 1 line

possible).

When reading the kernel's code and testing, I find there are three ca=
se:

A. network partition:
Becasue the client can't communicate with server's rpcbind,=20 so there is no influence.

B. server's nfs service stop:
The client call xprt_connect to conncet, but get err(111: Connecti=
on refused).

C. server's nfs service sotp, and ifdown the NIC after about 60s:
At first, when the NIC is up, xprt_connect get err(111: Connection=
refused) as 2.

After NIC is down, xprt_connect get err(113: No route to host).

When connecting fail, the sunrpc level only get a ETIMEDOUT or EAGAIN =
err, it will also
call xprt_connect to reconnect.
If we make the network layer to keep sending SYN requests, but there w=
ill be more request=20
be delayed at the request queue, and the reestablish_timeout also be i=
ncreased.

Can we distinguish those refusal at sunrpc level, but not at xprt leve=
l ?
If we can do that, the problem will solved easily.
=20
[NOTE]
the testing process:
client server
1. mount nfs (OK)=20
2. df (OK)
3. nfs stop
4. df (hang)

I get message through rpcdebug.

Quoted 2 lines

=20
The second scenario might disable the reconnect timer so that only on=

e

Quoted 1 line

->connect() call would be outstanding until the network layer tells u=

s

Quoted 1 line

it's given up on SYN retries.

=20
I think that's a good idea, but implementation may be a great work.
=20
thanks,
Mi Jinlong

#3 lore raw
Quoted 9 lines · Mi Jinlong

Hi Chuck,

Sorry for replying your message so later.

Chuck Lever =E5=86=99=E9=81=93:

Quoted 4 lines

Hi Mi-

Quoted 1 line · Mi Jinlong

If network partition or some other reason cause a reconnect, it can=

not

Quoted 1 line
Quoted 1 line
Quoted 1 line

succeed immediately when environment recover, but client want to co=

nnect

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

timely sometimes.

This patch can provide a proc file(/proc/sys/fs/nfs/nfs_disable_reconnect_delay)
to allow client disable the reconnect delay(reestablish_timeout) wh=

en

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

using NFS.

It's only useful for NFS.

There's a good reason for the connection re-establishment delay, and only very few instances where you'd want to disable it. A sysctl is=

the

Quoted 1 line
Quoted 1 line

wrong place for this, as it would disable the reconnect delay across=

the

Quoted 1 line
Quoted 1 line

board, instead of for just those occasions when it is actually neces=

sary

Quoted 6 lines
Quoted 1 line

to connect immediately.

Yes, I agree with you.

Quoted 2 lines

I assume that because the grace period has a time limit, you would w=

ant

Quoted 7 lines
Quoted 7 lines

the client to reconnect at all costs? I think that this is actually when a client should take care not to spuriously reconnect: during a server reboot, a server may be sluggish or not completely ready to accept client requests. It's not a time when a client should be showering a server with connection attempts.

The reconnect delay is an exponential backoff that starts at 3 secon=

ds,

Quoted 5 lines
Quoted 5 lines

so if the server is really ready to accept connections, the actual connection delay ought to be quick.

We're already considering shortening the maximum amount of time the client can wait before trying a reconnect. And, it might possibly b=

e

Quoted 1 line
Quoted 1 line

that the network layer itself is interfering with the backoff logic =

that

Quoted 1 line
Quoted 1 line

is already built into the RPC client. (If true, that would be the r=

eal

Quoted 1 line
Quoted 1 line

bug in this case). I'm not interested in a workaround when we reall=

y

Quoted 3 lines
Quoted 3 lines

should fix any underlying issues to make this work correctly.

Perhaps the RPC client needs to distinguish between connection refus=

al

Quoted 2 lines
Quoted 2 lines

(where a lengthening exponential backoff between connection attempts makes sense) and no server response (where we want the client's netw=

ork

Quoted 1 line
Quoted 1 line

layer to keep sending SYN requests so that it can reconnect as soon =

as

Quoted 3 lines
Quoted 1 line

possible).

When reading the kernel's code and testing, I find there are three=

case:

Quoted 7 lines

A. network partition:
Becasue the client can't communicate with server's rpcbind, so there is no influence.

B. server's nfs service stop:
The client call xprt_connect to conncet, but get err(111: Conne=

ction refused).

Quoted 3 lines

C. server's nfs service sotp, and ifdown the NIC after about 60s:
At first, when the NIC is up, xprt_connect get err(111: Connect=

ion refused) as 2.

Quoted 4 lines

After NIC is down, xprt_connect get err(113: No route to host).

When connecting fail, the sunrpc level only get a ETIMEDOUT or EAGA=

IN err, it will also

Quoted 2 lines

call xprt_connect to reconnect.
If we make the network layer to keep sending SYN requests, but ther=

e will be more request

Quoted 1 line

be delayed at the request queue, and the reestablish_timeout also b=

e increased.

Quoted 2 lines

Can we distinguish those refusal at sunrpc level, but not at xprt l=

evel ?

Quoted 11 lines

If we can do that, the problem will solved easily.

[NOTE]
the testing process:
client server
1. mount nfs (OK)
2. df (OK)
3. nfs stop
4. df (hang)

I get message through rpcdebug.

We have a matrix of cases. "soft" v. "hard" RPCs, ECONNREFUSED v. no=20 response, connection previously closed by server disconnect v. client=20 idle timeout.

I've found at least one major bug in this logic, and that is that the 6= 0=20
second transport connect timer is clobbered in the ECONNREFUSED case, s= o=20
soft RPCs never time out if the server refuses a connection, for=20 example. I handed all of this off to Trond.

Quoted 1 line
Quoted 1 line

The second scenario might disable the reconnect timer so that only o=

ne

Quoted 1 line
Quoted 1 line

->connect() call would be outstanding until the network layer tells =

us

Quoted 3 lines
Quoted 1 line

it's given up on SYN retries.

I think that's a good idea, but implementation may be a great work=

=2E

Quoted 4 lines

thanks,
Mi Jinlong

--=20
chuck[dot]lever[at]oracle[dot]com

author
#4 lore raw

Chuck Lever =E5=86=99=E9=81=93:

Quoted 10 lines · Mi Jinlong
Quoted 9 lines

Hi Chuck,

Sorry for replying your message so later.

Chuck Lever =E5=86=99=E9=81=93:

Quoted 4 lines

Hi Mi-

Quoted 1 line · Mi Jinlong

If network partition or some other reason cause a reconnect, it ca=

nnot

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

succeed immediately when environment recover, but client want to connect
timely sometimes.

This patch can provide a proc file(/proc/sys/fs/nfs/nfs_disable_reconnect_delay)
to allow client disable the reconnect delay(reestablish_timeout) w=

hen

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

using NFS.

It's only useful for NFS.

There's a good reason for the connection re-establishment delay, an=

d

Quoted 1 line
Quoted 1 line
Quoted 1 line

only very few instances where you'd want to disable it. A sysctl i=

s the

Quoted 1 line
Quoted 1 line
Quoted 1 line

wrong place for this, as it would disable the reconnect delay acros=

s the

Quoted 1 line
Quoted 1 line
Quoted 1 line

board, instead of for just those occasions when it is actually nece=

ssary

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

to connect immediately.

Yes, I agree with you.

Quoted 2 lines

I assume that because the grace period has a time limit, you would =

want

Quoted 1 line
Quoted 1 line
Quoted 1 line

the client to reconnect at all costs? I think that this is actuall=

y

Quoted 1 line
Quoted 1 line
Quoted 1 line

when a client should take care not to spuriously reconnect: during =

a

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

server reboot, a server may be sluggish or not completely ready to accept client requests. It's not a time when a client should be showering a server with connection attempts.

The reconnect delay is an exponential backoff that starts at 3 seco=

nds,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

so if the server is really ready to accept connections, the actual connection delay ought to be quick.

We're already considering shortening the maximum amount of time the client can wait before trying a reconnect. And, it might possibly =

be

Quoted 1 line
Quoted 1 line
Quoted 1 line

that the network layer itself is interfering with the backoff logic=

that

Quoted 1 line
Quoted 1 line
Quoted 1 line

is already built into the RPC client. (If true, that would be the =

real

Quoted 1 line
Quoted 1 line
Quoted 1 line

bug in this case). I'm not interested in a workaround when we real=

ly

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

should fix any underlying issues to make this work correctly.

Perhaps the RPC client needs to distinguish between connection refu=

sal

Quoted 1 line
Quoted 1 line
Quoted 1 line

(where a lengthening exponential backoff between connection attempt=

s

Quoted 1 line
Quoted 1 line
Quoted 1 line

makes sense) and no server response (where we want the client's net=

work

Quoted 1 line
Quoted 1 line
Quoted 1 line

layer to keep sending SYN requests so that it can reconnect as soon=

as

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

possible).

When reading the kernel's code and testing, I find there are thre=

e

Quoted 10 lines
Quoted 10 lines

case:

A. network partition:
Becasue the client can't communicate with server's rpcbind, so there is no influence.

B. server's nfs service stop:
The client call xprt_connect to conncet, but get err(111:
Connection refused).

Quoted 5 lines
Quoted 5 lines

C. server's nfs service sotp, and ifdown the NIC after about 60s:
At first, when the NIC is up, xprt_connect get err(111:
Connection refused) as 2.

After NIC is down, xprt_connect get err(113: No route to host)=

=2E

Quoted 5 lines
Quoted 5 lines

When connecting fail, the sunrpc level only get a ETIMEDOUT or
EAGAIN err, it will also
call xprt_connect to reconnect.
If we make the network layer to keep sending SYN requests, but the=

re

Quoted 2 lines
Quoted 2 lines

will be more request
be delayed at the request queue, and the reestablish_timeout also =

be

Quoted 4 lines
Quoted 4 lines

increased.

Can we distinguish those refusal at sunrpc level, but not at xprt
level ?

What do you think that I show yesterday?

Quoted 10 lines
Quoted 10 lines

If we can do that, the problem will solved easily.

[NOTE]
the testing process:
client server
1. mount nfs (OK)
2. df (OK)
3. nfs stop
4. df (hang)

Quoted 5 lines
Quoted 1 line

I get message through rpcdebug.

=20
We have a matrix of cases. "soft" v. "hard" RPCs, ECONNREFUSED v. no response, connection previously closed by server disconnect v. client idle timeout.

connection previously closed by server disconnect v. client idle time=
out?
Can you explain to me in some sort? Maybe it's useful for me. Thanks.

Quoted 2 lines

=20
I've found at least one major bug in this logic, and that is that the=

60

Quoted 1 line

second transport connect timer is clobbered in the ECONNREFUSED case,=

so

Quoted 2 lines

soft RPCs never time out if the server refuses a connection, for example. I handed all of this off to Trond.

Really?=20
I mount the nfs file through soft(-o soft), and then I using "df" com=
mand
to see the mount information after server's nfs stop.
The "df" will return with error -5(Input/output error), maybe it's RP=
Cs=20
timeout cause the df return?

thanks,
Mi Jinlong

#5 lore raw
Quoted 11 lines · Mi Jinlong

Chuck Lever =E5=86=99=E9=81=93:

Quoted 10 lines · Mi Jinlong
Quoted 9 lines

Hi Chuck,
=20
Sorry for replying your message so later.
=20
Chuck Lever =E5=86=99=E9=81=93:

Quoted 4 lines

Hi Mi-
=20

Quoted 1 line · Mi Jinlong

If network partition or some other reason cause a reconnect, it c=

annot

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

succeed immediately when environment recover, but client want to connect
timely sometimes.
=20
This patch can provide a proc file(/proc/sys/fs/nfs/nfs_disable_reconnect_delay)
to allow client disable the reconnect delay(reestablish_timeout) =

when

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

using NFS.
=20
It's only useful for NFS.

=20
There's a good reason for the connection re-establishment delay, a=

nd

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

only very few instances where you'd want to disable it. A sysctl =

is the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

wrong place for this, as it would disable the reconnect delay acro=

ss the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

board, instead of for just those occasions when it is actually nec=

essary

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

to connect immediately.

=20
Yes, I agree with you.
=20

Quoted 2 lines

=20
I assume that because the grace period has a time limit, you would=

want

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

the client to reconnect at all costs? I think that this is actual=

ly

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

when a client should take care not to spuriously reconnect: during=

a

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

server reboot, a server may be sluggish or not completely ready to accept client requests. It's not a time when a client should be showering a server with connection attempts.
=20
The reconnect delay is an exponential backoff that starts at 3 sec=

onds,

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

so if the server is really ready to accept connections, the actual connection delay ought to be quick.
=20
We're already considering shortening the maximum amount of time th=

e

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

client can wait before trying a reconnect. And, it might possibly=

be

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

that the network layer itself is interfering with the backoff logi=

c that

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

is already built into the RPC client. (If true, that would be the=

real

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

bug in this case). I'm not interested in a workaround when we rea=

lly

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

should fix any underlying issues to make this work correctly. =20
Perhaps the RPC client needs to distinguish between connection ref=

usal

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

(where a lengthening exponential backoff between connection attemp=

ts

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

makes sense) and no server response (where we want the client's ne=

twork

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

layer to keep sending SYN requests so that it can reconnect as soo=

n as

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

possible).

=20
When reading the kernel's code and testing, I find there are thre=

e

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

case:
=20
A. network partition:
Becasue the client can't communicate with server's rpcbind, so there is no influence.
=20
B. server's nfs service stop:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

The client call xprt_connect to conncet, but get err(111:
Connection refused).
=20
C. server's nfs service sotp, and ifdown the NIC after about 60s:
At first, when the NIC is up, xprt_connect get err(111:
Connection refused) as 2.
=20
After NIC is down, xprt_connect get err(113: No route to host)=

=2E

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

=20
When connecting fail, the sunrpc level only get a ETIMEDOUT or
EAGAIN err, it will also
call xprt_connect to reconnect.
If we make the network layer to keep sending SYN requests, but the=

re

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

will be more request
be delayed at the request queue, and the reestablish_timeout also =

be

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

increased.
=20
Can we distinguish those refusal at sunrpc level, but not at xprt
level ?

=20
What do you think that I show yesterday?

In xprtsock.c, these reconnection errors are distinguished. In the gen= eric sunrpc client (xprt.c) they are not -- when an RPC transmission is=
sent, xprtsock.c returns ENOTCONN for any connection error. Trond mad=
e this change after 2.6.18. The differences matter in how the client r= e-establishes the connection, and that logic is all in xprtsock.c. So,=
the RPC client already makes this distinction, but the logic may have =
bugs.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

If we can do that, the problem will solved easily.
=20
[NOTE]
the testing process:
client server
1. mount nfs (OK)
2. df (OK)
3. nfs stop
4. df (hang)
=20
I get message through rpcdebug.

Quoted 2 lines
Quoted 2 lines

=20
We have a matrix of cases. "soft" v. "hard" RPCs, ECONNREFUSED v. n=

o

Quoted 1 line
Quoted 1 line

response, connection previously closed by server disconnect v. clien=

t

Quoted 3 lines
Quoted 1 line

idle timeout.

=20
connection previously closed by server disconnect v. client idle tim=

eout?

Quoted 1 line

Can you explain to me in some sort? Maybe it's useful for me. Thanks=

=2E

If the server closed the connection, the client should use the re-estab= lish timeout to delay the reconnection in order to prevent a hard loop = of client connection retries. If the client idled the connection out, = then the client should reconnect immediately.

Quoted 1 line
Quoted 1 line

I've found at least one major bug in this logic, and that is that th=

e 60

Quoted 1 line
Quoted 1 line

second transport connect timer is clobbered in the ECONNREFUSED case=

, so

Quoted 5 lines
Quoted 2 lines

soft RPCs never time out if the server refuses a connection, for example. I handed all of this off to Trond.

=20
Really?=20
I mount the nfs file through soft(-o soft), and then I using "df" co=

mmand

Quoted 2 lines

to see the mount information after server's nfs stop.
The "df" will return with error -5(Input/output error), maybe it's R=

PCs=20

Quoted 1 line

timeout cause the df return?

RPC timeouts generally cause an EIO. However, if the server continues = to refuse a connection, the timeout never occurs.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com