Skip to contentSkip to conversation

SUNRPC: increase max timeout for rebind to handle NFS server restart

PATCH In review Dai Ngo 4 participants 17 messages Apr 8, 2023 – Apr 18, 2023

Subsystems (3)
lore
Share & tools
lore

Dai Ngo

author reviewer
#0 lore raw

PATCH 1/1 SUNRPC: increase max timeout for rebind to handle NFS server restart

Currently call_bind_status places a hard limit of 9 seconds for retries on EACCES error. This limit was done to prevent the RPC request from being retried forever if the remote server has problem and never comes up

However this 9 seconds timeout is too short, comparing to other RPC timeouts which are generally in minutes. This causes intermittent failure with EIO on the client side when there are lots of NLM activity and the NFS server is restarted.

Instead of removing the max timeout for retry and relying on the RPC timeout mechanism to handle the retry, which can lead to the RPC being retried forever if the remote NLM service fails to come up. This patch simply increases the max timeout of call_bind_status from 9 to 90 seconds which should allow enough time for NLM to register after a restart, and not retrying forever if there is real problem with the remote system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM unlock requests")
---
 include/linux/sunrpc/clnt.h  | 3 +++
 include/linux/sunrpc/sched.h | 4 ++--
 net/sunrpc/clnt.c            | 2 +-
 net/sunrpc/sched.c           | 3 ++-
 4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +3 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
 #define RPC_CLNT_CREATE_REUSEPORT	(1UL << 11)
 #define RPC_CLNT_CREATE_CONNECTED	(1UL << 12)
 
+#define	RPC_CLNT_REBIND_DELAY		3
+#define	RPC_CLNT_REBIND_MAX_TIMEOUT	90
+
 struct rpc_clnt *rpc_create(struct rpc_create_args *args);
 struct rpc_clnt	*rpc_bind_new_program(struct rpc_clnt *,
 				const struct rpc_program *, u32);
include/linux/sunrpc/sched.h +2 -2
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
 #endif
 	unsigned char		tk_priority : 2,/* Task priority */
 				tk_garb_retry : 2,
-				tk_cred_retry : 2,
-				tk_rebind_retry : 2;
+				tk_cred_retry : 2;
+	unsigned char		tk_rebind_retry;
 };
 
 typedef void			(*rpc_action)(struct rpc_task *);
net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
 		if (task->tk_rebind_retry == 0)
 			break;
 		task->tk_rebind_retry--;
-		rpc_delay(task, 3*HZ);
+		rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
 		goto retry_timeout;
 	case -ENOBUFS:
 		rpc_delay(task, HZ >> 2);
net/sunrpc/sched.c +2 -1
index be587a308e05..5c18a35752aa 100644
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task *task)
 	/* Initialize retry counters */
 	task->tk_garb_retry = 2;
 	task->tk_cred_retry = 2;
-	task->tk_rebind_retry = 2;
+	task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
+					RPC_CLNT_REBIND_DELAY;
 
 	/* starting timestamp */
 	task->tk_start = ktime_get();
Signature · 1 line
2.9.5
maintainer Reviewed-by on v1 1/1
#1 lore raw
Quoted 75 lines · Dai Ngo

Currently call_bind_status places a hard limit of 9 seconds for retries on EACCES error. This limit was done to prevent the RPC request from being retried forever if the remote server has problem and never comes up

However this 9 seconds timeout is too short, comparing to other RPC timeouts which are generally in minutes. This causes intermittent failure with EIO on the client side when there are lots of NLM activity and the NFS server is restarted.

Instead of removing the max timeout for retry and relying on the RPC timeout mechanism to handle the retry, which can lead to the RPC being retried forever if the remote NLM service fails to come up. This patch simply increases the max timeout of call_bind_status from 9 to 90 seconds which should allow enough time for NLM to register after a restart, and not retrying forever if there is real problem with the remote system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM unlock requests")
---
 include/linux/sunrpc/clnt.h  | 3 +++
 include/linux/sunrpc/sched.h | 4 ++--
 net/sunrpc/clnt.c            | 2 +-
 net/sunrpc/sched.c           | 3 ++-
 4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +3 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
 #define RPC_CLNT_CREATE_REUSEPORT	(1UL << 11)
 #define RPC_CLNT_CREATE_CONNECTED	(1UL << 12)
 
+#define	RPC_CLNT_REBIND_DELAY		3
+#define	RPC_CLNT_REBIND_MAX_TIMEOUT	90
+
 struct rpc_clnt *rpc_create(struct rpc_create_args *args);
 struct rpc_clnt	*rpc_bind_new_program(struct rpc_clnt *,
 				const struct rpc_program *, u32);
include/linux/sunrpc/sched.h +2 -2
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
 #endif
 	unsigned char		tk_priority : 2,/* Task priority */
 				tk_garb_retry : 2,
-				tk_cred_retry : 2,
-				tk_rebind_retry : 2;
+				tk_cred_retry : 2;
+	unsigned char		tk_rebind_retry;
 };
 
 typedef void			(*rpc_action)(struct rpc_task *);
net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
 		if (task->tk_rebind_retry == 0)
 			break;
 		task->tk_rebind_retry--;
-		rpc_delay(task, 3*HZ);
+		rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
 		goto retry_timeout;
 	case -ENOBUFS:
 		rpc_delay(task, HZ >> 2);
net/sunrpc/sched.c +0 -0
index be587a308e05..5c18a35752aa 100644

net/sunrpc/sched.c rpc_init_task_statistics(struct rpc_task *task) · line 824

Quoted 9 lines
net/sunrpc/sched.c +2 -1
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task *task)
 	/* Initialize retry counters */
 	task->tk_garb_retry = 2;
 	task->tk_cred_retry = 2;
-	task->tk_rebind_retry = 2;
+	task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
+					RPC_CLNT_REBIND_DELAY;
 
 	/* starting timestamp */
 	task->tk_start = ktime_get();
#2 lore raw
Quoted 87 lines · Jeff Layton
Quoted 82 lines · Dai Ngo

Currently call_bind_status places a hard limit of 9 seconds for retries on EACCES error. This limit was done to prevent the RPC request from being retried forever if the remote server has problem and never comes up

However this 9 seconds timeout is too short, comparing to other RPC timeouts which are generally in minutes. This causes intermittent failure with EIO on the client side when there are lots of NLM activity and the NFS server is restarted.

Instead of removing the max timeout for retry and relying on the RPC timeout mechanism to handle the retry, which can lead to the RPC being retried forever if the remote NLM service fails to come up. This patch simply increases the max timeout of call_bind_status from 9 to 90 seconds which should allow enough time for NLM to register after a restart, and not retrying forever if there is real problem with the remote system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM unlock requests")

---

include/linux/sunrpc/clnt.h  | 3 +++
include/linux/sunrpc/sched.h | 4 ++--
net/sunrpc/clnt.c            | 2 +-
net/sunrpc/sched.c           | 3 ++-
4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +3 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
 #define RPC_CLNT_CREATE_REUSEPORT (1UL << 11)
 #define RPC_CLNT_CREATE_CONNECTED (1UL << 12)
 
+#define RPC_CLNT_REBIND_DELAY 3
+#define RPC_CLNT_REBIND_MAX_TIMEOUT 90
+
 struct rpc_clnt *rpc_create(struct rpc_create_args *args);
 struct rpc_clnt *rpc_bind_new_program(struct rpc_clnt *,
 const struct rpc_program *, u32);
include/linux/sunrpc/sched.h +0 -0
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
 #endif

unsigned char tk_priority : 2,/* Task priority */
tk_garb_retry : 2,
- tk_cred_retry : 2,
- tk_rebind_retry : 2;
+ tk_cred_retry : 2;
+ unsigned char tk_rebind_retry;
};

typedef void (*rpc_action)(struct rpc_task *);

net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
 if (task->tk_rebind_retry == 0)
 break;
 task->tk_rebind_retry--;
- rpc_delay(task, 3*HZ);
+ rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
 goto retry_timeout;

case -ENOBUFS:
rpc_delay(task, HZ >> 2);

net/sunrpc/sched.c +2 -1
index be587a308e05..5c18a35752aa 100644
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task *task)
 /* Initialize retry counters */
 task->tk_garb_retry = 2;
 task->tk_cred_retry = 2;
- task->tk_rebind_retry = 2;
+ task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
+ RPC_CLNT_REBIND_DELAY;
 
Quoted 4 lines

net/sunrpc/sched.c rpc_init_task_statistics(struct rpc_task *task) · line 824

Quoted 2 lines
net/sunrpc/sched.c +0 -0
 /* starting timestamp */
 task->tk_start = ktime_get();

Thanks!

Since we still haven't heard from the client folks on this, I can take this fix through the nfsd tree.

--
Chuck Lever

author reviewer
#3 lore raw
Quoted 86 lines · Chuck Lever III
Quoted 86 lines · Jeff Layton
Quoted 82 lines · Dai Ngo

Currently call_bind_status places a hard limit of 9 seconds for retries on EACCES error. This limit was done to prevent the RPC request from being retried forever if the remote server has problem and never comes up

However this 9 seconds timeout is too short, comparing to other RPC timeouts which are generally in minutes. This causes intermittent failure with EIO on the client side when there are lots of NLM activity and the NFS server is restarted.

Instead of removing the max timeout for retry and relying on the RPC timeout mechanism to handle the retry, which can lead to the RPC being retried forever if the remote NLM service fails to come up. This patch simply increases the max timeout of call_bind_status from 9 to 90 seconds which should allow enough time for NLM to register after a restart, and not retrying forever if there is real problem with the remote system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM unlock requests")

---

include/linux/sunrpc/clnt.h  | 3 +++
include/linux/sunrpc/sched.h | 4 ++--
net/sunrpc/clnt.c            | 2 +-
net/sunrpc/sched.c           | 3 ++-
4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +3 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
 #define RPC_CLNT_CREATE_REUSEPORT (1UL << 11)
 #define RPC_CLNT_CREATE_CONNECTED (1UL << 12)
 
+#define RPC_CLNT_REBIND_DELAY 3
+#define RPC_CLNT_REBIND_MAX_TIMEOUT 90
+
 struct rpc_clnt *rpc_create(struct rpc_create_args *args);
 struct rpc_clnt *rpc_bind_new_program(struct rpc_clnt *,
 const struct rpc_program *, u32);
include/linux/sunrpc/sched.h +0 -0
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
 #endif

unsigned char tk_priority : 2,/* Task priority */
tk_garb_retry : 2,
- tk_cred_retry : 2,
- tk_rebind_retry : 2;
+ tk_cred_retry : 2;
+ unsigned char tk_rebind_retry;
};

typedef void (*rpc_action)(struct rpc_task *);

net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
 if (task->tk_rebind_retry == 0)
 break;
 task->tk_rebind_retry--;
- rpc_delay(task, 3*HZ);
+ rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
 goto retry_timeout;

case -ENOBUFS:
rpc_delay(task, HZ >> 2);

net/sunrpc/sched.c +2 -1
index be587a308e05..5c18a35752aa 100644
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task *task)
 /* Initialize retry counters */
 task->tk_garb_retry = 2;
 task->tk_cred_retry = 2;
- task->tk_rebind_retry = 2;
+ task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
+ RPC_CLNT_REBIND_DELAY;
 
Quoted 7 lines
Quoted 3 lines

net/sunrpc/sched.c rpc_init_task_statistics(struct rpc_task *task) · line 824

Quoted 2 lines
net/sunrpc/sched.c +0 -0
 /* starting timestamp */
 task->tk_start = ktime_get();

Thanks!

Since we still haven't heard from the client folks on this, I can take this fix through the nfsd tree.

Thank you Jeff and Chuck!

-Dai

#4 lore raw
In reply to Dai Ngo
Quoted 86 lines · Dai Ngo

Currently call_bind_status places a hard limit of 9 seconds for retries
on EACCES error. This limit was done to prevent the RPC request from being retried forever if the remote server has problem and never comes
up

However this 9 seconds timeout is too short, comparing to other RPC timeouts which are generally in minutes. This causes intermittent failure
with EIO on the client side when there are lots of NLM activity and the
NFS server is restarted.

Instead of removing the max timeout for retry and relying on the RPC timeout mechanism to handle the retry, which can lead to the RPC being
retried forever if the remote NLM service fails to come up. This patch
simply increases the max timeout of call_bind_status from 9 to 90 seconds
which should allow enough time for NLM to register after a restart, and
not retrying forever if there is real problem with the remote system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM unlock

requests")

---
 include/linux/sunrpc/clnt.h  | 3 +++
 include/linux/sunrpc/sched.h | 4 ++--
 net/sunrpc/clnt.c            | 2 +-
 net/sunrpc/sched.c           | 3 ++-

 4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +0 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
  #define RPC_CLNT_CREATE_REUSEPORT      (1UL << 11)
  #define RPC_CLNT_CREATE_CONNECTED      (1UL << 12)

include/linux/sunrpc/clnt.h +3 -0
+#define        RPC_CLNT_REBIND_DELAY           3
+#define        RPC_CLNT_REBIND_MAX_TIMEOUT     90
+

 struct rpc_clnt *rpc_create(struct rpc_create_args *args); struct rpc_clnt        *rpc_bind_new_program(struct rpc_clnt *, const struct rpc_program *, u32);

include/linux/sunrpc/sched.h +0 -0
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
  #endif

        unsigned char           tk_priority : 2,/* Task priority */ tk_garb_retry : 2,
-                               tk_cred_retry : 2,
-                               tk_rebind_retry : 2;
+                               tk_cred_retry : 2;
+       unsigned char           tk_rebind_retry;
 };

 typedef void                   (*rpc_action)(struct rpc_task *);

net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
                 if (task->tk_rebind_retry == 0)
                         break;
                 task->tk_rebind_retry--;
-               rpc_delay(task, 3*HZ);
+               rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
                 goto retry_timeout;

        case -ENOBUFS:
                rpc_delay(task, HZ >> 2);

net/sunrpc/sched.c +0 -0
index be587a308e05..5c18a35752aa 100644

net/sunrpc/sched.c rpc_init_task_statistics(struct rpc_task *task) · line 821

Quoted 6 lines
net/sunrpc/sched.c +2 -1
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task *task)
         /* Initialize retry counters */
         task->tk_garb_retry = 2;
         task->tk_cred_retry = 2;
-       task->tk_rebind_retry = 2;
+       task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
+                                       RPC_CLNT_REBIND_DELAY;

Why not just implement an exponential back off? If the server is slow to come up, then pounding the rpcbind service every 3 seconds isn't going to help.

Quoted 3 lines


        /* starting timestamp */
        task->tk_start = ktime_get();

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#5 lore raw
Quoted 93 lines · Trond Myklebust
Quoted 88 lines · Dai Ngo

Currently call_bind_status places a hard limit of 9 seconds for retries
on EACCES error. This limit was done to prevent the RPC request from being retried forever if the remote server has problem and never comes
up

However this 9 seconds timeout is too short, comparing to other RPC timeouts which are generally in minutes. This causes intermittent failure
with EIO on the client side when there are lots of NLM activity and the
NFS server is restarted.

Instead of removing the max timeout for retry and relying on the RPC timeout mechanism to handle the retry, which can lead to the RPC being
retried forever if the remote NLM service fails to come up. This patch
simply increases the max timeout of call_bind_status from 9 to 90 seconds
which should allow enough time for NLM to register after a restart, and
not retrying forever if there is real problem with the remote system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM unlock

requests")

---
 include/linux/sunrpc/clnt.h  | 3 +++
 include/linux/sunrpc/sched.h | 4 ++--
 net/sunrpc/clnt.c            | 2 +-
 net/sunrpc/sched.c           | 3 ++-
 4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +3 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
 #define RPC_CLNT_CREATE_REUSEPORT      (1UL << 11)
 #define RPC_CLNT_CREATE_CONNECTED      (1UL << 12)
 
+#define        RPC_CLNT_REBIND_DELAY           3
+#define        RPC_CLNT_REBIND_MAX_TIMEOUT     90
+
 struct rpc_clnt *rpc_create(struct rpc_create_args *args);
 struct rpc_clnt        *rpc_bind_new_program(struct rpc_clnt *,
                                const struct rpc_program *, u32);
include/linux/sunrpc/sched.h +2 -2
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
 #endif
        unsigned char           tk_priority : 2,/* Task priority */
                                tk_garb_retry : 2,
-                               tk_cred_retry : 2,
-                               tk_rebind_retry : 2;
+                               tk_cred_retry : 2;
+       unsigned char           tk_rebind_retry;
 };
 
 typedef void                   (*rpc_action)(struct rpc_task *);
net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
                if (task->tk_rebind_retry == 0)
                        break;
                task->tk_rebind_retry--;
-               rpc_delay(task, 3*HZ);
+               rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
                goto retry_timeout;
        case -ENOBUFS:
                rpc_delay(task, HZ >> 2);
net/sunrpc/sched.c +0 -0
index be587a308e05..5c18a35752aa 100644
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task *task)
        /* Initialize retry counters */
        task->tk_garb_retry = 2;
Quoted 8 lines

net/sunrpc/sched.c rpc_init_task_statistics(struct rpc_task *task) · line 821

Quoted 4 lines
net/sunrpc/sched.c +2 -1
        task->tk_cred_retry = 2;
-       task->tk_rebind_retry = 2;
+       task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
+                                       RPC_CLNT_REBIND_DELAY;

Why not just implement an exponential back off? If the server is slow to come up, then pounding the rpcbind service every 3 seconds isn't going to help.

Exponential backoff has the disadvantage that once we've gotten to the longer retry times, it's easy for a client to wait quite some time before it notices the rebind service is available.

But I have to wonder if retrying every 3 seconds is useful if the request is going over TCP.

Quoted 3 lines

net/sunrpc/sched.c rpc_init_task_statistics(struct rpc_task *task) · line 824

Quoted 3 lines
net/sunrpc/sched.c +0 -0
        /* starting timestamp */
        task->tk_start = ktime_get();

--
Chuck Lever

#6 lore raw
Quoted 112 lines · Chuck Lever III
Quoted 103 lines

On Apr 17, 2023, at 4:49 PM, Trond Myklebust
<[email protected]> wrote:

Quoted 97 lines · Dai Ngo

Currently call_bind_status places a hard limit of 9 seconds for retries
on EACCES error. This limit was done to prevent the RPC request from
being retried forever if the remote server has problem and never comes
up

However this 9 seconds timeout is too short, comparing to other RPC
timeouts which are generally in minutes. This causes intermittent failure
with EIO on the client side when there are lots of NLM activity and
the
NFS server is restarted.

Instead of removing the max timeout for retry and relying on the RPC
timeout mechanism to handle the retry, which can lead to the RPC being
retried forever if the remote NLM service fails to come up. This patch
simply increases the max timeout of call_bind_status from 9 to 90 seconds
which should allow enough time for NLM to register after a restart,
and
not retrying forever if there is real problem with the remote system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM unlock

requests")

---
 include/linux/sunrpc/clnt.h  | 3 +++
 include/linux/sunrpc/sched.h | 4 ++--
 net/sunrpc/clnt.c            | 2 +-
 net/sunrpc/sched.c           | 3 ++-

 4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +0 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
  #define RPC_CLNT_CREATE_REUSEPORT      (1UL << 11)
  #define RPC_CLNT_CREATE_CONNECTED      (1UL << 12)

include/linux/sunrpc/clnt.h +3 -0
+#define        RPC_CLNT_REBIND_DELAY           3
+#define        RPC_CLNT_REBIND_MAX_TIMEOUT     90
+

 struct rpc_clnt *rpc_create(struct rpc_create_args *args); struct rpc_clnt        *rpc_bind_new_program(struct rpc_clnt *, const struct rpc_program *, u32);

include/linux/sunrpc/sched.h +0 -0
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
  #endif

        unsigned char           tk_priority : 2,/* Task priority */
                                tk_garb_retry : 2,
-                               tk_cred_retry : 2,
-                               tk_rebind_retry : 2;
+                               tk_cred_retry : 2;
+       unsigned char           tk_rebind_retry;
 };

 typedef void                   (*rpc_action)(struct rpc_task *);

net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
                 if (task->tk_rebind_retry == 0)
                         break;
                 task->tk_rebind_retry--;
-               rpc_delay(task, 3*HZ);
+               rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
                 goto retry_timeout;

        case -ENOBUFS:
                rpc_delay(task, HZ >> 2);

net/sunrpc/sched.c +0 -0
index be587a308e05..5c18a35752aa 100644
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task
 *task)
         /* Initialize retry counters */
         task->tk_garb_retry = 2;
         task->tk_cred_retry = 2;
Quoted 8 lines

net/sunrpc/sched.c rpc_init_task_statistics(struct rpc_task *task) · line 821

Quoted 3 lines
net/sunrpc/sched.c +2 -1
-       task->tk_rebind_retry = 2;
+       task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
+                                       RPC_CLNT_REBIND_DELAY;

Why not just implement an exponential back off? If the server is slow
to come up, then pounding the rpcbind service every 3 seconds isn't going to help.

Quoted 7 lines

Exponential backoff has the disadvantage that once we've gotten to the longer retry times, it's easy for a client to wait quite some time before it notices the rebind service is available.

But I have to wonder if retrying every 3 seconds is useful if the request is going over TCP.

The problem isn't reliability: this is handling a case where we _are_ getting a reply from the server, just not one we wanted. EACCES here means that the rpcbind server didn't return a valid entry for the service we requested, presumably because the service hasn't been registered yet.

So yes, an exponential back off is appropriate here.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
author reviewer
#7 lore raw
Quoted 114 lines · Trond Myklebust
Quoted 113 lines · Chuck Lever III
Quoted 101 lines

On Apr 17, 2023, at 4:49 PM, Trond Myklebust
<[email protected]> wrote:

Quoted 96 lines · Dai Ngo

Currently call_bind_status places a hard limit of 9 seconds for retries
on EACCES error. This limit was done to prevent the RPC request from
being retried forever if the remote server has problem and never comes
up

However this 9 seconds timeout is too short, comparing to other RPC
timeouts which are generally in minutes. This causes intermittent failure
with EIO on the client side when there are lots of NLM activity and
the
NFS server is restarted.

Instead of removing the max timeout for retry and relying on the RPC
timeout mechanism to handle the retry, which can lead to the RPC being
retried forever if the remote NLM service fails to come up. This patch
simply increases the max timeout of call_bind_status from 9 to 90 seconds
which should allow enough time for NLM to register after a restart,
and
not retrying forever if there is real problem with the remote system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM unlock

requests")

---
  include/linux/sunrpc/clnt.h  | 3 +++
  include/linux/sunrpc/sched.h | 4 ++--
  net/sunrpc/clnt.c            | 2 +-
  net/sunrpc/sched.c           | 3 ++-
  4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +3 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
  #define RPC_CLNT_CREATE_REUSEPORT      (1UL << 11)
  #define RPC_CLNT_CREATE_CONNECTED      (1UL << 12)
 
+#define        RPC_CLNT_REBIND_DELAY           3
+#define        RPC_CLNT_REBIND_MAX_TIMEOUT     90
+
  struct rpc_clnt *rpc_create(struct rpc_create_args *args);
  struct rpc_clnt        *rpc_bind_new_program(struct rpc_clnt *,
                                 const struct rpc_program *, u32);
include/linux/sunrpc/sched.h +2 -2
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
  #endif
         unsigned char           tk_priority : 2,/* Task priority
 */
                                 tk_garb_retry : 2,
-                               tk_cred_retry : 2,
-                               tk_rebind_retry : 2;
+                               tk_cred_retry : 2;
+       unsigned char           tk_rebind_retry;
  };
 

 typedef void                   (*rpc_action)(struct rpc_task *);

net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
                 if (task->tk_rebind_retry == 0)
                         break;
                 task->tk_rebind_retry--;
-               rpc_delay(task, 3*HZ);
+               rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
                 goto retry_timeout;
         case -ENOBUFS:
                 rpc_delay(task, HZ >> 2);
net/sunrpc/sched.c +0 -0
index be587a308e05..5c18a35752aa 100644
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task
 *task)
         /* Initialize retry counters */
         task->tk_garb_retry = 2;
Quoted 8 lines

net/sunrpc/sched.c rpc_init_task_statistics(struct rpc_task *task) · line 821

Quoted 4 lines
net/sunrpc/sched.c +2 -1
         task->tk_cred_retry = 2;
-       task->tk_rebind_retry = 2;
+       task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
+                                       RPC_CLNT_REBIND_DELAY;

Why not just implement an exponential back off? If the server is slow
to come up, then pounding the rpcbind service every 3 seconds isn't going to help.

Exponential backoff has the disadvantage that once we've gotten to the longer retry times, it's easy for a client to wait quite some time before it notices the rebind service is available.

Quoted 8 lines
Quoted 3 lines

But I have to wonder if retrying every 3 seconds is useful if the request is going over TCP.

The problem isn't reliability: this is handling a case where we _are_ getting a reply from the server, just not one we wanted. EACCES here means that the rpcbind server didn't return a valid entry for the service we requested, presumably because the service hasn't been registered yet.

That's correct.

Quoted 2 lines

So yes, an exponential back off is appropriate here.

I think Chuck's point is still valid. It makes the client a little more responsive; does not have to wait that long, and the overhead of a RPC request every 3 seconds is not that significant.

-Dai

Quoted 1 line
#8 lore raw
Quoted 147 lines · On Mon, 2023-04-17 at 14:51 -0700, [email protected] wrote:
Quoted 134 lines · Trond Myklebust
Quoted 126 lines · Chuck Lever III
Quoted 118 lines

On Apr 17, 2023, at 4:49 PM, Trond Myklebust
<[email protected]> wrote:

Quoted 113 lines · Dai Ngo

Currently call_bind_status places a hard limit of 9 seconds for
retries
on EACCES error. This limit was done to prevent the RPC request
from
being retried forever if the remote server has problem and never
comes
up

However this 9 seconds timeout is too short, comparing to other
RPC
timeouts which are generally in minutes. This causes intermittent
failure
with EIO on the client side when there are lots of NLM activity
and
the
NFS server is restarted.

Instead of removing the max timeout for retry and relying on the
RPC
timeout mechanism to handle the retry, which can lead to the RPC
being
retried forever if the remote NLM service fails to come up. This
patch
simply increases the max timeout of call_bind_status from 9 to 90
seconds
which should allow enough time for NLM to register after a restart,
and
not retrying forever if there is real problem with the remote system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM unlock

requests")

---
  include/linux/sunrpc/clnt.h  | 3 +++
  include/linux/sunrpc/sched.h | 4 ++--
  net/sunrpc/clnt.c            | 2 +-
  net/sunrpc/sched.c           | 3 ++-

  4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +0 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
   #define RPC_CLNT_CREATE_REUSEPORT      (1UL << 11)
   #define RPC_CLNT_CREATE_CONNECTED      (1UL << 12)

include/linux/sunrpc/clnt.h +3 -0
+#define        RPC_CLNT_REBIND_DELAY           3
+#define        RPC_CLNT_REBIND_MAX_TIMEOUT     90
+

  struct rpc_clnt *rpc_create(struct rpc_create_args *args); struct rpc_clnt        *rpc_bind_new_program(struct rpc_clnt *,
                                 const struct rpc_program *, u32);

include/linux/sunrpc/sched.h +0 -0
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
   #endif

         unsigned char           tk_priority : 2,/* Task priority
*/
                                 tk_garb_retry : 2, -                               tk_cred_retry : 2, -                               tk_rebind_retry : 2; +                               tk_cred_retry : 2; +       unsigned char           tk_rebind_retry; };

  typedef void                   (*rpc_action)(struct rpc_task *);

net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
                  if (task->tk_rebind_retry == 0)
                          break;
                  task->tk_rebind_retry--;
-               rpc_delay(task, 3*HZ);
+               rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
                  goto retry_timeout;

         case -ENOBUFS:
                 rpc_delay(task, HZ >> 2);

net/sunrpc/sched.c +1 -1
index be587a308e05..5c18a35752aa 100644
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task
 *task)
          /* Initialize retry counters */
          task->tk_garb_retry = 2;
          task->tk_cred_retry = 2;
-       task->tk_rebind_retry = 2;
+       task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
Quoted 8 lines

net/sunrpc/sched.c rpc_init_task_statistics(struct rpc_task · line 823

Quoted 2 lines
net/sunrpc/sched.c +1 -0
+                                      
 RPC_CLNT_REBIND_DELAY;

Why not just implement an exponential back off? If the server is
slow
to come up, then pounding the rpcbind service every 3 seconds isn't
going to help.

Quoted 7 lines

Exponential backoff has the disadvantage that once we've gotten to the longer retry times, it's easy for a client to wait quite some time before it notices the rebind service is available.

But I have to wonder if retrying every 3 seconds is useful if the request is going over TCP.

Quoted 7 lines

The problem isn't reliability: this is handling a case where we _are_
getting a reply from the server, just not one we wanted. EACCES here
means that the rpcbind server didn't return a valid entry for the service we requested, presumably because the service hasn't been registered yet.

That's correct.

Quoted 1 line
Quoted 8 lines
Quoted 1 line

So yes, an exponential back off is appropriate here.

I think Chuck's point is still valid. It makes the client a little more
responsive; does not have to wait that long, and the overhead of a RPC
request every 3 seconds is not that significant.

Quoted 1 line

It is when you do it 30 times before giving up.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
author reviewer
#9 lore raw
Quoted 144 lines · Trond Myklebust
Quoted 143 lines · On Mon, 2023-04-17 at 14:51 -0700, [email protected] wrote:
Quoted 133 lines · Trond Myklebust
Quoted 125 lines · Chuck Lever III
Quoted 117 lines

On Apr 17, 2023, at 4:49 PM, Trond Myklebust
<[email protected]> wrote:

Quoted 113 lines · Dai Ngo

Currently call_bind_status places a hard limit of 9 seconds for
retries
on EACCES error. This limit was done to prevent the RPC request
from
being retried forever if the remote server has problem and never
comes
up

However this 9 seconds timeout is too short, comparing to other
RPC
timeouts which are generally in minutes. This causes intermittent
failure
with EIO on the client side when there are lots of NLM activity
and
the
NFS server is restarted.

Instead of removing the max timeout for retry and relying on the
RPC
timeout mechanism to handle the retry, which can lead to the RPC
being
retried forever if the remote NLM service fails to come up. This
patch
simply increases the max timeout of call_bind_status from 9 to 90
seconds
which should allow enough time for NLM to register after a restart,
and
not retrying forever if there is real problem with the remote system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM unlock

requests")

---
   include/linux/sunrpc/clnt.h  | 3 +++
   include/linux/sunrpc/sched.h | 4 ++--
   net/sunrpc/clnt.c            | 2 +-
   net/sunrpc/sched.c           | 3 ++-
   4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +3 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
   #define RPC_CLNT_CREATE_REUSEPORT      (1UL << 11)
   #define RPC_CLNT_CREATE_CONNECTED      (1UL << 12)
 
+#define        RPC_CLNT_REBIND_DELAY           3
+#define        RPC_CLNT_REBIND_MAX_TIMEOUT     90
+
   struct rpc_clnt *rpc_create(struct rpc_create_args *args);
   struct rpc_clnt        *rpc_bind_new_program(struct
 rpc_clnt *,

                                 const struct rpc_program *,
u32);

include/linux/sunrpc/sched.h +0 -0
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
   #endif
          unsigned char           tk_priority : 2,/* Task

priority
*/
                                 tk_garb_retry : 2,
-                               tk_cred_retry : 2, -                               tk_rebind_retry : 2; +                               tk_cred_retry : 2; +       unsigned char           tk_rebind_retry;
  };

  typedef void                   (*rpc_action)(struct
rpc_task *);

net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
                  if (task->tk_rebind_retry == 0)
                          break;
                  task->tk_rebind_retry--;
-               rpc_delay(task, 3*HZ);
+               rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
                  goto retry_timeout;
          case -ENOBUFS:
                  rpc_delay(task, HZ >> 2);
net/sunrpc/sched.c +1 -1
index be587a308e05..5c18a35752aa 100644
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task
 *task)
          /* Initialize retry counters */
          task->tk_garb_retry = 2;
          task->tk_cred_retry = 2;
-       task->tk_rebind_retry = 2;
+       task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
Quoted 8 lines

net/sunrpc/sched.c rpc_init_task_statistics(struct rpc_task · line 823

Quoted 2 lines
net/sunrpc/sched.c +1 -0
+
 RPC_CLNT_REBIND_DELAY;

Why not just implement an exponential back off? If the server is
slow
to come up, then pounding the rpcbind service every 3 seconds isn't
going to help.

Quoted 7 lines

Exponential backoff has the disadvantage that once we've gotten to the longer retry times, it's easy for a client to wait quite some time before it notices the rebind service is available.

But I have to wonder if retrying every 3 seconds is useful if the request is going over TCP.

Quoted 7 lines

The problem isn't reliability: this is handling a case where we _are_
getting a reply from the server, just not one we wanted. EACCES here
means that the rpcbind server didn't return a valid entry for the service we requested, presumably because the service hasn't been registered yet.

That's correct.

Quoted 7 lines
Quoted 6 lines
Quoted 1 line

So yes, an exponential back off is appropriate here.

I think Chuck's point is still valid. It makes the client a little more
responsive; does not have to wait that long, and the overhead of a RPC
request every 3 seconds is not that significant.

It is when you do it 30 times before giving up.

This is true for a dead NFS server, we save 25 RPC calls in a period of ~90 seconds.

In the case of a head failover and the NFS service on the takeover head can take up to 15 seconds to restart to pick up the new exports from the fail head, then the client can potentially wait up to ~20 seconds, after the NFS server is ready, to resume normal operation. IMHO, it's a 'long' time and I'd prioritize speed over saving some overhead.

Having said that, I will make the change that Trond suggested if Chuck and Jeff don't have any objection.

-Dai

Quoted 1 line
#10 lore raw
Quoted 180 lines · On Mon, 2023-04-17 at 16:14 -0700, [email protected] wrote:
Quoted 166 lines · Trond Myklebust
Quoted 165 lines · On Mon, 2023-04-17 at 14:51 -0700, [email protected] wrote:
Quoted 152 lines · Trond Myklebust
Quoted 142 lines · Chuck Lever III
Quoted 134 lines

On Apr 17, 2023, at 4:49 PM, Trond Myklebust
<[email protected]> wrote:

Quoted 122 lines · Dai Ngo

Currently call_bind_status places a hard limit of 9 seconds
for
retries
on EACCES error. This limit was done to prevent the RPC request
from
being retried forever if the remote server has problem and
never
comes
up

However this 9 seconds timeout is too short, comparing to other
RPC
timeouts which are generally in minutes. This causes intermittent
failure
with EIO on the client side when there are lots of NLM activity
and
the
NFS server is restarted.

Instead of removing the max timeout for retry and relying on
the
RPC
timeout mechanism to handle the retry, which can lead to the
RPC
being
retried forever if the remote NLM service fails to come up.
This
patch
simply increases the max timeout of call_bind_status from 9
to 90
seconds
which should allow enough time for NLM to register after a
restart,
and
not retrying forever if there is real problem with the remote
system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM

unlock
requests")

---
   include/linux/sunrpc/clnt.h  | 3 +++
   include/linux/sunrpc/sched.h | 4 ++--
   net/sunrpc/clnt.c            | 2 +-
   net/sunrpc/sched.c           | 3 ++-

   4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +0 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
    #define RPC_CLNT_CREATE_REUSEPORT      (1UL << 11)
    #define RPC_CLNT_CREATE_CONNECTED      (1UL << 12)

include/linux/sunrpc/clnt.h +3 -0
+#define        RPC_CLNT_REBIND_DELAY           3
+#define        RPC_CLNT_REBIND_MAX_TIMEOUT     90
+

   struct rpc_clnt *rpc_create(struct rpc_create_args *args);
   struct rpc_clnt        *rpc_bind_new_program(struct rpc_clnt *,
                                  const struct rpc_program *,
u32);

include/linux/sunrpc/sched.h +0 -0
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
    #endif

          unsigned char           tk_priority : 2,/* Task priority
*/
                                  tk_garb_retry : 2, -                               tk_cred_retry : 2, -                               tk_rebind_retry : 2; +                               tk_cred_retry : 2; +       unsigned char           tk_rebind_retry; };

   typedef void                   (*rpc_action)(struct rpc_task *);

net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task
 *task)
                   if (task->tk_rebind_retry == 0)
                           break;
                   task->tk_rebind_retry--;
-               rpc_delay(task, 3*HZ);
+               rpc_delay(task, RPC_CLNT_REBIND_DELAY *
 HZ);
                   goto retry_timeout;

          case -ENOBUFS:
                  rpc_delay(task, HZ >> 2);

net/sunrpc/sched.c +0 -0
index be587a308e05..5c18a35752aa 100644
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct

rpc_task
*task)

Quoted 8 lines

          /* Initialize retry counters */
          task->tk_garb_retry = 2;
          task->tk_cred_retry = 2;
-       task->tk_rebind_retry = 2;
+       task->tk_rebind_retry =
RPC_CLNT_REBIND_MAX_TIMEOUT /
+
RPC_CLNT_REBIND_DELAY;

Quoted 8 lines

Why not just implement an exponential back off? If the server
is
slow
to come up, then pounding the rpcbind service every 3 seconds
isn't
going to help.

Quoted 9 lines

Exponential backoff has the disadvantage that once we've gotten
to the longer retry times, it's easy for a client to wait quite
some time before it notices the rebind service is available.

But I have to wonder if retrying every 3 seconds is useful if the request is going over TCP.

Quoted 9 lines

The problem isn't reliability: this is handling a case where we _are_
getting a reply from the server, just not one we wanted. EACCES here
means that the rpcbind server didn't return a valid entry for the
service we requested, presumably because the service hasn't been
registered yet.

That's correct.

Quoted 1 line

So yes, an exponential back off is appropriate here.

Quoted 8 lines
Quoted 7 lines

I think Chuck's point is still valid. It makes the client a little
more
responsive; does not have to wait that long, and the overhead of a
RPC
request every 3 seconds is not that significant.

It is when you do it 30 times before giving up.

This is true for a dead NFS server, we save 25 RPC calls in a period of ~90 seconds.

Quoted 7 lines

In the case of a head failover and the NFS service on the takeover head can take up to 15 seconds to restart to pick up the new exports from the fail head, then the client can potentially wait up to ~20 seconds, after the NFS server is ready, to resume normal operation. IMHO, it's a 'long' time and I'd prioritize speed over saving some overhead.

task->tk_rebind_retry is _only_ changed if the rpcbind server is up and running, and returns an empty reply because the service we're looking up isn't registered.
task->tk_rebind_retry isn't changed on any request timeout. It isn't changed on any connection failure. It isn't changed by any other code path in the RPC client.

So none of this applies to the case of a dead server.

It applies to the case of a live server, where rpcbind is running and accessible to the client and where, for some reason or another, it is taking an exceptionally long time to register the service we are looking up the port for (either NLM or NFSv3).

So where are you seeing this process take 90 seconds? Why do we need to wait for that long before we can finally conclude that the particular service in question is not going to come back up?

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
author reviewer
#11 lore raw
Quoted 186 lines · Trond Myklebust
Quoted 178 lines · On Mon, 2023-04-17 at 16:14 -0700, [email protected] wrote:
Quoted 166 lines · Trond Myklebust
Quoted 165 lines · On Mon, 2023-04-17 at 14:51 -0700, [email protected] wrote:
Quoted 152 lines · Trond Myklebust
Quoted 142 lines · Chuck Lever III
Quoted 134 lines

On Apr 17, 2023, at 4:49 PM, Trond Myklebust
<[email protected]> wrote:

Quoted 122 lines · Dai Ngo

Currently call_bind_status places a hard limit of 9 seconds
for
retries
on EACCES error. This limit was done to prevent the RPC request
from
being retried forever if the remote server has problem and
never
comes
up

However this 9 seconds timeout is too short, comparing to other
RPC
timeouts which are generally in minutes. This causes intermittent
failure
with EIO on the client side when there are lots of NLM activity
and
the
NFS server is restarted.

Instead of removing the max timeout for retry and relying on
the
RPC
timeout mechanism to handle the retry, which can lead to the
RPC
being
retried forever if the remote NLM service fails to come up.
This
patch
simply increases the max timeout of call_bind_status from 9
to 90
seconds
which should allow enough time for NLM to register after a
restart,
and
not retrying forever if there is real problem with the remote
system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM

unlock
requests")

---
    include/linux/sunrpc/clnt.h  | 3 +++
    include/linux/sunrpc/sched.h | 4 ++--
    net/sunrpc/clnt.c            | 2 +-
    net/sunrpc/sched.c           | 3 ++-
    4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +3 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
    #define RPC_CLNT_CREATE_REUSEPORT      (1UL << 11)
    #define RPC_CLNT_CREATE_CONNECTED      (1UL << 12)
 
+#define        RPC_CLNT_REBIND_DELAY           3
+#define        RPC_CLNT_REBIND_MAX_TIMEOUT     90
+
    struct rpc_clnt *rpc_create(struct rpc_create_args
 *args);
    struct rpc_clnt        *rpc_bind_new_program(struct

rpc_clnt *,
                                  const struct
rpc_program *,
u32);

include/linux/sunrpc/sched.h +0 -0
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
    #endif
           unsigned char           tk_priority : 2,/* Task

priority
*/
                                  tk_garb_retry : 2,
-                               tk_cred_retry : 2, -                               tk_rebind_retry : 2; +                               tk_cred_retry : 2; +       unsigned char           tk_rebind_retry;
   };

   typedef void                   (*rpc_action)(struct
rpc_task *);

net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task
 *task)
                   if (task->tk_rebind_retry == 0)
                           break;
                   task->tk_rebind_retry--;
-               rpc_delay(task, 3*HZ);
+               rpc_delay(task, RPC_CLNT_REBIND_DELAY *
 HZ);
                   goto retry_timeout;

          case -ENOBUFS:
                  rpc_delay(task, HZ >> 2);

net/sunrpc/sched.c +0 -0
index be587a308e05..5c18a35752aa 100644
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct

rpc_task
*task)

Quoted 8 lines

          /* Initialize retry counters */
          task->tk_garb_retry = 2;
          task->tk_cred_retry = 2;
-       task->tk_rebind_retry = 2;
+       task->tk_rebind_retry =
RPC_CLNT_REBIND_MAX_TIMEOUT /
+
RPC_CLNT_REBIND_DELAY;

Quoted 8 lines

Why not just implement an exponential back off? If the server
is
slow
to come up, then pounding the rpcbind service every 3 seconds
isn't
going to help.

Quoted 9 lines

Exponential backoff has the disadvantage that once we've gotten
to the longer retry times, it's easy for a client to wait quite
some time before it notices the rebind service is available.

But I have to wonder if retrying every 3 seconds is useful if the request is going over TCP.

Quoted 9 lines

The problem isn't reliability: this is handling a case where we _are_
getting a reply from the server, just not one we wanted. EACCES here
means that the rpcbind server didn't return a valid entry for the
service we requested, presumably because the service hasn't been
registered yet.

That's correct.

Quoted 1 line

So yes, an exponential back off is appropriate here.

Quoted 8 lines
Quoted 7 lines

I think Chuck's point is still valid. It makes the client a little
more
responsive; does not have to wait that long, and the overhead of a
RPC
request every 3 seconds is not that significant.

It is when you do it 30 times before giving up.

This is true for a dead NFS server, we save 25 RPC calls in a period of ~90 seconds.

Quoted 7 lines

In the case of a head failover and the NFS service on the takeover head can take up to 15 seconds to restart to pick up the new exports from the fail head, then the client can potentially wait up to ~20 seconds, after the NFS server is ready, to resume normal operation. IMHO, it's a 'long' time and I'd prioritize speed over saving some overhead.

Quoted 8 lines

task->tk_rebind_retry is _only_ changed if the rpcbind server is up and running, and returns an empty reply because the service we're looking up isn't registered.
task->tk_rebind_retry isn't changed on any request timeout. It isn't changed on any connection failure. It isn't changed by any other code path in the RPC client.

So none of this applies to the case of a dead server.

Sorry if I'm not clear. What I meant by a dead server is a dead NFS server and not rpcbind service. So in this case we get EACCES from rpcbind and we retry.

Quoted 5 lines

It applies to the case of a live server, where rpcbind is running and accessible to the client and where, for some reason or another, it is taking an exceptionally long time to register the service we are looking up the port for (either NLM or NFSv3).

Yes, this is the problem that I'm facing.

Quoted 4 lines

So where are you seeing this process take 90 seconds? Why do we need to wait for that long before we can finally conclude that the particular service in question is not going to come back up?

90 secs wait is for when the NFS server never come up and we keep getting EACCES from rpcbind for this whole time.

-Dai

#12 lore raw
Quoted 36 lines · On Mon, 2023-04-17 at 18:04 -0700, [email protected] wrote:
Quoted 11 lines · Trond Myklebust

task->tk_rebind_retry is _only_ changed if the rpcbind server is up and
running, and returns an empty reply because the service we're looking
up isn't registered.
task->tk_rebind_retry isn't changed on any request timeout. It isn't
changed on any connection failure. It isn't changed by any other code
path in the RPC client.

Quoted 1 line

So none of this applies to the case of a dead server.

Sorry if I'm not clear. What I meant by a dead server is a dead NFS server and not rpcbind service. So in this case we get EACCES from rpcbind and we retry.

Quoted 7 lines

It applies to the case of a live server, where rpcbind is running and
accessible to the client and where, for some reason or another, it is
taking an exceptionally long time to register the service we are looking up the port for (either NLM or NFSv3).

Yes, this is the problem that I'm facing.

Quoted 6 lines

So where are you seeing this process take 90 seconds? Why do we need to
wait for that long before we can finally conclude that the particular
service in question is not going to come back up?

Quoted 4 lines

90 secs wait is for when the NFS server never come up and we keep getting
EACCES from rpcbind for this whole time.

OK, so the 90s is completely arbitrary then, and was only chosen because it fits your particular server?

To me, that appears to invalidate the entire premise of commit 0b760113a3a1 that we can rely on rpcbind to tell us if the service is present or not.
In that case, I'd rather rip out the task->tk_rebind_retry counter, and just rely on standard hard/soft task semantics.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
author reviewer
#13 lore raw
Quoted 35 lines · Trond Myklebust
Quoted 25 lines · On Mon, 2023-04-17 at 18:04 -0700, [email protected] wrote:
Quoted 11 lines · Trond Myklebust

task->tk_rebind_retry is _only_ changed if the rpcbind server is up and
running, and returns an empty reply because the service we're looking
up isn't registered.
task->tk_rebind_retry isn't changed on any request timeout. It isn't
changed on any connection failure. It isn't changed by any other code
path in the RPC client.

Quoted 1 line

So none of this applies to the case of a dead server.

Sorry if I'm not clear. What I meant by a dead server is a dead NFS server and not rpcbind service. So in this case we get EACCES from rpcbind and we retry.

Quoted 6 lines

It applies to the case of a live server, where rpcbind is running and
accessible to the client and where, for some reason or another, it is
taking an exceptionally long time to register the service we are looking up the port for (either NLM or NFSv3).

Yes, this is the problem that I'm facing.

Quoted 9 lines
Quoted 5 lines

So where are you seeing this process take 90 seconds? Why do we need to
wait for that long before we can finally conclude that the particular
service in question is not going to come back up?

90 secs wait is for when the NFS server never come up and we keep getting
EACCES from rpcbind for this whole time.

Quoted 8 lines

OK, so the 90s is completely arbitrary then, and was only chosen because it fits your particular server?

To me, that appears to invalidate the entire premise of commit 0b760113a3a1 that we can rely on rpcbind to tell us if the service is present or not.
In that case, I'd rather rip out the task->tk_rebind_retry counter, and just rely on standard hard/soft task semantics.

Okay, I will resend the patch that relies on standard hard/soft task semantics.

Thanks,
-Dai

#14 lore raw
In reply to Trond Myklebust
Quoted 125 lines · Trond Myklebust
Quoted 112 lines · Chuck Lever III
Quoted 103 lines

On Apr 17, 2023, at 4:49 PM, Trond Myklebust
<[email protected]> wrote:

Quoted 97 lines · Dai Ngo

Currently call_bind_status places a hard limit of 9 seconds for retries
on EACCES error. This limit was done to prevent the RPC request from
being retried forever if the remote server has problem and never comes
up

However this 9 seconds timeout is too short, comparing to other RPC
timeouts which are generally in minutes. This causes intermittent failure
with EIO on the client side when there are lots of NLM activity and
the
NFS server is restarted.

Instead of removing the max timeout for retry and relying on the RPC
timeout mechanism to handle the retry, which can lead to the RPC being
retried forever if the remote NLM service fails to come up. This patch
simply increases the max timeout of call_bind_status from 9 to 90 seconds
which should allow enough time for NLM to register after a restart,
and
not retrying forever if there is real problem with the remote system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM unlock

requests")

---
 include/linux/sunrpc/clnt.h  | 3 +++
 include/linux/sunrpc/sched.h | 4 ++--
 net/sunrpc/clnt.c            | 2 +-
 net/sunrpc/sched.c           | 3 ++-
 4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +3 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
 #define RPC_CLNT_CREATE_REUSEPORT      (1UL << 11)
 #define RPC_CLNT_CREATE_CONNECTED      (1UL << 12)
 
+#define        RPC_CLNT_REBIND_DELAY           3
+#define        RPC_CLNT_REBIND_MAX_TIMEOUT     90
+
 struct rpc_clnt *rpc_create(struct rpc_create_args *args);
 struct rpc_clnt        *rpc_bind_new_program(struct rpc_clnt *,
                                const struct rpc_program *, u32);
include/linux/sunrpc/sched.h +2 -2
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
 #endif
        unsigned char           tk_priority : 2,/* Task priority
 */
                                tk_garb_retry : 2,
-                               tk_cred_retry : 2,
-                               tk_rebind_retry : 2;
+                               tk_cred_retry : 2;
+       unsigned char           tk_rebind_retry;
 };
 

typedef void (*rpc_action)(struct rpc_task *);

net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
                if (task->tk_rebind_retry == 0)
                        break;
                task->tk_rebind_retry--;
-               rpc_delay(task, 3*HZ);
+               rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
                goto retry_timeout;
        case -ENOBUFS:
                rpc_delay(task, HZ >> 2);
net/sunrpc/sched.c +0 -0
index be587a308e05..5c18a35752aa 100644
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task
 *task)
        /* Initialize retry counters */
        task->tk_garb_retry = 2;
        task->tk_cred_retry = 2;
Quoted 8 lines

net/sunrpc/sched.c rpc_init_task_statistics(struct rpc_task *task) · line 821

Quoted 3 lines
net/sunrpc/sched.c +2 -1
-       task->tk_rebind_retry = 2;
+       task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
+                                       RPC_CLNT_REBIND_DELAY;

Why not just implement an exponential back off? If the server is slow
to come up, then pounding the rpcbind service every 3 seconds isn't going to help.

Quoted 7 lines

Exponential backoff has the disadvantage that once we've gotten to the longer retry times, it's easy for a client to wait quite some time before it notices the rebind service is available.

But I have to wonder if retrying every 3 seconds is useful if the request is going over TCP.

Quoted 7 lines

The problem isn't reliability: this is handling a case where we _are_ getting a reply from the server, just not one we wanted. EACCES here means that the rpcbind server didn't return a valid entry for the service we requested, presumably because the service hasn't been registered yet.

So yes, an exponential back off is appropriate here.

OK, rpcbind is responding, but the registered service is not ready yet.

But you've missed my point: exponential backoff is not desirable if we want clients to recover quickly from a service restart.

If we keep the longest retry time short enough to keep recovery responsive, it won't be all that much longer than 3 seconds. Say, it might be 10 seconds, or 15 seconds. I don't see any value in complicating the client's logic for that. Just make RPC_CLNT_REBIND_DELAY longer.

--
Chuck Lever

#15 lore raw
Quoted 156 lines · Chuck Lever III
Quoted 151 lines

On Apr 17, 2023, at 5:48 PM, Trond Myklebust
<[email protected]> wrote:

Quoted 129 lines · Chuck Lever III
Quoted 120 lines

On Apr 17, 2023, at 4:49 PM, Trond Myklebust
<[email protected]> wrote:

Quoted 114 lines · Dai Ngo

Currently call_bind_status places a hard limit of 9 seconds for
retries
on EACCES error. This limit was done to prevent the RPC request
from
being retried forever if the remote server has problem and never
comes
up

However this 9 seconds timeout is too short, comparing to other
RPC
timeouts which are generally in minutes. This causes intermittent
failure
with EIO on the client side when there are lots of NLM activity
and
the
NFS server is restarted.

Instead of removing the max timeout for retry and relying on the
RPC
timeout mechanism to handle the retry, which can lead to the RPC
being
retried forever if the remote NLM service fails to come up. This
patch
simply increases the max timeout of call_bind_status from 9 to 90
seconds
which should allow enough time for NLM to register after a restart,
and
not retrying forever if there is real problem with the remote system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM unlock

requests")

---
 include/linux/sunrpc/clnt.h  | 3 +++
 include/linux/sunrpc/sched.h | 4 ++--
 net/sunrpc/clnt.c            | 2 +-
 net/sunrpc/sched.c           | 3 ++-

 4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +0 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
  #define RPC_CLNT_CREATE_REUSEPORT      (1UL << 11)
  #define RPC_CLNT_CREATE_CONNECTED      (1UL << 12)

include/linux/sunrpc/clnt.h +3 -0
+#define        RPC_CLNT_REBIND_DELAY           3
+#define        RPC_CLNT_REBIND_MAX_TIMEOUT     90
+

 struct rpc_clnt *rpc_create(struct rpc_create_args *args); struct rpc_clnt        *rpc_bind_new_program(struct rpc_clnt *,
                                const struct rpc_program *, u32);

include/linux/sunrpc/sched.h +0 -0
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
  #endif

        unsigned char           tk_priority : 2,/* Task priority
*/
                                tk_garb_retry : 2, -                               tk_cred_retry : 2, -                               tk_rebind_retry : 2; +                               tk_cred_retry : 2; +       unsigned char           tk_rebind_retry;
 };

 typedef void                   (*rpc_action)(struct rpc_task *);

net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
                 if (task->tk_rebind_retry == 0)
                         break;
                 task->tk_rebind_retry--;
-               rpc_delay(task, 3*HZ);
+               rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
                 goto retry_timeout;

        case -ENOBUFS:
                rpc_delay(task, HZ >> 2);

net/sunrpc/sched.c +2 -1
index be587a308e05..5c18a35752aa 100644
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task
 *task)
         /* Initialize retry counters */
         task->tk_garb_retry = 2;
         task->tk_cred_retry = 2;
-       task->tk_rebind_retry = 2;
+       task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
+                                      
Quoted 8 lines

net/sunrpc/sched.c rpc_init_task_statistics(struct rpc_task · line 823

Quoted 1 line
net/sunrpc/sched.c +0 -0
 RPC_CLNT_REBIND_DELAY;

Why not just implement an exponential back off? If the server is
slow
to come up, then pounding the rpcbind service every 3 seconds isn't
going to help.

Quoted 7 lines

Exponential backoff has the disadvantage that once we've gotten to the longer retry times, it's easy for a client to wait quite some time before it notices the rebind service is available.

But I have to wonder if retrying every 3 seconds is useful if the request is going over TCP.

The problem isn't reliability: this is handling a case where we _are_
getting a reply from the server, just not one we wanted. EACCES here
means that the rpcbind server didn't return a valid entry for the service we requested, presumably because the service hasn't been registered yet.

Quoted 1 line

So yes, an exponential back off is appropriate here.

OK, rpcbind is responding, but the registered service is not ready yet.

Quoted 9 lines

But you've missed my point: exponential backoff is not desirable if we want clients to recover quickly from a service restart.

If we keep the longest retry time short enough to keep recovery responsive, it won't be all that much longer than 3 seconds. Say, it might be 10 seconds, or 15 seconds. I don't see any value in complicating the client's logic for that. Just make RPC_CLNT_REBIND_DELAY longer.

What's so complicated about exponential back off?

delay = RPC_CLNT_REBIND_DELAY * (1U << nretries);

...and then limit nretries to take values between 0 and 4. That gives you a 93 second total wait, with the largest wait between retries being 48 seconds.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#16 lore raw
Quoted 164 lines · Trond Myklebust
Quoted 159 lines · Chuck Lever III
Quoted 151 lines

On Apr 17, 2023, at 5:48 PM, Trond Myklebust
<[email protected]> wrote:

Quoted 129 lines · Chuck Lever III
Quoted 120 lines

On Apr 17, 2023, at 4:49 PM, Trond Myklebust
<[email protected]> wrote:

Quoted 114 lines · Dai Ngo

Currently call_bind_status places a hard limit of 9 seconds for
retries
on EACCES error. This limit was done to prevent the RPC request
from
being retried forever if the remote server has problem and never
comes
up

However this 9 seconds timeout is too short, comparing to other
RPC
timeouts which are generally in minutes. This causes intermittent
failure
with EIO on the client side when there are lots of NLM activity
and
the
NFS server is restarted.

Instead of removing the max timeout for retry and relying on the
RPC
timeout mechanism to handle the retry, which can lead to the RPC
being
retried forever if the remote NLM service fails to come up. This
patch
simply increases the max timeout of call_bind_status from 9 to 90
seconds
which should allow enough time for NLM to register after a restart,
and
not retrying forever if there is real problem with the remote system.

Fixes 0b760113a3a1 ("NLM: Don't hang forever on NLM unlock

requests")

---
 include/linux/sunrpc/clnt.h  | 3 +++
 include/linux/sunrpc/sched.h | 4 ++--
 net/sunrpc/clnt.c            | 2 +-
 net/sunrpc/sched.c           | 3 ++-
 4 files changed, 8 insertions(+), 4 deletions(-)
include/linux/sunrpc/clnt.h +3 -0
index 770ef2cb5775..81afc5ea2665 100644
@@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
 #define RPC_CLNT_CREATE_REUSEPORT      (1UL << 11)
 #define RPC_CLNT_CREATE_CONNECTED      (1UL << 12)
 
+#define        RPC_CLNT_REBIND_DELAY           3
+#define        RPC_CLNT_REBIND_MAX_TIMEOUT     90
+
 struct rpc_clnt *rpc_create(struct rpc_create_args *args);
 struct rpc_clnt        *rpc_bind_new_program(struct rpc_clnt
 *,

const struct rpc_program *,
u32);

include/linux/sunrpc/sched.h +0 -0
index b8ca3ecaf8d7..e9dc142f10bb 100644
@@ -90,8 +90,8 @@ struct rpc_task {
 #endif
        unsigned char           tk_priority : 2,/* Task

priority
*/

                                tk_garb_retry : 2,
include/linux/sunrpc/sched.h +2 -2
-                               tk_cred_retry : 2,
-                               tk_rebind_retry : 2;
+                               tk_cred_retry : 2;
+       unsigned char           tk_rebind_retry;
 };
 
 typedef void                   (*rpc_action)(struct rpc_task

*);

net/sunrpc/clnt.c +1 -1
index fd7e1c630493..222578af6b01 100644
@@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
                if (task->tk_rebind_retry == 0)
                        break;
                task->tk_rebind_retry--;
-               rpc_delay(task, 3*HZ);
+               rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
                goto retry_timeout;
        case -ENOBUFS:
                rpc_delay(task, HZ >> 2);
net/sunrpc/sched.c +2 -1
index be587a308e05..5c18a35752aa 100644
@@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task
 *task)
        /* Initialize retry counters */
        task->tk_garb_retry = 2;
        task->tk_cred_retry = 2;
-       task->tk_rebind_retry = 2;
+       task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
+                                      
Quoted 8 lines

net/sunrpc/sched.c rpc_init_task_statistics(struct rpc_task · line 823

Quoted 1 line
net/sunrpc/sched.c +0 -0
 RPC_CLNT_REBIND_DELAY;

Why not just implement an exponential back off? If the server is
slow
to come up, then pounding the rpcbind service every 3 seconds isn't
going to help.

Quoted 7 lines

Exponential backoff has the disadvantage that once we've gotten to the longer retry times, it's easy for a client to wait quite some time before it notices the rebind service is available.

But I have to wonder if retrying every 3 seconds is useful if the request is going over TCP.

The problem isn't reliability: this is handling a case where we _are_
getting a reply from the server, just not one we wanted. EACCES here
means that the rpcbind server didn't return a valid entry for the service we requested, presumably because the service hasn't been registered yet.

Quoted 1 line

So yes, an exponential back off is appropriate here.

OK, rpcbind is responding, but the registered service is not ready yet.

But you've missed my point: exponential backoff is not desirable if we want clients to recover quickly from a service restart.

Quoted 8 lines
Quoted 6 lines

If we keep the longest retry time short enough to keep recovery responsive, it won't be all that much longer than 3 seconds. Say, it might be 10 seconds, or 15 seconds. I don't see any value in complicating the client's logic for that. Just make RPC_CLNT_REBIND_DELAY longer.

What's so complicated about exponential back off?

I didn't claim that exponential backoff is /impossibly/ complicated. My point is that it is /unnecessarily/ complicated.

That extra bit of logic there doesn't buy you anything... and the worst case behavior of exp. backoff is noticeably worse than a fixed backoff, as Dai has pointed out twice already.

Quoted 5 lines

delay = RPC_CLNT_REBIND_DELAY * (1U << nretries);

...and then limit nretries to take values between 0 and 4. That gives you a 93 second total wait, with the largest wait between retries being 48 seconds.

48 seconds is actually quite a long time. I know some customers who would find that amount of recovery time to be unacceptable.

This is not a hill I care to die on, however. Dai should implement whatever you prefer. Thanks for your input.

--
Chuck Lever