Skip to contentSkip to conversation

SUNRPC: Fix backchannel reply, again

PATCH In review [email protected] 3 participants 3 messages Jun 19, 2024 – Jun 20, 2024

Subsystems (3)
lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

PATCH v2 1/1 SUNRPC: Fix backchannel reply, again

From: Chuck Lever <[email protected]>

I still see "RPC: Could not send backchannel reply error: -110" quite often, along with slow-running tests. Debugging shows that the backchannel is still stumbling when it has to queue a callback reply on a busy transport.

Note that every one of these timeouts causes a connection loss by virtue of the xprt_conditional_disconnect() call in that arm of call_cb_transmit_status().

I found that setting to_maxval is necessary to get the RPC timeout logic to behave whenever to_exponential is not set.

Fixes 57331a59ac0d ("NFSv4.1: Use the nfs_client's rpc timeouts for backchannel")
---
 net/sunrpc/svc.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
net/sunrpc/svc.c +4 -1
index 965a27806bfd..e03f14024e47 100644
@@ -1588,9 +1588,11 @@ void svc_process(struct svc_rqst *rqstp)
  */
 void svc_process_bc(struct rpc_rqst *req, struct svc_rqst *rqstp)
 {
+	struct rpc_timeout timeout = {
+		.to_increment		= 0,
+	};
 	struct rpc_task *task;
 	int proc_error;
-	struct rpc_timeout timeout;
 
 	/* Build the svc_rqst used by the common processing routine */
 	rqstp->rq_xid = req->rq_xid;
@@ -1643,6 +1645,7 @@ void svc_process_bc(struct rpc_rqst *req, struct svc_rqst *rqstp)
 		timeout.to_initval = req->rq_xprt->timeout->to_initval;
 		timeout.to_retries = req->rq_xprt->timeout->to_retries;
 	}
+	timeout.to_maxval = timeout.to_initval;
 	memcpy(&req->rq_snd_buf, &rqstp->rq_res, sizeof(req->rq_snd_buf));
 	task = rpc_run_bc_task(req, &timeout);
 
Signature · 1 line
2.45.1
Reviewed-by on v2 1/1
#1 lore raw
Quoted 11 lines · On 19 Jun 2024, at 9:51, [email protected] wrote:

From: Chuck Lever <[email protected]>

I still see "RPC: Could not send backchannel reply error: -110" quite often, along with slow-running tests. Debugging shows that the backchannel is still stumbling when it has to queue a callback reply on a busy transport.

Note that every one of these timeouts causes a connection loss by virtue of the xprt_conditional_disconnect() call in that arm of call_cb_transmit_status().

Quoted 5 lines

I found that setting to_maxval is necessary to get the RPC timeout logic to behave whenever to_exponential is not set.

Fixes 57331a59ac0d ("NFSv4.1: Use the nfs_client's rpc timeouts for backchannel")

That makes sense - I guess we were getting some random stack value in there?

Ben

Quoted 31 lines
---
 net/sunrpc/svc.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
net/sunrpc/svc.c +4 -1
index 965a27806bfd..e03f14024e47 100644
@@ -1588,9 +1588,11 @@ void svc_process(struct svc_rqst *rqstp)
  */
 void svc_process_bc(struct rpc_rqst *req, struct svc_rqst *rqstp)
 {
+	struct rpc_timeout timeout = {
+		.to_increment		= 0,
+	};
 	struct rpc_task *task;
 	int proc_error;
-	struct rpc_timeout timeout;
 
 	/* Build the svc_rqst used by the common processing routine */
 	rqstp->rq_xid = req->rq_xid;
@@ -1643,6 +1645,7 @@ void svc_process_bc(struct rpc_rqst *req, struct svc_rqst *rqstp)
 		timeout.to_initval = req->rq_xprt->timeout->to_initval;
 		timeout.to_retries = req->rq_xprt->timeout->to_retries;
 	}
+	timeout.to_maxval = timeout.to_initval;
 	memcpy(&req->rq_snd_buf, &rqstp->rq_res, sizeof(req->rq_snd_buf));
 	task = rpc_run_bc_task(req, &timeout);
 
Signature · 1 line
2.45.1
#2 lore raw
Quoted 13 lines · Benjamin Coddington
Quoted 11 lines · On 19 Jun 2024, at 9:51, [email protected] wrote:

From: Chuck Lever <[email protected]>

I still see "RPC: Could not send backchannel reply error: -110" quite often, along with slow-running tests. Debugging shows that the backchannel is still stumbling when it has to queue a callback reply on a busy transport.

Note that every one of these timeouts causes a connection loss by virtue of the xprt_conditional_disconnect() call in that arm of call_cb_transmit_status().

Quoted 7 lines
Quoted 5 lines

I found that setting to_maxval is necessary to get the RPC timeout logic to behave whenever to_exponential is not set.

Fixes 57331a59ac0d ("NFSv4.1: Use the nfs_client's rpc timeouts for backchannel")

That makes sense - I guess we were getting some random stack value in there?

Hi Ben-

On my systems it was always zero (which is why v1 of this patch did not clear the other fields in @timeout before using it).

A zero to_maxval value results in the same timeout-on-sleep behavior as you saw before 57331a59ac0d was applied.

A random non-zero value will behave correctly as long as the transport is making forward progress, so we never noticed a problem.

Quoted 36 lines

Ben

Quoted 31 lines
---
 net/sunrpc/svc.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
net/sunrpc/svc.c +4 -1
index 965a27806bfd..e03f14024e47 100644
@@ -1588,9 +1588,11 @@ void svc_process(struct svc_rqst *rqstp)
  */
 void svc_process_bc(struct rpc_rqst *req, struct svc_rqst *rqstp)
 {
+	struct rpc_timeout timeout = {
+		.to_increment		= 0,
+	};
 	struct rpc_task *task;
 	int proc_error;
-	struct rpc_timeout timeout;
 
 	/* Build the svc_rqst used by the common processing routine */
 	rqstp->rq_xid = req->rq_xid;
@@ -1643,6 +1645,7 @@ void svc_process_bc(struct rpc_rqst *req, struct svc_rqst *rqstp)
 		timeout.to_initval = req->rq_xprt->timeout->to_initval;
 		timeout.to_retries = req->rq_xprt->timeout->to_retries;
 	}
+	timeout.to_maxval = timeout.to_initval;
 	memcpy(&req->rq_snd_buf, &rqstp->rq_res, sizeof(req->rq_snd_buf));
 	task = rpc_run_bc_task(req, &timeout);
 
Signature · 1 line
2.45.1
Signature · 1 line
Chuck Lever