Skip to contentSkip to conversation

SUNRPC: Fix backchannel reply, again

PATCH Superseded [email protected] 4 participants 4 messages Jun 14, 2024 – Jun 18, 2024

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

PATCH 1/1 SUNRPC: Fix backchannel reply, again

From: Chuck Lever <[email protected]>

I still see "RPC: Could not send backchannel reply error: -110" quite often, along with slow-running tests. Debugging shows that the backchannel is still stumbling when it has to queue a callback reply on a busy transport.

Note that every one of these timeouts causes a connection loss by virtue of the xprt_conditional_disconnect() call in that arm of call_cb_transmit_status().

I found that setting to_maxval is necessary to get the RPC timeout logic to behave whenever to_exponential is not set.

Fixes 57331a59ac0d ("NFSv4.1: Use the nfs_client's rpc timeouts for backchannel")
---
 net/sunrpc/svc.c | 1 +
 1 file changed, 1 insertion(+)
net/sunrpc/svc.c +1 -0
index 965a27806bfd..f4ddb2961042 100644
@@ -1643,6 +1643,7 @@ void svc_process_bc(struct rpc_rqst *req, struct svc_rqst *rqstp)
 		timeout.to_initval = req->rq_xprt->timeout->to_initval;
 		timeout.to_retries = req->rq_xprt->timeout->to_retries;
 	}
+	timeout.to_maxval = timeout.to_initval;
 	memcpy(&req->rq_snd_buf, &rqstp->rq_res, sizeof(req->rq_snd_buf));
 	task = rpc_run_bc_task(req, &timeout);
 
Signature · 1 line
2.45.1
#1 lore raw
Quoted 35 lines · On Fri, Jun 14, 2024 at 10:18:52AM -0400, [email protected] wrote:

From: Chuck Lever <[email protected]>

I still see "RPC: Could not send backchannel reply error: -110" quite often, along with slow-running tests. Debugging shows that the backchannel is still stumbling when it has to queue a callback reply on a busy transport.

Note that every one of these timeouts causes a connection loss by virtue of the xprt_conditional_disconnect() call in that arm of call_cb_transmit_status().

I found that setting to_maxval is necessary to get the RPC timeout logic to behave whenever to_exponential is not set.

Fixes 57331a59ac0d ("NFSv4.1: Use the nfs_client's rpc timeouts for backchannel")
---
 net/sunrpc/svc.c | 1 +
 1 file changed, 1 insertion(+)
net/sunrpc/svc.c +1 -0
index 965a27806bfd..f4ddb2961042 100644
@@ -1643,6 +1643,7 @@ void svc_process_bc(struct rpc_rqst *req, struct svc_rqst *rqstp)
 		timeout.to_initval = req->rq_xprt->timeout->to_initval;
 		timeout.to_retries = req->rq_xprt->timeout->to_retries;
 	}
+	timeout.to_maxval = timeout.to_initval;
 	memcpy(&req->rq_snd_buf, &rqstp->rq_res, sizeof(req->rq_snd_buf));
 	task = rpc_run_bc_task(req, &timeout);
 
Signature · 1 line
2.45.1

Hi - would love to see this in 6.10-rc. Is there a chance that could happen?

Signature · 1 line
Chuck Lever
#2 lore raw
Quoted 38 lines · Chuck Lever
Quoted 37 lines · On Fri, Jun 14, 2024 at 10:18:52AM -0400, [email protected] wrote:

From: Chuck Lever <[email protected]>

I still see "RPC: Could not send backchannel reply error: -110" quite often, along with slow-running tests. Debugging shows that the
backchannel is still stumbling when it has to queue a callback reply
on a busy transport.

Note that every one of these timeouts causes a connection loss by virtue of the xprt_conditional_disconnect() call in that arm of call_cb_transmit_status().

I found that setting to_maxval is necessary to get the RPC timeout logic to behave whenever to_exponential is not set.

Fixes 57331a59ac0d ("NFSv4.1: Use the nfs_client's rpc timeouts

for backchannel")

---
 net/sunrpc/svc.c | 1 +

 1 file changed, 1 insertion(+)
net/sunrpc/svc.c +0 -0
index 965a27806bfd..f4ddb2961042 100644
@@ -1643,6 +1643,7 @@ void svc_process_bc(struct rpc_rqst *req,
 struct svc_rqst *rqstp)

  timeout.to_initval = req->rq_xprt->timeout-

Quoted 1 line

to_initval;

  timeout.to_retries = req->rq_xprt->timeout-

Quoted 1 line

to_retries;

  }

net/sunrpc/svc.c +1 -0
+	timeout.to_maxval = timeout.to_initval;

  memcpy(&req->rq_snd_buf, &rqstp->rq_res, sizeof(req-

Quoted 1 line

rq_snd_buf));

Quoted 8 lines
Quoted 5 lines

  task = rpc_run_bc_task(req, &timeout);

Signature · 1 line
2.45.1

Hi - would love to see this in 6.10-rc. Is there a chance that could happen?

Hmm... Can we please also set the remaining fields in timeout to 0? Otherwise, we're still playing roulette with what actually ends up happening in xprt_calc_majortimeo(). If to_increment happens to be large enough, we could overflow and end up with a silly small timeout value on a retry.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
maintainer Reviewed-by on v1 1/1
#3 lore raw
In reply to [email protected]
Quoted 34 lines · On Fri, 2024-06-14 at 10:18 -0400, [email protected] wrote:

From: Chuck Lever <[email protected]>

I still see "RPC: Could not send backchannel reply error: -110" quite often, along with slow-running tests. Debugging shows that the backchannel is still stumbling when it has to queue a callback reply on a busy transport.

Note that every one of these timeouts causes a connection loss by virtue of the xprt_conditional_disconnect() call in that arm of call_cb_transmit_status().

I found that setting to_maxval is necessary to get the RPC timeout logic to behave whenever to_exponential is not set.

Fixes 57331a59ac0d ("NFSv4.1: Use the nfs_client's rpc timeouts for

backchannel")

---
 net/sunrpc/svc.c | 1 +

 1 file changed, 1 insertion(+)
net/sunrpc/svc.c +0 -0
index 965a27806bfd..f4ddb2961042 100644
@@ -1643,6 +1643,7 @@ void svc_process_bc(struct rpc_rqst *req,
 struct svc_rqst *rqstp)

  timeout.to_initval = req->rq_xprt->timeout-

Quoted 1 line

to_initval;

  timeout.to_retries = req->rq_xprt->timeout-

Quoted 1 line

to_retries;

  }

net/sunrpc/svc.c +1 -0
+	timeout.to_maxval = timeout.to_initval;

  memcpy(&req->rq_snd_buf, &rqstp->rq_res, sizeof(req-

Quoted 3 lines
Quoted 1 line

rq_snd_buf));

  task = rpc_run_bc_task(req, &timeout);