Make RPCRDMA_MAX_RECV_BATCH configurable.
Subsystems (4)
PATCH 1/1 Make RPCRDMA_MAX_RECV_BATCH configurable.
Bumped up rpcrdma_max_recv_batch to 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.
--- net/sunrpc/xprtrdma/frwr_ops.c | 2 +- net/sunrpc/xprtrdma/module.c | 6 ++++++ net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 +- net/sunrpc/xprtrdma/verbs.c | 2 +- net/sunrpc/xprtrdma/xprt_rdma.h | 4 +--- 5 files changed, 10 insertions(+), 6 deletions(-)
net/sunrpc/xprtrdma/frwr_ops.c +1 -1
@@ -246,7 +246,7 @@ int frwr_query_device(struct rpcrdma_ep *ep, const struct ib_device *device) ep->re_attr.cap.max_send_wr +=1 ;/* for ib_drain_sq */ ep->re_attr.cap.max_recv_wr = ep->re_max_requests; ep->re_attr.cap.max_recv_wr +=RPCRDMA_BACKWARD_WRS ; - ep->re_attr.cap.max_recv_wr +=RPCRDMA_MAX_RECV_BATCH ; + ep->re_attr.cap.max_recv_wr += rpcrdma_max_recv_batch; ep->re_attr.cap.max_recv_wr +=1 ;/* for ib_drain_rq */ ep->re_max_rdma_segs =
net/sunrpc/xprtrdma/module.c +6 -0
@@ -27,6 +27,12 @@ MODULE_ALIAS("svcrdma");MODULE_ALIAS ("xprtrdma" );MODULE_ALIAS ("rpcrdma6" ); +unsigned int rpcrdma_max_recv_batch =64 ; +module_param_named (max_recv_batch, rpcrdma_max_recv_batch, uint,0644 ); +MODULE_PARM_DESC (max_recv_batch, +"Maximum number of Receive WRs to post in a batch " +"(default: 64, set to 0 to disable batching)" ); + static void __exit rpc_rdma_cleanup(void) {xprt_rdma_cleanup ();
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -1
@@ -440,7 +440,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt) newxprt->sc_max_req_size = svcrdma_max_req_size; newxprt->sc_max_requests = svcrdma_max_requests; newxprt->sc_max_bc_requests = svcrdma_max_bc_requests; - newxprt->sc_recv_batch =RPCRDMA_MAX_RECV_BATCH ; + newxprt->sc_recv_batch = rpcrdma_max_recv_batch; newxprt->sc_fc_credits =cpu_to_be32 (newxprt->sc_max_requests);/* Qualify the transport's resource defaults with the
net/sunrpc/xprtrdma/verbs.c +1 -1
@@ -1359,7 +1359,7 @@ void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed)if (likely (ep->re_receive_count > needed)) goto out; needed -= ep->re_receive_count; - needed +=RPCRDMA_MAX_RECV_BATCH ; + needed += rpcrdma_max_recv_batch;if (atomic_inc_return (&ep->re_receiving) >1 ) goto out;
net/sunrpc/xprtrdma/xprt_rdma.h +1 -3
@@ -216,9 +216,7 @@ struct rpcrdma_rep {* * Setting this to zero disables Receive post batching. */ -enum { -RPCRDMA_MAX_RECV_BATCH =7 , -}; +extern unsigned int rpcrdma_max_recv_batch;/* struct rpcrdma_sendctx - DMA mapped SGEs to unmap after Send completes */
Signature · 1 line
2.43.7
Quoted 3 lines · Gaurav Gangalwar
Bumped up
rpcrdma_max_recv_batchto 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.
Hi Gaurav -
Adding an administrative setting is generally a last resort. First, we want a full root-cause analysis to understand the symptoms you are trying to address.
Quoted 80 lines
Signed-off-by Gaurav Gangalwar <[email protected]>--- net/sunrpc/xprtrdma/frwr_ops.c | 2 +- net/sunrpc/xprtrdma/module.c | 6 ++++++ net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 +- net/sunrpc/xprtrdma/verbs.c | 2 +- net/sunrpc/xprtrdma/xprt_rdma.h | 4 +--- 5 files changed, 10 insertions(+), 6 deletions(-)net/sunrpc/xprtrdma/frwr_ops.c +1
-1@@ -246,7 +246,7 @@ int frwr_query_device(struct rpcrdma_ep *ep, const struct ib_device *device) ep->re_attr.cap.max_send_wr +=1 ;/* for ib_drain_sq */ ep->re_attr.cap.max_recv_wr = ep->re_max_requests; ep->re_attr.cap.max_recv_wr +=RPCRDMA_BACKWARD_WRS ; - ep->re_attr.cap.max_recv_wr +=RPCRDMA_MAX_RECV_BATCH ; + ep->re_attr.cap.max_recv_wr += rpcrdma_max_recv_batch; ep->re_attr.cap.max_recv_wr +=1 ;/* for ib_drain_rq */ ep->re_max_rdma_segs =net/sunrpc/xprtrdma/module.c +6
-0@@ -27,6 +27,12 @@ MODULE_ALIAS("svcrdma");MODULE_ALIAS ("xprtrdma" );MODULE_ALIAS ("rpcrdma6" ); +unsigned int rpcrdma_max_recv_batch =64 ; +module_param_named (max_recv_batch, rpcrdma_max_recv_batch, uint,0644 ); +MODULE_PARM_DESC (max_recv_batch, +"Maximum number of Receive WRs to post in a batch " +"(default: 64, set to 0 to disable batching)" ); + static void __exit rpc_rdma_cleanup(void) {xprt_rdma_cleanup ();net/sunrpc/xprtrdma/svc_rdma_transport.c +1
-1@@ -440,7 +440,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt) newxprt->sc_max_req_size = svcrdma_max_req_size; newxprt->sc_max_requests = svcrdma_max_requests; newxprt->sc_max_bc_requests = svcrdma_max_bc_requests; - newxprt->sc_recv_batch =RPCRDMA_MAX_RECV_BATCH ; + newxprt->sc_recv_batch = rpcrdma_max_recv_batch; newxprt->sc_fc_credits =cpu_to_be32 (newxprt->sc_max_requests);/* Qualify the transport's resource defaults with the net/sunrpc/xprtrdma/verbs.c +1
-1@@ -1359,7 +1359,7 @@ void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed)if (likely (ep->re_receive_count > needed)) goto out; needed -= ep->re_receive_count; - needed +=RPCRDMA_MAX_RECV_BATCH ; + needed += rpcrdma_max_recv_batch;if (atomic_inc_return (&ep->re_receiving) >1 ) goto out;net/sunrpc/xprtrdma/xprt_rdma.h +1
-3@@ -216,9 +216,7 @@ struct rpcrdma_rep {* * Setting this to zero disables Receive post batching. */ -enum { -RPCRDMA_MAX_RECV_BATCH =7 , -}; +extern unsigned int rpcrdma_max_recv_batch;/* struct rpcrdma_sendctx - DMA mapped SGEs to unmap after Send completes */
Signature · 1 line
Chuck Lever
Quoted 3 lines · Gaurav Gangalwar
Bumped up
rpcrdma_max_recv_batchto 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.
[ Resend with correct NFSD reviewer email addresses and linux-rdma@ ]
Hi Gaurav -
Adding an administrative setting is generally a last resort. First, we want a full root-cause analysis to understand the symptoms you are trying to address. Do you have an RCA or a simple reproducer to share with us?
Quoted 80 lines
Signed-off-by Gaurav Gangalwar <[email protected]>--- net/sunrpc/xprtrdma/frwr_ops.c | 2 +- net/sunrpc/xprtrdma/module.c | 6 ++++++ net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 +- net/sunrpc/xprtrdma/verbs.c | 2 +- net/sunrpc/xprtrdma/xprt_rdma.h | 4 +--- 5 files changed, 10 insertions(+), 6 deletions(-)net/sunrpc/xprtrdma/frwr_ops.c +1
-1@@ -246,7 +246,7 @@ int frwr_query_device(struct rpcrdma_ep *ep, const struct ib_device *device) ep->re_attr.cap.max_send_wr +=1 ;/* for ib_drain_sq */ ep->re_attr.cap.max_recv_wr = ep->re_max_requests; ep->re_attr.cap.max_recv_wr +=RPCRDMA_BACKWARD_WRS ; - ep->re_attr.cap.max_recv_wr +=RPCRDMA_MAX_RECV_BATCH ; + ep->re_attr.cap.max_recv_wr += rpcrdma_max_recv_batch; ep->re_attr.cap.max_recv_wr +=1 ;/* for ib_drain_rq */ ep->re_max_rdma_segs =net/sunrpc/xprtrdma/module.c +6
-0@@ -27,6 +27,12 @@ MODULE_ALIAS("svcrdma");MODULE_ALIAS ("xprtrdma" );MODULE_ALIAS ("rpcrdma6" ); +unsigned int rpcrdma_max_recv_batch =64 ; +module_param_named (max_recv_batch, rpcrdma_max_recv_batch, uint,0644 ); +MODULE_PARM_DESC (max_recv_batch, +"Maximum number of Receive WRs to post in a batch " +"(default: 64, set to 0 to disable batching)" ); + static void __exit rpc_rdma_cleanup(void) {xprt_rdma_cleanup ();net/sunrpc/xprtrdma/svc_rdma_transport.c +1
-1@@ -440,7 +440,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt) newxprt->sc_max_req_size = svcrdma_max_req_size; newxprt->sc_max_requests = svcrdma_max_requests; newxprt->sc_max_bc_requests = svcrdma_max_bc_requests; - newxprt->sc_recv_batch =RPCRDMA_MAX_RECV_BATCH ; + newxprt->sc_recv_batch = rpcrdma_max_recv_batch; newxprt->sc_fc_credits =cpu_to_be32 (newxprt->sc_max_requests);/* Qualify the transport's resource defaults with the net/sunrpc/xprtrdma/verbs.c +1
-1@@ -1359,7 +1359,7 @@ void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed)if (likely (ep->re_receive_count > needed)) goto out; needed -= ep->re_receive_count; - needed +=RPCRDMA_MAX_RECV_BATCH ; + needed += rpcrdma_max_recv_batch;if (atomic_inc_return (&ep->re_receiving) >1 ) goto out;net/sunrpc/xprtrdma/xprt_rdma.h +1
-3@@ -216,9 +216,7 @@ struct rpcrdma_rep {* * Setting this to zero disables Receive post batching. */ -enum { -RPCRDMA_MAX_RECV_BATCH =7 , -}; +extern unsigned int rpcrdma_max_recv_batch;/* struct rpcrdma_sendctx - DMA mapped SGEs to unmap after Send completes */
Signature · 1 line
Chuck Lever
Quoted 6 lines · Chuck Lever
Quoted 3 lines · Gaurav Gangalwar
Bumped up
rpcrdma_max_recv_batchto 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.
Quoted 8 lines
[ Resend with correct NFSD reviewer email addresses and linux-rdma@ ]
Hi Gaurav -
Adding an administrative setting is generally a last resort. First, we want a full root-cause analysis to understand the symptoms you are trying to address. Do you have an RCA or a simple reproducer to share with us?
Issue found while testing fio workload over RDMA
Client: Ubuntu 24.04
Server: Ganesha NFS server
We have seen intermittent hung on client with buffered IO workload at large scale with around 30 RDMA connections, client was under memory pressure.
Ganesha log shows
10/11/2025 16:39:12Z : ntnx-10-57-210-224-a-fsvm 1309416[none] [0x7f49a6c3fe80] rpc :TIRPC :EVENT :rpc_rdma_cq_event_handler() cq completion status: RNR retry counter exceeded (13) rdma_xprt state 5 opcode 2 cbc 0x7f4996688000 inline 1
Which points to lack of posted recv buffers on client. Once we increased rpcrdma_max_recv_batch to 64, issue was resolved.
Quoted 86 lines
Quoted 72 lines
Signed-off-by Gaurav Gangalwar <[email protected]>--- net/sunrpc/xprtrdma/frwr_ops.c | 2 +- net/sunrpc/xprtrdma/module.c | 6 ++++++ net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 +- net/sunrpc/xprtrdma/verbs.c | 2 +- net/sunrpc/xprtrdma/xprt_rdma.h | 4 +--- 5 files changed, 10 insertions(+), 6 deletions(-)net/sunrpc/xprtrdma/frwr_ops.c +1
-1@@ -246,7 +246,7 @@ int frwr_query_device(struct rpcrdma_ep *ep, const struct ib_device *device) ep->re_attr.cap.max_send_wr +=1 ;/* for ib_drain_sq */ ep->re_attr.cap.max_recv_wr = ep->re_max_requests; ep->re_attr.cap.max_recv_wr +=RPCRDMA_BACKWARD_WRS ; - ep->re_attr.cap.max_recv_wr +=RPCRDMA_MAX_RECV_BATCH ; + ep->re_attr.cap.max_recv_wr += rpcrdma_max_recv_batch; ep->re_attr.cap.max_recv_wr +=1 ;/* for ib_drain_rq */ ep->re_max_rdma_segs =net/sunrpc/xprtrdma/module.c +6
-0@@ -27,6 +27,12 @@ MODULE_ALIAS("svcrdma");MODULE_ALIAS ("xprtrdma" );MODULE_ALIAS ("rpcrdma6" ); +unsigned int rpcrdma_max_recv_batch =64 ; +module_param_named (max_recv_batch, rpcrdma_max_recv_batch, uint,0644 ); +MODULE_PARM_DESC (max_recv_batch, +"Maximum number of Receive WRs to post in a batch " +"(default: 64, set to 0 to disable batching)" ); + static void __exit rpc_rdma_cleanup(void) {xprt_rdma_cleanup ();net/sunrpc/xprtrdma/svc_rdma_transport.c +1
-1@@ -440,7 +440,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt) newxprt->sc_max_req_size = svcrdma_max_req_size; newxprt->sc_max_requests = svcrdma_max_requests; newxprt->sc_max_bc_requests = svcrdma_max_bc_requests; - newxprt->sc_recv_batch =RPCRDMA_MAX_RECV_BATCH ; + newxprt->sc_recv_batch = rpcrdma_max_recv_batch; newxprt->sc_fc_credits =cpu_to_be32 (newxprt->sc_max_requests);/* Qualify the transport's resource defaults with the net/sunrpc/xprtrdma/verbs.c +1
-1@@ -1359,7 +1359,7 @@ void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed)if (likely (ep->re_receive_count > needed)) goto out; needed -= ep->re_receive_count; - needed +=RPCRDMA_MAX_RECV_BATCH ; + needed += rpcrdma_max_recv_batch;if (atomic_inc_return (&ep->re_receiving) >1 ) goto out;net/sunrpc/xprtrdma/xprt_rdma.h +0
-0net/sunrpc/xprtrdma/xprt_rdma.h struct rpcrdma_rep { · line 222
Quoted 10 lines
net/sunrpc/xprtrdma/xprt_rdma.h +1
-3@@ -216,9 +216,7 @@ struct rpcrdma_rep {* * Setting this to zero disables Receive post batching. */ -enum { -RPCRDMA_MAX_RECV_BATCH =7 , -}; +extern unsigned int rpcrdma_max_recv_batch;/* struct rpcrdma_sendctx - DMA mapped SGEs to unmap after Send completes */ --
Chuck Lever
Quoted 24 lines · gaurav gangalwar
Quoted 6 lines · Chuck Lever
Quoted 3 lines · Gaurav Gangalwar
Bumped up
rpcrdma_max_recv_batchto 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.Quoted 8 lines
[ Resend with correct NFSD reviewer email addresses and linux-rdma@ ]
Hi Gaurav -
Adding an administrative setting is generally a last resort. First, we want a full root-cause analysis to understand the symptoms you are trying to address. Do you have an RCA or a simple reproducer to share with us?
Issue found while testing fio workload over RDMA
Client: Ubuntu 24.04
Server: Ganesha NFS server
We have seen intermittent hung on client with buffered IO workload at large scale with around 30 RDMA connections, client was under memory pressure.
Ganesha log shows
Quoted 7 lines
10/11/2025 16:39:12Z : ntnx-10-57-210-224-a-fsvm 1309416[none] [
0x7f49a6c3fe80] rpc :TIRPC :EVENT :rpc_rdma_cq_event_handler() cq completion status: RNR retry counter exceeded (13)rdma_xprtstate 5 opcode 2 cbc0x7f4996688000inline 1Which points to lack of posted recv buffers on client. Once we increased
rpcrdma_max_recv_batchto 64, issue was resolved.
That still doesn't convince me that increasing the receive batch count is a good fix, though it's certainly a workaround.
The client's RPC/RDMA code is supposed to track the number of Sends and keep the correct number of Receives on the Receive Queue. The goal of the implementation is to never encounter an RNR.
Therefore, if it's not doing that (and the RNR retries suggests that's the case) there is an actual bug somewhere. The extra batch Receives are an optimization, and should have no impact on correct operation.
If you can't reproduce this with the Linux NFS server, the place to start looking for misbehavior is NFS/Ganesha, as it is the newer NFS over RDMA implementation of the two servers. Maybe it's not handling credit accounting correctly, or perhaps it's putting more Sends on the wire than the credit limit allows.
Signature · 1 line
Chuck Lever
Quoted 45 lines · Chuck Lever
Quoted 24 lines · gaurav gangalwar
Quoted 6 lines · Chuck Lever
Quoted 3 lines · Gaurav Gangalwar
Bumped up
rpcrdma_max_recv_batchto 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.Quoted 8 lines
[ Resend with correct NFSD reviewer email addresses and linux-rdma@ ]
Hi Gaurav -
Adding an administrative setting is generally a last resort. First, we want a full root-cause analysis to understand the symptoms you are trying to address. Do you have an RCA or a simple reproducer to share with us?
Issue found while testing fio workload over RDMA
Client: Ubuntu 24.04
Server: Ganesha NFS server
We have seen intermittent hung on client with buffered IO workload at large scale with around 30 RDMA connections, client was under memory pressure.
Ganesha log showsQuoted 7 lines
10/11/2025 16:39:12Z : ntnx-10-57-210-224-a-fsvm 1309416[none] [
0x7f49a6c3fe80] rpc :TIRPC :EVENT :rpc_rdma_cq_event_handler() cq completion status: RNR retry counter exceeded (13)rdma_xprtstate 5 opcode 2 cbc0x7f4996688000inline 1Which points to lack of posted recv buffers on client. Once we increased
rpcrdma_max_recv_batchto 64, issue was resolved.That still doesn't convince me that increasing the receive batch count is a good fix, though it's certainly a workaround.
The client's RPC/RDMA code is supposed to track the number of Sends and keep the correct number of Receives on the Receive Queue. The goal of the implementation is to never encounter an RNR.
Therefore, if it's not doing that (and the RNR retries suggests that's the case) there is an actual bug somewhere. The extra batch Receives are an optimization, and should have no impact on correct operation.
Quoted 5 lines
If you can't reproduce this with the Linux NFS server, the place to start looking for misbehavior is NFS/Ganesha, as it is the newer NFS over RDMA implementation of the two servers. Maybe it's not handling credit accounting correctly, or perhaps it's putting more Sends on the wire than the credit limit allows.
Sure I will try to get more details.
Issue is specific to pNFS, for non pNFS shares we don't see this issue. Even for pNFS we see hung when number of ds connections are high (30 ds connections)
But this work around is definitely helping.
Quoted 4 lines
--
Chuck Lever
Tom Talpey
reviewerQuoted 30 lines · Chuck Lever
Quoted 29 lines · gaurav gangalwar
Quoted 6 lines · Chuck Lever
Quoted 3 lines · Gaurav Gangalwar
Bumped up
rpcrdma_max_recv_batchto 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.Quoted 8 lines
[ Resend with correct NFSD reviewer email addresses and linux-rdma@ ]
Hi Gaurav -
Adding an administrative setting is generally a last resort. First, we want a full root-cause analysis to understand the symptoms you are trying to address. Do you have an RCA or a simple reproducer to share with us?
Issue found while testing fio workload over RDMA
Client: Ubuntu 24.04
Server: Ganesha NFS server
We have seen intermittent hung on client with buffered IO workload at large scale with around 30 RDMA connections, client was under memory pressure.
Ganesha log shows10/11/2025 16:39:12Z : ntnx-10-57-210-224-a-fsvm 1309416[none] [
0x7f49a6c3fe80] rpc :TIRPC :EVENT :rpc_rdma_cq_event_handler() cq completion status: RNR retry counter exceeded (13)rdma_xprtstate 5 opcode 2 cbc0x7f4996688000inline 1
Quoted 5 lines
Quoted 2 lines
Which points to lack of posted recv buffers on client. Once we increased
rpcrdma_max_recv_batchto 64, issue was resolved.That still doesn't convince me that increasing the receive batch count is a good fix, though it's certainly a workaround.
It's not a workaround, this will fail on any RDMA provider that doesn't perform RNR retry, for example iWARP. But more importantly, RNR retry is unnecessary because the rpcrdma protocol implements a strict crediting exchange. A proper rpcrdma implementation will never trigger RNR.
This is almost certainly an rpcrdma protocol violation in the sender, which is failing to honor the credit limit granted by the receiving peer and is overrunning the peer's receive queue. A wireshark trace would prove it. Please do this research.
Tom.
Quoted 16 lines
The client's RPC/RDMA code is supposed to track the number of Sends and keep the correct number of Receives on the Receive Queue. The goal of the implementation is to never encounter an RNR.
Therefore, if it's not doing that (and the RNR retries suggests that's the case) there is an actual bug somewhere. The extra batch Receives are an optimization, and should have no impact on correct operation.
If you can't reproduce this with the Linux NFS server, the place to start looking for misbehavior is NFS/Ganesha, as it is the newer NFS over RDMA implementation of the two servers. Maybe it's not handling credit accounting correctly, or perhaps it's putting more Sends on the wire than the credit limit allows.
Quoted 43 lines · Tom Talpey
Quoted 30 lines · Chuck Lever
Quoted 29 lines · gaurav gangalwar
Quoted 6 lines · Chuck Lever
Quoted 3 lines · Gaurav Gangalwar
Bumped up
rpcrdma_max_recv_batchto 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.Quoted 8 lines
[ Resend with correct NFSD reviewer email addresses and linux-rdma@ ]
Hi Gaurav -
Adding an administrative setting is generally a last resort. First, we want a full root-cause analysis to understand the symptoms you are trying to address. Do you have an RCA or a simple reproducer to share with us?
Issue found while testing fio workload over RDMA
Client: Ubuntu 24.04
Server: Ganesha NFS server
We have seen intermittent hung on client with buffered IO workload at large scale with around 30 RDMA connections, client was under memory pressure.
Ganesha log shows10/11/2025 16:39:12Z : ntnx-10-57-210-224-a-fsvm 1309416[none] [
0x7f49a6c3fe80] rpc :TIRPC :EVENT :rpc_rdma_cq_event_handler() cq completion status: RNR retry counter exceeded (13)rdma_xprtstate 5 opcode 2 cbc0x7f4996688000inline 1Quoted 5 lines
Quoted 2 lines
Which points to lack of posted recv buffers on client. Once we increased
rpcrdma_max_recv_batchto 64, issue was resolved.That still doesn't convince me that increasing the receive batch count is a good fix, though it's certainly a workaround.
It's not a workaround, this will fail on any RDMA provider that doesn't perform RNR retry, for example iWARP. But more importantly, RNR retry is unnecessary because the rpcrdma protocol implements a strict crediting exchange. A proper rpcrdma implementation will never trigger RNR.
Quoted 25 lines
This is almost certainly an rpcrdma protocol violation in the sender, which is failing to honor the credit limit granted by the receiving peer and is overrunning the peer's receive queue. A wireshark trace would prove it. Please do this research.
Tom.
Quoted 16 lines
The client's RPC/RDMA code is supposed to track the number of Sends and keep the correct number of Receives on the Receive Queue. The goal of the implementation is to never encounter an RNR.
Therefore, if it's not doing that (and the RNR retries suggests that's the case) there is an actual bug somewhere. The extra batch Receives are an optimization, and should have no impact on correct operation.
If you can't reproduce this with the Linux NFS server, the place to start looking for misbehavior is NFS/Ganesha, as it is the newer NFS over RDMA implementation of the two servers. Maybe it's not handling credit accounting correctly, or perhaps it's putting more Sends on the wire than the credit limit allows.
Thanks for the review, I was going through server implementation for NFS Ganesha, we strictly honor read_chunks, write_chunks and reply_chunks, so the credit limit should be client driven only. Only in case of callbacks from server to client, as of now there is no credits check, we advertise server_credits in cb call similar to rb_bc_max_requests in linux NFS server, but don't check for client_credits from client in reply. We need to limit callbacks to MIN(server_credits, client_credits)
Regards,
Gaurav