Skip to contentSkip to conversation

Make RPCRDMA_MAX_RECV_BATCH configurable.

PATCH In review Gaurav Gangalwar 3 participants 8 messages Nov 13, 2025 – Nov 28, 2025

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Make RPCRDMA_MAX_RECV_BATCH configurable.

Bumped up rpcrdma_max_recv_batch to 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.

---
 net/sunrpc/xprtrdma/frwr_ops.c           | 2 +-
 net/sunrpc/xprtrdma/module.c             | 6 ++++++
 net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 +-
 net/sunrpc/xprtrdma/verbs.c              | 2 +-
 net/sunrpc/xprtrdma/xprt_rdma.h          | 4 +---
 5 files changed, 10 insertions(+), 6 deletions(-)
net/sunrpc/xprtrdma/frwr_ops.c +1 -1
index 31434aeb8e29..863a0c567915 100644
@@ -246,7 +246,7 @@ int frwr_query_device(struct rpcrdma_ep *ep, const struct ib_device *device)
 	ep->re_attr.cap.max_send_wr += 1; /* for ib_drain_sq */
 	ep->re_attr.cap.max_recv_wr = ep->re_max_requests;
 	ep->re_attr.cap.max_recv_wr += RPCRDMA_BACKWARD_WRS;
-	ep->re_attr.cap.max_recv_wr += RPCRDMA_MAX_RECV_BATCH;
+	ep->re_attr.cap.max_recv_wr += rpcrdma_max_recv_batch;
 	ep->re_attr.cap.max_recv_wr += 1; /* for ib_drain_rq */
 
 	ep->re_max_rdma_segs =
net/sunrpc/xprtrdma/module.c +6 -0
index 697f571d4c01..afeec5a68151 100644
@@ -27,6 +27,12 @@ MODULE_ALIAS("svcrdma");
 MODULE_ALIAS("xprtrdma");
 MODULE_ALIAS("rpcrdma6");
 
+unsigned int rpcrdma_max_recv_batch = 64;
+module_param_named(max_recv_batch, rpcrdma_max_recv_batch, uint, 0644);
+MODULE_PARM_DESC(max_recv_batch,
+		 "Maximum number of Receive WRs to post in a batch "
+		 "(default: 64, set to 0 to disable batching)");
+
 static void __exit rpc_rdma_cleanup(void)
 {
 	xprt_rdma_cleanup();
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -1
index 3d7f1413df02..32a9ceb18389 100644
@@ -440,7 +440,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt)
 	newxprt->sc_max_req_size = svcrdma_max_req_size;
 	newxprt->sc_max_requests = svcrdma_max_requests;
 	newxprt->sc_max_bc_requests = svcrdma_max_bc_requests;
-	newxprt->sc_recv_batch = RPCRDMA_MAX_RECV_BATCH;
+	newxprt->sc_recv_batch = rpcrdma_max_recv_batch;
 	newxprt->sc_fc_credits = cpu_to_be32(newxprt->sc_max_requests);
 
 	/* Qualify the transport's resource defaults with the
net/sunrpc/xprtrdma/verbs.c +1 -1
index 63262ef0c2e3..7cd0a2c152e6 100644
@@ -1359,7 +1359,7 @@ void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed)
 	if (likely(ep->re_receive_count > needed))
 		goto out;
 	needed -= ep->re_receive_count;
-	needed += RPCRDMA_MAX_RECV_BATCH;
+	needed += rpcrdma_max_recv_batch;
 
 	if (atomic_inc_return(&ep->re_receiving) > 1)
 		goto out;
net/sunrpc/xprtrdma/xprt_rdma.h +1 -3
index 8147d2b41494..1051aa612f36 100644
@@ -216,9 +216,7 @@ struct rpcrdma_rep {
  *
  * Setting this to zero disables Receive post batching.
  */
-enum {
-	RPCRDMA_MAX_RECV_BATCH = 7,
-};
+extern unsigned int rpcrdma_max_recv_batch;
 
 /* struct rpcrdma_sendctx - DMA mapped SGEs to unmap after Send completes
  */
Signature · 1 line
2.43.7
#1 lore raw
Quoted 3 lines · Gaurav Gangalwar

Bumped up rpcrdma_max_recv_batch to 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.

Hi Gaurav -

Adding an administrative setting is generally a last resort. First, we want a full root-cause analysis to understand the symptoms you are trying to address.

Quoted 80 lines
---
 net/sunrpc/xprtrdma/frwr_ops.c           | 2 +-
 net/sunrpc/xprtrdma/module.c             | 6 ++++++
 net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 +-
 net/sunrpc/xprtrdma/verbs.c              | 2 +-
 net/sunrpc/xprtrdma/xprt_rdma.h          | 4 +---
 5 files changed, 10 insertions(+), 6 deletions(-)
net/sunrpc/xprtrdma/frwr_ops.c +1 -1
index 31434aeb8e29..863a0c567915 100644
@@ -246,7 +246,7 @@ int frwr_query_device(struct rpcrdma_ep *ep, const struct ib_device *device)
 	ep->re_attr.cap.max_send_wr += 1; /* for ib_drain_sq */
 	ep->re_attr.cap.max_recv_wr = ep->re_max_requests;
 	ep->re_attr.cap.max_recv_wr += RPCRDMA_BACKWARD_WRS;
-	ep->re_attr.cap.max_recv_wr += RPCRDMA_MAX_RECV_BATCH;
+	ep->re_attr.cap.max_recv_wr += rpcrdma_max_recv_batch;
 	ep->re_attr.cap.max_recv_wr += 1; /* for ib_drain_rq */
 
 	ep->re_max_rdma_segs =
net/sunrpc/xprtrdma/module.c +6 -0
index 697f571d4c01..afeec5a68151 100644
@@ -27,6 +27,12 @@ MODULE_ALIAS("svcrdma");
 MODULE_ALIAS("xprtrdma");
 MODULE_ALIAS("rpcrdma6");
 
+unsigned int rpcrdma_max_recv_batch = 64;
+module_param_named(max_recv_batch, rpcrdma_max_recv_batch, uint, 0644);
+MODULE_PARM_DESC(max_recv_batch,
+		 "Maximum number of Receive WRs to post in a batch "
+		 "(default: 64, set to 0 to disable batching)");
+
 static void __exit rpc_rdma_cleanup(void)
 {
 	xprt_rdma_cleanup();
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -1
index 3d7f1413df02..32a9ceb18389 100644
@@ -440,7 +440,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt)
 	newxprt->sc_max_req_size = svcrdma_max_req_size;
 	newxprt->sc_max_requests = svcrdma_max_requests;
 	newxprt->sc_max_bc_requests = svcrdma_max_bc_requests;
-	newxprt->sc_recv_batch = RPCRDMA_MAX_RECV_BATCH;
+	newxprt->sc_recv_batch = rpcrdma_max_recv_batch;
 	newxprt->sc_fc_credits = cpu_to_be32(newxprt->sc_max_requests);
 
 	/* Qualify the transport's resource defaults with the
net/sunrpc/xprtrdma/verbs.c +1 -1
index 63262ef0c2e3..7cd0a2c152e6 100644
@@ -1359,7 +1359,7 @@ void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed)
 	if (likely(ep->re_receive_count > needed))
 		goto out;
 	needed -= ep->re_receive_count;
-	needed += RPCRDMA_MAX_RECV_BATCH;
+	needed += rpcrdma_max_recv_batch;
 
 	if (atomic_inc_return(&ep->re_receiving) > 1)
 		goto out;
net/sunrpc/xprtrdma/xprt_rdma.h +1 -3
index 8147d2b41494..1051aa612f36 100644
@@ -216,9 +216,7 @@ struct rpcrdma_rep {
  *
  * Setting this to zero disables Receive post batching.
  */
-enum {
-	RPCRDMA_MAX_RECV_BATCH = 7,
-};
+extern unsigned int rpcrdma_max_recv_batch;
 
 /* struct rpcrdma_sendctx - DMA mapped SGEs to unmap after Send completes
  */
Signature · 1 line
Chuck Lever
#2 lore raw
In reply to Gaurav Gangalwar
Quoted 3 lines · Gaurav Gangalwar

Bumped up rpcrdma_max_recv_batch to 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.

[ Resend with correct NFSD reviewer email addresses and linux-rdma@ ]

Hi Gaurav -

Adding an administrative setting is generally a last resort. First, we want a full root-cause analysis to understand the symptoms you are trying to address. Do you have an RCA or a simple reproducer to share with us?

Quoted 80 lines
---
 net/sunrpc/xprtrdma/frwr_ops.c           | 2 +-
 net/sunrpc/xprtrdma/module.c             | 6 ++++++
 net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 +-
 net/sunrpc/xprtrdma/verbs.c              | 2 +-
 net/sunrpc/xprtrdma/xprt_rdma.h          | 4 +---
 5 files changed, 10 insertions(+), 6 deletions(-)
net/sunrpc/xprtrdma/frwr_ops.c +1 -1
index 31434aeb8e29..863a0c567915 100644
@@ -246,7 +246,7 @@ int frwr_query_device(struct rpcrdma_ep *ep, const struct ib_device *device)
 	ep->re_attr.cap.max_send_wr += 1; /* for ib_drain_sq */
 	ep->re_attr.cap.max_recv_wr = ep->re_max_requests;
 	ep->re_attr.cap.max_recv_wr += RPCRDMA_BACKWARD_WRS;
-	ep->re_attr.cap.max_recv_wr += RPCRDMA_MAX_RECV_BATCH;
+	ep->re_attr.cap.max_recv_wr += rpcrdma_max_recv_batch;
 	ep->re_attr.cap.max_recv_wr += 1; /* for ib_drain_rq */
 
 	ep->re_max_rdma_segs =
net/sunrpc/xprtrdma/module.c +6 -0
index 697f571d4c01..afeec5a68151 100644
@@ -27,6 +27,12 @@ MODULE_ALIAS("svcrdma");
 MODULE_ALIAS("xprtrdma");
 MODULE_ALIAS("rpcrdma6");
 
+unsigned int rpcrdma_max_recv_batch = 64;
+module_param_named(max_recv_batch, rpcrdma_max_recv_batch, uint, 0644);
+MODULE_PARM_DESC(max_recv_batch,
+		 "Maximum number of Receive WRs to post in a batch "
+		 "(default: 64, set to 0 to disable batching)");
+
 static void __exit rpc_rdma_cleanup(void)
 {
 	xprt_rdma_cleanup();
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -1
index 3d7f1413df02..32a9ceb18389 100644
@@ -440,7 +440,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt)
 	newxprt->sc_max_req_size = svcrdma_max_req_size;
 	newxprt->sc_max_requests = svcrdma_max_requests;
 	newxprt->sc_max_bc_requests = svcrdma_max_bc_requests;
-	newxprt->sc_recv_batch = RPCRDMA_MAX_RECV_BATCH;
+	newxprt->sc_recv_batch = rpcrdma_max_recv_batch;
 	newxprt->sc_fc_credits = cpu_to_be32(newxprt->sc_max_requests);
 
 	/* Qualify the transport's resource defaults with the
net/sunrpc/xprtrdma/verbs.c +1 -1
index 63262ef0c2e3..7cd0a2c152e6 100644
@@ -1359,7 +1359,7 @@ void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed)
 	if (likely(ep->re_receive_count > needed))
 		goto out;
 	needed -= ep->re_receive_count;
-	needed += RPCRDMA_MAX_RECV_BATCH;
+	needed += rpcrdma_max_recv_batch;
 
 	if (atomic_inc_return(&ep->re_receiving) > 1)
 		goto out;
net/sunrpc/xprtrdma/xprt_rdma.h +1 -3
index 8147d2b41494..1051aa612f36 100644
@@ -216,9 +216,7 @@ struct rpcrdma_rep {
  *
  * Setting this to zero disables Receive post batching.
  */
-enum {
-	RPCRDMA_MAX_RECV_BATCH = 7,
-};
+extern unsigned int rpcrdma_max_recv_batch;
 
 /* struct rpcrdma_sendctx - DMA mapped SGEs to unmap after Send completes
  */
Signature · 1 line
Chuck Lever
#3 lore raw
Quoted 6 lines · Chuck Lever
Quoted 3 lines · Gaurav Gangalwar

Bumped up rpcrdma_max_recv_batch to 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.

Quoted 8 lines

[ Resend with correct NFSD reviewer email addresses and linux-rdma@ ]

Hi Gaurav -

Adding an administrative setting is generally a last resort. First, we want a full root-cause analysis to understand the symptoms you are trying to address. Do you have an RCA or a simple reproducer to share with us?

Issue found while testing fio workload over RDMA
Client: Ubuntu 24.04
Server: Ganesha NFS server
We have seen intermittent hung on client with buffered IO workload at large scale with around 30 RDMA connections, client was under memory pressure.
Ganesha log shows

10/11/2025 16:39:12Z : ntnx-10-57-210-224-a-fsvm 1309416[none] [0x7f49a6c3fe80] rpc :TIRPC :EVENT :rpc_rdma_cq_event_handler() cq completion status: RNR retry counter exceeded (13) rdma_xprt state 5 opcode 2 cbc 0x7f4996688000 inline 1

Which points to lack of posted recv buffers on client. Once we increased rpcrdma_max_recv_batch to 64, issue was resolved.

Quoted 86 lines
Quoted 72 lines
---
 net/sunrpc/xprtrdma/frwr_ops.c           | 2 +-
 net/sunrpc/xprtrdma/module.c             | 6 ++++++
 net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 +-
 net/sunrpc/xprtrdma/verbs.c              | 2 +-
 net/sunrpc/xprtrdma/xprt_rdma.h          | 4 +---
 5 files changed, 10 insertions(+), 6 deletions(-)
net/sunrpc/xprtrdma/frwr_ops.c +1 -1
index 31434aeb8e29..863a0c567915 100644
@@ -246,7 +246,7 @@ int frwr_query_device(struct rpcrdma_ep *ep, const struct ib_device *device)
      ep->re_attr.cap.max_send_wr += 1; /* for ib_drain_sq */
      ep->re_attr.cap.max_recv_wr = ep->re_max_requests;
      ep->re_attr.cap.max_recv_wr += RPCRDMA_BACKWARD_WRS;
-     ep->re_attr.cap.max_recv_wr += RPCRDMA_MAX_RECV_BATCH;
+     ep->re_attr.cap.max_recv_wr += rpcrdma_max_recv_batch;
      ep->re_attr.cap.max_recv_wr += 1; /* for ib_drain_rq */
 
      ep->re_max_rdma_segs =
net/sunrpc/xprtrdma/module.c +6 -0
index 697f571d4c01..afeec5a68151 100644
@@ -27,6 +27,12 @@ MODULE_ALIAS("svcrdma");
 MODULE_ALIAS("xprtrdma");
 MODULE_ALIAS("rpcrdma6");
 
+unsigned int rpcrdma_max_recv_batch = 64;
+module_param_named(max_recv_batch, rpcrdma_max_recv_batch, uint, 0644);
+MODULE_PARM_DESC(max_recv_batch,
+              "Maximum number of Receive WRs to post in a batch "
+              "(default: 64, set to 0 to disable batching)");
+
 static void __exit rpc_rdma_cleanup(void)
 {
      xprt_rdma_cleanup();
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -1
index 3d7f1413df02..32a9ceb18389 100644
@@ -440,7 +440,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt)
      newxprt->sc_max_req_size = svcrdma_max_req_size;
      newxprt->sc_max_requests = svcrdma_max_requests;
      newxprt->sc_max_bc_requests = svcrdma_max_bc_requests;
-     newxprt->sc_recv_batch = RPCRDMA_MAX_RECV_BATCH;
+     newxprt->sc_recv_batch = rpcrdma_max_recv_batch;
      newxprt->sc_fc_credits = cpu_to_be32(newxprt->sc_max_requests);
 
      /* Qualify the transport's resource defaults with the
net/sunrpc/xprtrdma/verbs.c +1 -1
index 63262ef0c2e3..7cd0a2c152e6 100644
@@ -1359,7 +1359,7 @@ void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed)
      if (likely(ep->re_receive_count > needed))
              goto out;
      needed -= ep->re_receive_count;
-     needed += RPCRDMA_MAX_RECV_BATCH;
+     needed += rpcrdma_max_recv_batch;
 
      if (atomic_inc_return(&ep->re_receiving) > 1)
              goto out;
net/sunrpc/xprtrdma/xprt_rdma.h +0 -0
index 8147d2b41494..1051aa612f36 100644

net/sunrpc/xprtrdma/xprt_rdma.h struct rpcrdma_rep { · line 222

Quoted 10 lines
net/sunrpc/xprtrdma/xprt_rdma.h +1 -3
@@ -216,9 +216,7 @@ struct rpcrdma_rep {
  *
  * Setting this to zero disables Receive post batching.
  */
-enum {
-     RPCRDMA_MAX_RECV_BATCH = 7,
-};
+extern unsigned int rpcrdma_max_recv_batch;
 
 /* struct rpcrdma_sendctx - DMA mapped SGEs to unmap after Send completes
  */

--
Chuck Lever

#4 lore raw
Quoted 24 lines · gaurav gangalwar
Quoted 6 lines · Chuck Lever
Quoted 3 lines · Gaurav Gangalwar

Bumped up rpcrdma_max_recv_batch to 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.

Quoted 8 lines

[ Resend with correct NFSD reviewer email addresses and linux-rdma@ ]

Hi Gaurav -

Adding an administrative setting is generally a last resort. First, we want a full root-cause analysis to understand the symptoms you are trying to address. Do you have an RCA or a simple reproducer to share with us?

Issue found while testing fio workload over RDMA
Client: Ubuntu 24.04
Server: Ganesha NFS server
We have seen intermittent hung on client with buffered IO workload at large scale with around 30 RDMA connections, client was under memory pressure.
Ganesha log shows

Quoted 7 lines

10/11/2025 16:39:12Z : ntnx-10-57-210-224-a-fsvm 1309416[none] [0x7f49a6c3fe80] rpc :TIRPC :EVENT :rpc_rdma_cq_event_handler() cq completion status: RNR retry counter exceeded (13) rdma_xprt state 5 opcode 2 cbc 0x7f4996688000 inline 1

Which points to lack of posted recv buffers on client. Once we increased rpcrdma_max_recv_batch to 64, issue was resolved.

That still doesn't convince me that increasing the receive batch count is a good fix, though it's certainly a workaround.

The client's RPC/RDMA code is supposed to track the number of Sends and keep the correct number of Receives on the Receive Queue. The goal of the implementation is to never encounter an RNR.

Therefore, if it's not doing that (and the RNR retries suggests that's the case) there is an actual bug somewhere. The extra batch Receives are an optimization, and should have no impact on correct operation.

If you can't reproduce this with the Linux NFS server, the place to start looking for misbehavior is NFS/Ganesha, as it is the newer NFS over RDMA implementation of the two servers. Maybe it's not handling credit accounting correctly, or perhaps it's putting more Sends on the wire than the credit limit allows.

Signature · 1 line
Chuck Lever
#5 lore raw
Quoted 45 lines · Chuck Lever
Quoted 24 lines · gaurav gangalwar
Quoted 6 lines · Chuck Lever
Quoted 3 lines · Gaurav Gangalwar

Bumped up rpcrdma_max_recv_batch to 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.

Quoted 8 lines

[ Resend with correct NFSD reviewer email addresses and linux-rdma@ ]

Hi Gaurav -

Adding an administrative setting is generally a last resort. First, we want a full root-cause analysis to understand the symptoms you are trying to address. Do you have an RCA or a simple reproducer to share with us?

Issue found while testing fio workload over RDMA
Client: Ubuntu 24.04
Server: Ganesha NFS server
We have seen intermittent hung on client with buffered IO workload at large scale with around 30 RDMA connections, client was under memory pressure.
Ganesha log shows

Quoted 7 lines

10/11/2025 16:39:12Z : ntnx-10-57-210-224-a-fsvm 1309416[none] [0x7f49a6c3fe80] rpc :TIRPC :EVENT :rpc_rdma_cq_event_handler() cq completion status: RNR retry counter exceeded (13) rdma_xprt state 5 opcode 2 cbc 0x7f4996688000 inline 1

Which points to lack of posted recv buffers on client. Once we increased rpcrdma_max_recv_batch to 64, issue was resolved.

That still doesn't convince me that increasing the receive batch count is a good fix, though it's certainly a workaround.

The client's RPC/RDMA code is supposed to track the number of Sends and keep the correct number of Receives on the Receive Queue. The goal of the implementation is to never encounter an RNR.

Therefore, if it's not doing that (and the RNR retries suggests that's the case) there is an actual bug somewhere. The extra batch Receives are an optimization, and should have no impact on correct operation.

Quoted 5 lines

If you can't reproduce this with the Linux NFS server, the place to start looking for misbehavior is NFS/Ganesha, as it is the newer NFS over RDMA implementation of the two servers. Maybe it's not handling credit accounting correctly, or perhaps it's putting more Sends on the wire than the credit limit allows.

Sure I will try to get more details.
Issue is specific to pNFS, for non pNFS shares we don't see this issue. Even for pNFS we see hung when number of ds connections are high (30 ds connections)
But this work around is definitely helping.

Quoted 4 lines

--
Chuck Lever

reviewer
#6 lore raw
In reply to Chuck Lever
Quoted 30 lines · Chuck Lever
Quoted 29 lines · gaurav gangalwar
Quoted 6 lines · Chuck Lever
Quoted 3 lines · Gaurav Gangalwar

Bumped up rpcrdma_max_recv_batch to 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.

Quoted 8 lines

[ Resend with correct NFSD reviewer email addresses and linux-rdma@ ]

Hi Gaurav -

Adding an administrative setting is generally a last resort. First, we want a full root-cause analysis to understand the symptoms you are trying to address. Do you have an RCA or a simple reproducer to share with us?

Issue found while testing fio workload over RDMA
Client: Ubuntu 24.04
Server: Ganesha NFS server
We have seen intermittent hung on client with buffered IO workload at large scale with around 30 RDMA connections, client was under memory pressure.
Ganesha log shows

10/11/2025 16:39:12Z : ntnx-10-57-210-224-a-fsvm 1309416[none] [0x7f49a6c3fe80] rpc :TIRPC :EVENT :rpc_rdma_cq_event_handler() cq completion status: RNR retry counter exceeded (13) rdma_xprt state 5 opcode 2 cbc 0x7f4996688000 inline 1

Quoted 5 lines
Quoted 2 lines

Which points to lack of posted recv buffers on client. Once we increased rpcrdma_max_recv_batch to 64, issue was resolved.

That still doesn't convince me that increasing the receive batch count is a good fix, though it's certainly a workaround.

It's not a workaround, this will fail on any RDMA provider that doesn't perform RNR retry, for example iWARP. But more importantly, RNR retry is unnecessary because the rpcrdma protocol implements a strict crediting exchange. A proper rpcrdma implementation will never trigger RNR.

This is almost certainly an rpcrdma protocol violation in the sender, which is failing to honor the credit limit granted by the receiving peer and is overrunning the peer's receive queue. A wireshark trace would prove it. Please do this research.

Tom.

Quoted 16 lines

The client's RPC/RDMA code is supposed to track the number of Sends and keep the correct number of Receives on the Receive Queue. The goal of the implementation is to never encounter an RNR.

Therefore, if it's not doing that (and the RNR retries suggests that's the case) there is an actual bug somewhere. The extra batch Receives are an optimization, and should have no impact on correct operation.

If you can't reproduce this with the Linux NFS server, the place to start looking for misbehavior is NFS/Ganesha, as it is the newer NFS over RDMA implementation of the two servers. Maybe it's not handling credit accounting correctly, or perhaps it's putting more Sends on the wire than the credit limit allows.

#7 lore raw
Quoted 43 lines · Tom Talpey
Quoted 30 lines · Chuck Lever
Quoted 29 lines · gaurav gangalwar
Quoted 6 lines · Chuck Lever
Quoted 3 lines · Gaurav Gangalwar

Bumped up rpcrdma_max_recv_batch to 64.
Added param to change to it, it becomes handy to use higher value to avoid hung.

Quoted 8 lines

[ Resend with correct NFSD reviewer email addresses and linux-rdma@ ]

Hi Gaurav -

Adding an administrative setting is generally a last resort. First, we want a full root-cause analysis to understand the symptoms you are trying to address. Do you have an RCA or a simple reproducer to share with us?

Issue found while testing fio workload over RDMA
Client: Ubuntu 24.04
Server: Ganesha NFS server
We have seen intermittent hung on client with buffered IO workload at large scale with around 30 RDMA connections, client was under memory pressure.
Ganesha log shows

10/11/2025 16:39:12Z : ntnx-10-57-210-224-a-fsvm 1309416[none] [0x7f49a6c3fe80] rpc :TIRPC :EVENT :rpc_rdma_cq_event_handler() cq completion status: RNR retry counter exceeded (13) rdma_xprt state 5 opcode 2 cbc 0x7f4996688000 inline 1

Quoted 5 lines
Quoted 2 lines

Which points to lack of posted recv buffers on client. Once we increased rpcrdma_max_recv_batch to 64, issue was resolved.

That still doesn't convince me that increasing the receive batch count is a good fix, though it's certainly a workaround.

It's not a workaround, this will fail on any RDMA provider that doesn't perform RNR retry, for example iWARP. But more importantly, RNR retry is unnecessary because the rpcrdma protocol implements a strict crediting exchange. A proper rpcrdma implementation will never trigger RNR.

Quoted 25 lines

This is almost certainly an rpcrdma protocol violation in the sender, which is failing to honor the credit limit granted by the receiving peer and is overrunning the peer's receive queue. A wireshark trace would prove it. Please do this research.

Tom.

Quoted 16 lines

The client's RPC/RDMA code is supposed to track the number of Sends and keep the correct number of Receives on the Receive Queue. The goal of the implementation is to never encounter an RNR.

Therefore, if it's not doing that (and the RNR retries suggests that's the case) there is an actual bug somewhere. The extra batch Receives are an optimization, and should have no impact on correct operation.

If you can't reproduce this with the Linux NFS server, the place to start looking for misbehavior is NFS/Ganesha, as it is the newer NFS over RDMA implementation of the two servers. Maybe it's not handling credit accounting correctly, or perhaps it's putting more Sends on the wire than the credit limit allows.

Thanks for the review, I was going through server implementation for NFS Ganesha, we strictly honor read_chunks, write_chunks and reply_chunks, so the credit limit should be client driven only. Only in case of callbacks from server to client, as of now there is no credits check, we advertise server_credits in cb call similar to rb_bc_max_requests in linux NFS server, but don't check for client_credits from client in reply. We need to limit callbacks to MIN(server_credits, client_credits)

Regards,
Gaurav