svcrdma performance scalability enhancements
Subsystems (4)
Chuck Lever
maintainercover svcrdma performance scalability enhancements
From: Chuck Lever <[email protected]>
When considering NFS I/O throughput and latency, typically the RPC transport is not the primary bottleneck. The CPU cost of the RPC/RDMA transport is insignificant in comparison to other resource utilization on the server.
However, when considered from a scalability point of view, the incremental costs of things like memory-per-connection and interrupt rate or doorbell rate per connection do add up.
The following series lowers the per-connection resource utilization of svcrdma in several areas. The main benefits are lower lock contention, lower interrupt and doorbell rate per RPC, and less CPU cache theft.
Profiling an 8KB NFSv3 read/write workload over RDMA identifies where overhead accumulates as connection count grows. Roughly 4% of total CPU time goes to contention on the svcrdma_wq unbound workqueue pool lock, driven by cascading work item re-queues through the Send completion path. Receive completion processing acquires a per-transport spinlock on every inbound message. Doorbell and completion event counts scale with Write chunk usage. svc_alloc_arg() scans ~259 rq_pages slots per receive even when only a few pages need replacement.
Three strategies recur throughout this series.
Lock-free lists replace spinlock-protected queues on the hottest paths. The receive completion queue, Read completion queue, and send context release path all convert to llist, eliminating producer-side locking. The global svcrdma_wq workqueue -- the single largest contention source -- is replaced by per-transport kthreads that drain completed send contexts from an llist in batches. The intermediate re-queue for write chunk resource release is thus removed as well. Those resources are now freed inline during send context teardown.
Work Request chaining reduces per-RPC doorbell and completion rates. Write chunk RDMA Write WRs are linked onto the Reply Send WR chain, so a single ib_post_send() covers both operations with one completion event. Receive Queue posting switches from small fixed batches to watermark-triggered bulk replenishment, provisioned at twice the negotiated credit limit. Ticket-based fair queuing for Send Queue slot allocation prevents starvation when the SQ fills under concurrent use.
Per-object caching and cache line separation reduce allocation cost and cross-CPU invalidation traffic. Each recv_ctxt includes a single-entry svc_rdma_chunk cache, covering the >99% common case without kmalloc. Cache line annotations on struct svcxprt_rdma place the Send context cache, R/W context cache, and SQ availability counter in separate cache lines.
XPT_DATA flag handling upon sc_read_complete_q consumption is corrected to clear and recompute the flag. Trace data from a 256KB write workload shows ~14 transport enqueue attempts per RPC; in 62% of cases, no work is pending. Clearing the flag on consumption eliminates the majority of these spurious dispatches.
Base commit: v6.19
URL: https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/log/?h=svcrdma-next
---
Chuck Lever (15):
svcrdma: Add fair queuing for Send Queue access
svcrdma: Clean up use of rdma->sc_pd->device in Receive paths
svcrdma: Clean up use of rdma->sc_pd->device
svcrdma: Add Write chunk WRs to the RPC's Send WR chain
svcrdma: Factor out WR chain linking into helper
svcrdma: Reduce false sharing in struct svcxprt_rdma
svcrdma: Use lock-free list for Receive Queue tracking
svcrdma: Convert Read completion queue to use lock-free list
svcrdma: Release write chunk resources without re-queuing
svcrdma: Use per-transport kthread for send context release
svcrdma: Use watermark-based Receive Queue replenishment
svcrdma: Add per-recv_ctxt chunk context cache
svcrdma: clear XPT_DATA on sc_read_complete_q consumption
svcrdma: retry when receive queues drain transiently
svcrdma: clear XPT_DATA on sc_rq_dto_q consumption
include/linux/sunrpc/svc_rdma.h | 80 ++++++--- include/linux/sunrpc/svc_rdma_pcl.h | 12 +- net/sunrpc/xprtrdma/svc_rdma.c | 18 +- net/sunrpc/xprtrdma/svc_rdma_pcl.c | 55 +++++- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 158 +++++++++++------ net/sunrpc/xprtrdma/svc_rdma_rw.c | 169 ++++++++++-------- net/sunrpc/xprtrdma/svc_rdma_sendto.c | 209 ++++++++++++++++------- net/sunrpc/xprtrdma/svc_rdma_transport.c | 28 +-- 8 files changed, 488 insertions(+), 241 deletions(-)
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 1/15 svcrdma: Add fair queuing for Send Queue access
From: Chuck Lever <[email protected]>
When the Send Queue fills, multiple threads may wait for SQ slots. The previous implementation had no ordering guarantee, allowing starvation when one thread repeatedly acquires slots while others wait indefinitely.
Introduce a ticket-based fair queuing system. Each waiter takes a ticket number and is served in FIFO order. This ensures forward progress for all waiters when SQ capacity is constrained.
The implementation has two phases:
1. Fast path: attempt to reserve SQ slots without waiting
2. Slow path: take a ticket, wait for turn, then wait for slots
The ticket system adds two atomic counters to the transport:
- sc_sq_ticket_head: next ticket to issue
- sc_sq_ticket_tail: ticket currently being served
When a waiter successfully reserves slots, it advances the tail counter and wakes the next waiter. This creates an orderly handoff that prevents starvation while maintaining good throughput on the fast path when contention is low.
--- include/linux/sunrpc/svc_rdma.h | 4 + net/sunrpc/xprtrdma/svc_rdma_rw.c | 42 ++++----- net/sunrpc/xprtrdma/svc_rdma_sendto.c | 113 +++++++++++++++-------- net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 + 4 files changed, 98 insertions(+), 63 deletions(-)
include/linux/sunrpc/svc_rdma.h +4 -0
@@ -84,6 +84,8 @@ struct svcxprt_rdma {atomic_t sc_sq_avail;/* SQEs ready to be consumed */ unsigned int sc_sq_depth;/* Depth of SQ */ +atomic_t sc_sq_ticket_head;/* Next ticket to issue */ +atomic_t sc_sq_ticket_tail;/* Ticket currently serving */ __be32 sc_fc_credits;/* Forward credits */ u32 sc_max_requests;/* Max requests */ u32 sc_max_bc_requests;/* Backward credits */
@@ -306,6 +308,8 @@ extern void svc_rdma_send_error_msg(struct svcxprt_rdma *rdma,struct svc_rdma_recv_ctxt *rctxt, int status);extern void svc_rdma_wake_send_waiters (struct svcxprt_rdma *rdma,int avail); +extern int svc_rdma_sq_wait (struct svcxprt_rdma *rdma, +const struct rpc_rdma_cid *cid,int sqecount);extern int svc_rdma_sendto (struct svc_rqst *);extern int svc_rdma_result_payload (struct svc_rqst *rqstp,unsigned int offset,unsigned int length);
net/sunrpc/xprtrdma/svc_rdma_rw.c +16 -26
50 lines not on this page · raw message
net/sunrpc/xprtrdma/svc_rdma_sendto.c +76 -37
135 lines not on this page · raw message
net/sunrpc/xprtrdma/svc_rdma_transport.c +2 -0
@@ -474,6 +474,8 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt)if (newxprt->sc_sq_depth > dev->attrs.max_qp_wr) newxprt->sc_sq_depth = dev->attrs.max_qp_wr;atomic_set (&newxprt->sc_sq_avail, newxprt->sc_sq_depth); +atomic_set (&newxprt->sc_sq_ticket_head,0 ); +atomic_set (&newxprt->sc_sq_ticket_tail,0 ); newxprt->sc_pd =ib_alloc_pd (dev,0 );if (IS_ERR (newxprt->sc_pd)) {
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 2/15 svcrdma: Clean up use of rdma->sc_pd->device in Receive paths
In reply to Chuck LeverFrom: Chuck Lever <[email protected]>
I can't think of a reason why svcrdma is using the PD's device. Most other consumers of the IB DMA API use the ib_device pointer from the connection's rdma_cm_id.
I don't believe there's any functional difference between the two, but it is a little confusing to see some uses of rdma_cm_id->device and some of ib_pd->device.
--- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-)
net/sunrpc/xprtrdma/svc_rdma_recvfrom.c +9 -8
@@ -118,7 +118,7 @@ svc_rdma_next_recv_ctxt(struct list_head *list)static struct svc_rdma_recv_ctxt *svc_rdma_recv_ctxt_alloc (struct svcxprt_rdma *rdma) { -int node =ibdev_to_node (rdma->sc_cm_id->device); +struct ib_device *device = rdma->sc_cm_id->device;struct svc_rdma_recv_ctxt *ctxt;unsigned long pages;dma_addr_t addr;
@@ -126,16 +126,17 @@ svc_rdma_recv_ctxt_alloc(struct svcxprt_rdma *rdma) pages =svc_serv_maxpages (rdma->sc_xprt.xpt_server); ctxt =kzalloc_node (struct_size (ctxt, rc_pages, pages), -GFP_KERNEL , node); +GFP_KERNEL ,ibdev_to_node (device));if (!ctxt) goto fail0; ctxt->rc_maxpages = pages; - buffer =kmalloc_node (rdma->sc_max_req_size,GFP_KERNEL , node); + buffer =kmalloc_node (rdma->sc_max_req_size,GFP_KERNEL , +ibdev_to_node (device));if (!buffer) goto fail1; - addr =ib_dma_map_single (rdma->sc_pd->device, buffer, - rdma->sc_max_req_size,DMA_FROM_DEVICE ); -if (ib_dma_mapping_error (rdma->sc_pd->device, addr)) + addr =ib_dma_map_single (device, buffer, rdma->sc_max_req_size, +DMA_FROM_DEVICE ); +if (ib_dma_mapping_error (device, addr)) goto fail2;svc_rdma_recv_cid_init (rdma, &ctxt->rc_cid);
@@ -167,7 +168,7 @@ svc_rdma_recv_ctxt_alloc(struct svcxprt_rdma *rdma)static void svc_rdma_recv_ctxt_destroy (struct svcxprt_rdma *rdma,struct svc_rdma_recv_ctxt *ctxt) { -ib_dma_unmap_single (rdma->sc_pd->device, ctxt->rc_recv_sge.addr, +ib_dma_unmap_single (rdma->sc_cm_id->device, ctxt->rc_recv_sge.addr, ctxt->rc_recv_sge.length,DMA_FROM_DEVICE );kfree (ctxt->rc_recv_buf);kfree (ctxt);
@@ -962,7 +963,7 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp)return 0 ;percpu_counter_inc (&svcrdma_stat_recv); -ib_dma_sync_single_for_cpu (rdma_xprt->sc_pd->device, +ib_dma_sync_single_for_cpu (rdma_xprt->sc_cm_id->device, ctxt->rc_recv_sge.addr, ctxt->rc_byte_len,DMA_FROM_DEVICE );svc_rdma_build_arg_xdr (rqstp, ctxt);
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 3/15 svcrdma: Clean up use of rdma->sc_pd->device
In reply to Chuck LeverFrom: Chuck Lever <[email protected]>
I can't think of a reason why svcrdma is using the PD's device. Most other consumers of the IB DMA API use the ib_device pointer from the connection's rdma_cm_id.
I don't think there's any functional difference between the two, but it is a little confusing to see some uses of rdma_cm_id and some of ib_pd.
--- net/sunrpc/xprtrdma/svc_rdma_sendto.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
net/sunrpc/xprtrdma/svc_rdma_sendto.c +12 -12
@@ -116,7 +116,7 @@ static void svc_rdma_wc_send(struct ib_cq *cq, struct ib_wc *wc);static struct svc_rdma_send_ctxt *svc_rdma_send_ctxt_alloc (struct svcxprt_rdma *rdma) { -int node =ibdev_to_node (rdma->sc_cm_id->device); +struct ib_device *device = rdma->sc_cm_id->device;struct svc_rdma_send_ctxt *ctxt;unsigned long pages;dma_addr_t addr;
@@ -124,21 +124,22 @@ svc_rdma_send_ctxt_alloc(struct svcxprt_rdma *rdma)int i; ctxt =kzalloc_node (struct_size (ctxt, sc_sges, rdma->sc_max_send_sges), -GFP_KERNEL , node); +GFP_KERNEL ,ibdev_to_node (device));if (!ctxt) goto fail0; pages =svc_serv_maxpages (rdma->sc_xprt.xpt_server); ctxt->sc_pages =kcalloc_node (pages,sizeof (struct page *), -GFP_KERNEL , node); +GFP_KERNEL ,ibdev_to_node (device));if (!ctxt->sc_pages) goto fail1; ctxt->sc_maxpages = pages; - buffer =kmalloc_node (rdma->sc_max_req_size,GFP_KERNEL , node); + buffer =kmalloc_node (rdma->sc_max_req_size,GFP_KERNEL , +ibdev_to_node (device));if (!buffer) goto fail2; - addr =ib_dma_map_single (rdma->sc_pd->device, buffer, - rdma->sc_max_req_size,DMA_TO_DEVICE ); -if (ib_dma_mapping_error (rdma->sc_pd->device, addr)) + addr =ib_dma_map_single (device, buffer, rdma->sc_max_req_size, +DMA_TO_DEVICE ); +if (ib_dma_mapping_error (device, addr)) goto fail3;svc_rdma_send_cid_init (rdma, &ctxt->sc_cid);
@@ -175,15 +176,14 @@ svc_rdma_send_ctxt_alloc(struct svcxprt_rdma *rdma)*/ void svc_rdma_send_ctxts_destroy (struct svcxprt_rdma *rdma) { +struct ib_device *device = rdma->sc_cm_id->device;struct svc_rdma_send_ctxt *ctxt;struct llist_node *node;while ((node =llist_del_first (&rdma->sc_send_ctxts)) !=NULL ) { ctxt = llist_entry(node, struct svc_rdma_send_ctxt, sc_node); -ib_dma_unmap_single (rdma->sc_pd->device, - ctxt->sc_sges[0 ].addr, - rdma->sc_max_req_size, -DMA_TO_DEVICE ); +ib_dma_unmap_single (device, ctxt->sc_sges[0 ].addr, + rdma->sc_max_req_size,DMA_TO_DEVICE );kfree (ctxt->sc_xprt_buf);kfree (ctxt->sc_pages);kfree (ctxt);
@@ -412,7 +412,7 @@ int svc_rdma_post_send(struct svcxprt_rdma *rdma,might_sleep ();/* Sync the transport header buffer */ -ib_dma_sync_single_for_device (rdma->sc_pd->device, +ib_dma_sync_single_for_device (rdma->sc_cm_id->device, send_wr->sg_list[0 ].addr, send_wr->sg_list[0 ].length,DMA_TO_DEVICE );
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 4/15 svcrdma: Add Write chunk WRs to the RPC's Send WR chain
In reply to Chuck LeverFrom: Chuck Lever <[email protected]>
Previously, Write chunk RDMA Writes were posted via a separate ib_post_send() call with their own completion handler. Each Write chunk incurred a doorbell and generated a completion event.
Link Write chunk WRs onto the RPC Reply's Send WR chain so that a single ib_post_send() call posts both the RDMA Writes and the Send WR. A single completion event signals that all operations have finished. This reduces both doorbell rate and completion rate, as well as eliminating the latency of a round-trip between the Write chunk completion and the subsequent Send WR posting.
The lifecycle of Write chunk resources changes: previously, the svc_rdma_write_done() completion handler released Write chunk resources when RDMA Writes completed. With WR chaining, resources remain live until the Send completion. A new sc_write_info_list tracks Write chunk metadata attached to each Send context, and svc_rdma_write_chunk_release() frees these resources when the Send context is released.
The svc_rdma_write_done() handler now handles only error cases. On success it returns immediately since the Send completion handles resource release. On failure (WR flush), it closes the connection to signal to the client that the RPC Reply is incomplete.
--- include/linux/sunrpc/svc_rdma.h | 13 +++- net/sunrpc/xprtrdma/svc_rdma_rw.c | 94 ++++++++++++++++++++------- net/sunrpc/xprtrdma/svc_rdma_sendto.c | 10 ++- 3 files changed, 91 insertions(+), 26 deletions(-)
include/linux/sunrpc/svc_rdma.h +10 -3
@@ -215,6 +215,7 @@ struct svc_rdma_recv_ctxt {*/ struct svc_rdma_write_info {struct svcxprt_rdma *wi_rdma; +struct list_head wi_list;const struct svc_rdma_chunk *wi_chunk;
@@ -243,7 +244,10 @@ struct svc_rdma_send_ctxt {struct ib_cqe sc_cqe;struct xdr_buf sc_hdrbuf;struct xdr_stream sc_stream; + +struct list_head sc_write_info_list;struct svc_rdma_write_info sc_reply_info; +void *sc_xprt_buf;int sc_page_count;int sc_cur_sge_no;
@@ -276,11 +280,14 @@ extern void svc_rdma_cc_init(struct svcxprt_rdma *rdma,extern void svc_rdma_cc_release (struct svcxprt_rdma *rdma,struct svc_rdma_chunk_ctxt *cc,enum dma_data_direction dir); +extern void svc_rdma_write_chunk_release (struct svcxprt_rdma *rdma, +struct svc_rdma_send_ctxt *ctxt);extern void svc_rdma_reply_chunk_release (struct svcxprt_rdma *rdma,struct svc_rdma_send_ctxt *ctxt); -extern int svc_rdma_send_write_list (struct svcxprt_rdma *rdma, -const struct svc_rdma_recv_ctxt *rctxt, -const struct xdr_buf *xdr); +extern int svc_rdma_prepare_write_list (struct svcxprt_rdma *rdma, +const struct svc_rdma_recv_ctxt *rctxt, +struct svc_rdma_send_ctxt *sctxt, +const struct xdr_buf *xdr); extern int svc_rdma_prepare_reply_chunk(struct svcxprt_rdma *rdma, const struct svc_rdma_pcl *write_pcl, const struct svc_rdma_pcl *reply_pcl,
net/sunrpc/xprtrdma/svc_rdma_rw.c +72 -22
150 lines not on this page · raw message
net/sunrpc/xprtrdma/svc_rdma_sendto.c +9 -1
@@ -150,6 +150,7 @@ svc_rdma_send_ctxt_alloc(struct svcxprt_rdma *rdma) ctxt->sc_send_wr.sg_list = ctxt->sc_sges; ctxt->sc_send_wr.send_flags =IB_SEND_SIGNALED ; ctxt->sc_cqe.done = svc_rdma_wc_send; +INIT_LIST_HEAD (&ctxt->sc_write_info_list); ctxt->sc_xprt_buf = buffer;xdr_buf_init (&ctxt->sc_hdrbuf, ctxt->sc_xprt_buf, rdma->sc_max_req_size);
@@ -237,6 +238,7 @@ static void svc_rdma_send_ctxt_release(struct svcxprt_rdma *rdma,struct ib_device *device = rdma->sc_cm_id->device;unsigned int i; +svc_rdma_write_chunk_release (rdma, ctxt);svc_rdma_reply_chunk_release (rdma, ctxt);if (ctxt->sc_page_count)
@@ -1015,6 +1017,12 @@ void svc_rdma_send_error_msg(struct svcxprt_rdma *rdma, sctxt->sc_send_wr.num_sge =1 ; sctxt->sc_send_wr.opcode =IB_WR_SEND ; sctxt->sc_sges[0 ].length = sctxt->sc_hdrbuf.len; + +/* Ensure only the error message is posted, not any previously +* prepared Write chunk WRs. +*/ + sctxt->sc_wr_chain = &sctxt->sc_send_wr; + sctxt->sc_sqecount =1 ;if (svc_rdma_post_send (rdma, sctxt)) goto put_ctxt;return ;
@@ -1062,7 +1070,7 @@ int svc_rdma_sendto(struct svc_rqst *rqstp)if (!p) goto put_ctxt; - ret =svc_rdma_send_write_list (rdma, rctxt, &rqstp->rq_res); + ret =svc_rdma_prepare_write_list (rdma, rctxt, sctxt, &rqstp->rq_res);if (ret <0 ) goto put_ctxt;
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 5/15 svcrdma: Factor out WR chain linking into helper
In reply to Chuck LeverFrom: Chuck Lever <[email protected]>
svc_rdma_prepare_write_chunk() and svc_rdma_prepare_reply_chunk() contain identical code for linking RDMA R/W work requests onto a Send context's WR chain. This duplication increases maintenance burden and risks divergent bug fixes.
Introduce svc_rdma_cc_link_wrs() to consolidate the WR chain linking logic. The helper walks the chunk context's rwctxts list, chains each WR via rdma_rw_ctx_wrs(), and updates the Send context's chain head and SQE count. Completion signaling is requested only for the tail WR (posted first).
No functional change.
--- net/sunrpc/xprtrdma/svc_rdma_rw.c | 67 +++++++++++++------------------ 1 file changed, 28 insertions(+), 39 deletions(-)
net/sunrpc/xprtrdma/svc_rdma_rw.c +28 -39
@@ -610,15 +610,32 @@ static int svc_rdma_xb_write(const struct xdr_buf *xdr, void *data)return xdr->len; } -/* -* svc_rdma_prepare_write_chunk - Link Write WRs for @chunk onto @sctxt's chain -* -* Write WRs are prepended to the Send WR chain so that a single -* ib_post_send() posts both RDMA Writes and the final Send. Only -* the first WR in each chunk gets a CQE for error detection; -* subsequent WRs complete without individual completion events. -* The Send WR's signaled completion indicates all chained -* operations have finished. +/* Link chunk WRs onto @sctxt's WR chain. Completion is requested +* for the tail WR, which is posted first. +*/ +static inline void svc_rdma_cc_link_wrs (struct svcxprt_rdma *rdma, +struct svc_rdma_send_ctxt *sctxt, +struct svc_rdma_chunk_ctxt *cc) +{ +struct ib_send_wr *first_wr; +struct list_head *pos; +struct ib_cqe *cqe; + + first_wr = sctxt->sc_wr_chain; + cqe = &cc->cc_cqe; +list_for_each (pos, &cc->cc_rwctxts) { +struct svc_rdma_rw_ctxt *rwc; + + rwc = list_entry(pos, struct svc_rdma_rw_ctxt, rw_list); + first_wr =rdma_rw_ctx_wrs (&rwc->rw_ctx, rdma->sc_qp, + rdma->sc_port_num, cqe, first_wr); + cqe =NULL ; + } + sctxt->sc_wr_chain = first_wr; + sctxt->sc_sqecount += cc->cc_sqecount; +} + +/* Link Write WRs for @chunk onto @sctxt's WR chain. */ static int svc_rdma_prepare_write_chunk(struct svcxprt_rdma *rdma, struct svc_rdma_send_ctxt *sctxt,
@@ -627,10 +644,7 @@ static int svc_rdma_prepare_write_chunk(struct svcxprt_rdma *rdma, {struct svc_rdma_write_info *info;struct svc_rdma_chunk_ctxt *cc; -struct ib_send_wr *first_wr;struct xdr_buf payload; -struct list_head *pos; -struct ib_cqe *cqe;int ret; if (xdr_buf_subsegment(xdr, &payload, chunk->ch_position,
@@ -650,18 +664,7 @@ static int svc_rdma_prepare_write_chunk(struct svcxprt_rdma *rdma,if (unlikely (sctxt->sc_sqecount + cc->cc_sqecount > rdma->sc_sq_depth)) goto out_err; - first_wr = sctxt->sc_wr_chain; - cqe = &cc->cc_cqe; -list_for_each (pos, &cc->cc_rwctxts) { -struct svc_rdma_rw_ctxt *rwc; - - rwc = list_entry(pos, struct svc_rdma_rw_ctxt, rw_list); - first_wr =rdma_rw_ctx_wrs (&rwc->rw_ctx, rdma->sc_qp, - rdma->sc_port_num, cqe, first_wr); - cqe =NULL ; - } - sctxt->sc_wr_chain = first_wr; - sctxt->sc_sqecount += cc->cc_sqecount; +svc_rdma_cc_link_wrs (rdma, sctxt, cc);list_add (&info->wi_list, &sctxt->sc_write_info_list);trace_svcrdma_post_write_chunk (&cc->cc_cid, cc->cc_sqecount);
@@ -723,9 +726,6 @@ int svc_rdma_prepare_reply_chunk(struct svcxprt_rdma *rdma, {struct svc_rdma_write_info *info = &sctxt->sc_reply_info;struct svc_rdma_chunk_ctxt *cc = &info->wi_cc; -struct ib_send_wr *first_wr; -struct list_head *pos; -struct ib_cqe *cqe;int ret; info->wi_rdma = rdma;
@@ -739,18 +739,7 @@ int svc_rdma_prepare_reply_chunk(struct svcxprt_rdma *rdma,if (ret <0 )return ret; - first_wr = sctxt->sc_wr_chain; - cqe = &cc->cc_cqe; -list_for_each (pos, &cc->cc_rwctxts) { -struct svc_rdma_rw_ctxt *rwc; - - rwc = list_entry(pos, struct svc_rdma_rw_ctxt, rw_list); - first_wr =rdma_rw_ctx_wrs (&rwc->rw_ctx, rdma->sc_qp, - rdma->sc_port_num, cqe, first_wr); - cqe =NULL ; - } - sctxt->sc_wr_chain = first_wr; - sctxt->sc_sqecount += cc->cc_sqecount; +svc_rdma_cc_link_wrs (rdma, sctxt, cc);trace_svcrdma_post_reply_chunk (&cc->cc_cid, cc->cc_sqecount);return xdr->len;
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 6/15 svcrdma: Reduce false sharing in struct svcxprt_rdma
In reply to Chuck LeverFrom: Chuck Lever <[email protected]>
Several frequently-modified fields in struct svcxprt_rdma reside in the same cache line, causing false sharing between independent code paths:
- sc_sq_avail: atomic, modified on every ib_post_send and completion
- sc_send_lock/sc_send_ctxts: Send context cache, accessed during reply construction
- sc_rw_ctxt_lock/sc_rw_ctxts: R/W context cache, accessed during Read/Write chunk processing
When any of these fields is modified, the entire cache line is invalidated on other CPUs. Under load, concurrent operations on different code paths cause the cache line to bounce between cores, degrading performance.
Insert ____cacheline_aligned_in_smp annotations to place the Send context cache, R/W context cache, and receive-path fields into separate cache lines. To utilize the padding this creates:
- Move sc_pd, sc_ord, sc_max_send_sges into the Send cache line (sc_pd is accessed during send context setup)
- Move sc_qp, sc_port_num, sc_rq_cq, sc_sq_cq into the R/W cache line (sc_qp and sc_port_num are accessed together during every rdma_rw_ctx_wrs call)
--- include/linux/sunrpc/svc_rdma.h | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-)
include/linux/sunrpc/svc_rdma.h +17 -12
@@ -78,8 +78,6 @@ struct svcxprt_rdma {struct rdma_cm_id *sc_cm_id;/* RDMA connection id */ struct list_head sc_accept_q;/* Conn. waiting accept */ struct rpcrdma_notification sc_rn;/* removal notification */ -int sc_ord;/* RDMA read limit */ -int sc_max_send_sges;bool sc_snd_w_inv;/* OK to use Send With Invalidate */ atomic_t sc_sq_avail;/* SQEs ready to be consumed */
@@ -90,23 +88,30 @@ struct svcxprt_rdma {u32 sc_max_requests;/* Max requests */ u32 sc_max_bc_requests;/* Backward credits */ int sc_max_req_size;/* Size of each RQ WR buf */ -u8 sc_port_num; -struct ib_pd *sc_pd; - -spinlock_t sc_send_lock; +/* Send context cache */ +spinlock_t sc_send_lock____cacheline_aligned_in_smp ;struct llist_head sc_send_ctxts; -spinlock_t sc_rw_ctxt_lock; -struct llist_head sc_rw_ctxts; +/* sc_pd accessed during send context alloc */ +struct ib_pd *sc_pd; +int sc_ord;/* RDMA read limit */ +int sc_max_send_sges; -u32 sc_pending_recvs; +/* R/W context cache */ +spinlock_t sc_rw_ctxt_lock____cacheline_aligned_in_smp ; +struct llist_head sc_rw_ctxts; +/* sc_qp and sc_port_num accessed together */ +struct ib_qp *sc_qp; +u8 sc_port_num; +struct ib_cq *sc_rq_cq; +struct ib_cq *sc_sq_cq; + +/* Receive path */ +u32 sc_pending_recvs____cacheline_aligned_in_smp ;u32 sc_recv_batch;struct list_head sc_rq_dto_q;struct list_head sc_read_complete_q;spinlock_t sc_rq_dto_lock; -struct ib_qp *sc_qp; -struct ib_cq *sc_rq_cq; -struct ib_cq *sc_sq_cq;spinlock_t sc_lock;/* transport lock */
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 7/15 svcrdma: Use lock-free list for Receive Queue tracking
In reply to Chuck LeverFrom: Chuck Lever <[email protected]>
The sc_rq_dto_lock spinlock is acquired on every receive completion to add the completed receive context to the sc_rq_dto_q list. Under high message rates this creates contention between softirq contexts processing completions.
Replace sc_rq_dto_q with a lock-free llist. Receive completions now use llist_add() which requires no locking. The consumer uses llist_del_first() to retrieve one item at a time.
The lock remains for sc_read_complete_q, but the primary hot path (receive completion and consumption) no longer requires it. This eliminates producer-side contention entirely.
Note that llist provides LIFO ordering rather than FIFO. For independent RPC requests this has no semantic impact and avoids the overhead of reversing the list on the consumer side.
--- include/linux/sunrpc/svc_rdma.h | 2 +- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 38 +++++++++++++++++------- net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 +- 3 files changed, 29 insertions(+), 13 deletions(-)
include/linux/sunrpc/svc_rdma.h +1 -1
@@ -109,7 +109,7 @@ struct svcxprt_rdma {/* Receive path */ u32 sc_pending_recvs____cacheline_aligned_in_smp ;u32 sc_recv_batch; -struct list_head sc_rq_dto_q; +struct llist_head sc_rq_dto_q;struct list_head sc_read_complete_q;spinlock_t sc_rq_dto_lock;
net/sunrpc/xprtrdma/svc_rdma_recvfrom.c +27 -11
@@ -361,11 +361,12 @@ static void svc_rdma_wc_receive(struct ib_cq *cq, struct ib_wc *wc)/* All wc fields are now known to be valid */ ctxt->rc_byte_len = wc->byte_len; -spin_lock (&rdma->sc_rq_dto_lock); -list_add_tail (&ctxt->rc_list, &rdma->sc_rq_dto_q); -/* Note the unlock pairs with the smp_rmb in svc_xprt_ready: */ +llist_add (&ctxt->rc_node, &rdma->sc_rq_dto_q); +/* +* The implicit barrier of llist_add's cmpxchg pairs with +* the smp_rmb in svc_xprt_ready. +*/ set_bit (XPT_DATA , &rdma->sc_xprt.xpt_flags); -spin_unlock (&rdma->sc_rq_dto_lock);if (!test_bit (RDMAXPRT_CONN_PENDING , &rdma->sc_flags))svc_xprt_enqueue (&rdma->sc_xprt);return ;
@@ -388,13 +389,16 @@ static void svc_rdma_wc_receive(struct ib_cq *cq, struct ib_wc *wc)void svc_rdma_flush_recv_queues (struct svcxprt_rdma *rdma) {struct svc_rdma_recv_ctxt *ctxt; +struct llist_node *node;while ((ctxt =svc_rdma_next_recv_ctxt (&rdma->sc_read_complete_q))) {list_del (&ctxt->rc_list);svc_rdma_recv_ctxt_put (rdma, ctxt); } -while ((ctxt =svc_rdma_next_recv_ctxt (&rdma->sc_rq_dto_q))) { -list_del (&ctxt->rc_list); + node =llist_del_all (&rdma->sc_rq_dto_q); +while (node) { + ctxt = llist_entry(node, struct svc_rdma_recv_ctxt, rc_node); + node = node->next;svc_rdma_recv_ctxt_put (rdma, ctxt); } }
@@ -930,6 +934,7 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp)struct svcxprt_rdma *rdma_xprt = container_of(xprt, struct svcxprt_rdma, sc_xprt);struct svc_rdma_recv_ctxt *ctxt; +struct llist_node *node;int ret;/* Prevent svc_xprt_release() from releasing pages in rq_pages
@@ -949,13 +954,24 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp)svc_rdma_read_complete (rqstp, ctxt); goto complete; } - ctxt =svc_rdma_next_recv_ctxt (&rdma_xprt->sc_rq_dto_q); -if (ctxt) -list_del (&ctxt->rc_list); -else +spin_unlock (&rdma_xprt->sc_rq_dto_lock); + + node =llist_del_first (&rdma_xprt->sc_rq_dto_q); +if (node) { + ctxt = llist_entry(node, struct svc_rdma_recv_ctxt, rc_node); + }else { + ctxt =NULL ;/* No new incoming requests, terminate the loop */ clear_bit (XPT_DATA , &xprt->xpt_flags); -spin_unlock (&rdma_xprt->sc_rq_dto_lock); + +/* +* If a completion arrived after llist_del_first but +* before clear_bit, the producer's set_bit would be +* cleared above. Recheck to close this race window. +*/ +if (!llist_empty (&rdma_xprt->sc_rq_dto_q)) +set_bit (XPT_DATA , &xprt->xpt_flags); + }/* Unblock the transport for the next receive */ svc_xprt_received (xprt);
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -1
@@ -173,7 +173,7 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,svc_xprt_init (net, &svc_rdma_class, &cma_xprt->sc_xprt, serv);INIT_LIST_HEAD (&cma_xprt->sc_accept_q); -INIT_LIST_HEAD (&cma_xprt->sc_rq_dto_q); +init_llist_head (&cma_xprt->sc_rq_dto_q);INIT_LIST_HEAD (&cma_xprt->sc_read_complete_q);init_llist_head (&cma_xprt->sc_send_ctxts);init_llist_head (&cma_xprt->sc_recv_ctxts);
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 8/15 svcrdma: Convert Read completion queue to use lock-free list
In reply to Chuck LeverFrom: Chuck Lever <[email protected]>
Extend the lock-free list conversion to sc_read_complete_q. This queue tracks receive contexts that have completed RDMA Read operations for handling Read chunks.
With both sc_rq_dto_q and sc_read_complete_q now using llist, the sc_rq_dto_lock spinlock is no longer needed and is removed. This eliminates all locking from the receive and Read completion paths.
Note that llist provides LIFO ordering rather than FIFO. For independent RPC requests this has no semantic impact.
--- include/linux/sunrpc/svc_rdma.h | 4 +-- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 34 +++++++++++------------- net/sunrpc/xprtrdma/svc_rdma_rw.c | 9 ++++--- net/sunrpc/xprtrdma/svc_rdma_transport.c | 5 +--- 4 files changed, 23 insertions(+), 29 deletions(-)
include/linux/sunrpc/svc_rdma.h +1 -3
@@ -110,8 +110,7 @@ struct svcxprt_rdma {u32 sc_pending_recvs____cacheline_aligned_in_smp ;u32 sc_recv_batch;struct llist_head sc_rq_dto_q; -struct list_head sc_read_complete_q; -spinlock_t sc_rq_dto_lock; +struct llist_head sc_read_complete_q;spinlock_t sc_lock;/* transport lock */
@@ -183,7 +182,6 @@ struct svc_rdma_chunk_ctxt {struct svc_rdma_recv_ctxt {struct llist_node rc_node; -struct list_head rc_list;struct ib_recv_wr rc_recv_wr;struct ib_cqe rc_cqe;struct rpc_rdma_cid rc_cid;
net/sunrpc/xprtrdma/svc_rdma_recvfrom.c +16 -18
@@ -108,13 +108,6 @@static void svc_rdma_wc_receive (struct ib_cq *cq,struct ib_wc *wc); -static inline struct svc_rdma_recv_ctxt * -svc_rdma_next_recv_ctxt (struct list_head *list) -{ - return list_first_entry_or_null(list, struct svc_rdma_recv_ctxt, - rc_list); -} -static struct svc_rdma_recv_ctxt *svc_rdma_recv_ctxt_alloc (struct svcxprt_rdma *rdma) {
@@ -385,14 +378,21 @@ static void svc_rdma_wc_receive(struct ib_cq *cq, struct ib_wc *wc)* svc_rdma_flush_recv_queues - Drain pending Receive work * @rdma: svcxprt_rdma being shut down * +* Called from svc_rdma_free() after ib_drain_qp() has blocked until +* completion queues are empty and flush_workqueue() has waited for +* pending work items. These preceding calls guarantee no concurrent +* producers (completion handlers) or consumers (svc_rdma_recvfrom) +* can be active, making unsynchronized llist_del_all() safe here. */ void svc_rdma_flush_recv_queues (struct svcxprt_rdma *rdma) {struct svc_rdma_recv_ctxt *ctxt;struct llist_node *node; -while ((ctxt =svc_rdma_next_recv_ctxt (&rdma->sc_read_complete_q))) { -list_del (&ctxt->rc_list); + node =llist_del_all (&rdma->sc_read_complete_q); +while (node) { + ctxt = llist_entry(node, struct svc_rdma_recv_ctxt, rc_node); + node = node->next;svc_rdma_recv_ctxt_put (rdma, ctxt); } node =llist_del_all (&rdma->sc_rq_dto_q);
@@ -945,17 +945,13 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp) rqstp->rq_xprt_ctxt =NULL ; -spin_lock (&rdma_xprt->sc_rq_dto_lock); - ctxt =svc_rdma_next_recv_ctxt (&rdma_xprt->sc_read_complete_q); -if (ctxt) { -list_del (&ctxt->rc_list); -spin_unlock (&rdma_xprt->sc_rq_dto_lock); + node =llist_del_first (&rdma_xprt->sc_read_complete_q); +if (node) { + ctxt = llist_entry(node, struct svc_rdma_recv_ctxt, rc_node);svc_xprt_received (xprt);svc_rdma_read_complete (rqstp, ctxt); goto complete; } -spin_unlock (&rdma_xprt->sc_rq_dto_lock); - node =llist_del_first (&rdma_xprt->sc_rq_dto_q);if (node) { ctxt = llist_entry(node, struct svc_rdma_recv_ctxt, rc_node);
@@ -967,9 +963,11 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp)/* * If a completion arrived after llist_del_first but * before clear_bit, the producer's set_bit would be -* cleared above. Recheck to close this race window. +* cleared above. Recheck both queues to close this +* race window. */ -if (!llist_empty (&rdma_xprt->sc_rq_dto_q)) +if (!llist_empty (&rdma_xprt->sc_rq_dto_q) || + !llist_empty (&rdma_xprt->sc_read_complete_q))set_bit (XPT_DATA , &xprt->xpt_flags); }
net/sunrpc/xprtrdma/svc_rdma_rw.c +5 -4
@@ -349,11 +349,12 @@ static void svc_rdma_wc_read_done(struct ib_cq *cq, struct ib_wc *wc)trace_svcrdma_wc_read (wc, &cc->cc_cid, ctxt->rc_readbytes, cc->cc_posttime); -spin_lock (&rdma->sc_rq_dto_lock); -list_add_tail (&ctxt->rc_list, &rdma->sc_read_complete_q); -/* the unlock pairs with the smp_rmb in svc_xprt_ready */ +llist_add (&ctxt->rc_node, &rdma->sc_read_complete_q); +/* +* The implicit barrier of llist_add's cmpxchg pairs with +* the smp_rmb in svc_xprt_ready. +*/ set_bit (XPT_DATA , &rdma->sc_xprt.xpt_flags); -spin_unlock (&rdma->sc_rq_dto_lock);svc_xprt_enqueue (&rdma->sc_xprt);return ;case IB_WC_WR_FLUSH_ERR :
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -4
@@ -164,7 +164,6 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv, {static struct lock_class_key svcrdma_rwctx_lock;static struct lock_class_key svcrdma_sctx_lock; -static struct lock_class_key svcrdma_dto_lock;struct svcxprt_rdma *cma_xprt; cma_xprt =kzalloc_node (sizeof (*cma_xprt),GFP_KERNEL , node);
@@ -174,15 +173,13 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,svc_xprt_init (net, &svc_rdma_class, &cma_xprt->sc_xprt, serv);INIT_LIST_HEAD (&cma_xprt->sc_accept_q);init_llist_head (&cma_xprt->sc_rq_dto_q); -INIT_LIST_HEAD (&cma_xprt->sc_read_complete_q); +init_llist_head (&cma_xprt->sc_read_complete_q);init_llist_head (&cma_xprt->sc_send_ctxts);init_llist_head (&cma_xprt->sc_recv_ctxts);init_llist_head (&cma_xprt->sc_rw_ctxts);init_waitqueue_head (&cma_xprt->sc_send_wait);spin_lock_init (&cma_xprt->sc_lock); -spin_lock_init (&cma_xprt->sc_rq_dto_lock); -lockdep_set_class (&cma_xprt->sc_rq_dto_lock, &svcrdma_dto_lock);spin_lock_init (&cma_xprt->sc_send_lock);lockdep_set_class (&cma_xprt->sc_send_lock, &svcrdma_sctx_lock);spin_lock_init (&cma_xprt->sc_rw_ctxt_lock);
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 9/15 svcrdma: Release write chunk resources without re-queuing
In reply to Chuck LeverFrom: Chuck Lever <[email protected]>
Each RDMA Send completion triggers a cascade of work items on the svcrdma_wq unbound workqueue:
ib_cq_poll_work (on ib_comp_wq, per-CPU)
-> svc_rdma_send_ctxt_put -> queue_work [work item 1]
-> svc_rdma_write_info_free -> queue_work [work item 2]Every transition through queue_work contends on the unbound pool's spinlock. Profiling an 8KB NFSv3 read/write workload over RDMA shows about 4% of total CPU cycles spent on this lock, with the cascading re-queue of write_info release contributing roughly 1%.
The initial queue_work in svc_rdma_send_ctxt_put is needed to move release work off the CQ completion context (which runs on a per-CPU bound workqueue). However, once executing on svcrdma_wq, there is no need to re-queue for each write_info structure. svc_rdma_reply_chunk_release already calls svc_rdma_cc_release inline, confirming these operations are safe in workqueue and nfsd thread context alike.
Release write chunk resources inline in svc_rdma_write_info_free, removing the intermediate svc_rdma_write_info_free_async work item and the wi_work field from struct svc_rdma_write_info.
--- include/linux/sunrpc/svc_rdma.h | 1 - net/sunrpc/xprtrdma/svc_rdma_rw.c | 13 ++----------- 2 files changed, 2 insertions(+), 12 deletions(-)
include/linux/sunrpc/svc_rdma.h +0 -1
@@ -232,7 +232,6 @@ struct svc_rdma_write_info {unsigned int wi_next_off;struct svc_rdma_chunk_ctxt wi_cc; -struct work_struct wi_work; };struct svc_rdma_send_ctxt {
net/sunrpc/xprtrdma/svc_rdma_rw.c +2 -11
@@ -215,19 +215,10 @@ svc_rdma_write_info_alloc(struct svcxprt_rdma *rdma,return info; } -static void svc_rdma_write_info_free_async (struct work_struct *work) -{ -struct svc_rdma_write_info *info; - - info = container_of(work, struct svc_rdma_write_info, wi_work); -svc_rdma_cc_release (info->wi_rdma, &info->wi_cc,DMA_TO_DEVICE ); -kfree (info); -} -static void svc_rdma_write_info_free (struct svc_rdma_write_info *info) { -INIT_WORK (&info->wi_work, svc_rdma_write_info_free_async); -queue_work (svcrdma_wq, &info->wi_work); +svc_rdma_cc_release (info->wi_rdma, &info->wi_cc,DMA_TO_DEVICE ); +kfree (info); }/**
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 10/15 svcrdma: Use per-transport kthread for send context release
In reply to Chuck LeverFrom: Chuck Lever <[email protected]>
Each RDMA Send completion queues a separate work item to the global svcrdma_wq (an unbound workqueue) to handle DMA unmapping and page release. Under load, many worker threads contend on the shared workqueue pool lock -- profiling an NFSv3 8KB read+write workload over RDMA shows ~2.6% of total CPU cycles spent in native_queued_spin_lock_slowpath on this lock.
The contention arises from three directions: CQ completion handlers acquiring the pool lock to enqueue work, a dozen unbound workers re-acquiring it after each work item completes, and XFS CIL flush callers hitting the same unbound pool lock.
Replace the workqueue with a per-transport kthread that drains a lock-free list. The CQ handler appends completed send contexts via llist_add() (a single cmpxchg) and wakes the kthread. The kthread collects all pending items with llist_del_all() (a single xchg) and releases them in a batch. Both operations are wait-free, eliminating the pool lock entirely.
This also removes the global svcrdma_wq workqueue, which has no remaining users.
--- include/linux/sunrpc/svc_rdma.h | 9 ++-- net/sunrpc/xprtrdma/svc_rdma.c | 18 +------ net/sunrpc/xprtrdma/svc_rdma_sendto.c | 62 +++++++++++++++++++++--- net/sunrpc/xprtrdma/svc_rdma_transport.c | 8 ++- 4 files changed, 69 insertions(+), 28 deletions(-)
include/linux/sunrpc/svc_rdma.h +6 -3
@@ -66,8 +66,6 @@ extern unsigned int svcrdma_ord;extern unsigned int svcrdma_max_requests;extern unsigned int svcrdma_max_bc_requests;extern unsigned int svcrdma_max_req_size; -extern struct workqueue_struct *svcrdma_wq; -extern struct percpu_counter svcrdma_stat_read;extern struct percpu_counter svcrdma_stat_recv;extern struct percpu_counter svcrdma_stat_sq_starve;
@@ -120,6 +118,10 @@ struct svcxprt_rdma {struct llist_head sc_recv_ctxts; +struct llist_head sc_send_release_list; +wait_queue_head_t sc_release_wait; +struct task_struct *sc_release_task; +atomic_t sc_completion_ids; };/* sc_flags */
@@ -237,7 +239,6 @@ struct svc_rdma_write_info {struct svc_rdma_send_ctxt {struct llist_node sc_node;struct rpc_rdma_cid sc_cid; -struct work_struct sc_work;struct svcxprt_rdma *sc_rdma;struct ib_send_wr sc_send_wr;
@@ -301,6 +302,8 @@ extern int svc_rdma_process_read_list(struct svcxprt_rdma *rdma,/* svc_rdma_sendto.c */ extern void svc_rdma_send_ctxts_destroy (struct svcxprt_rdma *rdma); +extern int svc_rdma_start_release_thread (struct svcxprt_rdma *rdma); +extern void svc_rdma_stop_release_thread (struct svcxprt_rdma *rdma);extern struct svc_rdma_send_ctxt *svc_rdma_send_ctxt_get (struct svcxprt_rdma *rdma); extern void svc_rdma_send_ctxt_put(struct svcxprt_rdma *rdma,
net/sunrpc/xprtrdma/svc_rdma.c +1 -17
@@ -264,38 +264,22 @@ static int svc_rdma_proc_init(void)return rc; } -struct workqueue_struct *svcrdma_wq; -void svc_rdma_cleanup (void ) {svc_unreg_xprt_class (&svc_rdma_class);svc_rdma_proc_cleanup (); -if (svcrdma_wq) { -struct workqueue_struct *wq = svcrdma_wq; - - svcrdma_wq =NULL ; -destroy_workqueue (wq); - }dprintk ("SVCRDMA Module Removed, deregister RPC RDMA transport\n" ); }int svc_rdma_init (void ) { -struct workqueue_struct *wq;int rc; - wq =alloc_workqueue ("svcrdma" ,WQ_UNBOUND ,0 ); -if (!wq) -return -ENOMEM ; - rc =svc_rdma_proc_init (); -if (rc) { -destroy_workqueue (wq); +if (rc)return rc; - } - svcrdma_wq = wq;svc_reg_xprt_class (&svc_rdma_class);dprintk ("SVCRDMA Module Init, register RPC RDMA transport\n" );
net/sunrpc/xprtrdma/svc_rdma_sendto.c +55 -7
86 lines not on this page · raw message
net/sunrpc/xprtrdma/svc_rdma_transport.c +7 -1
@@ -177,7 +177,9 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,init_llist_head (&cma_xprt->sc_send_ctxts);init_llist_head (&cma_xprt->sc_recv_ctxts);init_llist_head (&cma_xprt->sc_rw_ctxts); +init_llist_head (&cma_xprt->sc_send_release_list);init_waitqueue_head (&cma_xprt->sc_send_wait); +init_waitqueue_head (&cma_xprt->sc_release_wait);spin_lock_init (&cma_xprt->sc_lock);spin_lock_init (&cma_xprt->sc_send_lock);
@@ -526,6 +528,10 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt)if (!svc_rdma_post_recvs (newxprt)) goto errout; + ret =svc_rdma_start_release_thread (newxprt); +if (ret) + goto errout; +/* Construct RDMA-CM private message */ pmsg.cp_magic = rpcrdma_cmp_magic; pmsg.cp_version =RPCRDMA_CMP_VERSION ;
@@ -605,7 +611,7 @@ static void svc_rdma_free(struct svc_xprt *xprt)/* This blocks until the Completion Queues are empty */ if (rdma->sc_qp && !IS_ERR (rdma->sc_qp))ib_drain_qp (rdma->sc_qp); -flush_workqueue (svcrdma_wq); +svc_rdma_stop_release_thread (rdma);svc_rdma_flush_recv_queues (rdma);
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 11/15 svcrdma: Use watermark-based Receive Queue replenishment
In reply to Chuck LeverFrom: Chuck Lever <[email protected]>
The current Receive posting strategy posts a small fixed batch of Receives on every completion when the queue depth drops below the maximum. At high message rates this results in frequent ib_post_recv() calls, each incurring doorbell overhead.
The Receive Queue is now provisioned with twice the negotiated credit limit (sc_max_requests). Replenishment is triggered when the number of posted Receives drops below the credit limit (the low watermark), posting enough Receives to refill the queue to capacity.
For a typical configuration with a credit limit of 128:
- Receive Queue depth: 256
- Low watermark: 128 (replenish when half consumed)
- Batch size: ~128 Receives per posting
Tying the watermark to the credit limit rather than a percentage of queue capacity ensures adequate buffering regardless of the configured credit limit. Even with a small credit limit, at least one full credit window remains posted, guaranteeing forward progress.
--- include/linux/sunrpc/svc_rdma.h | 22 ++++++++++++- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 41 ++++++++++++++++-------- net/sunrpc/xprtrdma/svc_rdma_transport.c | 11 +++---- 3 files changed, 54 insertions(+), 20 deletions(-)
include/linux/sunrpc/svc_rdma.h +21 -1
@@ -106,7 +106,6 @@ struct svcxprt_rdma {/* Receive path */ u32 sc_pending_recvs____cacheline_aligned_in_smp ; -u32 sc_recv_batch;struct llist_head sc_rq_dto_q;struct llist_head sc_read_complete_q;
@@ -143,6 +142,27 @@ enum { RPCRDMA_MAX_BC_REQUESTS = 2, }; +/* +* Receive Queue provisioning constants for watermark-based replenishment. +* +* The Receive Queue is sized at twice the credit limit to enable +* batched posting that reduces doorbell overhead. Replenishment +* occurs when posted receives drop below the credit limit (the +* low watermark), refilling to full capacity. +*/ +enum { +/* Queue depth = sc_max_requests * multiplier */ +SVCRDMA_RQ_DEPTH_MULT =2 , + +/* Total recv_ctxt pool = sc_max_requests * multiplier +* (RQ_DEPTH_MULT for posted receives + 1 for RPCs in process) +*/ +SVCRDMA_RECV_CTXT_MULT =3 , + +/* Overhead entries in RQ: sc_max_bc_requests + drain sentinel */ +SVCRDMA_RQ_OVERHEAD =3 , +}; +#define RPCSVC_MAXPAYLOAD_RDMA RPCSVC_MAXPAYLOAD/**
net/sunrpc/xprtrdma/svc_rdma_recvfrom.c +28 -13
63 lines not on this page · raw message
net/sunrpc/xprtrdma/svc_rdma_transport.c +5 -6
@@ -439,7 +439,6 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt) newxprt->sc_max_req_size = svcrdma_max_req_size; newxprt->sc_max_requests = svcrdma_max_requests; newxprt->sc_max_bc_requests = svcrdma_max_bc_requests; - newxprt->sc_recv_batch =RPCRDMA_MAX_RECV_BATCH ; newxprt->sc_fc_credits =cpu_to_be32 (newxprt->sc_max_requests);/* Qualify the transport's resource defaults with the
@@ -452,12 +451,12 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt) newxprt->sc_max_send_sges += (svcrdma_max_req_size /PAGE_SIZE ) +1 ;if (newxprt->sc_max_send_sges > dev->attrs.max_send_sge) newxprt->sc_max_send_sges = dev->attrs.max_send_sge; - rq_depth = newxprt->sc_max_requests + newxprt->sc_max_bc_requests + - newxprt->sc_recv_batch +1 /* drain */ ; + rq_depth = (newxprt->sc_max_requests *SVCRDMA_RQ_DEPTH_MULT ) + + newxprt->sc_max_bc_requests +1 /* drain */ ;if (rq_depth > dev->attrs.max_qp_wr) { rq_depth = dev->attrs.max_qp_wr; - newxprt->sc_recv_batch =1 ; - newxprt->sc_max_requests = rq_depth -2 ; + newxprt->sc_max_requests = + (rq_depth -SVCRDMA_RQ_OVERHEAD ) /SVCRDMA_RQ_DEPTH_MULT ; newxprt->sc_max_bc_requests =2 ; }
@@ -465,7 +464,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt)*/ maxpayload =min (xprt->xpt_server->sv_max_payload,RPCSVC_MAXPAYLOAD_RDMA ); - ctxts = newxprt->sc_max_requests *3 * + ctxts = newxprt->sc_max_requests *SVCRDMA_RECV_CTXT_MULT *rdma_rw_mr_factor (dev, newxprt->sc_port_num, maxpayload >>PAGE_SHIFT );
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 12/15 svcrdma: Add per-recv_ctxt chunk context cache
In reply to Chuck LeverFrom: Chuck Lever <[email protected]>
Parsed chunk list (PCL) processing currently allocates a new svc_rdma_chunk structure via kmalloc for each chunk in every incoming RPC. These allocations add overhead to the receive path.
Introduce a per-recv_ctxt single-entry cache. Over 99% of RPC Calls that specify RPC/RDMA chunks provide only a single chunk, so a single cached chunk handles the common case. Chunks with up to SVC_RDMA_CHUNK_SEGMAX (4) segments are eligible for caching; larger chunks fall back to dynamic allocation.
Using per-recv_ctxt caching instead of a per-transport pool avoids the need for locking or atomic operations, since a recv_ctxt is used by only one thread at a time.
--- include/linux/sunrpc/svc_rdma.h | 2 + include/linux/sunrpc/svc_rdma_pcl.h | 12 +++++- net/sunrpc/xprtrdma/svc_rdma_pcl.c | 55 +++++++++++++++++++++---- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 10 +++-- 4 files changed, 67 insertions(+), 12 deletions(-)
include/linux/sunrpc/svc_rdma.h +2 -0
@@ -204,6 +204,8 @@ struct svc_rdma_chunk_ctxt {struct svc_rdma_recv_ctxt {struct llist_node rc_node; +struct svcxprt_rdma *rc_rdma; +struct svc_rdma_chunk *rc_chunk_cache;struct ib_recv_wr rc_recv_wr;struct ib_cqe rc_cqe;struct rpc_rdma_cid rc_cid;
include/linux/sunrpc/svc_rdma_pcl.h +11 -1
@@ -22,6 +22,7 @@ struct svc_rdma_chunk {u32 ch_payload_length;u32 ch_segcount; +u32 ch_segmax;struct svc_rdma_segment ch_segments[]; };
@@ -114,7 +115,16 @@ pcl_chunk_end_offset(const struct svc_rdma_chunk *chunk)struct svc_rdma_recv_ctxt ; -extern void pcl_free (struct svc_rdma_pcl *pcl); +/* +* Cached chunks have capacity for this many segments. +* Typical clients can register up to 120KB per segment, so 4 +* segments covers most NFS I/O operations. Larger chunks fall +* back to kmalloc. +*/ +#define SVC_RDMA_CHUNK_SEGMAX 4 + +extern void pcl_free (struct svc_rdma_recv_ctxt *rctxt, +struct svc_rdma_pcl *pcl);extern bool pcl_alloc_call (struct svc_rdma_recv_ctxt *rctxt,__be32 *p);extern bool pcl_alloc_read (struct svc_rdma_recv_ctxt *rctxt,__be32 *p); extern bool pcl_alloc_write(struct svc_rdma_recv_ctxt *rctxt,
net/sunrpc/xprtrdma/svc_rdma_pcl.c +48 -7
99 lines not on this page · raw message
net/sunrpc/xprtrdma/svc_rdma_recvfrom.c +6 -4
@@ -122,6 +122,7 @@ svc_rdma_recv_ctxt_alloc(struct svcxprt_rdma *rdma) GFP_KERNEL, ibdev_to_node(device));if (!ctxt) goto fail0; + ctxt->rc_rdma = rdma; ctxt->rc_maxpages = pages; buffer =kmalloc_node (rdma->sc_max_req_size,GFP_KERNEL ,ibdev_to_node (device));
@@ -161,6 +162,7 @@ svc_rdma_recv_ctxt_alloc(struct svcxprt_rdma *rdma)static void svc_rdma_recv_ctxt_destroy (struct svcxprt_rdma *rdma,struct svc_rdma_recv_ctxt *ctxt) { +kfree (ctxt->rc_chunk_cache);ib_dma_unmap_single (rdma->sc_cm_id->device, ctxt->rc_recv_sge.addr, ctxt->rc_recv_sge.length,DMA_FROM_DEVICE );kfree (ctxt->rc_recv_buf);
@@ -219,10 +221,10 @@ void svc_rdma_recv_ctxt_put(struct svcxprt_rdma *rdma,*/ release_pages (ctxt->rc_pages, ctxt->rc_page_count); -pcl_free (&ctxt->rc_call_pcl); -pcl_free (&ctxt->rc_read_pcl); -pcl_free (&ctxt->rc_write_pcl); -pcl_free (&ctxt->rc_reply_pcl); +pcl_free (ctxt, &ctxt->rc_call_pcl); +pcl_free (ctxt, &ctxt->rc_read_pcl); +pcl_free (ctxt, &ctxt->rc_write_pcl); +pcl_free (ctxt, &ctxt->rc_reply_pcl);llist_add (&ctxt->rc_node, &rdma->sc_recv_ctxts); }
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 13/15 svcrdma: clear XPT_DATA on sc_read_complete_q consumption
In reply to Chuck LeverFrom: Chuck Lever <[email protected]>
svc_rdma_wc_read_done() sets XPT_DATA when adding a completed RDMA Read context to sc_read_complete_q. The consumer in svc_rdma_recvfrom() takes the context but leaves XPT_DATA set. The subsequent svc_xprt_received() clears XPT_BUSY and re-enqueues the transport; because XPT_DATA remains set, a second thread awakens. That thread finds both queues empty, accomplishes nothing, and releases its slot and reservation.
Trace data from a 256KB NFSv3 WRITE workload over RDMA shows approximately 14 enqueue attempts per RPC, with 62% returning immediately due to no pending data. The majority originate from this spurious dispatch path.
After clearing XPT_DATA to acknowledge consumption, the XPT_DATA state must be recomputed from both queue states. A concurrent producer may call llist_add and then set_bit(XPT_DATA) between this consumer's llist_del_first and the clear_bit, causing clear_bit to erase the producer's signal. An smp_mb__after_atomic() barrier after clear_bit pairs with the implicit barrier in each producer's llist_add cmpxchg, ensuring llist_empty rechecks observe any add whose set_bit was erased. This barrier requirement applies at both call sites: the new sc_read_complete_q path and the pre-existing sc_rq_dto_q "both queues empty" path.
A new helper svc_rdma_update_xpt_data() centralizes this clear/barrier/recheck/set pattern to ensure both locations maintain the required memory ordering.
--- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 33 ++++++++++++++++--------- 1 file changed, 22 insertions(+), 11 deletions(-)
net/sunrpc/xprtrdma/svc_rdma_recvfrom.c +22 -11
@@ -917,6 +917,25 @@ static noinline void svc_rdma_read_complete(struct svc_rqst *rqstp,trace_svcrdma_read_finished (&ctxt->rc_cid); } +/* +* Recompute XPT_DATA from queue state after consuming a completion. A +* concurrent producer may have called llist_add and then set_bit(XPT_DATA) +* between this consumer's llist_del_first and the clear_bit below, causing +* clear_bit to erase the producer's signal. The barrier pairs with the +* implicit barrier in each producer's llist_add so that the llist_empty +* rechecks observe any add whose set_bit was erased. +*/ +static void svc_rdma_update_xpt_data (struct svcxprt_rdma *rdma) +{ +struct svc_xprt *xprt = &rdma->sc_xprt; + +clear_bit (XPT_DATA , &xprt->xpt_flags); +smp_mb__after_atomic (); +if (!llist_empty (&rdma->sc_rq_dto_q) || + !llist_empty (&rdma->sc_read_complete_q)) +set_bit (XPT_DATA , &xprt->xpt_flags); +} +/** * svc_rdma_recvfrom - Receive an RPC call * @rqstp: request structure into which to receive an RPC Call
@@ -965,6 +984,8 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp) node =llist_del_first (&rdma_xprt->sc_read_complete_q);if (node) { ctxt = llist_entry(node, struct svc_rdma_recv_ctxt, rc_node); + +svc_rdma_update_xpt_data (rdma_xprt);svc_xprt_received (xprt);svc_rdma_read_complete (rqstp, ctxt); goto complete;
@@ -975,17 +996,7 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp) }else { ctxt =NULL ;/* No new incoming requests, terminate the loop */ -clear_bit (XPT_DATA , &xprt->xpt_flags); - -/* -* If a completion arrived after llist_del_first but -* before clear_bit, the producer's set_bit would be -* cleared above. Recheck both queues to close this -* race window. -*/ -if (!llist_empty (&rdma_xprt->sc_rq_dto_q) || - !llist_empty (&rdma_xprt->sc_read_complete_q)) -set_bit (XPT_DATA , &xprt->xpt_flags); +svc_rdma_update_xpt_data (rdma_xprt); }/* Unblock the transport for the next receive */
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 14/15 svcrdma: retry when receive queues drain transiently
In reply to Chuck LeverFrom: Chuck Lever <[email protected]>
When svc_rdma_recvfrom finds both sc_read_complete_q and sc_rq_dto_q empty, svc_rdma_update_xpt_data clears XPT_DATA, executes a barrier, and rechecks the queues. If a completion arrived between the llist_del_first and the recheck, XPT_DATA is re-set, but recvfrom returns zero regardless. The thread then traverses the full svc_recv cycle -- page allocation, dequeue, recvfrom, release -- only to find the item that was already available at the time of the recheck.
Trace data from a 256KB NFSv3 workload over RDMA shows 267,848 of 464,355 transport dequeues (57.7%) are these empty bounces. Each bounce costs roughly 37 us. During the READ phase, empty bounces consume 8.6% of thread capacity and inflate inter-RPC gaps by an average of 87 us.
The calling thread holds XPT_BUSY for the duration, so no other consumer can drain the queue between the recheck and the retry. A retry is therefore guaranteed to find data on its first iteration.
--- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
net/sunrpc/xprtrdma/svc_rdma_recvfrom.c +11 -1
@@ -981,6 +981,7 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp) rqstp->rq_xprt_ctxt =NULL ; +retry: node =llist_del_first (&rdma_xprt->sc_read_complete_q);if (node) { ctxt = llist_entry(node, struct svc_rdma_recv_ctxt, rc_node);
@@ -995,8 +996,17 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp) ctxt = llist_entry(node, struct svc_rdma_recv_ctxt, rc_node); }else { ctxt =NULL ; -/* No new incoming requests, terminate the loop */ svc_rdma_update_xpt_data (rdma_xprt); +/* +* A completion may have arrived between the +* llist_del_first above and the queue recheck +* inside svc_rdma_update_xpt_data. This thread +* holds XPT_BUSY, preventing any other consumer +* from draining the queue in the meantime. +* Retry to avoid a full svc_recv round-trip. +*/ +if (test_bit (XPT_DATA , &xprt->xpt_flags)) + goto retry; }/* Unblock the transport for the next receive */
Signature · 1 line
2.52.0
Chuck Lever
maintainerRFC PATCH 15/15 svcrdma: clear XPT_DATA on sc_rq_dto_q consumption
In reply to Chuck LeverFrom: Chuck Lever <[email protected]>
svc_rdma_wc_receive() sets XPT_DATA when adding a completed Receive to sc_rq_dto_q. When svc_rdma_recvfrom() consumes the item from sc_rq_dto_q, XPT_DATA is left set. The subsequent svc_xprt_received() clears XPT_BUSY and re-enqueues the transport; because stale XPT_DATA remains set, svc_xprt_enqueue() dispatches a second thread. That thread finds both queues empty, accomplishes nothing, and returns zero.
Trace data from a 256KB NFSv3 workload over RDMA shows 172,280 of 467,171 transport dequeues (36.9%) are these spurious dispatches. The READ phase averages 1.99 dequeues per RPC (expected 1.0) and the WRITE phase averages 2.77 (expected 2.0). Each wasted cycle traverses svc_alloc_arg, svc_thread_wait_for_work, svc_rdma_recvfrom, and svc_xprt_release before the thread can accept new work.
Add svc_rdma_update_xpt_data() on the sc_rq_dto_q success path, matching the existing call on the sc_read_complete_q path added by commit 6807f36a39b7 ("svcrdma: clear XPT_DATA on sc_read_complete_q consumption"). The same barrier semantics apply: the clear/recheck pattern in svc_rdma_update_xpt_data() ensures a concurrent producer's llist_add + set_bit is not lost.
--- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 1 + 1 file changed, 1 insertion(+)
net/sunrpc/xprtrdma/svc_rdma_recvfrom.c +1 -0
@@ -994,6 +994,7 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp) node =llist_del_first (&rdma_xprt->sc_rq_dto_q);if (node) { ctxt = llist_entry(node, struct svc_rdma_recv_ctxt, rc_node); +svc_rdma_update_xpt_data (rdma_xprt); }else { ctxt =NULL ;svc_rdma_update_xpt_data (rdma_xprt);
Signature · 1 line
2.52.0