Skip to content

svcrdma performance scalability enhancements

Chuck Lever linux-nfs 2 versions – New

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest18 patches 0 repliesNew
  2. v1 RFC15 patches 0 repliesSuperseded

Changes since RFC from the cover letter

- Rebase on v7.0-rc1
- Address thundering herd in SQ fairness implementation
- Release Send contexts during xpo_release_ctxt
- Skip xpt_reserved accounting for RDMA
- Close several races

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/18svcrdma: Add fair queuing for Send Queue accessinclude/linux/sunrpc/svc_rdma.h net/sunrpc/xprtrdma/svc_rdma_rw.c net/sunrpc/xprtrdma/svc_rdma_sendto.c +1 more+145 -680 replies
2/18svcrdma: Clean up use of rdma->sc_pd->device in Receive pathsnet/sunrpc/xprtrdma/svc_rdma_recvfrom.c+7 -60 replies
3/18svcrdma: Clean up use of rdma->sc_pd->devicenet/sunrpc/xprtrdma/svc_rdma_sendto.c+9 -90 replies
4/18svcrdma: Add Write chunk WRs to the RPC's Send WR chaininclude/linux/sunrpc/svc_rdma.h net/sunrpc/xprtrdma/svc_rdma_rw.c net/sunrpc/xprtrdma/svc_rdma_sendto.c+91 -260 replies
5/18svcrdma: Factor out WR chain linking into helpernet/sunrpc/xprtrdma/svc_rdma_rw.c+28 -390 replies
6/18svcrdma: Reduce false sharing in struct svcxprt_rdmainclude/linux/sunrpc/svc_rdma.h+36 -120 replies
7/18svcrdma: Use lock-free list for Receive Queue trackinginclude/linux/sunrpc/svc_rdma.h net/sunrpc/xprtrdma/svc_rdma_recvfrom.c net/sunrpc/xprtrdma/svc_rdma_transport.c+40 -130 replies
8/18svcrdma: Convert Read completion queue to use lock-free listinclude/linux/sunrpc/svc_rdma.h net/sunrpc/xprtrdma/svc_rdma_recvfrom.c net/sunrpc/xprtrdma/svc_rdma_rw.c +1 more+24 -390 replies
9/18svcrdma: Release write chunk resources without re-queuinginclude/linux/sunrpc/svc_rdma.h net/sunrpc/xprtrdma/svc_rdma_rw.c+2 -120 replies
10/18svcrdma: Defer send context release to xpo_release_ctxtinclude/linux/sunrpc/svc_rdma.h net/sunrpc/xprtrdma/svc_rdma.c net/sunrpc/xprtrdma/svc_rdma_recvfrom.c +2 more+59 -500 replies
11/18svcrdma: Use watermark-based Receive Queue replenishmentinclude/linux/sunrpc/svc_rdma.h net/sunrpc/xprtrdma/svc_rdma_recvfrom.c net/sunrpc/xprtrdma/svc_rdma_transport.c+56 -200 replies
12/18svcrdma: Add per-recv_ctxt chunk context cacheinclude/linux/sunrpc/svc_rdma.h include/linux/sunrpc/svc_rdma_pcl.h net/sunrpc/xprtrdma/svc_rdma_pcl.c +1 more+65 -110 replies
13/18svcrdma: clear XPT_DATA on sc_read_complete_q consumptionnet/sunrpc/xprtrdma/svc_rdma_recvfrom.c+22 -110 replies
14/18svcrdma: retry when receive queues drain transientlynet/sunrpc/xprtrdma/svc_rdma_recvfrom.c+13 -10 replies
15/18svcrdma: clear XPT_DATA on sc_rq_dto_q consumptionnet/sunrpc/xprtrdma/svc_rdma_recvfrom.c+1 -00 replies
16/18sunrpc: skip svc_xprt_enqueue when no work is pendingnet/sunrpc/svc_xprt.c+20 -50 replies
17/18sunrpc: skip svc_xprt_enqueue in svc_xprt_received when idlenet/sunrpc/svc_xprt.c+13 -10 replies
18/18sunrpc: Skip xpt_reserved accounting for non-UDP transportsinclude/linux/sunrpc/svc_xprt.h net/sunrpc/svc_xprt.c net/sunrpc/svcsock.c+16 -90 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH v2 00/18] svcrdma performance scalability enhancements

From: Chuck Lever <[email protected]>

When considering NFS I/O throughput and latency, typically the RPC transport is not the primary bottleneck. The CPU cost of the RPC/RDMA transport is insignificant in comparison to other resource utilization on the server.

However, when considered from a scalability point of view, the incremental costs of things like memory-per-connection and interrupt rate or doorbell rate per connection do add up.

The following series lowers the per-connection resource utilization of svcrdma in several areas. The main benefits are lower lock contention, lower interrupt and doorbell rate per RPC, and less CPU cache theft.

Profiling an 8KB NFSv3 read/write workload over RDMA identifies where overhead accumulates as connection count grows. Roughly 4% of total CPU time goes to contention on the svcrdma_wq unbound workqueue pool lock, driven by cascading work item re-queues through the Send completion path. Receive completion processing acquires a per-transport spinlock on every inbound message. Doorbell and completion event counts scale with Write chunk usage. svc_alloc_arg() scans ~259 rq_pages slots per receive even when only a few pages need replacement.

Three strategies recur throughout this series.

Lock-free lists replace spinlock-protected queues on the hottest paths. The receive completion queue, Read completion queue, and send context release path all convert to llist, eliminating producer-side locking. The global svcrdma_wq workqueue -- the single largest contention source -- is replaced by per-transport kthreads that drain completed send contexts from an llist in batches. The intermediate re-queue for write chunk resource release is thus removed as well. Those resources are now freed inline during send context teardown.

Work Request chaining reduces per-RPC doorbell and completion rates. Write chunk RDMA Write WRs are linked onto the Reply Send WR chain, so a single ib_post_send() covers both operations with one completion event. Receive Queue posting switches from small fixed batches to watermark-triggered bulk replenishment, provisioned at twice the negotiated credit limit. Ticket-based fair queuing for Send Queue slot allocation prevents starvation when the SQ fills under concurrent use.

Per-object caching and cache line separation reduce allocation cost and cross-CPU invalidation traffic. Each recv_ctxt includes a single-entry svc_rdma_chunk cache, covering the >99% common case without kmalloc. Cache line annotations on struct svcxprt_rdma place the Send context cache, R/W context cache, and SQ availability counter in separate cache lines.

XPT_DATA flag handling upon sc_read_complete_q consumption is corrected to clear and recompute the flag. Trace data from a 256KB write workload shows ~14 transport enqueue attempts per RPC; in 62% of cases, no work is pending. Clearing the flag on consumption eliminates the majority of these spurious dispatches.

Base commit: v7.0-rc1
URL: https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/log/?h=svcrdma-next

---

Changes since RFC:
- Rebase on v7.0-rc1
- Address thundering herd in SQ fairness implementation
- Release Send contexts during xpo_release_ctxt
- Skip xpt_reserved accounting for RDMA
- Close several races

Chuck Lever (18):
svcrdma: Add fair queuing for Send Queue access
svcrdma: Clean up use of rdma->sc_pd->device in Receive paths
svcrdma: Clean up use of rdma->sc_pd->device
svcrdma: Add Write chunk WRs to the RPC's Send WR chain
svcrdma: Factor out WR chain linking into helper
svcrdma: Reduce false sharing in struct svcxprt_rdma
svcrdma: Use lock-free list for Receive Queue tracking
svcrdma: Convert Read completion queue to use lock-free list
svcrdma: Release write chunk resources without re-queuing
svcrdma: Defer send context release to xpo_release_ctxt
svcrdma: Use watermark-based Receive Queue replenishment
svcrdma: Add per-recv_ctxt chunk context cache
svcrdma: clear XPT_DATA on sc_read_complete_q consumption
svcrdma: retry when receive queues drain transiently
svcrdma: clear XPT_DATA on sc_rq_dto_q consumption
sunrpc: skip svc_xprt_enqueue when no work is pending
sunrpc: skip svc_xprt_enqueue in svc_xprt_received when idle
sunrpc: Skip xpt_reserved accounting for non-UDP transports

 include/linux/sunrpc/svc_rdma.h          | 103 +++++++--
 include/linux/sunrpc/svc_rdma_pcl.h      |  12 +-
 include/linux/sunrpc/svc_xprt.h          |   2 +
 net/sunrpc/svc_xprt.c                    |  55 +++--
 net/sunrpc/svcsock.c                     |   1 +
 net/sunrpc/xprtrdma/svc_rdma.c           |  18 +-
 net/sunrpc/xprtrdma/svc_rdma_pcl.c       |  52 ++++-
 net/sunrpc/xprtrdma/svc_rdma_recvfrom.c  | 156 +++++++++-----
 net/sunrpc/xprtrdma/svc_rdma_rw.c        | 165 ++++++++-------
 net/sunrpc/xprtrdma/svc_rdma_sendto.c    | 256 ++++++++++++++++-------
 net/sunrpc/xprtrdma/svc_rdma_transport.c |  29 +--
 11 files changed, 582 insertions(+), 267 deletions(-)
Signature · 1 line
2.53.0