RDMA/rw: Fix MR pool exhaustion in bvec RDMA READ path
Subsystems (4)
Chuck Lever
maintainerPATCH 1/1 RDMA/rw: Fix MR pool exhaustion in bvec RDMA READ path
From: Chuck Lever <[email protected]>
When IOVA-based DMA mapping is unavailable (eg IOMMU passthrough mode), rdma_rw_ctx_init_bvec() falls back to checking rdma_rw_io_needs_mr() with the raw bvec count. Unlike the scatterlist path in rdma_rw_ctx_init(), which passes a post-DMA-mapping entry count that reflects coalescing of physically contiguous pages, the bvec path passes the pre-mapping page count. This overstates the number of DMA entries, causing every multi-bvec RDMA READ to consume an MR from the QP's pool.
Under NFS WRITE workloads the server performs RDMA READs to pull data from the client. With the inflated MR demand, the pool is rapidly exhausted, ib_mr_pool_get() returns NULL, and rdma_rw_init_one_mr() returns -EAGAIN. svcrdma treats this as a DMA mapping failure, closes the connection, and the client reconnects -- producing a cycle of 71% RPC retransmissions and ~100 reconnections per test run. RDMA WRITEs (NFS READ direction) are unaffected because DMA_TO_DEVICE never triggers the max_sgl_rd check.
--- drivers/infiniband/core/rw.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
drivers/infiniband/core/rw.c +8 -7
@@ -686,14 +686,15 @@ int rdma_rw_ctx_init_bvec(struct rdma_rw_ctx *ctx, struct ib_qp *qp,return ret;/* -* IOVA mapping not available. Check if MR registration provides -* better performance than multiple SGE entries. +* IOVA not available; map each bvec individually. Do not +* check max_sgl_rd here: nr_bvec is a raw page count that +* overstates DMA entry demand and exhausts the MR pool. +* +* TODO: A bulk DMA mapping API for bvecs analogous to +* dma_map_sgtable() would provide a proper post-DMA- +* coalescing segment count here, enabling the map_wrs +* path in more cases. */ -if (rdma_rw_io_needs_mr (dev, port_num, dir, nr_bvec)) -return rdma_rw_init_mr_wrs_bvec (ctx, qp, port_num, bvecs, - nr_bvec, &iter, remote_addr, - rkey, dir); -return rdma_rw_init_map_wrs_bvec (ctx, qp, bvecs, nr_bvec, &iter, remote_addr, rkey, dir); }
Signature · 1 line
2.53.0
Quoted 9 lines · Chuck Lever
Under NFS WRITE workloads the server performs RDMA READs to pull data from the client. With the inflated MR demand, the pool is rapidly exhausted,
ib_mr_pool_get()returns NULL, andrdma_rw_init_one_mr()returns-EAGAIN. svcrdma treats this as a DMA mapping failure, closes the connection, and the client reconnects -- producing a cycle of 71% RPC retransmissions and ~100 reconnections per test run. RDMA WRITEs (NFS READ direction) are unaffected becauseDMA_TO_DEVICEnever triggers themax_sgl_rdcheck.
So this changelog extensively describes the problem, but it doesn't actually say how you fix it.
drivers/infiniband/core/rw.c int rdma_rw_ctx_init_bvec(struct rdma_rw_ctx *ctx, struct ib_qp *qp, · line 691
Quoted 4 lines
drivers/infiniband/core/rw.c +3
-0/* +* IOVA not available; map each bvec individually. Do not +* check max_sgl_rd here: nr_bvec is a raw page count that +* overstates DMA entry demand and exhausts the MR pool.
It fails to explain why that is fine?
drivers/infiniband/core/rw.c int rdma_rw_ctx_init_bvec(struct rdma_rw_ctx *ctx, struct ib_qp *qp, · line 696
Quoted 5 lines
drivers/infiniband/core/rw.c +5
-0+* +* TODO: A bulk DMA mapping API for bvecs analogous to +* dma_map_sgtable() would provide a proper post-DMA- +* coalescing segment count here, enabling the map_wrs +* path in more cases.
This isn't really something the DMA layer can easily do without getting as inefficient as the sgtable based path. What the block layer does here is to simply keep a higher level count of merged segments. The other option would be to not create multiple bvecs for continguous regions, which is what modern file system do in general, and why the above block layer nr_phys_segments based optimization isn't actually used all that much these days.
Why can't NFS send a single bvec for contiguous ranges?
drivers/infiniband/core/rw.c int rdma_rw_ctx_init_bvec(struct rdma_rw_ctx *ctx, struct ib_qp *qp, · line 699
Quoted 7 lines
drivers/infiniband/core/rw.c +0
-5-if (rdma_rw_io_needs_mr (dev, port_num, dir, nr_bvec)) -return rdma_rw_init_mr_wrs_bvec (ctx, qp, port_num, bvecs, - nr_bvec, &iter, remote_addr, - rkey, dir); -return rdma_rw_init_map_wrs_bvec (ctx, qp, bvecs, nr_bvec, &iter, remote_addr, rkey, dir);
First I thought this breaks iWarp reads, but they are handled earlier, which might have been usefu in the commit log.
Chuck Lever
maintainerQuoted 11 lines · Christoph Hellwig
Quoted 9 lines · Chuck Lever
Under NFS WRITE workloads the server performs RDMA READs to pull data from the client. With the inflated MR demand, the pool is rapidly exhausted,
ib_mr_pool_get()returns NULL, andrdma_rw_init_one_mr()returns-EAGAIN. svcrdma treats this as a DMA mapping failure, closes the connection, and the client reconnects -- producing a cycle of 71% RPC retransmissions and ~100 reconnections per test run. RDMA WRITEs (NFS READ direction) are unaffected becauseDMA_TO_DEVICEnever triggers themax_sgl_rdcheck.
Quoted 2 lines
So this changelog extensively describes the problem, but it doesn't actually say how you fix it.
I didn't want to waste everyone's time, but I can add that.
Quoted 6 lines
drivers/infiniband/core/rw.c int rdma_rw_ctx_init_bvec(struct rdma_rw_ctx *ctx, struct ib_qp *qp, · line 696
Quoted 5 lines
drivers/infiniband/core/rw.c +5
-0+* +* TODO: A bulk DMA mapping API for bvecs analogous to +* dma_map_sgtable() would provide a proper post-DMA- +* coalescing segment count here, enabling the map_wrs +* path in more cases.
Quoted 7 lines
This isn't really something the DMA layer can easily do without getting as inefficient as the sgtable based path. What the block layer does here is to simply keep a higher level count of merged segments. The other option would be to not create multiple bvecs for continguous regions, which is what modern file system do in general, and why the above block layer
nr_phys_segmentsbased optimization isn't actually used all that much these days.
Technically, NFSD isn't a file system, it's a protocol adapter.
Quoted 1 line
Why can't NFS send a single bvec for contiguous ranges?
Have a look at svc_rdma_build_read_segment(). The RDMA READ path builds bvecs from rqstp->rq_pages[], which is an array of individual struct page pointers. Each bvec entry covers at most one page.
This is because I/O payloads arrive in an xdr_buf, which represents its page data as a struct page ** array (xdr->pages), and svc_rqst::rq_pages is likewise a flat array of single-page pointers. These pages are allocated individually (typically from the page allocator via alloc_pages()), so there's no guarantee of physical contiguity. Even if adjacent pages happen to be contiguous, the code has no way to know that without inspecting PFNs (which is exactly what the DMA mapping layer does).
So currently svcrdma can't send a single bvec for contiguous ranges because the contiguity information doesn't exist at the NFSD or RPC layer. Contiguity is (re)discovered only at DMA map time.
The alternative is to build an SGL for mapping the bvec so that rw.c can get the real contiguity of the pages before proceeding. But that seems icky.
Long term, I expect that NFSD will need to preserve the folios it gets from file systems and pass those to the RPC transports without translating them to an array of page pointers.
Signature · 1 line
Chuck Lever
Quoted 54 lines · Chuck Lever
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · Chuck Lever
Under NFS WRITE workloads the server performs RDMA READs to pull data from the client. With the inflated MR demand, the pool is rapidly exhausted,
ib_mr_pool_get()returns NULL, andrdma_rw_init_one_mr()returns-EAGAIN. svcrdma treats this as a DMA mapping failure, closes the connection, and the client reconnects -- producing a cycle of 71% RPC retransmissions and ~100 reconnections per test run. RDMA WRITEs (NFS READ direction) are unaffected becauseDMA_TO_DEVICEnever triggers themax_sgl_rdcheck.Quoted 2 lines
So this changelog extensively describes the problem, but it doesn't actually say how you fix it.
I didn't want to waste everyone's time, but I can add that.
Quoted 6 lines
drivers/infiniband/core/rw.c int rdma_rw_ctx_init_bvec(struct rdma_rw_ctx *ctx, struct ib_qp *qp, · line 696
Quoted 5 lines
drivers/infiniband/core/rw.c +5
-0+* +* TODO: A bulk DMA mapping API for bvecs analogous to +* dma_map_sgtable() would provide a proper post-DMA- +* coalescing segment count here, enabling the map_wrs +* path in more cases. Quoted 7 lines
This isn't really something the DMA layer can easily do without getting as inefficient as the sgtable based path. What the block layer does here is to simply keep a higher level count of merged segments. The other option would be to not create multiple bvecs for continguous regions, which is what modern file system do in general, and why the above block layer
nr_phys_segmentsbased optimization isn't actually used all that much these days.Technically, NFSD isn't a file system, it's a protocol adapter.
Quoted 1 line
Why can't NFS send a single bvec for contiguous ranges?
Have a look at
svc_rdma_build_read_segment(). The RDMA READ path builds bvecs from rqstp->rq_pages[], which is an array of individual struct page pointers. Each bvec entry covers at most one page.This is because I/O payloads arrive in an
xdr_buf, which represents its page data as a struct page ** array (xdr->pages), andsvc_rqst::rq_pagesis likewise a flat array of single-page pointers. These pages are allocated individually (typically from the page allocator viaalloc_pages()), so there's no guarantee of physical contiguity. Even if adjacent pages happen to be contiguous, the code has no way to know that without inspecting PFNs (which is exactly what the DMA mapping layer does).So currently svcrdma can't send a single bvec for contiguous ranges because the contiguity information doesn't exist at the NFSD or RPC layer. Contiguity is (re)discovered only at DMA map time.
Quoted 7 lines
The alternative is to build an SGL for mapping the bvec so that
rw.ccan get the real contiguity of the pages before proceeding. But that seems icky.Long term, I expect that NFSD will need to preserve the folios it gets from file systems and pass those to the RPC transports without translating them to an array of page pointers.
Folio sounds like a correct approach to me, why do you mark it as "long term"?
Thanks
Quoted 4 lines
Signature · 2 lines
-- Chuck Lever
Chuck Lever
maintainerQuoted 59 lines · Leon Romanovsky
Quoted 58 lines · Chuck Lever
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · Chuck Lever
Under NFS WRITE workloads the server performs RDMA READs to pull data from the client. With the inflated MR demand, the pool is rapidly exhausted,
ib_mr_pool_get()returns NULL, andrdma_rw_init_one_mr()returns-EAGAIN. svcrdma treats this as a DMA mapping failure, closes the connection, and the client reconnects -- producing a cycle of 71% RPC retransmissions and ~100 reconnections per test run. RDMA WRITEs (NFS READ direction) are unaffected becauseDMA_TO_DEVICEnever triggers themax_sgl_rdcheck.Quoted 2 lines
So this changelog extensively describes the problem, but it doesn't actually say how you fix it.
I didn't want to waste everyone's time, but I can add that.
Quoted 6 lines
drivers/infiniband/core/rw.c int rdma_rw_ctx_init_bvec(struct rdma_rw_ctx *ctx, struct ib_qp *qp, · line 696
Quoted 5 lines
drivers/infiniband/core/rw.c +5
-0+* +* TODO: A bulk DMA mapping API for bvecs analogous to +* dma_map_sgtable() would provide a proper post-DMA- +* coalescing segment count here, enabling the map_wrs +* path in more cases. Quoted 7 lines
This isn't really something the DMA layer can easily do without getting as inefficient as the sgtable based path. What the block layer does here is to simply keep a higher level count of merged segments. The other option would be to not create multiple bvecs for continguous regions, which is what modern file system do in general, and why the above block layer
nr_phys_segmentsbased optimization isn't actually used all that much these days.Technically, NFSD isn't a file system, it's a protocol adapter.
Quoted 1 line
Why can't NFS send a single bvec for contiguous ranges?
Have a look at
svc_rdma_build_read_segment(). The RDMA READ path builds bvecs from rqstp->rq_pages[], which is an array of individual struct page pointers. Each bvec entry covers at most one page.This is because I/O payloads arrive in an
xdr_buf, which represents its page data as a struct page ** array (xdr->pages), andsvc_rqst::rq_pagesis likewise a flat array of single-page pointers. These pages are allocated individually (typically from the page allocator viaalloc_pages()), so there's no guarantee of physical contiguity. Even if adjacent pages happen to be contiguous, the code has no way to know that without inspecting PFNs (which is exactly what the DMA mapping layer does).So currently svcrdma can't send a single bvec for contiguous ranges because the contiguity information doesn't exist at the NFSD or RPC layer. Contiguity is (re)discovered only at DMA map time.
The alternative is to build an SGL for mapping the bvec so that
rw.ccan get the real contiguity of the pages before proceeding. But that seems icky.
Quoted 5 lines
Quoted 3 lines
Long term, I expect that NFSD will need to preserve the folios it gets from file systems and pass those to the RPC transports without translating them to an array of page pointers.
Folio sounds like a correct approach to me, why do you mark it as "long term"?
All four NFS maintainers are aware of this need, and I /think/ we are all on board with the "folio/bvec" approach. But there are a number of reasons this is not just a "go write the code" kind of problem:
- xdr_buf is used by both the NFS client and NFS server stacks, and they are separately maintained.
- xdr_buf is used from nearly the top to the bottom of these stacks, so making this kind of change will be painstaking.
- The XDR layer is full of helper APIs that deal with xdr_buf page arrays that would need attention.
- I need to understand whether addressing the DMA map problem has benefits for more broadly-deployed RPC transports such as TCP.
I'll have to think if we can just add something clever to the xdr_buf that only NFSD can use and then have that act as the conduit from file system to RPC transport. That could act as a prototype.
Signature · 1 line
Chuck Lever
Quoted 79 lines · Chuck Lever
Quoted 59 lines · Leon Romanovsky
Quoted 58 lines · Chuck Lever
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · Chuck Lever
Under NFS WRITE workloads the server performs RDMA READs to pull data from the client. With the inflated MR demand, the pool is rapidly exhausted,
ib_mr_pool_get()returns NULL, andrdma_rw_init_one_mr()returns-EAGAIN. svcrdma treats this as a DMA mapping failure, closes the connection, and the client reconnects -- producing a cycle of 71% RPC retransmissions and ~100 reconnections per test run. RDMA WRITEs (NFS READ direction) are unaffected becauseDMA_TO_DEVICEnever triggers themax_sgl_rdcheck.Quoted 2 lines
So this changelog extensively describes the problem, but it doesn't actually say how you fix it.
I didn't want to waste everyone's time, but I can add that.
Quoted 6 lines
drivers/infiniband/core/rw.c int rdma_rw_ctx_init_bvec(struct rdma_rw_ctx *ctx, struct ib_qp *qp, · line 696
Quoted 5 lines
drivers/infiniband/core/rw.c +5
-0+* +* TODO: A bulk DMA mapping API for bvecs analogous to +* dma_map_sgtable() would provide a proper post-DMA- +* coalescing segment count here, enabling the map_wrs +* path in more cases. Quoted 7 lines
This isn't really something the DMA layer can easily do without getting as inefficient as the sgtable based path. What the block layer does here is to simply keep a higher level count of merged segments. The other option would be to not create multiple bvecs for continguous regions, which is what modern file system do in general, and why the above block layer
nr_phys_segmentsbased optimization isn't actually used all that much these days.Technically, NFSD isn't a file system, it's a protocol adapter.
Quoted 1 line
Why can't NFS send a single bvec for contiguous ranges?
Have a look at
svc_rdma_build_read_segment(). The RDMA READ path builds bvecs from rqstp->rq_pages[], which is an array of individual struct page pointers. Each bvec entry covers at most one page.This is because I/O payloads arrive in an
xdr_buf, which represents its page data as a struct page ** array (xdr->pages), andsvc_rqst::rq_pagesis likewise a flat array of single-page pointers. These pages are allocated individually (typically from the page allocator viaalloc_pages()), so there's no guarantee of physical contiguity. Even if adjacent pages happen to be contiguous, the code has no way to know that without inspecting PFNs (which is exactly what the DMA mapping layer does).So currently svcrdma can't send a single bvec for contiguous ranges because the contiguity information doesn't exist at the NFSD or RPC layer. Contiguity is (re)discovered only at DMA map time.
The alternative is to build an SGL for mapping the bvec so that
rw.ccan get the real contiguity of the pages before proceeding. But that seems icky.Quoted 5 lines
Quoted 3 lines
Long term, I expect that NFSD will need to preserve the folios it gets from file systems and pass those to the RPC transports without translating them to an array of page pointers.
Folio sounds like a correct approach to me, why do you mark it as "long term"?
All four NFS maintainers are aware of this need, and I /think/ we are all on board with the "folio/bvec" approach. But there are a number of reasons this is not just a "go write the code" kind of problem:
-
xdr_bufis used by both the NFS client and NFS server stacks, and they are separately maintained.-
xdr_bufis used from nearly the top to the bottom of these stacks, so making this kind of change will be painstaking.- The XDR layer is full of helper APIs that deal with
xdr_bufpage arrays that would need attention.
Quoted 7 lines
- I need to understand whether addressing the DMA map problem has benefits for more broadly-deployed RPC transports such as TCP.
I'll have to think if we can just add something clever to the
xdr_bufthat only NFSD can use and then have that act as the conduit from file system to RPC transport. That could act as a prototype.
Thanks for the summary. It explains.
Quoted 4 lines
Signature · 2 lines
-- Chuck Lever
Chuck Lever
maintainerRFC PATCH 1/1 svcrdma: Use compound pages for RDMA Read sink buffers
From: Chuck Lever <[email protected]>
svc_rdma_build_read_segment() constructs RDMA Read sink buffers by consuming pages one-at-a-time from rq_pages[] and building one bvec per page. A 64KB NFS READ payload produces 16 separate bvecs, 16 DMA mappings, and potentially multiple RDMA Read WRs.
A single higher-order allocation followed by split_page() yields physically contiguous memory while preserving per-page refcounts. A single bvec spanning the contiguous range causes rdma_rw_ctx_init_bvec() to take the rdma_rw_init_single_wr_bvec() fast path: one DMA mapping, one SGE, one WR.
The split sub-pages replace the original rq_pages[] entries, so all downstream page tracking, completion handling, and xdr_buf assembly remain unchanged.
Allocation uses __GFP_NORETRY | __GFP_NOWARN and falls back through decreasing orders. If even order-1 fails, the existing per-page path handles the segment.
The compound path is attempted only when the segment starts page-aligned (rc_pageoff == 0) and spans at least two pages.
--- net/sunrpc/xprtrdma/svc_rdma_rw.c | 120 ++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+)
What if svcrdma did something derpy like this?
net/sunrpc/xprtrdma/svc_rdma_rw.c +120 -0
@@ -754,6 +754,118 @@ int svc_rdma_prepare_reply_chunk(struct svcxprt_rdma *rdma,return xdr->len; } +#define SVC_RDMA_COMPOUND_MAX_ORDER 4/* 64KB max */ + +/** +* svc_rdma_alloc_read_pages - Allocate physically contiguous pages +* @nr_pages: number of pages needed +* @order: on success, set to the allocation order +* +* Attempts a higher-order allocation, falling back to smaller orders. +* The returned pages are split immediately so each sub-page has its +* own refcount and can be freed independently. +* +* Returns a pointer to the first page on success, or NULL if even +* order-1 allocation fails. +*/ +static struct page * +svc_rdma_alloc_read_pages (unsigned int nr_pages,unsigned int *order) +{ +unsigned int o; +struct page *page; + + o =get_order (nr_pages <<PAGE_SHIFT ); +if (o >SVC_RDMA_COMPOUND_MAX_ORDER ) + o =SVC_RDMA_COMPOUND_MAX_ORDER ; + +while (o >=1 ) { + page =alloc_pages (GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN, + o); +if (page) { +split_page (page, o); + *order = o; +return page; + } + o--; + } +return NULL ; +} + +/** +* svc_rdma_build_read_segment_compound - Build a single RDMA Read WR using compound pages +* @rqstp: RPC transaction context +* @head: context for ongoing I/O +* @segment: co-ordinates of remote memory to be read +* +* Allocates a higher-order page and splits it, then builds a single +* bvec spanning the contiguous physical range. The split sub-pages +* replace entries in rq_pages[] so downstream cleanup is unchanged. +* +* Returns: +* %0: the Read WR was constructed successfully +* %-EINVAL: not enough rq_pages slots +* %-ENOMEM: compound allocation or rw_ctxt allocation failed +* %-EIO: a DMA mapping error occurred +*/ +static int svc_rdma_build_read_segment_compound (struct svc_rqst *rqstp, +struct svc_rdma_recv_ctxt *head, +const struct svc_rdma_segment *segment) +{ +struct svcxprt_rdma *rdma =svc_rdma_rqst_rdma (rqstp); +struct svc_rdma_chunk_ctxt *cc = &head->rc_cc; +unsigned int order, alloc_nr, nr_data_pages, i; +struct svc_rdma_rw_ctxt *ctxt; +struct page *page; +int ret; + + nr_data_pages =PAGE_ALIGN (segment->rs_length) >>PAGE_SHIFT ; + + page =svc_rdma_alloc_read_pages (nr_data_pages, &order); +if (!page) +return -ENOMEM ; + alloc_nr =1 << order; + +if (alloc_nr < nr_data_pages || + head->rc_curpage + alloc_nr > rqstp->rq_maxpages) { +for (i =0 ; i < alloc_nr; i++) +__free_page (page + i); +return -ENOMEM ; + } + + ctxt =svc_rdma_get_rw_ctxt (rdma,1 ); +if (!ctxt) { +for (i =0 ; i < alloc_nr; i++) +__free_page (page + i); +return -ENOMEM ; + } + +for (i =0 ; i < alloc_nr; i++) { +put_page (rqstp->rq_pages[head->rc_curpage + i]); + rqstp->rq_pages[head->rc_curpage + i] = page + i; + } + +bvec_set_page (&ctxt->rw_bvec[0 ], page, segment->rs_length,0 ); + ctxt->rw_nents =1 ; + + head->rc_page_count += nr_data_pages; + head->rc_pageoff =offset_in_page (segment->rs_length); +if (head->rc_pageoff) + head->rc_curpage += nr_data_pages -1 ; +else + head->rc_curpage += nr_data_pages; + + ret =svc_rdma_rw_ctx_init (rdma, ctxt, segment->rs_offset, + segment->rs_handle, segment->rs_length, +DMA_FROM_DEVICE ); +if (ret <0 ) +return -EIO ; +percpu_counter_inc (&svcrdma_stat_read); + +list_add (&ctxt->rw_list, &cc->cc_rwctxts); + cc->cc_sqecount += ret; +return 0 ; +} +/** * svc_rdma_build_read_segment - Build RDMA Read WQEs to pull one RDMA segment * @rqstp: RPC transaction context
@@ -780,6 +892,14 @@ static int svc_rdma_build_read_segment(struct svc_rqst *rqstp,if (check_add_overflow (head->rc_pageoff, len, &total))return -EINVAL ; nr_bvec =PAGE_ALIGN (total) >>PAGE_SHIFT ; + +if (head->rc_pageoff ==0 && nr_bvec >=2 ) { + ret =svc_rdma_build_read_segment_compound (rqstp, head, + segment); +if (ret != -ENOMEM ) +return ret; + } + ctxt =svc_rdma_get_rw_ctxt (rdma, nr_bvec);if (!ctxt)return -ENOMEM ;
Signature · 1 line
2.53.0
Re: [RFC PATCH] svcrdma: Use compound pages for RDMA Read sink buffers
Quoted 8 lines · Chuck Lever
From: Chuck Lever <[email protected]>
svc_rdma_build_read_segment()constructs RDMA Read sink buffers by consuming pages one-at-a-time from rq_pages[] and building one bvec per page. A 64KB NFS READ payload produces 16 separate bvecs, 16 DMA mappings, and potentially multiple RDMA Read WRs.
Quoted 6 lines
A single higher-order allocation followed by
split_page()yields physically contiguous memory while preserving per-page refcounts. A single bvec spanning the contiguous range causesrdma_rw_ctx_init_bvec()to take therdma_rw_init_single_wr_bvec()fast path: one DMA mapping, one SGE, one WR.
Nice, clever.
Quoted 11 lines
The split sub-pages replace the original rq_pages[] entries, so all downstream page tracking, completion handling, and
xdr_bufassembly remain unchanged.Allocation uses
__GFP_NORETRY|__GFP_NOWARNand falls back through decreasing orders. If even order-1 fails, the existing per-page path handles the segment.The compound path is attempted only when the segment starts page-aligned (
rc_pageoff== 0) and spans at least two pages.
In cases where you request three pages (order‑2), one page will be wasted. This can lead to problems later, especially under stress testing.
Thanks
Quoted 149 lines
Signed-off-by Chuck Lever <[email protected]>--- net/sunrpc/xprtrdma/svc_rdma_rw.c | 120 ++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+)What if svcrdma did something derpy like this?
net/sunrpc/xprtrdma/svc_rdma_rw.c +120
-0@@ -754,6 +754,118 @@ int svc_rdma_prepare_reply_chunk(struct svcxprt_rdma *rdma,return xdr->len; } +#define SVC_RDMA_COMPOUND_MAX_ORDER 4/* 64KB max */ + +/** +* svc_rdma_alloc_read_pages - Allocate physically contiguous pages +* @nr_pages: number of pages needed +* @order: on success, set to the allocation order +* +* Attempts a higher-order allocation, falling back to smaller orders. +* The returned pages are split immediately so each sub-page has its +* own refcount and can be freed independently. +* +* Returns a pointer to the first page on success, or NULL if even +* order-1 allocation fails. +*/ +static struct page * +svc_rdma_alloc_read_pages (unsigned int nr_pages,unsigned int *order) +{ +unsigned int o; +struct page *page; + + o =get_order (nr_pages <<PAGE_SHIFT ); +if (o >SVC_RDMA_COMPOUND_MAX_ORDER ) + o =SVC_RDMA_COMPOUND_MAX_ORDER ; + +while (o >=1 ) { + page =alloc_pages (GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN, + o); +if (page) { +split_page (page, o); + *order = o; +return page; + } + o--; + } +return NULL ; +} + +/** +* svc_rdma_build_read_segment_compound - Build a single RDMA Read WR using compound pages +* @rqstp: RPC transaction context +* @head: context for ongoing I/O +* @segment: co-ordinates of remote memory to be read +* +* Allocates a higher-order page and splits it, then builds a single +* bvec spanning the contiguous physical range. The split sub-pages +* replace entries in rq_pages[] so downstream cleanup is unchanged. +* +* Returns: +* %0: the Read WR was constructed successfully +* %-EINVAL: not enough rq_pages slots +* %-ENOMEM: compound allocation or rw_ctxt allocation failed +* %-EIO: a DMA mapping error occurred +*/ +static int svc_rdma_build_read_segment_compound (struct svc_rqst *rqstp, +struct svc_rdma_recv_ctxt *head, +const struct svc_rdma_segment *segment) +{ +struct svcxprt_rdma *rdma =svc_rdma_rqst_rdma (rqstp); +struct svc_rdma_chunk_ctxt *cc = &head->rc_cc; +unsigned int order, alloc_nr, nr_data_pages, i; +struct svc_rdma_rw_ctxt *ctxt; +struct page *page; +int ret; + + nr_data_pages =PAGE_ALIGN (segment->rs_length) >>PAGE_SHIFT ; + + page =svc_rdma_alloc_read_pages (nr_data_pages, &order); +if (!page) +return -ENOMEM ; + alloc_nr =1 << order; + +if (alloc_nr < nr_data_pages || + head->rc_curpage + alloc_nr > rqstp->rq_maxpages) { +for (i =0 ; i < alloc_nr; i++) +__free_page (page + i); +return -ENOMEM ; + } + + ctxt =svc_rdma_get_rw_ctxt (rdma,1 ); +if (!ctxt) { +for (i =0 ; i < alloc_nr; i++) +__free_page (page + i); +return -ENOMEM ; + } + +for (i =0 ; i < alloc_nr; i++) { +put_page (rqstp->rq_pages[head->rc_curpage + i]); + rqstp->rq_pages[head->rc_curpage + i] = page + i; + } + +bvec_set_page (&ctxt->rw_bvec[0 ], page, segment->rs_length,0 ); + ctxt->rw_nents =1 ; + + head->rc_page_count += nr_data_pages; + head->rc_pageoff =offset_in_page (segment->rs_length); +if (head->rc_pageoff) + head->rc_curpage += nr_data_pages -1 ; +else + head->rc_curpage += nr_data_pages; + + ret =svc_rdma_rw_ctx_init (rdma, ctxt, segment->rs_offset, + segment->rs_handle, segment->rs_length, +DMA_FROM_DEVICE ); +if (ret <0 ) +return -EIO ; +percpu_counter_inc (&svcrdma_stat_read); + +list_add (&ctxt->rw_list, &cc->cc_rwctxts); + cc->cc_sqecount += ret; +return 0 ; +} +/** * svc_rdma_build_read_segment - Build RDMA Read WQEs to pull one RDMA segment * @rqstp: RPC transaction context @@ -780,6 +892,14 @@ static int svc_rdma_build_read_segment(struct svc_rqst *rqstp,if (check_add_overflow (head->rc_pageoff, len, &total))return -EINVAL ; nr_bvec =PAGE_ALIGN (total) >>PAGE_SHIFT ; + +if (head->rc_pageoff ==0 && nr_bvec >=2 ) { + ret =svc_rdma_build_read_segment_compound (rqstp, head, + segment); +if (ret != -ENOMEM ) +return ret; + } + ctxt =svc_rdma_get_rw_ctxt (rdma, nr_bvec);if (!ctxt)return -ENOMEM ;Signature · 1 line
2.53.0