SUNRPC: fire svc_xprt_free tracepoint before dropping the netns
Subsystems (4)
Chuck Lever
maintainerPATCH 1/5 SUNRPC: fire svc_xprt_free tracepoint before dropping the netns
svc_xprt_free() releases the transport's network namespace reference and then fires trace_svc_xprt_free(), which reads xpt_net->ns.inum through SVC_XPRT_ENDPOINT_ASSIGNMENTS. When the transport holds the last reference, the namespace is already on the cleanup list by the time the tracepoint reads it. The window is narrow, since cleanup_net() runs from a workqueue, but nothing holds the namespace alive across it.
Fire the tracepoint before any of the teardown steps, while every field it records is still valid.
--- net/sunrpc/svc_xprt.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
net/sunrpc/svc_xprt.c +2 -1
@@ -170,6 +170,8 @@ static void svc_xprt_free(struct kref *kref)struct svc_xprt *xprt = container_of(kref, struct svc_xprt, xpt_ref);struct module *owner = xprt->xpt_class->xcl_owner; + +trace_svc_xprt_free (xprt);if (test_bit (XPT_CACHE_AUTH , &xprt->xpt_flags))svcauth_unix_info_release (xprt);put_cred (xprt->xpt_cred);
@@ -179,7 +181,6 @@ static void svc_xprt_free(struct kref *kref)xprt_put (xprt->xpt_bc_xprt);if (xprt->xpt_bc_xps)xprt_switch_put (xprt->xpt_bc_xps); -trace_svc_xprt_free (xprt); xprt->xpt_ops->xpo_free (xprt);module_put (owner); }
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH 2/5 SUNRPC: fire svc_defer_queue tracepoint before publishing the request
svc_revisit() adds the deferred request to xpt_deferred, drops xpt_lock, and then fires trace_svc_defer_queue(), which reads the request's XID and source address. Once the request is on the list, an nfsd thread already servicing that transport can dequeue it in svc_deferred_dequeue(), process it, and free it in svc_xprt_release() before the tracepoint reads it. The window is narrow, but nothing keeps the request alive across it.
Fire the tracepoint before the list_add(), while svc_revisit() still owns the request.
--- net/sunrpc/svc_xprt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/svc_xprt.c +1 -1
@@ -1271,9 +1271,9 @@ static void svc_revisit(struct cache_deferred_req *dreq, int too_many)return ; } dr->xprt =NULL ; +trace_svc_defer_queue (dr);list_add (&dr->handle.recent, &xprt->xpt_deferred);spin_unlock (&xprt->xpt_lock); -trace_svc_defer_queue (dr);svc_xprt_enqueue (xprt);svc_xprt_put (xprt); }
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH v1 2/5] SUNRPC: fire svc_defer_queue tracepoint before publishing the request
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 8 lines
svc_revisit()adds the deferred request toxpt_deferred, dropsxpt_lock, and then firestrace_svc_defer_queue(), which reads the request's XID and source address. Once the request is on the list, an nfsd thread already servicing that transport can dequeue it insvc_deferred_dequeue(), process it, and free it insvc_xprt_release()before the tracepoint reads it. The window is narrow, but nothing keeps the request alive across it.
Quoted 5 lines
Fire the tracepoint before the
list_add(), whilesvc_revisit()still owns the request.Fixes 8954c5c212d3 ("SUNRPC: Clean up request deferral tracepoints")Signed-off-by Chuck Lever <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Chuck Lever
maintainerPATCH 3/5 svcrdma: fix a page leak in backchannel sends
In reply to Chuck Leversvc_rdma_bc_sendto() takes an extra reference on the page that holds the backchannel Call message, so that the page is not returned to the allocator while the Send that reads it is still posted. Send completion once dropped that reference, back when the page sat in the send context's page array. Commit 99722fe4d5a6 ("svcrdma: Persistently allocate and DMA-map Send buffers") switched the backchannel to svc_rdma_map_reply_msg(), which DMA-maps the buffer and records no pages. Send completion no longer touches the page, but the get_page() stayed.
xprt_rdma_bc_free() drops the allocation reference and nothing drops the extra one, so every backchannel Call sent over RPC/RDMA leaks its send buffer page.
The extra reference is still needed. A Call at or above RPCRDMA_PULLUP_THRESH is DMA-mapped in place. The RPC client frees rq_buffer when the callback task times out or is killed, whether or not the Send has completed. Record the page in the send context's page array so that svc_rdma_send_ctxt_release() drops the reference after Send completion.
--- net/sunrpc/xprtrdma/svc_rdma_backchannel.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
net/sunrpc/xprtrdma/svc_rdma_backchannel.c +7 -3
@@ -85,10 +85,14 @@ static int svc_rdma_bc_sendto(struct svcxprt_rdma *rdma,if (ret <0 )return -EIO ; -/* Bump page refcnt so Send completion doesn't release -* the rq_buffer before all retransmits are complete. +/* The RPC client frees rq_buffer when the callback task ends, +* whether or not the Send has completed. Hold the page until the +* send context is released after Send completion. */ -get_page (virt_to_page (rqst->rq_buffer)); + sctxt->sc_pages[0 ] =virt_to_page (rqst->rq_buffer); +get_page (sctxt->sc_pages[0 ]); + sctxt->sc_page_count =1 ; + sctxt->sc_send_wr.opcode =IB_WR_SEND ;return svc_rdma_post_send (rdma, sctxt); }
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH v1 3/5] svcrdma: fix a page leak in backchannel sends
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 21 lines
svc_rdma_bc_sendto()takes an extra reference on the page that holds the backchannel Call message, so that the page is not returned to the allocator while the Send that reads it is still posted. Send completion once dropped that reference, back when the page sat in the send context's page array. Commit99722fe4d5a6("svcrdma: Persistently allocate and DMA-map Send buffers") switched the backchannel tosvc_rdma_map_reply_msg(), which DMA-maps the buffer and records no pages. Send completion no longer touches the page, but theget_page()stayed.
xprt_rdma_bc_free()drops the allocation reference and nothing drops the extra one, so every backchannel Call sent over RPC/RDMA leaks its send buffer page.The extra reference is still needed. A Call at or above
RPCRDMA_PULLUP_THRESHis DMA-mapped in place. The RPC client freesrq_bufferwhen the callback task times out or is killed, whether or not the Send has completed. Record the page in the send context's page array so thatsvc_rdma_send_ctxt_release()drops the reference after Send completion.
Quoted 2 lines
Fixes 99722fe4d5a6 ("svcrdma: Persistently allocate and DMA-map Send buffers")Signed-off-by Chuck Lever <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Chuck Lever
maintainerPATCH 4/5 svcrdma: release a send context stranded by a partial post
In reply to Chuck LeverWhen ib_post_send() rejects a WR mid-chain, svc_rdma_post_send_err() returns zero to say that a completion will clean up. The Write and Reply chunk WRs in a reply's chain do carry completions, but those only trace or close the transport. The Send at the chain's tail is the one whose completion puts the send context, so a chain cut before the Send leaves the context unreleased. svc_rdma_sendto() reports success and never puts it either. No list the transport destructor walks holds the context, so it leaks past teardown along with the DMA mappings, page references, and rw contexts it owns.
The error path cannot release the context itself. The posted WRs still reference its rw contexts and pages, and the HCA may still be reading them. Nor can it drain the Send Queue to wait for them. A drain needs a free SQ slot for its marker WR. A provider that rejects a WR after accepting earlier ones has most likely diverged from svcrdma's SQ accounting, so no slot svcrdma believes it holds can be trusted.
Transfer the context to the transport instead. On a partial post, svc_rdma_post_send() parks the context on a per-transport list and reports success, the same ownership a fully posted chain has. svc_rdma_free() releases the list after its own drain and before the rw contexts are destroyed. The SQ reservation stays debited, since the Send that refunds it never completes and the transport is closing.
Reaching this path requires a provider to reject a WR after accepting earlier ones in the same chain. That has not been observed. The Fixes: tag names the commit that wrote the mistaken contract. The chain became long enough to cut only with commit 10e6fc1054d9 ("svcrdma: Post the Reply chunk and Send WR together").
--- include/linux/sunrpc/svc_rdma.h | 2 + net/sunrpc/xprtrdma/svc_rdma_rw.c | 1 + net/sunrpc/xprtrdma/svc_rdma_sendto.c | 47 +++++++++++++++++++----- net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 + 4 files changed, 43 insertions(+), 9 deletions(-)
include/linux/sunrpc/svc_rdma.h +2 -0
@@ -117,6 +117,7 @@ struct svcxprt_rdma {struct llist_head sc_recv_ctxts;struct llist_head sc_send_release_list; +struct llist_head sc_send_stranded_ctxts;atomic_t sc_completion_ids; };
@@ -300,6 +301,7 @@ extern int svc_rdma_process_read_list(struct svcxprt_rdma *rdma,/* svc_rdma_sendto.c */ extern void svc_rdma_send_ctxts_destroy (struct svcxprt_rdma *rdma);extern void svc_rdma_send_ctxts_drain (struct svcxprt_rdma *rdma); +extern void svc_rdma_send_ctxts_stranded_release (struct svcxprt_rdma *rdma);extern struct svc_rdma_send_ctxt *svc_rdma_send_ctxt_get (struct svcxprt_rdma *rdma); extern void svc_rdma_send_ctxt_put(struct svcxprt_rdma *rdma,
net/sunrpc/xprtrdma/svc_rdma_rw.c +1 -0
@@ -420,6 +420,7 @@ static int svc_rdma_post_chunk_ctxt(struct svcxprt_rdma *rdma,return ret; cc->cc_posttime =ktime_get (); + bad_wr = first_wr; ret =ib_post_send (rdma->sc_qp, first_wr, &bad_wr);if (ret) return svc_rdma_post_send_err(rdma, &cc->cc_cid, bad_wr,
net/sunrpc/xprtrdma/svc_rdma_sendto.c +38 -9
78 lines not on this page · raw message
net/sunrpc/xprtrdma/svc_rdma_transport.c +2 -0
@@ -197,6 +197,7 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,init_llist_head (&cma_xprt->sc_recv_ctxts);init_llist_head (&cma_xprt->sc_rw_ctxts);init_llist_head (&cma_xprt->sc_send_release_list); +init_llist_head (&cma_xprt->sc_send_stranded_ctxts);init_waitqueue_head (&cma_xprt->sc_send_wait);init_waitqueue_head (&cma_xprt->sc_sq_ticket_wait);
@@ -673,6 +674,7 @@ static void svc_rdma_free(struct svc_xprt *xprt)if (rdma->sc_qp && !IS_ERR (rdma->sc_qp))ib_drain_qp (rdma->sc_qp);svc_rdma_send_ctxts_drain (rdma); +svc_rdma_send_ctxts_stranded_release (rdma);svc_rdma_flush_recv_queues (rdma);
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH v1 4/5] svcrdma: release a send context stranded by a partial post
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines
When
ib_post_send()rejects a WR mid-chain,svc_rdma_post_send_err()returns zero to say that a completion will clean up. The Write and Reply chunk WRs in a reply's chain do carry completions, but those only trace or close the transport. The Send at the chain's tail is the one whose completion puts the send context, so a chain cut before the Send leaves the context unreleased.svc_rdma_sendto()reports success and never puts it either. No list the transport destructor walks holds the context, so it leaks past teardown along with the DMA mappings, page references, and rw contexts it owns.The error path cannot release the context itself. The posted WRs still reference its rw contexts and pages, and the HCA may still be reading them. Nor can it drain the Send Queue to wait for them. A drain needs a free SQ slot for its marker WR. A provider that rejects a WR after accepting earlier ones has most likely diverged [ ... ]
contract. The chain became long enough to cut only with commit10e6fc1054d9("svcrdma: Post the Reply chunk and Send WR together").
Quoted 2 lines
Fixes 71b43531ee0b ("svcrdma: Post Send WR chain")Signed-off-by Chuck Lever <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Chuck Lever
maintainerPATCH 5/5 svcrdma: release a receive context stranded by a partial Read post
In reply to Chuck LeverWhen ib_post_send() rejects a WR mid-chain, svc_rdma_post_send_err() returns zero and svc_rdma_post_chunk_ctxt() reports the Read chain as posted. The only signaled WR in the chain is its tail, so a chain cut before the tail posts nothing that completes. svc_rdma_process_read_list() then reports the Read in progress, and the receive context waits for svc_rdma_wc_read_done(), which never runs. No list the transport destructor walks holds the context, so it leaks past teardown along with its rw contexts and their DMA mappings.
The caller cannot release the context on the error, because the posted Read WRs still write into pages it owns. Nor can the error path drain the Send Queue. The drain's marker WR needs an SQ slot, and a provider that has just rejected a WR cannot be trusted to have one.
Park the receive context on a per-transport list, as the Send path does, and release it from svc_rdma_free() after the drain and before the rw contexts are destroyed. svc_rdma_post_chunk_ctxt() posts only Read chains, so pass it the receive context itself rather than the embedded chunk context.
Reaching this path needs a provider to reject a WR after accepting earlier ones in the same chain. That has not been observed.
--- include/linux/sunrpc/svc_rdma.h | 2 ++ net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 18 ++++++++++++++ net/sunrpc/xprtrdma/svc_rdma_rw.c | 31 ++++++++++++++++++------ net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 ++ 4 files changed, 46 insertions(+), 7 deletions(-)
include/linux/sunrpc/svc_rdma.h +2 -0
@@ -118,6 +118,7 @@ struct svcxprt_rdma {struct llist_head sc_send_release_list;struct llist_head sc_send_stranded_ctxts; +struct llist_head sc_recv_stranded_ctxts;atomic_t sc_completion_ids; };
@@ -263,6 +264,7 @@ extern void svc_rdma_handle_bc_reply(struct svc_rqst *rqstp,/* svc_rdma_recvfrom.c */ extern void svc_rdma_recv_ctxts_destroy (struct svcxprt_rdma *rdma); +extern void svc_rdma_recv_ctxts_stranded_release (struct svcxprt_rdma *rdma);extern bool svc_rdma_post_recvs (struct svcxprt_rdma *rdma);extern struct svc_rdma_recv_ctxt *svc_rdma_recv_ctxt_get (struct svcxprt_rdma *rdma);
net/sunrpc/xprtrdma/svc_rdma_recvfrom.c +18 -0
@@ -234,6 +234,24 @@ void svc_rdma_recv_ctxt_put(struct svcxprt_rdma *rdma,llist_add (&ctxt->rc_node, &rdma->sc_recv_ctxts); } +/** +* svc_rdma_recv_ctxts_stranded_release - Release stranded recv_ctxts +* @rdma: svcxprt_rdma being torn down +* +* Context: transport destructor only, after the QP has been drained +* and before its rw contexts and recv_ctxts are destroyed. +*/ +void svc_rdma_recv_ctxts_stranded_release (struct svcxprt_rdma *rdma) +{ +struct svc_rdma_recv_ctxt *ctxt; +struct llist_node *node; + +while ((node =llist_del_first (&rdma->sc_recv_stranded_ctxts)) !=NULL ) { + ctxt = llist_entry(node, struct svc_rdma_recv_ctxt, rc_node); +svc_rdma_recv_ctxt_put (rdma, ctxt); + } +} +/** * svc_rdma_release_ctxt - Release transport-specific per-rqst resources * @xprt: the transport which owned the context
net/sunrpc/xprtrdma/svc_rdma_rw.c +24 -7
@@ -389,10 +389,18 @@ static void svc_rdma_wc_read_done(struct ib_cq *cq, struct ib_wc *wc)* - If ib_post_send() succeeds, only one completion is expected, * even if one or more WRs are flushed. This is true when posting * an rdma_rw_ctx or when posting a single signaled WR. +* +* Return values: +* %0: The chain was posted; svc_rdma_wc_read_done() releases +* @head. Or a prefix of the chain was posted; no completion +* follows, and the transport destructor releases @head. +* %-ENOTCONN: No WR was posted; the caller still owns @head. +* %-EINVAL: @head's Read chain needs more SQ entries than exist. */ static int svc_rdma_post_chunk_ctxt (struct svcxprt_rdma *rdma, -struct svc_rdma_chunk_ctxt *cc) +struct svc_rdma_recv_ctxt *head) { +struct svc_rdma_chunk_ctxt *cc = &head->rc_cc;struct ib_send_wr *first_wr;const struct ib_send_wr *bad_wr;struct list_head *tmp;
@@ -422,10 +430,18 @@ static int svc_rdma_post_chunk_ctxt(struct svcxprt_rdma *rdma, cc->cc_posttime =ktime_get (); bad_wr = first_wr; ret =ib_post_send (rdma->sc_qp, first_wr, &bad_wr); -if (ret) -return svc_rdma_post_send_err (rdma, &cc->cc_cid, bad_wr, - first_wr, cc->cc_sqecount, - ret); +if (ret) { + ret =svc_rdma_post_send_err (rdma, &cc->cc_cid, bad_wr, + first_wr, cc->cc_sqecount, + ret); +if (ret) +return ret; + +/* The posted prefix still references @head. Only the +* transport destructor may release it. +*/ +llist_add (&head->rc_node, &rdma->sc_recv_stranded_ctxts); + }return 0 ; }
@@ -1169,7 +1185,8 @@ static void svc_rdma_clear_rqst_pages(struct svc_rqst *rqstp,* RDMA Reads have completed. * * Return values: -* %1: all needed RDMA Reads were posted successfully, +* %1: RDMA Reads were posted; the transport now owns @head, and +* a Read completion or the transport destructor releases it, * %-EINVAL: client provided too many chunks or segments, * %-ENOMEM: rdma_rw context pool was exhausted, * %-ENOTCONN: posting failed (connection is lost),
@@ -1200,6 +1217,6 @@ int svc_rdma_process_read_list(struct svcxprt_rdma *rdma,return ret;trace_svcrdma_post_read_chunk (&cc->cc_cid, cc->cc_sqecount); - ret =svc_rdma_post_chunk_ctxt (rdma, cc); + ret =svc_rdma_post_chunk_ctxt (rdma, head);return ret <0 ? ret :1 ; }
net/sunrpc/xprtrdma/svc_rdma_transport.c +2 -0
@@ -198,6 +198,7 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,init_llist_head (&cma_xprt->sc_rw_ctxts);init_llist_head (&cma_xprt->sc_send_release_list);init_llist_head (&cma_xprt->sc_send_stranded_ctxts); +init_llist_head (&cma_xprt->sc_recv_stranded_ctxts);init_waitqueue_head (&cma_xprt->sc_send_wait);init_waitqueue_head (&cma_xprt->sc_sq_ticket_wait);
@@ -677,6 +678,7 @@ static void svc_rdma_free(struct svc_xprt *xprt)svc_rdma_send_ctxts_stranded_release (rdma);svc_rdma_flush_recv_queues (rdma); +svc_rdma_recv_ctxts_stranded_release (rdma);svc_rdma_destroy_rw_ctxts (rdma);svc_rdma_send_ctxts_destroy (rdma);
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH v1 5/5] svcrdma: release a receive context stranded by a partial Read post
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 22 lines
When
ib_post_send()rejects a WR mid-chain,svc_rdma_post_send_err()returns zero andsvc_rdma_post_chunk_ctxt()reports the Read chain as posted. The only signaled WR in the chain is its tail, so a chain cut before the tail posts nothing that completes.svc_rdma_process_read_list()then reports the Read in progress, and the receive context waits forsvc_rdma_wc_read_done(), which never runs. No list the transport destructor walks holds the context, so it leaks past teardown along with its rw contexts and their DMA mappings.The caller cannot release the context on the error, because the posted Read WRs still write into pages it owns. Nor can the error path drain the Send Queue. The drain's marker WR needs an SQ slot, and a provider that has just rejected a WR cannot be trusted to have one.
Park the receive context on a per-transport list, as the Send path does, and release it from
svc_rdma_free()after the drain and before the rw contexts are destroyed.svc_rdma_post_chunk_ctxt()posts only Read chains, so pass it the receive context itself rather than the embedded chunk context.
Quoted 5 lines
Reaching this path needs a provider to reject a WR after accepting earlier ones in the same chain. That has not been observed.
Fixes f13193f50b64 ("svcrdma: Introduce local rdma_rw API helpers")Signed-off-by Chuck Lever <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
[email protected]
AI reviewSashiko has reviewed this patch and found no issues. It looks great!
Quoted 8 lines
svc_xprt_free()releases the transport's network namespace reference and then firestrace_svc_xprt_free(), which readsxpt_net->ns.inumthroughSVC_XPRT_ENDPOINT_ASSIGNMENTS. When the transport holds the last reference, the namespace is already on the cleanup list by the time the tracepoint reads it. The window is narrow, sincecleanup_net()runs from a workqueue, but nothing holds the namespace alive across it.
Quoted 5 lines
Fire the tracepoint before any of the teardown steps, while every field it records is still valid.
Fixes 11bbb0f76e99 ("SUNRPC: Trace a few more generic svc_xprt events")Signed-off-by Chuck Lever <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1