NFS/RDMA client patches for v4.20
Subsystems (4)
cover NFS/RDMA client patches for v4.20
A mixed bag of fixes and clean ups, the important ones being:
- Ensure credit grant is reduced to 1 after a reconnect
- Delete MRs instead of recovering them
- Support sending RPCs with both Write and Reply chunks
- Modernize the connect logic
Details can be found in the patch descriptions.
---
Chuck Lever (22):
xprtrdma: Reset credit grant properly after a disconnect
xprtrdma: Create more MRs at a time
xprtrdma: Explicitly resetting MRs is no longer necessary
xprtrdma: Name MR trace events consistently
xprtrdma: Refactor chunk encoding
xprtrdma: Refactor chunktype handling
xprtrdma: Support Write+Reply Replies
sunrpc: Fix connect metrics
sunrpc: Report connect_time in seconds
xprtrdma: Rename rpcrdma_conn_upcall
xprtrdma: Conventional variable names in rpcrdma_conn_upcall
xprtrdma: Eliminate "connstate" variable from rpcrdma_conn_upcall()
xprtrdma: Re-organize the switch() in rpcrdma_conn_upcall
xprtrdma: Simplify RPC wake-ups on connect
xprtrdma: Rename rpcrdma_qp_async_error_upcall
xprtrdma: Remove memory address of "ep" from an error message
svcrdma: Don't disable BH's in backchannel
xprtrdma: Move rb_flags initialization
xprtrdma: Report when there were zero posted Receives
xprtrdma: Add documenting comments
xprtrdma: Clean up xprt_rdma_disconnect_inject
xprtrdma: Squelch a sparse warning
include/trace/events/rpcrdma.h | 40 ++++---- net/sunrpc/xprt.c | 10 +- net/sunrpc/xprtrdma/backchannel.c | 16 ++- net/sunrpc/xprtrdma/fmr_ops.c | 131 ++++++++++++--------------- net/sunrpc/xprtrdma/frwr_ops.c | 137 +++++++++++----------------- net/sunrpc/xprtrdma/rpc_rdma.c | 168 +++++++++++++++++++---------------- net/sunrpc/xprtrdma/transport.c | 103 +++++++++++---------- net/sunrpc/xprtrdma/verbs.c | 178 +++++++++++++++++++------------------ net/sunrpc/xprtrdma/xprt_rdma.h | 23 ++--- net/sunrpc/xprtsock.c | 14 ++- 10 files changed, 400 insertions(+), 420 deletions(-)
--
Chuck Lever
PATCH 1/22 xprtrdma: Reset credit grant properly after a disconnect
On a fresh connection, an RPC/RDMA client is supposed to send only one RPC Call until it gets a credit grant in the first RPC Reply from the server [RFC 8166, Section 3.3.3].
There is a bug in the Linux client's credit accounting mechanism introduced by commit e7ce710a8802 ("xprtrdma: Avoid deadlock when credit window is reset"). On connect, it simply dumps all pending RPC Calls onto the new connection.
Servers have been tolerant of this bad behavior. Currently no server implementation ever changes its credit grant over reconnects, and servers always repost enough Receives before connections are fully established.
To correct this issue, ensure that the client resets both the credit grant _and_ the congestion window when handling a reconnect.
--- net/sunrpc/xprtrdma/transport.c | 6 ++++++ 1 file changed, 6 insertions(+)
net/sunrpc/xprtrdma/transport.c +6 -0
@@ -468,6 +468,12 @@ xprt->reestablish_timeout =0 ;xprt_disconnect_done (xprt);rpcrdma_ep_disconnect (ep, ia); + +/* Prepare @xprt for the next connection by reinitializing +* its credit grant to one (see RFC 8166, Section 3.3.3). +*/ + r_xprt->rx_buf.rb_credits =1 ; + xprt->cwnd =RPC_CWNDSHIFT ; }/**
PATCH 2/22 xprtrdma: Create more MRs at a time
In reply to Chuck LeverSome devices require more than 3 MRs to build a single 1MB I/O. Ensure that rpcrdma_mrs_create() will add enough MRs to build that I/O. Otherwise it's possible for ->send_request to loop acquiring some MRs, not getting enough, getting called again, recycling the previous MRs, then not getting enough, lather rinse repeat.
I'm "reusing" ia->ri_max_segs. All of its accessors seem to want the maximum number of data segments plus two, so I'm going to bake that into the initial calculation.
--- net/sunrpc/xprtrdma/fmr_ops.c | 1 + net/sunrpc/xprtrdma/frwr_ops.c | 1 + net/sunrpc/xprtrdma/rpc_rdma.c | 2 -- net/sunrpc/xprtrdma/verbs.c | 2 +- 4 files changed, 3 insertions(+), 3 deletions(-)
net/sunrpc/xprtrdma/fmr_ops.c +1 -0
@@ -187,6 +187,7 @@ enum { ia->ri_max_segs = max_t(unsigned int, 1, RPCRDMA_MAX_DATA_SEGS /RPCRDMA_MAX_FMR_SGES ); + ia->ri_max_segs +=2 ;/* segments for head and tail buffers */ return 0 ; }
net/sunrpc/xprtrdma/frwr_ops.c +1 -0
@@ -276,6 +276,7 @@ ia->ri_max_segs = max_t(unsigned int, 1, RPCRDMA_MAX_DATA_SEGS / ia->ri_max_frwr_depth); + ia->ri_max_segs +=2 ;/* segments for head and tail buffers */ return 0 ; }
net/sunrpc/xprtrdma/rpc_rdma.c +0 -2
@@ -71,7 +71,6 @@ static unsigned int rpcrdma_max_call_header_size(unsigned int maxsegs) size =RPCRDMA_HDRLEN_MIN ;/* Maximum Read list size */ - maxsegs +=2 ;/* segment for head and tail buffers */ size = maxsegs * rpcrdma_readchunk_maxsz *sizeof (__be32);/* Minimal Read chunk size */
@@ -97,7 +96,6 @@ static unsigned int rpcrdma_max_reply_header_size(unsigned int maxsegs) size =RPCRDMA_HDRLEN_MIN ;/* Maximum Write list size */ - maxsegs +=2 ;/* segment for head and tail buffers */ size =sizeof (__be32);/* segment count */ size += maxsegs * rpcrdma_segment_maxsz *sizeof (__be32); size +=sizeof (__be32);/* list discriminator */
net/sunrpc/xprtrdma/verbs.c +1 -1
@@ -1019,7 +1019,7 @@ struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_buffer *buf)LIST_HEAD (free);LIST_HEAD (all); -for (count =0 ; count <3 ; count++) { +for (count =0 ; count < ia->ri_max_segs; count++) {struct rpcrdma_mr *mr;int rc;
PATCH 3/22 xprtrdma: Explicitly resetting MRs is no longer necessary
In reply to Chuck LeverWhen a memory operation fails, the MR's driver state might not match its hardware state. The only reliable recourse is to dereg the MR. This is done in ->ro_recover_mr, which then attempts to allocate a fresh MR to replace the released MR.
Since commit e2ac236c0b651 ("xprtrdma: Allocate MRs on demand"), xprtrdma dynamically allocates MRs. It can add more MRs whenever they are needed.
That makes it possible to simply release an MR when a memory operation fails, instead of "recovering" it. It will automatically be replaced by the on-demand MR allocator.
This commit is a little larger than I wanted, but it replaces ->ro_recover_mr, rb_recovery_lock, rb_recovery_worker, and the rb_stale_mrs list with a generic work queue.
Since MRs are no longer orphaned, the mrs_orphaned metric is no longer used.
--- include/trace/events/rpcrdma.h | 2 - net/sunrpc/xprtrdma/fmr_ops.c | 124 +++++++++++++++++-------------------- net/sunrpc/xprtrdma/frwr_ops.c | 130 ++++++++++++++------------------------- net/sunrpc/xprtrdma/rpc_rdma.c | 2 - net/sunrpc/xprtrdma/transport.c | 2 - net/sunrpc/xprtrdma/verbs.c | 38 ----------- net/sunrpc/xprtrdma/xprt_rdma.h | 14 ++-- 7 files changed, 114 insertions(+), 198 deletions(-)
include/trace/events/rpcrdma.h +1 -1
@@ -655,7 +655,7 @@DEFINE_MR_EVENT (xprtrdma_dma_map);DEFINE_MR_EVENT (xprtrdma_dma_unmap);DEFINE_MR_EVENT (xprtrdma_remoteinv); -DEFINE_MR_EVENT (xprtrdma_recover_mr); +DEFINE_MR_EVENT (xprtrdma_mr_recycle);/** ** Reply events
net/sunrpc/xprtrdma/fmr_ops.c +56 -68
191 lines not on this page · raw message
net/sunrpc/xprtrdma/frwr_ops.c +47 -83
188 lines not on this page · raw message
net/sunrpc/xprtrdma/rpc_rdma.c +1 -1
@@ -803,7 +803,7 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt,struct rpcrdma_mr *mr; mr =rpcrdma_mr_pop (&req->rl_registered); -rpcrdma_mr_defer_recovery (mr); +rpcrdma_mr_recycle (mr); }/* This implementation supports the following combinations
net/sunrpc/xprtrdma/transport.c +1 -1
@@ -792,7 +792,7 @@ void xprt_rdma_print_stats(struct rpc_xprt *xprt, struct seq_file *seq) r_xprt->rx_stats.bad_reply_count, r_xprt->rx_stats.nomsg_call_count); seq_printf(seq, "%lu %lu %lu %lu %lu %lu\n", - r_xprt->rx_stats.mrs_recovered, + r_xprt->rx_stats.mrs_recycled, r_xprt->rx_stats.mrs_orphaned, r_xprt->rx_stats.mrs_allocated, r_xprt->rx_stats.local_inv_needed,
net/sunrpc/xprtrdma/verbs.c +0 -38
@@ -978,39 +978,6 @@ struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_buffer *buf) }static void -rpcrdma_mr_recovery_worker (struct work_struct *work) -{ - struct rpcrdma_buffer *buf = container_of(work, struct rpcrdma_buffer, - rb_recovery_worker.work); -struct rpcrdma_mr *mr; - -spin_lock (&buf->rb_recovery_lock); -while (!list_empty (&buf->rb_stale_mrs)) { - mr =rpcrdma_mr_pop (&buf->rb_stale_mrs); -spin_unlock (&buf->rb_recovery_lock); - -trace_xprtrdma_recover_mr (mr); - mr->mr_xprt->rx_ia.ri_ops->ro_recover_mr (mr); - -spin_lock (&buf->rb_recovery_lock); - } -spin_unlock (&buf->rb_recovery_lock); -} - -void -rpcrdma_mr_defer_recovery (struct rpcrdma_mr *mr) -{ -struct rpcrdma_xprt *r_xprt = mr->mr_xprt; -struct rpcrdma_buffer *buf = &r_xprt->rx_buf; - -spin_lock (&buf->rb_recovery_lock); -rpcrdma_mr_push (mr, &buf->rb_stale_mrs); -spin_unlock (&buf->rb_recovery_lock); - -schedule_delayed_work (&buf->rb_recovery_worker,0 ); -} - -static void rpcrdma_mrs_create (struct rpcrdma_xprt *r_xprt) {struct rpcrdma_buffer *buf = &r_xprt->rx_buf;
@@ -1142,14 +1109,10 @@ struct rpcrdma_req * buf->rb_bc_srv_max_requests =0 ;spin_lock_init (&buf->rb_mrlock);spin_lock_init (&buf->rb_lock); -spin_lock_init (&buf->rb_recovery_lock);INIT_LIST_HEAD (&buf->rb_mrs);INIT_LIST_HEAD (&buf->rb_all); -INIT_LIST_HEAD (&buf->rb_stale_mrs);INIT_DELAYED_WORK (&buf->rb_refresh_worker, rpcrdma_mr_refresh_worker); -INIT_DELAYED_WORK (&buf->rb_recovery_worker, - rpcrdma_mr_recovery_worker);rpcrdma_mrs_create (r_xprt);
@@ -1233,7 +1196,6 @@ struct rpcrdma_req *void rpcrdma_buffer_destroy (struct rpcrdma_buffer *buf) { -cancel_delayed_work_sync (&buf->rb_recovery_worker);cancel_delayed_work_sync (&buf->rb_refresh_worker);rpcrdma_sendctxs_destroy (buf);
net/sunrpc/xprtrdma/xprt_rdma.h +8 -6
@@ -280,6 +280,7 @@ struct rpcrdma_mr {u32 mr_handle;u32 mr_length;u64 mr_offset; +struct work_struct mr_recycle;struct list_head mr_all; };
@@ -411,9 +412,6 @@ struct rpcrdma_buffer {u32 rb_bc_max_requests; -spinlock_t rb_recovery_lock;/* protect rb_stale_mrs */ -struct list_head rb_stale_mrs; -struct delayed_work rb_recovery_worker;struct delayed_work rb_refresh_worker; };#define rdmab_to_ia (b) (&container_of((b), struct rpcrdma_xprt, rx_buf)->rx_ia)
@@ -452,7 +450,7 @@ struct rpcrdma_stats {unsigned long hardway_register_count;unsigned long failed_marshal_count;unsigned long bad_reply_count; -unsigned long mrs_recovered; +unsigned long mrs_recycled;unsigned long mrs_orphaned;unsigned long mrs_allocated;unsigned long empty_sendctx_q;
@@ -481,7 +479,6 @@ struct rpcrdma_memreg_ops { struct list_head *mrs);void (*ro_unmap_sync)(struct rpcrdma_xprt *,struct list_head *); -void (*ro_recover_mr)(struct rpcrdma_mr *mr);int (*ro_open)(struct rpcrdma_ia *,struct rpcrdma_ep *,struct rpcrdma_create_data_internal *);
@@ -578,7 +575,12 @@ int rpcrdma_ep_post(struct rpcrdma_ia *, struct rpcrdma_ep *,struct rpcrdma_mr *rpcrdma_mr_get (struct rpcrdma_xprt *r_xprt);void rpcrdma_mr_put (struct rpcrdma_mr *mr);void rpcrdma_mr_unmap_and_put (struct rpcrdma_mr *mr); -void rpcrdma_mr_defer_recovery (struct rpcrdma_mr *mr); + +static inline void +rpcrdma_mr_recycle (struct rpcrdma_mr *mr) +{ +schedule_work (&mr->mr_recycle); +}struct rpcrdma_req *rpcrdma_buffer_get (struct rpcrdma_buffer *);void rpcrdma_buffer_put (struct rpcrdma_req *);
PATCH 4/22 xprtrdma: Name MR trace events consistently
In reply to Chuck LeverClean up the names of trace events related to MRs so that it's easy to enable these with a glob.
--- include/trace/events/rpcrdma.h | 12 ++++++------ net/sunrpc/xprtrdma/fmr_ops.c | 6 +++--- net/sunrpc/xprtrdma/frwr_ops.c | 8 ++++---- net/sunrpc/xprtrdma/verbs.c | 2 +- 4 files changed, 14 insertions(+), 14 deletions(-)
include/trace/events/rpcrdma.h +6 -6
@@ -263,7 +263,7 @@ );#define DEFINE_MR_EVENT (name) \ - DEFINE_EVENT(xprtrdma_mr, name, \ + DEFINE_EVENT(xprtrdma_mr, xprtrdma_mr_##name, \ TP_PROTO( \ const struct rpcrdma_mr *mr \ ), \
@@ -651,11 +651,11 @@DEFINE_FRWR_DONE_EVENT (xprtrdma_wc_li);DEFINE_FRWR_DONE_EVENT (xprtrdma_wc_li_wake); -DEFINE_MR_EVENT (xprtrdma_localinv); -DEFINE_MR_EVENT (xprtrdma_dma_map); -DEFINE_MR_EVENT (xprtrdma_dma_unmap); -DEFINE_MR_EVENT (xprtrdma_remoteinv); -DEFINE_MR_EVENT (xprtrdma_mr_recycle); +DEFINE_MR_EVENT (localinv); +DEFINE_MR_EVENT (map); +DEFINE_MR_EVENT (unmap); +DEFINE_MR_EVENT (remoteinv); +DEFINE_MR_EVENT (recycle);/** ** Reply events
net/sunrpc/xprtrdma/fmr_ops.c +3 -3
@@ -97,7 +97,7 @@ enum {trace_xprtrdma_mr_recycle (mr); -trace_xprtrdma_dma_unmap (mr); +trace_xprtrdma_mr_unmap (mr);ib_dma_unmap_sg (r_xprt->rx_ia.ri_device, mr->mr_sg, mr->mr_nents, mr->mr_dir);
@@ -234,7 +234,7 @@ enum { mr->mr_sg, i, mr->mr_dir);if (!mr->mr_nents) goto out_dmamap_err; -trace_xprtrdma_dma_map (mr); +trace_xprtrdma_mr_map (mr);for (i =0 , dma_pages = mr->fmr.fm_physaddrs; i < mr->mr_nents; i++) dma_pages[i] =sg_dma_address (&mr->mr_sg[i]);
@@ -295,7 +295,7 @@ enum {list_for_each_entry (mr, mrs, mr_list) {dprintk ("RPC: %s: unmapping fmr %p\n" , __func__, &mr->fmr); -trace_xprtrdma_localinv (mr); +trace_xprtrdma_mr_localinv (mr);list_add_tail (&mr->fmr.fm_mr->list, &unmap_list); } r_xprt->rx_stats.local_inv_needed++;
net/sunrpc/xprtrdma/frwr_ops.c +4 -4
@@ -123,7 +123,7 @@trace_xprtrdma_mr_recycle (mr);if (state !=FRWR_FLUSHED_LI ) { -trace_xprtrdma_dma_unmap (mr); +trace_xprtrdma_mr_unmap (mr);ib_dma_unmap_sg (r_xprt->rx_ia.ri_device, mr->mr_sg, mr->mr_nents, mr->mr_dir); }
@@ -384,7 +384,7 @@ mr->mr_nents =ib_dma_map_sg (ia->ri_device, mr->mr_sg, i, mr->mr_dir);if (!mr->mr_nents) goto out_dmamap_err; -trace_xprtrdma_dma_map (mr); +trace_xprtrdma_mr_map (mr); ibmr = frwr->fr_mr; n =ib_map_mr_sg (ibmr, mr->mr_sg, mr->mr_nents,NULL ,PAGE_SIZE );
@@ -466,7 +466,7 @@list_for_each_entry (mr, mrs, mr_list)if (mr->mr_handle == rep->rr_inv_rkey) {list_del_init (&mr->mr_list); -trace_xprtrdma_remoteinv (mr); +trace_xprtrdma_mr_remoteinv (mr); mr->frwr.fr_state =FRWR_IS_INVALID ;rpcrdma_mr_unmap_and_put (mr);break ;/* only one invalidated MR per RPC */
@@ -503,7 +503,7 @@ mr->frwr.fr_state =FRWR_IS_INVALID ; frwr = &mr->frwr; -trace_xprtrdma_localinv (mr); +trace_xprtrdma_mr_localinv (mr); frwr->fr_cqe.done = frwr_wc_localinv; last = &frwr->fr_invwr;
net/sunrpc/xprtrdma/verbs.c +1 -1
@@ -1288,7 +1288,7 @@ struct rpcrdma_mr * {struct rpcrdma_xprt *r_xprt = mr->mr_xprt; -trace_xprtrdma_dma_unmap (mr); +trace_xprtrdma_mr_unmap (mr);ib_dma_unmap_sg (r_xprt->rx_ia.ri_device, mr->mr_sg, mr->mr_nents, mr->mr_dir);__rpcrdma_mr_put (&r_xprt->rx_buf, mr);
PATCH 5/22 xprtrdma: Refactor chunk encoding
In reply to Chuck LeverMove the "not present" case into the individual chunk encoders. This improves code organization and readability, and prepares the marsh- aling code path for handling cases like "Write + Reply".
The reason for the original organization was to optimize this code path for the case where there there are no chunks. The optimization was inconsequential.
--- net/sunrpc/xprtrdma/rpc_rdma.c | 37 ++++++++++++++++--------------------- 1 file changed, 16 insertions(+), 21 deletions(-)
net/sunrpc/xprtrdma/rpc_rdma.c +16 -21
@@ -346,6 +346,9 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt,unsigned int pos;int nsegs; +if (rtype == rpcrdma_noch) + goto done; + pos = rqst->rq_snd_buf.head[0 ].iov_len;if (rtype == rpcrdma_areadch) pos =0 ;
@@ -370,7 +373,8 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt, nsegs -= mr->mr_nents; }while (nsegs); -return 0 ; +done: +return encode_item_not_present (xdr); }/* Register and XDR encode the Write list. Supports encoding a list
@@ -398,6 +402,9 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt,int nsegs, nchunks;__be32 *segcount; +if (wtype != rpcrdma_writech) + goto done; + seg = req->rl_segments; nsegs = rpcrdma_convert_iovs(r_xprt, &rqst->rq_rcv_buf, rqst->rq_rcv_buf.head[0].iov_len,
@@ -433,7 +440,8 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt,/* Update count of segments in this Write chunk */ *segcount =cpu_to_be32 (nchunks); -return 0 ; +done: +return encode_item_not_present (xdr); }/* Register and XDR encode the Reply chunk. Supports encoding an array
@@ -458,6 +466,9 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt,int nsegs, nchunks;__be32 *segcount; +if (wtype != rpcrdma_replych) +return encode_item_not_present (xdr); + seg = req->rl_segments; nsegs =rpcrdma_convert_iovs (r_xprt, &rqst->rq_rcv_buf,0 , wtype, seg);if (nsegs <0 )
@@ -828,31 +839,15 @@ static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt,* send a Call message with a Position Zero Read chunk and a * regular Read chunk at the same time. */ -if (rtype != rpcrdma_noch) { - ret =rpcrdma_encode_read_list (r_xprt, req, rqst, rtype); -if (ret) - goto out_err; - } - ret =encode_item_not_present (xdr); + ret =rpcrdma_encode_read_list (r_xprt, req, rqst, rtype);if (ret) goto out_err; - -if (wtype == rpcrdma_writech) { - ret =rpcrdma_encode_write_list (r_xprt, req, rqst, wtype); -if (ret) - goto out_err; - } - ret =encode_item_not_present (xdr); + ret =rpcrdma_encode_write_list (r_xprt, req, rqst, wtype);if (ret) goto out_err; - -if (wtype != rpcrdma_replych) - ret =encode_item_not_present (xdr); -else - ret =rpcrdma_encode_reply_chunk (r_xprt, req, rqst, wtype); + ret =rpcrdma_encode_reply_chunk (r_xprt, req, rqst, wtype);if (ret) goto out_err; -trace_xprtrdma_marshal (rqst,xdr_stream_pos (xdr), rtype, wtype); ret = rpcrdma_prepare_send_sges(r_xprt, req, xdr_stream_pos(xdr),
PATCH 6/22 xprtrdma: Refactor chunktype handling
In reply to Chuck LeverThis is a pre-requisite for adding support for sending both a Write chunk and a Reply chunk in the same RPC. It is a refactoring/clean up patch, so no functional change is expected.
rpcrdma_convert_iovs is a helper function that is invoked by each chunk encoder. It's passed an enum rpcrdma_chunktype variable that indicates either the set of chunks needed for the Call arguments or the Reply results.
While marshaling the same RPC, rpcrdma_convert_iovs is invoked once for the Write chunk and once for the Reply chunk. But it's passed the same @type both times. Thus it can't distinguish between when it's called to encode the Write chunk versus when it's called to encode the Reply chunk.
Let's refactor rpcrdma_convert_iovs so it needs just a detail about how the chunk should be handled. That detail is different for Write chunks and Reply chunks.
Looking at this change I realized that "wtype" is unhelpfully named in the entire marshaling code path, since it will include both the Write chunk list and the Reply chunk. As part of refactoring rpcrdma_convert_iovs, rename wtype to better fit its purpose. Update "rtype" as well to match this convention.
--- include/trace/events/rpcrdma.h | 18 ++++--- net/sunrpc/xprtrdma/rpc_rdma.c | 100 +++++++++++++++++++++------------------ net/sunrpc/xprtrdma/xprt_rdma.h | 2 - 3 files changed, 63 insertions(+), 57 deletions(-)
include/trace/events/rpcrdma.h +9 -9
@@ -459,11 +459,11 @@TP_PROTO ( const struct rpc_rqst *rqst, unsigned int hdrlen, - unsigned int rtype, - unsigned int wtype + unsigned int argtype, + unsigned int restype ), -TP_ARGS (rqst, hdrlen, rtype, wtype), +TP_ARGS (rqst, hdrlen, argtype, restype),TP_STRUCT__entry ( __field(unsigned int, task_id)
@@ -473,8 +473,8 @@ __field(unsigned int, headlen) __field(unsigned int, pagelen) __field(unsigned int, taillen) - __field(unsigned int, rtype) - __field(unsigned int, wtype) + __field(unsigned int, argtype) + __field(unsigned int, restype) ), TP_fast_assign(
@@ -485,16 +485,16 @@ __entry->headlen = rqst->rq_snd_buf.head[0 ].iov_len; __entry->pagelen = rqst->rq_snd_buf.page_len; __entry->taillen = rqst->rq_snd_buf.tail[0 ].iov_len; - __entry->rtype = rtype; - __entry->wtype = wtype; + __entry->argtype = argtype; + __entry->restype = restype; ),TP_printk ("task:%u@%u xid=0x%08x: hdr=%u xdr=%u/%u/%u %s/%s" , __entry->task_id, __entry->client_id, __entry->xid, __entry->hdrlen, __entry->headlen, __entry->pagelen, __entry->taillen, -xprtrdma_show_chunktype (__entry->rtype), -xprtrdma_show_chunktype (__entry->wtype) +xprtrdma_show_chunktype (__entry->argtype), +xprtrdma_show_chunktype (__entry->restype) ) );
net/sunrpc/xprtrdma/rpc_rdma.c +53 -47
231 lines not on this page · raw message
net/sunrpc/xprtrdma/xprt_rdma.h +1 -1
@@ -633,7 +633,7 @@ enum rpcrdma_chunktype {int rpcrdma_prepare_send_sges (struct rpcrdma_xprt *r_xprt,struct rpcrdma_req *req,u32 hdrlen,struct xdr_buf *xdr, -enum rpcrdma_chunktype rtype); +enum rpcrdma_chunktype argtype);void rpcrdma_unmap_sendctx (struct rpcrdma_sendctx *sc);int rpcrdma_marshal_req (struct rpcrdma_xprt *r_xprt,struct rpc_rqst *rqst);void rpcrdma_set_max_header_sizes (struct rpcrdma_xprt *);
PATCH 7/22 xprtrdma: Support Write+Reply Replies
In reply to Chuck LeverCurrently the client handles a large NFS READ request by providing the server with a Write chunk, and expecting that the non-payload part of the RPC Reply will always fit inline.
When the inline threshold is small (for instance, when talking to a server that uses a 1024-byte threshold) the non-payload part of the Reply might not fit inline in certain rare cases. The server has to drop the Reply or return an ERR_CHUNK and the RPC transaction fails.
Let's add a little logic to recognize when the non-payload part of an NFS READ might be large, and marshal both a Write chunk and a Reply chunk to enable the server to send the payload in the Write chunk and the large non-payload part in the Reply chunk.
I've never seen this failure in the wild.
--- include/trace/events/rpcrdma.h | 4 ++ net/sunrpc/xprtrdma/rpc_rdma.c | 63 +++++++++++++++++++++++++-------------- net/sunrpc/xprtrdma/xprt_rdma.h | 3 +- 3 files changed, 46 insertions(+), 24 deletions(-)
include/trace/events/rpcrdma.h +3 -1
@@ -446,6 +446,7 @@TRACE_DEFINE_ENUM (rpcrdma_areadch);TRACE_DEFINE_ENUM (rpcrdma_writech);TRACE_DEFINE_ENUM (rpcrdma_replych); +TRACE_DEFINE_ENUM (rpcrdma_writereply);#define xprtrdma_show_chunktype (x) \ __print_symbolic(x, \
@@ -453,7 +454,8 @@ { rpcrdma_readch, "read list" }, \ { rpcrdma_areadch, "*read list" }, \ { rpcrdma_writech, "write list" }, \ - { rpcrdma_replych, "reply chunk" }) + { rpcrdma_replych, "reply chunk" }, \ + { rpcrdma_writereply, "write+reply" }) TRACE_EVENT(xprtrdma_marshal, TP_PROTO(
net/sunrpc/xprtrdma/rpc_rdma.c +41 -22
138 lines not on this page · raw message
net/sunrpc/xprtrdma/xprt_rdma.h +2 -1
@@ -627,7 +627,8 @@ enum rpcrdma_chunktype { rpcrdma_readch, rpcrdma_areadch, rpcrdma_writech, - rpcrdma_replych + rpcrdma_replych, + rpcrdma_writereply, }; int rpcrdma_prepare_send_sges(struct rpcrdma_xprt *r_xprt,
PATCH 8/22 sunrpc: Fix connect metrics
In reply to Chuck LeverFor TCP, the logic in xprt_connect_status is currently never invoked to record a successful connection. Commit 2a4919919a97 ("SUNRPC:
Return EAGAIN instead of ENOTCONN when waking up xprt->pending") changed the way TCP xprt's are awoken after a connect succeeds.
Instead, change connection-oriented transports to bump connect_count and compute connect_time the moment that XPRT_CONNECTED is set.
--- net/sunrpc/xprt.c | 10 +++------- net/sunrpc/xprtrdma/transport.c | 6 +++++- net/sunrpc/xprtsock.c | 10 ++++++---- 3 files changed, 14 insertions(+), 12 deletions(-)
net/sunrpc/xprt.c +3 -7
@@ -791,15 +791,11 @@ static void xprt_connect_status(struct rpc_task *task) {struct rpc_xprt *xprt = task->tk_rqstp->rq_xprt; -if (task->tk_status ==0 ) { - xprt->stat.connect_count++; - xprt->stat.connect_time += (long )jiffies - xprt->stat.connect_start; +switch (task->tk_status) { +case 0 :dprintk ("RPC: %5u xprt_connect_status: connection established\n" , task->tk_pid); -return ; - } - -switch (task->tk_status) { +break ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :
net/sunrpc/xprtrdma/transport.c +5 -1
@@ -242,8 +242,12 @@spin_lock_bh (&xprt->transport_lock);if (ep->rep_connected >0 ) { -if (!xprt_test_and_set_connected (xprt)) +if (!xprt_test_and_set_connected (xprt)) { + xprt->stat.connect_count++; + xprt->stat.connect_time += (long )jiffies - + xprt->stat.connect_start;xprt_wake_pending_tasks (xprt,0 ); + } }else {if (xprt_test_and_clear_connected (xprt))xprt_wake_pending_tasks (xprt, -ENOTCONN );
net/sunrpc/xprtsock.c +6 -4
@@ -1611,6 +1611,9 @@ static void xs_tcp_state_change(struct sock *sk)clear_bit (XPRT_SOCK_CONNECTING , &transport->sock_state);xprt_clear_connecting (xprt); + xprt->stat.connect_count++; + xprt->stat.connect_time += (long )jiffies - + xprt->stat.connect_start;xprt_wake_pending_tasks (xprt, -EAGAIN ); }spin_unlock (&xprt->transport_lock);
@@ -2029,8 +2032,6 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt, }/* Tell the socket layer to start connecting... */ - xprt->stat.connect_count++; - xprt->stat.connect_start = jiffies;return kernel_connect (sock,xs_addr (xprt), xprt->addrlen,0 ); }
@@ -2062,6 +2063,9 @@ static int xs_local_setup_socket(struct sock_xprt *transport)case 0 :dprintk ("RPC: xprt %p connected to %s\n" , xprt, xprt->address_strings[RPC_DISPLAY_ADDR ]); + xprt->stat.connect_count++; + xprt->stat.connect_time += (long )jiffies - + xprt->stat.connect_start;xprt_set_connected (xprt);case -ENOBUFS :break ;
@@ -2387,8 +2391,6 @@ static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock)xs_set_memalloc (xprt);/* Tell the socket layer to start connecting... */ - xprt->stat.connect_count++; - xprt->stat.connect_start = jiffies;set_bit (XPRT_SOCK_CONNECTING , &transport->sock_state); ret =kernel_connect (sock,xs_addr (xprt), xprt->addrlen,O_NONBLOCK );switch (ret) {
Re: [PATCH v1 08/22] sunrpc: Fix connect metrics
Hi Chuck,
Quoted 20 lines · Chuck Lever
For TCP, the logic in
xprt_connect_statusis currently never invoked to record a successful connection. Commit2a4919919a97("SUNRPC:
Return EAGAIN instead of ENOTCONN when waking upxprt->pending") changed the way TCP xprt's are awoken after a connect succeeds.Instead, change connection-oriented transports to bump
connect_countand computeconnect_timethe moment thatXPRT_CONNECTEDis set.Signed-off-by Chuck Lever <[email protected]>--- net/sunrpc/xprt.c | 10 +++------- net/sunrpc/xprtrdma/transport.c | 6 +++++- net/sunrpc/xprtsock.c | 10 ++++++---- 3 files changed, 14 insertions(+), 12 deletions(-)net/sunrpc/xprt.c +0
-0
net/sunrpc/xprt.c static void xprt_connect_status(struct rpc_task *task) · line 792
Quoted 2 lines
net/sunrpc/xprt.c +0
-0@@ -791,15 +791,11 @@ static void xprt_connect_status(struct rpc_task *task) {struct rpc_xprt *xprt = task->tk_rqstp->rq_xprt;
Looks like the only remaining user of the xprt is in a dprintk() towards the bottom of this function to get the servername. This is giving me an unused variable warning when CONFIG_SUNRPC_DEBUG=n, so I'm wondering if you can take out the variable and just access the server name the long way (task->tk_rqstp-
Quoted 1 line
rq_xprt->servername)?
Thanks,
Anna
Quoted 82 lines
net/sunrpc/xprt.c +0
-3-if (task->tk_status ==0 ) { - xprt->stat.connect_count++; - xprt->stat.connect_time += (long)jiffies - xprt-Quoted 1 line
stat.connect_start;
net/sunrpc/xprt.c +2
-0+switch (task->tk_status) { +case 0 : dprintk("RPC: %5u xprt_connect_status: connectionestablished\n",
net/sunrpc/xprt.c +1
-4task->tk_pid); -return ; - } - -switch (task->tk_status) { +break ;case -ECONNREFUSED :case -ECONNRESET :case -ECONNABORTED :net/sunrpc/xprtrdma/transport.c +5
-1@@ -242,8 +242,12 @@spin_lock_bh (&xprt->transport_lock);if (ep->rep_connected >0 ) { -if (!xprt_test_and_set_connected (xprt)) +if (!xprt_test_and_set_connected (xprt)) { + xprt->stat.connect_count++; + xprt->stat.connect_time += (long )jiffies - + xprt->stat.connect_start;xprt_wake_pending_tasks (xprt,0 ); + } }else {if (xprt_test_and_clear_connected (xprt))xprt_wake_pending_tasks (xprt, -ENOTCONN );net/sunrpc/xprtsock.c +6
-2@@ -1611,6 +1611,9 @@ static void xs_tcp_state_change(struct sock *sk)clear_bit (XPRT_SOCK_CONNECTING , &transport->sock_state);xprt_clear_connecting (xprt); + xprt->stat.connect_count++; + xprt->stat.connect_time += (long )jiffies - + xprt->stat.connect_start;xprt_wake_pending_tasks (xprt, -EAGAIN ); }spin_unlock (&xprt->transport_lock);@@ -2029,8 +2032,6 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt, }/* Tell the socket layer to start connecting... */ - xprt->stat.connect_count++; - xprt->stat.connect_start = jiffies;return kernel_connect (sock,xs_addr (xprt), xprt->addrlen,0 ); }@@ -2062,6 +2063,9 @@ static int xs_local_setup_socket(struct sock_xprt *transport)case 0 :dprintk ("RPC: xprt %p connected to %s\n" , xprt, xprt->address_strings[RPC_DISPLAY_ADDR ]); + xprt->stat.connect_count++; + xprt->stat.connect_time += (long )jiffies - + xprt->stat.connect_start;xprt_set_connected (xprt);case -ENOBUFS :net/sunrpc/xprtsock.c +0
-2break ;@@ -2387,8 +2391,6 @@ static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock)xs_set_memalloc (xprt);/* Tell the socket layer to start connecting... */ - xprt->stat.connect_count++; - xprt->stat.connect_start = jiffies;set_bit (XPRT_SOCK_CONNECTING , &transport->sock_state); ret =kernel_connect (sock,xs_addr (xprt), xprt->addrlen,O_NONBLOCK );switch (ret) {
PATCH 9/22 sunrpc: Report connect_time in seconds
In reply to Chuck LeverThe way connection-oriented transports report connect_time is wrong:
it's supposed to be in seconds, not in jiffies.
--- net/sunrpc/xprtrdma/transport.c | 2 +- net/sunrpc/xprtsock.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
net/sunrpc/xprtrdma/transport.c +1 -1
@@ -776,7 +776,7 @@ void xprt_rdma_print_stats(struct rpc_xprt *xprt, struct seq_file *seq)0 ,/* need a local port? */ xprt->stat.bind_count, xprt->stat.connect_count, - xprt->stat.connect_time, + xprt->stat.connect_time /HZ , idle_time, xprt->stat.sends, xprt->stat.recvs,
net/sunrpc/xprtsock.c +2 -2
@@ -2563,7 +2563,7 @@ static void xs_local_print_stats(struct rpc_xprt *xprt, struct seq_file *seq)"%llu %llu %lu %llu %llu\n" , xprt->stat.bind_count, xprt->stat.connect_count, - xprt->stat.connect_time, + xprt->stat.connect_time /HZ , idle_time, xprt->stat.sends, xprt->stat.recvs,
@@ -2618,7 +2618,7 @@ static void xs_tcp_print_stats(struct rpc_xprt *xprt, struct seq_file *seq) transport->srcport, xprt->stat.bind_count, xprt->stat.connect_count, - xprt->stat.connect_time, + xprt->stat.connect_time /HZ , idle_time, xprt->stat.sends, xprt->stat.recvs,
PATCH 10/22 xprtrdma: Rename rpcrdma_conn_upcall
In reply to Chuck LeverClean up: Use a function name that is consistent with the RDMA core API and with other consumers. Because this is a function that is invoked from outside the rpcrdma.ko module, add an appropriate documenting comment.
--- include/trace/events/rpcrdma.h | 2 +- net/sunrpc/xprtrdma/verbs.c | 16 +++++++++++++--- 2 files changed, 14 insertions(+), 4 deletions(-)
include/trace/events/rpcrdma.h +1 -1
@@ -306,7 +306,7 @@** Connection events **/ -TRACE_EVENT(xprtrdma_conn_upcall, +TRACE_EVENT(xprtrdma_cm_event, TP_PROTO( const struct rpcrdma_xprt *r_xprt, struct rdma_cm_event *event
net/sunrpc/xprtrdma/verbs.c +13 -3
@@ -219,15 +219,25 @@rpcrdma_set_max_header_sizes (r_xprt); } +/** +* rpcrdma_cm_event_handler - Handle RDMA CM events +* @id: rdma_cm_id on which an event has occurred +* @event: details of the event +* +* Called with @id's mutex held. Returns 1 if caller should +* destroy @id, otherwise 0. +*/ static int -rpcrdma_conn_upcall (struct rdma_cm_id *id,struct rdma_cm_event *event) +rpcrdma_cm_event_handler (struct rdma_cm_id *id,struct rdma_cm_event *event) {struct rpcrdma_xprt *xprt = id->context;struct rpcrdma_ia *ia = &xprt->rx_ia;struct rpcrdma_ep *ep = &xprt->rx_ep;int connstate =0 ; -trace_xprtrdma_conn_upcall (xprt, event); +might_sleep (); + +trace_xprtrdma_cm_event (xprt, event);switch (event->event) {case RDMA_CM_EVENT_ADDR_RESOLVED :case RDMA_CM_EVENT_ROUTE_RESOLVED :
@@ -308,7 +318,7 @@init_completion (&ia->ri_done);init_completion (&ia->ri_remove_done); - id =rdma_create_id (xprt->rx_xprt.xprt_net, rpcrdma_conn_upcall, + id =rdma_create_id (xprt->rx_xprt.xprt_net, rpcrdma_cm_event_handler, xprt,RDMA_PS_TCP ,IB_QPT_RC );if (IS_ERR (id)) { rc =PTR_ERR (id);
PATCH 11/22 xprtrdma: Conventional variable names in rpcrdma_conn_upcall
In reply to Chuck LeverClean up: The convention throughout other parts of xprtrdma is to name variables of type struct rpcrdma_xprt "r_xprt", not "xprt". This convention enables the use of the name "xprt" for a "struct rpc_xprt" type variable, as in other parts of the RPC client.
--- net/sunrpc/xprtrdma/verbs.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)
net/sunrpc/xprtrdma/verbs.c +12 -11
@@ -230,14 +230,15 @@static int rpcrdma_cm_event_handler (struct rdma_cm_id *id,struct rdma_cm_event *event) { -struct rpcrdma_xprt *xprt = id->context; -struct rpcrdma_ia *ia = &xprt->rx_ia; -struct rpcrdma_ep *ep = &xprt->rx_ep; +struct rpcrdma_xprt *r_xprt = id->context; +struct rpcrdma_ia *ia = &r_xprt->rx_ia; +struct rpcrdma_ep *ep = &r_xprt->rx_ep; +struct rpc_xprt *xprt = &r_xprt->rx_xprt;int connstate =0 ;might_sleep (); -trace_xprtrdma_cm_event (xprt, event); +trace_xprtrdma_cm_event (r_xprt, event);switch (event->event) {case RDMA_CM_EVENT_ADDR_RESOLVED :case RDMA_CM_EVENT_ROUTE_RESOLVED :
@@ -256,11 +257,11 @@#if IS_ENABLED (CONFIG_SUNRPC_DEBUG )pr_info ("rpcrdma: removing device %s for %s:%s\n" , ia->ri_device->name, -rpcrdma_addrstr (xprt),rpcrdma_portstr (xprt)); +rpcrdma_addrstr (r_xprt),rpcrdma_portstr (r_xprt));#endif set_bit (RPCRDMA_IAF_REMOVING , &ia->ri_flags); ep->rep_connected = -ENODEV ; -xprt_force_disconnect (&xprt->rx_xprt); +xprt_force_disconnect (xprt);wait_for_completion (&ia->ri_remove_done); ia->ri_id =NULL ;
@@ -268,9 +269,9 @@/* Return 1 to ensure the core destroys the id. */ return 1 ;case RDMA_CM_EVENT_ESTABLISHED : - ++xprt->rx_xprt.connect_cookie; + ++xprt->connect_cookie; connstate =1 ; -rpcrdma_update_connect_private (xprt, &event->param.conn); +rpcrdma_update_connect_private (r_xprt, &event->param.conn); goto connected;case RDMA_CM_EVENT_CONNECT_ERROR : connstate = -ENOTCONN ;
@@ -280,14 +281,14 @@ goto connected;case RDMA_CM_EVENT_REJECTED :dprintk ("rpcrdma: connection to %s:%s rejected: %s\n" , -rpcrdma_addrstr (xprt),rpcrdma_portstr (xprt), +rpcrdma_addrstr (r_xprt),rpcrdma_portstr (r_xprt),rdma_reject_msg (id, event->status)); connstate = -ECONNREFUSED ;if (event->status ==IB_CM_REJ_STALE_CONN ) connstate = -EAGAIN ; goto connected;case RDMA_CM_EVENT_DISCONNECTED : - ++xprt->rx_xprt.connect_cookie; + ++xprt->connect_cookie; connstate = -ECONNABORTED ; connected: ep->rep_connected = connstate;
@@ -297,7 +298,7 @@default :dprintk ("RPC: %s: %s:%s on %s/%s (ep 0x%p): %s\n" , __func__, -rpcrdma_addrstr (xprt),rpcrdma_portstr (xprt), +rpcrdma_addrstr (r_xprt),rpcrdma_portstr (r_xprt), ia->ri_device->name, ia->ri_ops->ro_displayname, ep,rdma_event_msg (event->event));break ;
PATCH 12/22 xprtrdma: Eliminate "connstate" variable from rpcrdma_conn_upcall()
In reply to Chuck LeverClean up.
Since commit 173b8f49b3af ("xprtrdma: Demote "connect" log messages") there has been no need to initialize connstat to zero. In fact, in this code path there's now no reason not to set rep_connected directly.
--- net/sunrpc/xprtrdma/verbs.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
net/sunrpc/xprtrdma/verbs.c +6 -8
@@ -234,7 +234,6 @@struct rpcrdma_ia *ia = &r_xprt->rx_ia;struct rpcrdma_ep *ep = &r_xprt->rx_ep;struct rpc_xprt *xprt = &r_xprt->rx_xprt; -int connstate =0 ;might_sleep ();
@@ -270,28 +269,27 @@return 1 ;case RDMA_CM_EVENT_ESTABLISHED : ++xprt->connect_cookie; - connstate =1 ; + ep->rep_connected =1 ;rpcrdma_update_connect_private (r_xprt, &event->param.conn); goto connected;case RDMA_CM_EVENT_CONNECT_ERROR : - connstate = -ENOTCONN ; + ep->rep_connected = -ENOTCONN ; goto connected;case RDMA_CM_EVENT_UNREACHABLE : - connstate = -ENETUNREACH ; + ep->rep_connected = -ENETUNREACH ; goto connected;case RDMA_CM_EVENT_REJECTED :dprintk ("rpcrdma: connection to %s:%s rejected: %s\n" ,rpcrdma_addrstr (r_xprt),rpcrdma_portstr (r_xprt),rdma_reject_msg (id, event->status)); - connstate = -ECONNREFUSED ; + ep->rep_connected = -ECONNREFUSED ;if (event->status ==IB_CM_REJ_STALE_CONN ) - connstate = -EAGAIN ; + ep->rep_connected = -EAGAIN ; goto connected;case RDMA_CM_EVENT_DISCONNECTED : ++xprt->connect_cookie; - connstate = -ECONNABORTED ; + ep->rep_connected = -ECONNABORTED ; connected: - ep->rep_connected = connstate;rpcrdma_conn_func (ep);wake_up_all (&ep->rep_connect_wait);/*FALLTHROUGH*/
PATCH 13/22 xprtrdma: Re-organize the switch() in rpcrdma_conn_upcall
In reply to Chuck LeverClean up: Eliminate the FALLTHROUGH into the default arm to make the switch easier to read.
Also, as long as I'm here, do not display the memory address of the target rpcrdma_ep. A hashed memory address is of marginal use here.
--- net/sunrpc/xprtrdma/verbs.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
net/sunrpc/xprtrdma/verbs.c +8 -9
@@ -243,15 +243,15 @@case RDMA_CM_EVENT_ROUTE_RESOLVED : ia->ri_async_rc =0 ;complete (&ia->ri_done); -break ; +return 0 ;case RDMA_CM_EVENT_ADDR_ERROR : ia->ri_async_rc = -EPROTO ;complete (&ia->ri_done); -break ; +return 0 ;case RDMA_CM_EVENT_ROUTE_ERROR : ia->ri_async_rc = -ENETUNREACH ;complete (&ia->ri_done); -break ; +return 0 ;case RDMA_CM_EVENT_DEVICE_REMOVAL :#if IS_ENABLED (CONFIG_SUNRPC_DEBUG ) pr_info("rpcrdma: removing device %s for %s:%s\n",
@@ -292,16 +292,15 @@ connected:rpcrdma_conn_func (ep);wake_up_all (&ep->rep_connect_wait); -/*FALLTHROUGH*/ +break ;default : -dprintk ("RPC: %s: %s:%s on %s/%s (ep 0x%p): %s\n" , - __func__, -rpcrdma_addrstr (r_xprt),rpcrdma_portstr (r_xprt), - ia->ri_device->name, ia->ri_ops->ro_displayname, - ep,rdma_event_msg (event->event));break ; } +dprintk ("RPC: %s: %s:%s on %s/%s: %s\n" , __func__, +rpcrdma_addrstr (r_xprt),rpcrdma_portstr (r_xprt), + ia->ri_device->name, ia->ri_ops->ro_displayname, +rdma_event_msg (event->event));return 0 ; }
PATCH 14/22 xprtrdma: Simplify RPC wake-ups on connect
In reply to Chuck LeverCurrently, when a connection is established, rpcrdma_conn_upcall invokes rpcrdma_conn_func and then wake_up_all(&ep->rep_connect_wait). The former wakes waiting RPCs, but the connect worker is not done yet, and that leads to races, double wakes, and difficulty understanding how this logic is supposed to work.
Instead, collect all the "connection established" logic in the connect worker (xprt_rdma_connect_worker). A disconnect worker is retained to handle provider upcalls safely.
--- net/sunrpc/xprtrdma/transport.c | 54 +++++++++++++-------------------------- net/sunrpc/xprtrdma/verbs.c | 42 ++++++++++++++++++++++++------ net/sunrpc/xprtrdma/xprt_rdma.h | 4 +-- 3 files changed, 52 insertions(+), 48 deletions(-)
net/sunrpc/xprtrdma/transport.c +18 -36
@@ -225,51 +225,35 @@ } } -void -rpcrdma_conn_func (struct rpcrdma_ep *ep) -{ -schedule_delayed_work (&ep->rep_connect_worker,0 ); -} - -void -rpcrdma_connect_worker (struct work_struct *work) +/** +* xprt_rdma_connect_worker - establish connection in the background +* @work: worker thread context +* +* Requester holds the xprt's send lock to prevent activity on this +* transport while a fresh connection is being established. RPC tasks +* sleep on the xprt's pending queue waiting for connect to complete. +*/ +static void +xprt_rdma_connect_worker (struct work_struct *work) { -struct rpcrdma_ep *ep = - container_of(work, struct rpcrdma_ep, rep_connect_worker.work); -struct rpcrdma_xprt *r_xprt = - container_of(ep, struct rpcrdma_xprt, rx_ep); + struct rpcrdma_xprt *r_xprt = container_of(work, struct rpcrdma_xprt, + rx_connect_worker.work);struct rpc_xprt *xprt = &r_xprt->rx_xprt; +int rc; -spin_lock_bh (&xprt->transport_lock); -if (ep->rep_connected >0 ) { + rc =rpcrdma_ep_connect (&r_xprt->rx_ep, &r_xprt->rx_ia); +xprt_clear_connecting (xprt); +if (r_xprt->rx_ep.rep_connected >0 ) {if (!xprt_test_and_set_connected (xprt)) { xprt->stat.connect_count++; xprt->stat.connect_time += (long )jiffies - xprt->stat.connect_start; -xprt_wake_pending_tasks (xprt,0 ); +xprt_wake_pending_tasks (xprt, -EAGAIN ); } }else {if (xprt_test_and_clear_connected (xprt)) -xprt_wake_pending_tasks (xprt, -ENOTCONN ); +xprt_wake_pending_tasks (xprt, rc); } -spin_unlock_bh (&xprt->transport_lock); -} - -static void -xprt_rdma_connect_worker (struct work_struct *work) -{ - struct rpcrdma_xprt *r_xprt = container_of(work, struct rpcrdma_xprt, - rx_connect_worker.work); -struct rpc_xprt *xprt = &r_xprt->rx_xprt; -int rc =0 ; - -xprt_clear_connected (xprt); - - rc =rpcrdma_ep_connect (&r_xprt->rx_ep, &r_xprt->rx_ia); -if (rc) -xprt_wake_pending_tasks (xprt, rc); - -xprt_clear_connecting (xprt); } static void
@@ -302,8 +286,6 @@cancel_delayed_work_sync (&r_xprt->rx_connect_worker); -xprt_clear_connected (xprt); -rpcrdma_ep_destroy (&r_xprt->rx_ep, &r_xprt->rx_ia);rpcrdma_buffer_destroy (&r_xprt->rx_buf);rpcrdma_ia_close (&r_xprt->rx_ia);
net/sunrpc/xprtrdma/verbs.c +33 -9
97 lines not on this page · raw message
net/sunrpc/xprtrdma/xprt_rdma.h +1 -3
@@ -101,7 +101,7 @@ struct rpcrdma_ep {wait_queue_head_t rep_connect_wait;struct rpcrdma_connect_private rep_cm_private;struct rdma_conn_param rep_remote_cma; -struct delayed_work rep_connect_worker; +struct delayed_work rep_disconnect_worker; };/* Pre-allocate extra Work Requests for handling backward receives
@@ -556,7 +556,6 @@ int rpcrdma_ep_create(struct rpcrdma_ep *, struct rpcrdma_ia *, struct rpcrdma_create_data_internal *); void rpcrdma_ep_destroy(struct rpcrdma_ep *, struct rpcrdma_ia *);int rpcrdma_ep_connect (struct rpcrdma_ep *,struct rpcrdma_ia *); -void rpcrdma_conn_func (struct rpcrdma_ep *ep);void rpcrdma_ep_disconnect (struct rpcrdma_ep *,struct rpcrdma_ia *); int rpcrdma_ep_post(struct rpcrdma_ia *, struct rpcrdma_ep *,
@@ -655,7 +654,6 @@ static inline void rpcrdma_set_xdrlen(struct xdr_buf *xdr, size_t len)extern unsigned int xprt_rdma_max_inline_read;void xprt_rdma_format_addresses (struct rpc_xprt *xprt,struct sockaddr *sap);void xprt_rdma_free_addresses (struct rpc_xprt *xprt); -void rpcrdma_connect_worker (struct work_struct *work);void xprt_rdma_print_stats (struct rpc_xprt *xprt,struct seq_file *seq);int xprt_rdma_init (void );void xprt_rdma_cleanup (void );
PATCH 15/22 xprtrdma: Rename rpcrdma_qp_async_error_upcall
In reply to Chuck LeverClean up: Use a function name that is consistent with the RDMA core API and with other consumers. Because this is a function that is invoked from outside the rpcrdma.ko module, add an appropriate documenting comment.
--- include/trace/events/rpcrdma.h | 2 +- net/sunrpc/xprtrdma/verbs.c | 14 +++++++++++--- 2 files changed, 12 insertions(+), 4 deletions(-)
include/trace/events/rpcrdma.h +1 -1
@@ -377,7 +377,7 @@DEFINE_RXPRT_EVENT (xprtrdma_reconnect);DEFINE_RXPRT_EVENT (xprtrdma_inject_dsc); -TRACE_EVENT(xprtrdma_qp_error, +TRACE_EVENT(xprtrdma_qp_event, TP_PROTO( const struct rpcrdma_xprt *r_xprt, const struct ib_event *event
net/sunrpc/xprtrdma/verbs.c +11 -3
@@ -127,14 +127,22 @@xprt_force_disconnect (&r_xprt->rx_xprt); } +/** +* rpcrdma_qp_event_handler - Handle one QP event (error notification) +* @event: details of the event +* @context: ep that owns QP where event occurred +* +* Called from RDMA provider (device driver) possibly in a non-blocking +* context. +*/ static void -rpcrdma_qp_async_error_upcall (struct ib_event *event,void *context) +rpcrdma_qp_event_handler (struct ib_event *event,void *context) {struct rpcrdma_ep *ep = context; struct rpcrdma_xprt *r_xprt = container_of(ep, struct rpcrdma_xprt, rx_ep); -trace_xprtrdma_qp_error (r_xprt, event); +trace_xprtrdma_qp_event (r_xprt, event);pr_err ("rpcrdma: %s on device %s ep %p\n" ,ib_event_msg (event->event), event->device->name, context);
@@ -547,7 +555,7 @@if (rc)return rc; - ep->rep_attr.event_handler = rpcrdma_qp_async_error_upcall; + ep->rep_attr.event_handler = rpcrdma_qp_event_handler; ep->rep_attr.qp_context = ep; ep->rep_attr.srq =NULL ; ep->rep_attr.cap.max_send_sge = max_sge;
PATCH 16/22 xprtrdma: Remove memory address of "ep" from an error message
In reply to Chuck LeverClean up: Replace the hashed memory address of the target rpcrdma_ep with the server's IP address and port. The server address is more useful in an administrative error message.
--- net/sunrpc/xprtrdma/verbs.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
net/sunrpc/xprtrdma/verbs.c +3 -2
@@ -143,8 +143,9 @@ rx_ep);trace_xprtrdma_qp_event (r_xprt, event); -pr_err ("rpcrdma: %s on device %s ep %p\n" , -ib_event_msg (event->event), event->device->name, context); +pr_err ("rpcrdma: %s on device %s connected to %s:%s\n" , +ib_event_msg (event->event), event->device->name, +rpcrdma_addrstr (r_xprt),rpcrdma_portstr (r_xprt));if (ep->rep_connected ==1 ) { ep->rep_connected = -EIO ;
PATCH 17/22 svcrdma: Don't disable BH's in backchannel
In reply to Chuck LeverClean up: This code was copied from xprtsock.c and backchannel_rqst.c. For rpcrdma, the backchannel runs exclusively in process context, thus disabling bottom-halves is unnecessary.
--- net/sunrpc/xprtrdma/backchannel.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
net/sunrpc/xprtrdma/backchannel.c +8 -8
@@ -54,9 +54,9 @@ static int rpcrdma_bc_setup_reqs(struct rpcrdma_xprt *r_xprt,INIT_LIST_HEAD (&rqst->rq_list);INIT_LIST_HEAD (&rqst->rq_bc_list);__set_bit (RPC_BC_PA_IN_USE , &rqst->rq_bc_pa_state); -spin_lock_bh (&xprt->bc_pa_lock); +spin_lock (&xprt->bc_pa_lock);list_add (&rqst->rq_bc_pa_list, &xprt->bc_pa_list); -spin_unlock_bh (&xprt->bc_pa_lock); +spin_unlock (&xprt->bc_pa_lock); size = r_xprt->rx_data.inline_rsize; rb =rpcrdma_alloc_regbuf (size,DMA_TO_DEVICE ,GFP_KERNEL );
@@ -228,16 +228,16 @@ void xprt_rdma_bc_destroy(struct rpc_xprt *xprt, unsigned int reqs)struct rpcrdma_xprt *r_xprt =rpcx_to_rdmax (xprt);struct rpc_rqst *rqst, *tmp; -spin_lock_bh (&xprt->bc_pa_lock); +spin_lock (&xprt->bc_pa_lock);list_for_each_entry_safe (rqst, tmp, &xprt->bc_pa_list, rq_bc_pa_list) {list_del (&rqst->rq_bc_pa_list); -spin_unlock_bh (&xprt->bc_pa_lock); +spin_unlock (&xprt->bc_pa_lock);rpcrdma_bc_free_rqst (r_xprt, rqst); -spin_lock_bh (&xprt->bc_pa_lock); +spin_lock (&xprt->bc_pa_lock); } -spin_unlock_bh (&xprt->bc_pa_lock); +spin_unlock (&xprt->bc_pa_lock); }/**
@@ -255,9 +255,9 @@ void xprt_rdma_bc_free_rqst(struct rpc_rqst *rqst)rpcrdma_recv_buffer_put (req->rl_reply); req->rl_reply =NULL ; -spin_lock_bh (&xprt->bc_pa_lock); +spin_lock (&xprt->bc_pa_lock);list_add_tail (&rqst->rq_bc_pa_list, &xprt->bc_pa_list); -spin_unlock_bh (&xprt->bc_pa_lock); +spin_unlock (&xprt->bc_pa_lock); }/**
PATCH 18/22 xprtrdma: Move rb_flags initialization
In reply to Chuck LeverClean up: rb_flags can be used for other things besides RPCRDMA_BUF_F_EMPTY_SCQ, so initialize it in a generic spot instead of in a send-completion-queue-related helper.
--- net/sunrpc/xprtrdma/verbs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/xprtrdma/verbs.c +1 -1
@@ -918,7 +918,6 @@ static int rpcrdma_sendctxs_create(struct rpcrdma_xprt *r_xprt) sc->sc_xprt = r_xprt; buf->rb_sc_ctxs[i] = sc; } - buf->rb_flags =0 ;return 0 ;
@@ -1146,6 +1145,7 @@ struct rpcrdma_req *struct rpcrdma_buffer *buf = &r_xprt->rx_buf;int i, rc; + buf->rb_flags =0 ; buf->rb_max_requests = r_xprt->rx_data.max_requests; buf->rb_bc_srv_max_requests =0 ;spin_lock_init (&buf->rb_mrlock);
PATCH 19/22 xprtrdma: Report when there were zero posted Receives
In reply to Chuck LeverTo show that a caller did attempt to allocate and post more Receive buffers, the trace point in rpcrdma_post_recvs() should report when rpcrdma_post_recvs() was invoked but no new Receive buffers were posted.
--- net/sunrpc/xprtrdma/verbs.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
net/sunrpc/xprtrdma/verbs.c +5 -2
@@ -1521,9 +1521,11 @@ struct rpcrdma_regbuf *struct ib_recv_wr *wr, *bad_wr;int needed, count, rc; + rc =0 ; + count =0 ; needed = buf->rb_credits + (buf->rb_bc_srv_max_requests <<1 );if (buf->rb_posted_receives > needed) -return ; + goto out; needed -= buf->rb_posted_receives; count =0 ;
@@ -1559,7 +1561,7 @@ struct rpcrdma_regbuf * --needed; }if (!count) -return ; + goto out; rc =ib_post_recv (r_xprt->rx_ia.ri_id->qp, wr, (const struct ib_recv_wr **)&bad_wr);
@@ -1573,5 +1575,6 @@ struct rpcrdma_regbuf * } } buf->rb_posted_receives += count; +out:trace_xprtrdma_post_recvs (r_xprt, count, rc); }
PATCH 20/22 xprtrdma: Add documenting comments
In reply to Chuck LeverClean up: fill in or update documenting comments for transport switch entry points.
--- net/sunrpc/xprtrdma/transport.c | 36 +++++++++++++++++++++++------------- 1 file changed, 23 insertions(+), 13 deletions(-)
net/sunrpc/xprtrdma/transport.c +23 -13
@@ -256,6 +256,13 @@ } } +/** +* xprt_rdma_inject_disconnect - inject a connection fault +* @xprt: transport context +* +* If @xprt is connected, disconnect it to simulate spurious connection +* loss. +*/ static void xprt_rdma_inject_disconnect (struct rpc_xprt *xprt) {
@@ -266,16 +273,12 @@rdma_disconnect (r_xprt->rx_ia.ri_id); } -/* -* xprt_rdma_destroy +/** +* xprt_rdma_destroy - Full tear down of transport +* @xprt: doomed transport context * -* Destroy the xprt. -* Free all memory associated with the object, including its own. -* NOTE: none of the *destroy methods free memory for their top-level -* objects, even though they may have allocated it (they do free -* private memory). It's up to the caller to handle it. In this -* case (RDMA transport), all structure memory is inlined with the -* struct rpcrdma_xprt. +* Caller guarantees there will be no more calls to us with +* this @xprt. */ static void xprt_rdma_destroy (struct rpc_xprt *xprt)
@@ -428,11 +431,12 @@ }/** -* xprt_rdma_close - Close down RDMA connection -* @xprt: generic transport to be closed +* xprt_rdma_close - close a transport connection +* @xprt: transport context * -* Called during transport shutdown reconnect, or device -* removal. Caller holds the transport's write lock. +* Called during transport shutdown, reconnect, or device removal. +* Caller holds @xprt's send lock to prevent activity on this +* transport while the connection is torn down. */ static void xprt_rdma_close (struct rpc_xprt *xprt)
@@ -511,6 +515,12 @@xprt_force_disconnect (xprt); } +/** +* xprt_rdma_connect - try to establish a transport connection +* @xprt: transport state +* @task: RPC scheduler context +* +*/ static void xprt_rdma_connect (struct rpc_xprt *xprt,struct rpc_task *task) {
PATCH 21/22 xprtrdma: Clean up xprt_rdma_disconnect_inject
In reply to Chuck LeverClean up: Use the appropriate C macro instead of open-coding container_of() .
--- net/sunrpc/xprtrdma/transport.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
net/sunrpc/xprtrdma/transport.c +1 -2
@@ -266,8 +266,7 @@static void xprt_rdma_inject_disconnect (struct rpc_xprt *xprt) { - struct rpcrdma_xprt *r_xprt = container_of(xprt, struct rpcrdma_xprt, - rx_xprt); +struct rpcrdma_xprt *r_xprt =rpcx_to_rdmax (xprt);trace_xprtrdma_inject_dsc (r_xprt);rdma_disconnect (r_xprt->rx_ia.ri_id);
PATCH 22/22 xprtrdma: Squelch a sparse warning
In reply to Chuck Leverlinux/include/trace/events/rpcrdma.h:501:1: warning: expression using sizeof bool linux/include/trace/events/rpcrdma.h:501:1: warning: odd constant _Bool cast (ffffffffffffffff becomes 1)
--- include/trace/events/rpcrdma.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/trace/events/rpcrdma.h +1 -1
@@ -511,7 +511,7 @@TP_STRUCT__entry (__field (const void *,req )__field (int, num_sge) -__field (bool, signaled) +__field (int, signaled)__field (int, status) ),