RDMA/rxe: Use acquire/release for memory ordering
Subsystems (2)
PATCH v2 1/1 RDMA/rxe: Use acquire/release for memory ordering
Changed work and completion queues to use smp_load_acquire() and smp_store_release() to synchronize between driver and users. This commit goes with a matching series of commits in the rxe user space provider.
v2: Addressed same issue for kernel ULPs which use rxe_post_send/recv().
--- drivers/infiniband/sw/rxe/rxe_cq.c | 5 -- drivers/infiniband/sw/rxe/rxe_queue.h | 94 +++++++++++++++++---------- drivers/infiniband/sw/rxe/rxe_verbs.c | 11 ---- include/uapi/rdma/rdma_user_rxe.h | 21 ++++++ 4 files changed, 81 insertions(+), 50 deletions(-)
drivers/infiniband/sw/rxe/rxe_cq.c +0 -5
@@ -123,11 +123,6 @@ int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited)memcpy (producer_addr (cq->queue), cqe,sizeof (*cqe)); -/* make sure all changes to the CQ are written before we update the -* producer pointer -*/ -smp_wmb (); -advance_producer (cq->queue);spin_unlock_irqrestore (&cq->cq_lock, flags);
drivers/infiniband/sw/rxe/rxe_queue.h +60 -34
142 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_verbs.c +0 -11
@@ -244,11 +244,6 @@ static int post_one_recv(struct rxe_rq *rq, const struct ib_recv_wr *ibwr) recv_wqe->dma.cur_sge =0 ; recv_wqe->dma.sge_offset =0 ; -/* make sure all changes to the work queue are written before we -* update the producer pointer -*/ -smp_wmb (); -advance_producer (rq->queue);return 0 ;
@@ -633,12 +628,6 @@ static int post_one_send(struct rxe_qp *qp, const struct ib_send_wr *ibwr,if (unlikely (err)) goto err1; -/* -* make sure all changes to the work queue are -* written before we update the producer pointer -*/ -smp_wmb (); -advance_producer (sq->queue);spin_unlock_irqrestore (&qp->sq.sq_lock, flags);
include/uapi/rdma/rdma_user_rxe.h +21 -0
@@ -181,4 +181,25 @@ struct rxe_modify_srq_cmd { __aligned_u64 mmap_info_addr; }; +/* This data structure is stored at the base of work and +* completion queues shared between user space and kernel space. +* It contains the producer and consumer indices. Is also +* contains a copy of the queue size parameters for user space +* to use but the kernel must use the parameters in the +* rxe_queue struct. For performance reasons arrange to have +* producer and consumer indices in separate cache lines +* the kernel should always mask the indices to avoid accessing +* memory outside of the data area +*/ +struct rxe_queue_buf { +__u32 log2_elem_size; +__u32 index_mask; +__u32 pad_1[30 ]; +__u32 producer_index; +__u32 pad_2[31 ]; +__u32 consumer_index; +__u32 pad_3[31 ]; +__u8 data[]; +}; +#endif /* RDMA_USER_RXE_H */
Signature · 1 line
2.27.0
Jason Gunthorpe
maintainerQuoted 5 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_queue.h +0
-1@@ -76,26 +56,56 @@ static inline int next_index(struct rxe_queue *q, int index)static inline int queue_empty (struct rxe_queue *q) { - return ((q->buf->producer_index - q->buf->consumer_index)
drivers/infiniband/sw/rxe/rxe_queue.h static inline int next_index(struct rxe_queue *q, int index) · line 67
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_queue.h +9
-1- & q->index_mask) == 0; +u32 prod; +u32 cons; + +/* make sure all changes to queue complete before +* testing queue empty +*/ + prod =smp_load_acquire (&q->buf->producer_index); +/* same */ + cons =smp_load_acquire (&q->buf->consumer_index);
This is not so sensible. The one written by the kernel should be just a normal load and there must be some lock held here to protect that
The one written by user space should be just READ_ONCE
acquire only has meaning if you go on to read additional data based on the result of the acquire - then acquire ensures the additional reads can't cross writes that occured before the matching release.
Jason
Jason Gunthorpe
maintainerQuoted 7 lines · Bob Pearson
Changed work and completion queues to use
smp_load_acquire()andsmp_store_release()to synchronize between driver and users. This commit goes with a matching series of commits in the rxe user space provider.v2: Addressed same issue for kernel ULPs which use rxe_post_send/recv().
Quoted 7 lines
Signed-off-by Bob Pearson <[email protected]>--- drivers/infiniband/sw/rxe/rxe_cq.c | 5 -- drivers/infiniband/sw/rxe/rxe_queue.h | 94 +++++++++++++++++---------- drivers/infiniband/sw/rxe/rxe_verbs.c | 11 ---- include/uapi/rdma/rdma_user_rxe.h | 21 ++++++ 4 files changed, 81 insertions(+), 50 deletions(-)
This really is a lot better than what was here, the extra barriers on empty/full can be fine tuned later
So applied to for-next
Thanks,
Jason