Skip to contentSkip to conversation

RDMA/rxe: Use acquire/release for memory ordering

PATCH In review Bob Pearson 2 participants 3 messages Dec 10, 2020 – Dec 12, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 RDMA/rxe: Use acquire/release for memory ordering

Changed work and completion queues to use smp_load_acquire() and smp_store_release() to synchronize between driver and users. This commit goes with a matching series of commits in the rxe user space provider.

v2: Addressed same issue for kernel ULPs which use rxe_post_send/recv().

---
 drivers/infiniband/sw/rxe/rxe_cq.c    |  5 --
 drivers/infiniband/sw/rxe/rxe_queue.h | 94 +++++++++++++++++----------
 drivers/infiniband/sw/rxe/rxe_verbs.c | 11 ----
 include/uapi/rdma/rdma_user_rxe.h     | 21 ++++++
 4 files changed, 81 insertions(+), 50 deletions(-)
drivers/infiniband/sw/rxe/rxe_cq.c +0 -5
index 43394c3f29d4..b315ebf041ac 100644
@@ -123,11 +123,6 @@ int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited)
 
 	memcpy(producer_addr(cq->queue), cqe, sizeof(*cqe));
 
-	/* make sure all changes to the CQ are written before we update the
-	 * producer pointer
-	 */
-	smp_wmb();
-
 	advance_producer(cq->queue);
 	spin_unlock_irqrestore(&cq->cq_lock, flags);
 
drivers/infiniband/sw/rxe/rxe_queue.h +60 -34

142 lines not on this page · raw message

drivers/infiniband/sw/rxe/rxe_verbs.c +0 -11
index 2fbea2b2d72a..a031514e2f41 100644
@@ -244,11 +244,6 @@ static int post_one_recv(struct rxe_rq *rq, const struct ib_recv_wr *ibwr)
 	recv_wqe->dma.cur_sge		= 0;
 	recv_wqe->dma.sge_offset	= 0;
 
-	/* make sure all changes to the work queue are written before we
-	 * update the producer pointer
-	 */
-	smp_wmb();
-
 	advance_producer(rq->queue);
 	return 0;
 
@@ -633,12 +628,6 @@ static int post_one_send(struct rxe_qp *qp, const struct ib_send_wr *ibwr,
 	if (unlikely(err))
 		goto err1;
 
-	/*
-	 * make sure all changes to the work queue are
-	 * written before we update the producer pointer
-	 */
-	smp_wmb();
-
 	advance_producer(sq->queue);
 	spin_unlock_irqrestore(&qp->sq.sq_lock, flags);
 
include/uapi/rdma/rdma_user_rxe.h +21 -0
index e591d8c1f3cf..068433e2229d 100644
@@ -181,4 +181,25 @@ struct rxe_modify_srq_cmd {
 	__aligned_u64 mmap_info_addr;
 };
 
+/* This data structure is stored at the base of work and
+ * completion queues shared between user space and kernel space.
+ * It contains the producer and consumer indices. Is also
+ * contains a copy of the queue size parameters for user space
+ * to use but the kernel must use the parameters in the
+ * rxe_queue struct. For performance reasons arrange to have
+ * producer and consumer indices in separate cache lines
+ * the kernel should always mask the indices to avoid accessing
+ * memory outside of the data area
+ */
+struct rxe_queue_buf {
+	__u32			log2_elem_size;
+	__u32			index_mask;
+	__u32			pad_1[30];
+	__u32			producer_index;
+	__u32			pad_2[31];
+	__u32			consumer_index;
+	__u32			pad_3[31];
+	__u8			data[];
+};
+
 #endif /* RDMA_USER_RXE_H */
Signature · 1 line
2.27.0
maintainer
#1 lore raw
Quoted 5 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_queue.h +0 -1
@@ -76,26 +56,56 @@ static inline int next_index(struct rxe_queue *q, int index)
 
 static inline int queue_empty(struct rxe_queue *q)
 {
-	return ((q->buf->producer_index - q->buf->consumer_index)

drivers/infiniband/sw/rxe/rxe_queue.h static inline int next_index(struct rxe_queue *q, int index) · line 67

Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_queue.h +9 -1
-			& q->index_mask) == 0;
+	u32 prod;
+	u32 cons;
+
+	/* make sure all changes to queue complete before
+	 * testing queue empty
+	 */
+	prod = smp_load_acquire(&q->buf->producer_index);
+	/* same */
+	cons = smp_load_acquire(&q->buf->consumer_index);

This is not so sensible. The one written by the kernel should be just a normal load and there must be some lock held here to protect that

The one written by user space should be just READ_ONCE

acquire only has meaning if you go on to read additional data based on the result of the acquire - then acquire ensures the additional reads can't cross writes that occured before the matching release.

Jason

maintainer
#2 lore raw
In reply to Bob Pearson
Quoted 7 lines · Bob Pearson

Changed work and completion queues to use smp_load_acquire() and smp_store_release() to synchronize between driver and users. This commit goes with a matching series of commits in the rxe user space provider.

v2: Addressed same issue for kernel ULPs which use rxe_post_send/recv().

Quoted 7 lines
---
 drivers/infiniband/sw/rxe/rxe_cq.c    |  5 --
 drivers/infiniband/sw/rxe/rxe_queue.h | 94 +++++++++++++++++----------
 drivers/infiniband/sw/rxe/rxe_verbs.c | 11 ----
 include/uapi/rdma/rdma_user_rxe.h     | 21 ++++++
 4 files changed, 81 insertions(+), 50 deletions(-)

This really is a lot better than what was here, the extra barriers on empty/full can be fine tuned later

So applied to for-next

Thanks,
Jason