Skip to contentSkip to conversation

RDMA/rxe: Fix memory ordering problem for resize cq

PATCH In review Bob Pearson 4 participants 8 messages May 25, 2021 – May 26, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/rxe: Fix memory ordering problem for resize cq

The rxe driver has recently begun exhibiting failures in the python tests that are due to stale values read from the completion queue producer or consumer indices. Unlike the other loads of these shared indices those in queue_count() were not protected by smp_load_acquire().

This patch replaces loads by smp_load_acquire() in queue_count(). The observed errors no longer occur.

Fixes d21a1240f516 ("RDMA/rxe: Use acquire/release for memory ordering")
---
 drivers/infiniband/sw/rxe/rxe_queue.h | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)
drivers/infiniband/sw/rxe/rxe_queue.h +16 -2
index 2902ca7b288c..5cb142282fa6 100644
@@ -161,8 +161,22 @@ static inline unsigned int index_from_addr(const struct rxe_queue *q,
 
 static inline unsigned int queue_count(const struct rxe_queue *q)
 {
-	return (q->buf->producer_index - q->buf->consumer_index)
-		& q->index_mask;
+	u32 prod;
+	u32 cons;
+	u32 count;
+
+	/* make sure all changes to queue complete before
+	 * changing producer index
+	 */
+	prod = smp_load_acquire(&q->buf->producer_index);
+
+	/* make sure all changes to queue complete before
+	 * changing consumer index
+	 */
+	cons = smp_load_acquire(&q->buf->consumer_index);
+	count = (prod - cons) % q->index_mask;
+
+	return count;
 }
 
 static inline void *queue_head(struct rxe_queue *q)
Signature · 1 line
2.30.2
maintainer
#1 lore raw
Quoted 30 lines · Bob Pearson

The rxe driver has recently begun exhibiting failures in the python tests that are due to stale values read from the completion queue producer or consumer indices. Unlike the other loads of these shared indices those in queue_count() were not protected by smp_load_acquire().

This patch replaces loads by smp_load_acquire() in queue_count(). The observed errors no longer occur.

Fixes d21a1240f516 ("RDMA/rxe: Use acquire/release for memory ordering")
 drivers/infiniband/sw/rxe/rxe_queue.h | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)
drivers/infiniband/sw/rxe/rxe_queue.h +8 -2
index 2902ca7b288c..5cb142282fa6 100644
+++ b/drivers/infiniband/sw/rxe/rxe_queue.h
@@ -161,8 +161,22 @@ static inline unsigned int index_from_addr(const struct rxe_queue *q,
 
 static inline unsigned int queue_count(const struct rxe_queue *q)
 {
-	return (q->buf->producer_index - q->buf->consumer_index)
-		& q->index_mask;
+	u32 prod;
+	u32 cons;
+	u32 count;
+
+	/* make sure all changes to queue complete before
+	 * changing producer index
+	 */

drivers/infiniband/sw/rxe/rxe_queue.h static inline unsigned int index_from_addr(const struct rxe_queue *q, · line 180

Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_queue.h +9 -0
+	prod = smp_load_acquire(&q->buf->producer_index);
+
+	/* make sure all changes to queue complete before
+	 * changing consumer index
+	 */
+	cons = smp_load_acquire(&q->buf->consumer_index);
+	count = (prod - cons) % q->index_mask;
+
+	return count;
 }

It doesn't quite make sense to load both?

Only the one written by userspace should require a load_acquire, the one written by the kernel should already be locked somehow with a kernel lock or there is a different problem

But yes, this does look like a bug to me

Jason

#2 lore raw
Quoted 43 lines · Jason Gunthorpe
Quoted 30 lines · Bob Pearson

The rxe driver has recently begun exhibiting failures in the python tests that are due to stale values read from the completion queue producer or consumer indices. Unlike the other loads of these shared indices those in queue_count() were not protected by smp_load_acquire().

This patch replaces loads by smp_load_acquire() in queue_count(). The observed errors no longer occur.

Fixes d21a1240f516 ("RDMA/rxe: Use acquire/release for memory ordering")
  drivers/infiniband/sw/rxe/rxe_queue.h | 18 ++++++++++++++++--
  1 file changed, 16 insertions(+), 2 deletions(-)
drivers/infiniband/sw/rxe/rxe_queue.h +8 -2
index 2902ca7b288c..5cb142282fa6 100644
+++ b/drivers/infiniband/sw/rxe/rxe_queue.h
@@ -161,8 +161,22 @@ static inline unsigned int index_from_addr(const struct rxe_queue *q,
 
  static inline unsigned int queue_count(const struct rxe_queue *q)
  {
-	return (q->buf->producer_index - q->buf->consumer_index)
-		& q->index_mask;
+	u32 prod;
+	u32 cons;
+	u32 count;
+
+	/* make sure all changes to queue complete before
+	 * changing producer index
+	 */

drivers/infiniband/sw/rxe/rxe_queue.h static inline unsigned int index_from_addr(const struct rxe_queue *q, · line 180

Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_queue.h +9 -0
+	prod = smp_load_acquire(&q->buf->producer_index);
+
+	/* make sure all changes to queue complete before
+	 * changing consumer index
+	 */
+	cons = smp_load_acquire(&q->buf->consumer_index);
+	count = (prod - cons) % q->index_mask;
+
+	return count;
  }

It doesn't quite make sense to load both?

Quoted 7 lines

Only the one written by userspace should require a load_acquire, the one written by the kernel should already be locked somehow with a kernel lock or there is a different problem

But yes, this does look like a bug to me

Jason

The rxe_queue.h API is used by completion and work queues so both directions occur. The question is the trade off between the logic to avoid the load_acquire vs the branching to get around. Same point could be applied to queue_empty() which already had both load_acquire() calls. Is it worth creating several versions of APIs with different choices made?

Bob

maintainer
#3 lore raw
Quoted 50 lines · Robert B
Quoted 48 lines · Jason Gunthorpe
Quoted 29 lines · Bob Pearson

The rxe driver has recently begun exhibiting failures in the python tests that are due to stale values read from the completion queue producer or consumer indices. Unlike the other loads of these shared indices those in queue_count() were not protected by smp_load_acquire().

This patch replaces loads by smp_load_acquire() in queue_count(). The observed errors no longer occur.

Fixes d21a1240f516 ("RDMA/rxe: Use acquire/release for memory ordering")
  drivers/infiniband/sw/rxe/rxe_queue.h | 18 ++++++++++++++++--
  1 file changed, 16 insertions(+), 2 deletions(-)
drivers/infiniband/sw/rxe/rxe_queue.h +8 -2
index 2902ca7b288c..5cb142282fa6 100644
+++ b/drivers/infiniband/sw/rxe/rxe_queue.h
@@ -161,8 +161,22 @@ static inline unsigned int index_from_addr(const struct rxe_queue *q,
  static inline unsigned int queue_count(const struct rxe_queue *q)
  {
-	return (q->buf->producer_index - q->buf->consumer_index)
-		& q->index_mask;
+	u32 prod;
+	u32 cons;
+	u32 count;
+
+	/* make sure all changes to queue complete before
+	 * changing producer index
+	 */

drivers/infiniband/sw/rxe/rxe_queue.h static inline unsigned int index_from_addr(const struct rxe_queue *q, · line 179

Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_queue.h +9 -0
+	prod = smp_load_acquire(&q->buf->producer_index);
+
+	/* make sure all changes to queue complete before
+	 * changing consumer index
+	 */
+	cons = smp_load_acquire(&q->buf->consumer_index);
+	count = (prod - cons) % q->index_mask;
+
+	return count;
  }

It doesn't quite make sense to load both?

Only the one written by userspace should require a load_acquire, the one written by the kernel should already be locked somehow with a kernel lock or there is a different problem

But yes, this does look like a bug to me

Quoted 7 lines
Quoted 1 line

Jason

The rxe_queue.h API is used by completion and work queues so both directions occur. The question is the trade off between the logic to avoid the load_acquire vs the branching to get around. Same point could be applied to queue_empty() which already had both load_acquire() calls. Is it worth creating several versions of APIs with different choices made?

Hard to say.. I would probably have made special types for 'user producer' and 'user consumer' queues and had actually different behavior - the kernel controlled data should be shadowed in the kernel and only copied to the user - have to consider the user corrupting the data, for instance.

Jason

maintainer
#4 lore raw
In reply to Bob Pearson
Quoted 10 lines · Bob Pearson

The rxe driver has recently begun exhibiting failures in the python tests that are due to stale values read from the completion queue producer or consumer indices. Unlike the other loads of these shared indices those in queue_count() were not protected by smp_load_acquire().

This patch replaces loads by smp_load_acquire() in queue_count(). The observed errors no longer occur.

My email address is wrong. It should be [email protected]

Zhu Yanjun

Quoted 38 lines
Fixes d21a1240f516 ("RDMA/rxe: Use acquire/release for memory ordering")
---
 drivers/infiniband/sw/rxe/rxe_queue.h | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)
drivers/infiniband/sw/rxe/rxe_queue.h +16 -3
index 2902ca7b288c..5cb142282fa6 100644
@@ -161,8 +161,22 @@ static inline unsigned int index_from_addr(const struct rxe_queue *q,
 
 static inline unsigned int queue_count(const struct rxe_queue *q)
 {
-       return (q->buf->producer_index - q->buf->consumer_index)
-               & q->index_mask;
+       u32 prod;
+       u32 cons;
+       u32 count;
+
+       /* make sure all changes to queue complete before
+        * changing producer index
+        */
+       prod = smp_load_acquire(&q->buf->producer_index);
+
+       /* make sure all changes to queue complete before
+        * changing consumer index
+        */
+       cons = smp_load_acquire(&q->buf->consumer_index);
+       count = (prod - cons) % q->index_mask;
+
+       return count;
 }
 
 static inline void *queue_head(struct rxe_queue *q)
--

2.30.2

maintainer
#5 lore raw
In reply to Bob Pearson
Quoted 30 lines · Bob Pearson

The rxe driver has recently begun exhibiting failures in the python tests that are due to stale values read from the completion queue producer or consumer indices. Unlike the other loads of these shared indices those in queue_count() were not protected by smp_load_acquire().

This patch replaces loads by smp_load_acquire() in queue_count(). The observed errors no longer occur.

Fixes d21a1240f516 ("RDMA/rxe: Use acquire/release for memory ordering")
---
 drivers/infiniband/sw/rxe/rxe_queue.h | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)
drivers/infiniband/sw/rxe/rxe_queue.h +4 -2
index 2902ca7b288c..5cb142282fa6 100644
@@ -161,8 +161,22 @@ static inline unsigned int index_from_addr(const struct rxe_queue *q,
 
 static inline unsigned int queue_count(const struct rxe_queue *q)
 {
-       return (q->buf->producer_index - q->buf->consumer_index)
-               & q->index_mask;
+       u32 prod;
+       u32 cons;
+       u32 count;
+

drivers/infiniband/sw/rxe/rxe_queue.h static inline unsigned int index_from_addr(const struct rxe_queue *q, · line 177

Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_queue.h +10 -0
+       /* make sure all changes to queue complete before
+        * changing producer index
+        */
+       prod = smp_load_acquire(&q->buf->producer_index);
+
+       /* make sure all changes to queue complete before
+        * changing consumer index
+        */
+       cons = smp_load_acquire(&q->buf->consumer_index);
+       count = (prod - cons) % q->index_mask;

% is different from &. Not sure it is correct to use % instead of & in the original source code.

Zhu Yanjun

Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_queue.h +2 -1
+
+       return count;
 }
 
 static inline void *queue_head(struct rxe_queue *q)
--

2.30.2

#6 lore raw

Thanks. Good catch. Should be &.

-----Original Message-----
From: Zhu Yanjun <[email protected]>
Sent: Wednesday, May 26, 2021 12:54 AM
To: Bob Pearson <[email protected]>
Cc: Jason Gunthorpe <[email protected]>; [email protected]; RDMA mailing list <[email protected]>
Subject: Re: [PATCH for-next] RDMA/rxe: Fix memory ordering problem for resize cq

Quoted 32 lines · Bob Pearson

The rxe driver has recently begun exhibiting failures in the python tests that are due to stale values read from the completion queue producer or consumer indices. Unlike the other loads of these shared indices those in queue_count() were not protected by smp_load_acquire().

This patch replaces loads by smp_load_acquire() in queue_count(). The observed errors no longer occur.

Fixes d21a1240f516 ("RDMA/rxe: Use acquire/release for memory

ordering")

---
 drivers/infiniband/sw/rxe/rxe_queue.h | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)
drivers/infiniband/sw/rxe/rxe_queue.h +4 -2
index 2902ca7b288c..5cb142282fa6 100644
@@ -161,8 +161,22 @@ static inline unsigned int index_from_addr(const 
 struct rxe_queue *q,
 
 static inline unsigned int queue_count(const struct rxe_queue *q)  {
-       return (q->buf->producer_index - q->buf->consumer_index)
-               & q->index_mask;
+       u32 prod;
+       u32 cons;
+       u32 count;
+

drivers/infiniband/sw/rxe/rxe_queue.h static inline unsigned int index_from_addr(const · line 177

Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_queue.h +10 -0
+       /* make sure all changes to queue complete before
+        * changing producer index
+        */
+       prod = smp_load_acquire(&q->buf->producer_index);
+
+       /* make sure all changes to queue complete before
+        * changing consumer index
+        */
+       cons = smp_load_acquire(&q->buf->consumer_index);
+       count = (prod - cons) % q->index_mask;

% is different from &. Not sure it is correct to use % instead of & in the original source code.

Zhu Yanjun

Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_queue.h +2 -1
+
+       return count;
 }
 
 static inline void *queue_head(struct rxe_queue *q)
--

2.30.2

#7 lore raw
In reply to Zhu Yanjun
Quoted 43 lines · Zhu Yanjun
Quoted 29 lines · Bob Pearson

The rxe driver has recently begun exhibiting failures in the python tests that are due to stale values read from the completion queue producer or consumer indices. Unlike the other loads of these shared indices those in queue_count() were not protected by smp_load_acquire().

This patch replaces loads by smp_load_acquire() in queue_count(). The observed errors no longer occur.

Fixes d21a1240f516 ("RDMA/rxe: Use acquire/release for memory ordering")
---
  drivers/infiniband/sw/rxe/rxe_queue.h | 18 ++++++++++++++++--
  1 file changed, 16 insertions(+), 2 deletions(-)
drivers/infiniband/sw/rxe/rxe_queue.h +4 -2
index 2902ca7b288c..5cb142282fa6 100644
@@ -161,8 +161,22 @@ static inline unsigned int index_from_addr(const struct rxe_queue *q,
 
  static inline unsigned int queue_count(const struct rxe_queue *q)
  {
-       return (q->buf->producer_index - q->buf->consumer_index)
-               & q->index_mask;
+       u32 prod;
+       u32 cons;
+       u32 count;
+

drivers/infiniband/sw/rxe/rxe_queue.h static inline unsigned int index_from_addr(const struct rxe_queue *q, · line 177

Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_queue.h +10 -0
+       /* make sure all changes to queue complete before
+        * changing producer index
+        */
+       prod = smp_load_acquire(&q->buf->producer_index);
+
+       /* make sure all changes to queue complete before
+        * changing consumer index
+        */
+       cons = smp_load_acquire(&q->buf->consumer_index);
+       count = (prod - cons) % q->index_mask;

% is different from &. Not sure it is correct to use % instead of & in the original source code.

Quoted 1 line

Zhu Yanjun

When I went to fix this I realized that this is the first version of this change. It is actually fixed in v2.

Bob

Quoted 9 lines
Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_queue.h +2 -1
+
+       return count;
  }
 
  static inline void *queue_head(struct rxe_queue *q)
--

2.30.2