Skip to contentSkip to conversation

ring: enforce reading the tails before ring operations

PATCH Superseded gavin hu 6 participants 27 messages Mar 6, 2019 – Mar 28, 2019

lore
Share & tools
lore

gavin hu

author
#0 lore raw

PATCH 1/1 ring: enforce reading the tails before ring operations

In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots. There must be a read fence before writing or reading the ring slots, rte_atomic32_cmpset() provides the same ordering for MP (and MC) case. This patch is to enforce this ordering for SP (and SC) case.

---
 lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
lib/librte_ring/rte_ring_generic.h +10 -6
index ea7dbe5..1bd3dfd 100644
@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sp)
-			r->prod.head = *new_head, success = 1;
-		else
+		if (is_sp) {
+			r->prod.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->prod.head,
 					*old_head, *new_head);
 	} while (unlikely(success == 0));
@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sc)
-			r->cons.head = *new_head, success = 1;
-		else
+		if (is_sc) {
+			r->cons.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->cons.head, *old_head,
 					*new_head);
 	} while (unlikely(success == 0));
Signature · 1 line
2.7.4
#1 lore raw
Quoted 7 lines · gavin hu

In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots. There must be a read fence before writing

Sorry, but the phrase "There must be a read fence before writing" makes no sense. Could you please rephrase or describe in details which reads you're trying to keep in exact order?

Quoted 46 lines

or reading the ring slots, rte_atomic32_cmpset() provides the same ordering for MP (and MC) case. This patch is to enforce this ordering for SP (and SC) case.

---
 lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
lib/librte_ring/rte_ring_generic.h +10 -6
index ea7dbe5..1bd3dfd 100644
@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sp)
-			r->prod.head = *new_head, success = 1;
-		else
+		if (is_sp) {
+			r->prod.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->prod.head,
 					*old_head, *new_head);
 	} while (unlikely(success == 0));
@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sc)
-			r->cons.head = *new_head, success = 1;
-		else
+		if (is_sc) {
+			r->cons.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->cons.head, *old_head,
 					*new_head);
 	} while (unlikely(success == 0));
#2 lore raw

Hi ilya,

Thanks for your comments, inline comments and new commit message in V2.

/Gavin

Quoted 19 lines

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Wednesday, March 6, 2019 7:49 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: Re: [v1] ring: enforce reading the tails before ring operations

Quoted 4 lines · gavin hu

In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is

most

Quoted 2 lines

likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to read

a

Quoted 8 lines
Quoted 1 line

stale value from the ring slots. There must be a read fence before

writing

Sorry, but the phrase "There must be a read fence before writing" makes no sense.
Could you please rephrase or describe in details which reads you're trying to
keep in exact order?

This patch is to keep in exact order for reading the tails(to calculate the available /free slots of ring when moving the heads) before reading or writing the ring slots.

I rephrased the commit message in V2, could you have a look?

Quoted 51 lines
Quoted 1 line

or reading the ring slots, rte_atomic32_cmpset() provides the same

ordering

Review

Quoted 11 lines

for MP (and MC) case. This patch is to enforce this ordering for SP (and SC) case.

---
 lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_generic.h

lib/librte_ring/rte_ring_generic.h __rte_ring_move_prod_head(struct rte_ring *r,

Quoted 4 lines

index ea7dbe5..1bd3dfd 100644

lib/librte_ring/rte_ring_generic.h +0 -0
@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r,

unsigned int is_sp,

lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r,

Quoted 15 lines
lib/librte_ring/rte_ring_generic.h +5 -3
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sp)
-			r->prod.head = *new_head, success = 1;
-		else
+		if (is_sp) {
+			r->prod.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->prod.head,
 					*old_head, *new_head);
 	} while (unlikely(success == 0));
@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring *r,

unsigned int is_sc,

lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,

Quoted 12 lines
lib/librte_ring/rte_ring_generic.h +5 -3
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sc)
-			r->cons.head = *new_head, success = 1;
-		else
+		if (is_sc) {
+			r->cons.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->cons.head,

*old_head,

Quoted 3 lines
lib/librte_ring/rte_ring_generic.h +0 -0
 					*new_head);
 	} while (unlikely(success == 0));
author
#3 lore raw

PATCH v2 1/1 ring: enforce reading the tails before ring operations

In reply to gavin hu

In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides the required ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.

---
 lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
lib/librte_ring/rte_ring_generic.h +10 -6
index ea7dbe5..1bd3dfd 100644
@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sp)
-			r->prod.head = *new_head, success = 1;
-		else
+		if (is_sp) {
+			r->prod.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->prod.head,
 					*old_head, *new_head);
 	} while (unlikely(success == 0));
@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sc)
-			r->cons.head = *new_head, success = 1;
-		else
+		if (is_sc) {
+			r->cons.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->cons.head, *old_head,
 					*new_head);
 	} while (unlikely(success == 0));
Signature · 1 line
2.7.4
#4 lore raw
Quoted 4 lines · gavin hu

In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.

Quoted 8 lines

This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides the required ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered before reading the tail. However, to prevent *writing* the ring get reordered *before reading* the tail you need a full memory barrier, i.e. rte_smp_mb().

Quoted 43 lines
---
 lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
lib/librte_ring/rte_ring_generic.h +10 -6
index ea7dbe5..1bd3dfd 100644
@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sp)
-			r->prod.head = *new_head, success = 1;
-		else
+		if (is_sp) {
+			r->prod.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->prod.head,
 					*old_head, *new_head);
 	} while (unlikely(success == 0));
@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sc)
-			r->cons.head = *new_head, success = 1;
-		else
+		if (is_sc) {
+			r->cons.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->cons.head, *old_head,
 					*new_head);
 	} while (unlikely(success == 0));
#5 lore raw
Quoted 30 lines

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 4:52 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 5 lines · gavin hu

In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The problem is

most

Quoted 2 lines

likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to read

a

Quoted 3 lines

stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides the

required

Quoted 2 lines

ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.

Quoted 6 lines

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e. rte_smp_mb().

ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST Option) orders ST/ST. For more details, please refer to: Table B2-1 Encoding of the DMB and DSB <option> parameter in
https://developer.arm.com/docs/ddi0487/latest/arm-architecture-reference-manual-armv8-for-armv8-a-architecture-profile

Quoted 48 lines

Review

Quoted 10 lines
---
 lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_generic.h

lib/librte_ring/rte_ring_generic.h __rte_ring_move_prod_head(struct rte_ring *r,

Quoted 4 lines

index ea7dbe5..1bd3dfd 100644

lib/librte_ring/rte_ring_generic.h +0 -0
@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r,

unsigned int is_sp,

lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r,

Quoted 15 lines
lib/librte_ring/rte_ring_generic.h +5 -3
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sp)
-			r->prod.head = *new_head, success = 1;
-		else
+		if (is_sp) {
+			r->prod.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->prod.head,
 					*old_head, *new_head);
 	} while (unlikely(success == 0));
@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring *r,

unsigned int is_sc,

lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,

Quoted 12 lines
lib/librte_ring/rte_ring_generic.h +5 -3
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sc)
-			r->cons.head = *new_head, success = 1;
-		else
+		if (is_sc) {
+			r->cons.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->cons.head,

*old_head,

Quoted 3 lines
lib/librte_ring/rte_ring_generic.h +0 -0
 					*new_head);
 	} while (unlikely(success == 0));
#6 lore raw
Quoted 37 lines · Gavin Hu (Arm Technology China)
Quoted 30 lines

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 4:52 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 5 lines · gavin hu

In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The problem is

most

Quoted 2 lines

likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to read

a

Quoted 3 lines

stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides the

required

Quoted 2 lines

ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.

Quoted 6 lines

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e. rte_smp_mb().

Quoted 3 lines

ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST Option) orders ST/ST. For more details, please refer to: Table B2-1 Encoding of the DMB and DSB <option> parameter in
https://developer.arm.com/docs/ddi0487/latest/arm-architecture-reference-manual-armv8-for-armv8-a-architecture-profile

I see. But you have to change the rte_smp_rmb() function definition in lib/librte_eal/common/include/generic/rte_atomic.h and assure that all other architectures follows same rules.
Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.

Quoted 49 lines
Quoted 49 lines

Review

Quoted 11 lines
---
 lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_generic.h

lib/librte_ring/rte_ring_generic.h __rte_ring_move_prod_head(struct rte_ring *r,

Quoted 4 lines

index ea7dbe5..1bd3dfd 100644

lib/librte_ring/rte_ring_generic.h +0 -0
@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r,

unsigned int is_sp,

lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r,

Quoted 15 lines
lib/librte_ring/rte_ring_generic.h +5 -3
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sp)
-			r->prod.head = *new_head, success = 1;
-		else
+		if (is_sp) {
+			r->prod.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->prod.head,
 					*old_head, *new_head);
 	} while (unlikely(success == 0));
@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring *r,

unsigned int is_sc,

lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,

Quoted 12 lines
lib/librte_ring/rte_ring_generic.h +5 -3
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sc)
-			r->cons.head = *new_head, success = 1;
-		else
+		if (is_sc) {
+			r->cons.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->cons.head,

*old_head,

Quoted 3 lines
lib/librte_ring/rte_ring_generic.h +0 -0
 					*new_head);
 	} while (unlikely(success == 0));
#7 lore raw
Quoted 55 lines

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 5:48 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 14 lines · Gavin Hu (Arm Technology China)
Quoted 14 lines

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 4:52 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operations

On 07.03.2019 9:45, gavin hu wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The problem is

most

Quoted 2 lines

likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to

read

Quoted 8 lines
Quoted 8 lines

a

Quoted 3 lines

stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides the

required

Quoted 2 lines

ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.

Quoted 8 lines
Quoted 6 lines

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e. rte_smp_mb().

ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST

Option) orders ST/ST.

Quoted 1 line

For more details, please refer to: Table B2-1 Encoding of the DMB and DSB

<option> parameter in

Quoted 9 lines
Quoted 1 line

https://developer.arm.com/docs/ddi0487/latest/arm-architecture-

reference-manual-armv8-for-armv8-a-architecture-profile

I see. But you have to change the rte_smp_rmb() function definition in lib/librte_eal/common/include/generic/rte_atomic.h and assure that all other architectures follows same rules.
Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.

Good points, let me re-think how to handle for other architectures.
Full MB is required for other architectures(x86? Ppc?), but for arm, read barrier(load/store and load/load) is enough.

Quoted 50 lines
Quoted 19 lines
Quoted 19 lines

Review

Quoted 11 lines
---
 lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_generic.h

lib/librte_ring/rte_ring_generic.h __rte_ring_move_prod_head(struct rte_ring *r,

Quoted 4 lines

index ea7dbe5..1bd3dfd 100644

lib/librte_ring/rte_ring_generic.h +0 -0
@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r,

unsigned int is_sp,

Quoted 2 lines
lib/librte_ring/rte_ring_generic.h +0 -0
 			return 0;
 
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
lib/librte_ring/rte_ring_generic.h +1 -3
 		*new_head = *old_head + n;
-		if (is_sp)
-			r->prod.head = *new_head, success = 1;
-		else
+		if (is_sp) {
Quoted 8 lines

lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring

Quoted 8 lines
lib/librte_ring/rte_ring_generic.h +4 -0
+			r->prod.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->prod.head,
 					*old_head, *new_head);
 	} while (unlikely(success == 0));
@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring

*r,

Quoted 17 lines
Quoted 17 lines

unsigned int is_sc,

lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,

Quoted 12 lines
lib/librte_ring/rte_ring_generic.h +5 -3
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sc)
-			r->cons.head = *new_head, success = 1;
-		else
+		if (is_sc) {
+			r->cons.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->cons.head,

*old_head,

Quoted 3 lines
lib/librte_ring/rte_ring_generic.h +0 -0
 					*new_head);
 	} while (unlikely(success == 0));
#8 lore raw

Hi Gavin,

Quoted 38 lines
Quoted 28 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The problem is

most

Quoted 2 lines

likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to

read

Quoted 8 lines
Quoted 8 lines

a

Quoted 3 lines

stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides the

required

Quoted 2 lines

ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.

Quoted 8 lines
Quoted 6 lines

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e. rte_smp_mb().

ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST

Option) orders ST/ST.

Quoted 1 line

For more details, please refer to: Table B2-1 Encoding of the DMB and DSB

<option> parameter in

Quoted 9 lines
Quoted 1 line

https://developer.arm.com/docs/ddi0487/latest/arm-architecture-

reference-manual-armv8-for-armv8-a-architecture-profile

I see. But you have to change the rte_smp_rmb() function definition in lib/librte_eal/common/include/generic/rte_atomic.h and assure that all other architectures follows same rules.
Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.

Quoted 2 lines

Good points, let me re-think how to handle for other architectures.
Full MB is required for other architectures(x86? Ppc?), but for arm, read barrier(load/store and load/load) is enough.

For x86, I don't think you need any barrier here, as with IA memory mode:
- Reads are not reordered with other reads.
- Writes are not reordered with older reads.

BTW, could you explain a bit more why barrier is necessary even on arm here? As I can see, there is a data dependency between the tail value and subsequent address calculations for ring writes/reads.
Isn't that sufficient to prevent re-ordering even for weak memory model? Konstantin

Quoted 51 lines
Quoted 51 lines
Quoted 20 lines
Quoted 20 lines
Quoted 4 lines

Review

Quoted 8 lines
---
 lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_generic.h

lib/librte_ring/rte_ring_generic.h __rte_ring_move_prod_head(struct rte_ring *r,

Quoted 4 lines

index ea7dbe5..1bd3dfd 100644

lib/librte_ring/rte_ring_generic.h +0 -0
@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r,

unsigned int is_sp,

Quoted 2 lines
lib/librte_ring/rte_ring_generic.h +0 -0
 			return 0;
 
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
lib/librte_ring/rte_ring_generic.h +1 -3
 		*new_head = *old_head + n;
-		if (is_sp)
-			r->prod.head = *new_head, success = 1;
-		else
+		if (is_sp) {
Quoted 8 lines

lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring

Quoted 8 lines
lib/librte_ring/rte_ring_generic.h +4 -0
+			r->prod.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->prod.head,
 					*old_head, *new_head);
 	} while (unlikely(success == 0));
@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring

*r,

Quoted 17 lines
Quoted 17 lines

unsigned int is_sc,

lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,

Quoted 12 lines
lib/librte_ring/rte_ring_generic.h +5 -3
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sc)
-			r->cons.head = *new_head, success = 1;
-		else
+		if (is_sc) {
+			r->cons.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->cons.head,

*old_head,

Quoted 3 lines
lib/librte_ring/rte_ring_generic.h +0 -0
 					*new_head);
 	} while (unlikely(success == 0));
#9 lore raw
Quoted 44 lines

Hi Gavin,

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines

In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The problem is

most

Quoted 3 lines

likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to

Quoted 8 lines
Quoted 8 lines

read

Quoted 7 lines
Quoted 7 lines

a

Quoted 3 lines

stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides the

required

Quoted 2 lines

ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC)

case.

Quoted 23 lines
Quoted 22 lines
Quoted 9 lines
Quoted 6 lines

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered before reading the tail. However, to prevent *writing* the ring get reordered *before reading* the tail you need a full memory barrier, i.e.
rte_smp_mb().

ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST

Option) orders ST/ST.

Quoted 2 lines

For more details, please refer to: Table B2-1 Encoding of the DMB and DSB

<option> parameter in

Quoted 1 line

https://developer.arm.com/docs/ddi0487/latest/arm-architecture-

reference-manual-armv8-for-armv8-a-architecture-profile

I see. But you have to change the rte_smp_rmb() function definition in lib/librte_eal/common/include/generic/rte_atomic.h and assure that all other architectures follows same rules. Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.

Quoted 7 lines
Quoted 2 lines

Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read

barrier(load/store and load/load) is enough.

For x86, I don't think you need any barrier here, as with IA memory mode:
- Reads are not reordered with other reads.
- Writes are not reordered with older reads.

Agree

Quoted 5 lines

BTW, could you explain a bit more why barrier is necessary even on arm here? As I can see, there is a data dependency between the tail value and subsequent address calculations for ring writes/reads.
Isn't that sufficient to prevent re-ordering even for weak memory model?

The tail value affects 'n'. But, the value of 'n' can be speculated because of the following 'if' statement:

if (unlikely(n > *free_entries))

                        n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 : *free_entries;

The address calculations for actual ring writes/reads do not depend on the tail value. Since 'n' can be speculated, the writes/reads can be moved up before the load of the tail value.

Quoted 3 lines

Konstantin

<snip>

#10 lore raw

Hi Konstantin,

Quoted 58 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Friday, March 8, 2019 11:21 AM
To: Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; Ilya Maximets <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 7 lines

Hi Gavin,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The

problem

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

is

most

Quoted 3 lines

likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to

read

Quoted 4 lines
Quoted 4 lines

a

Quoted 3 lines

stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides

the

Quoted 27 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

required

Quoted 2 lines

ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC)

case.

Quoted 23 lines
Quoted 22 lines
Quoted 9 lines
Quoted 6 lines

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered before reading the tail. However, to prevent *writing* the ring get reordered *before reading* the tail you need a full memory barrier, i.e.
rte_smp_mb().

ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST

Option) orders ST/ST.

Quoted 2 lines

For more details, please refer to: Table B2-1 Encoding of the DMB and DSB

<option> parameter in

Quoted 1 line

https://developer.arm.com/docs/ddi0487/latest/arm-architecture-

reference-manual-armv8-for-armv8-a-architecture-profile

I see. But you have to change the rte_smp_rmb() function definition in lib/librte_eal/common/include/generic/rte_atomic.h and assure that all other architectures follows same rules. Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read

barrier(load/store and load/load) is enough.

For x86, I don't think you need any barrier here, as with IA memory mode:
- Reads are not reordered with other reads.
- Writes are not reordered with older reads.

Agree

I understand herein no instruction level barriers are required for IA, but how about the compiler barrier: rte_compiler_barrier?

Quoted 10 lines
Quoted 3 lines

BTW, could you explain a bit more why barrier is necessary even on arm

here?

Quoted 3 lines

As I can see, there is a data dependency between the tail value and subsequent address calculations for ring writes/reads.
Isn't that sufficient to prevent re-ordering even for weak memory model?

The tail value affects 'n'. But, the value of 'n' can be speculated because of the following 'if' statement:

Quoted 6 lines

if (unlikely(n > *free_entries))

                        n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 : *free_entries;

The address calculations for actual ring writes/reads do not depend on the tail value. Since 'n' can be speculated, the writes/reads can be moved up before the load of the tail value.

Good explanation. The address calculations does not depend on tail/n, only the limit/last one depends on it, while it can be speculated.

Quoted 4 lines
Quoted 3 lines

Konstantin

<snip>

#11 lore raw
Quoted 49 lines
Quoted 49 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The

problem

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

is

most

Quoted 3 lines

likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to

read

Quoted 4 lines
Quoted 4 lines

a

Quoted 3 lines

stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides

the

Quoted 31 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

required

Quoted 2 lines

ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC)

case.

Quoted 23 lines
Quoted 13 lines
Quoted 9 lines
Quoted 6 lines

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered before reading the tail. However, to prevent *writing* the ring get reordered *before reading* the tail you need a full memory barrier, i.e.
rte_smp_mb().

ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST

Option) orders ST/ST.

Quoted 2 lines

For more details, please refer to: Table B2-1 Encoding of the DMB and DSB

<option> parameter in

Quoted 9 lines
Quoted 1 line

https://developer.arm.com/docs/ddi0487/latest/arm-architecture-

reference-manual-armv8-for-armv8-a-architecture-profile

I see. But you have to change the rte_smp_rmb() function definition in lib/librte_eal/common/include/generic/rte_atomic.h and assure that all other architectures follows same rules. Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.

Quoted 2 lines

Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read

barrier(load/store and load/load) is enough.

Quoted 23 lines
Quoted 4 lines
Quoted 3 lines

For x86, I don't think you need any barrier here, as with IA memory mode:
- Reads are not reordered with other reads.
- Writes are not reordered with older reads.

Agree

I understand herein no instruction level barriers are required for IA, but how about the compiler barrier: rte_compiler_barrier?

Quoted 15 lines
Quoted 3 lines

BTW, could you explain a bit more why barrier is necessary even on arm

here?

Quoted 3 lines

As I can see, there is a data dependency between the tail value and subsequent address calculations for ring writes/reads.
Isn't that sufficient to prevent re-ordering even for weak memory model?

The tail value affects 'n'. But, the value of 'n' can be speculated because of the following 'if' statement:

if (unlikely(n > *free_entries))

                        n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 : *free_entries;

The address calculations for actual ring writes/reads do not depend on the tail value.

Ok, agree I formulated it wrongly, only limit value is dependent on cons.tail. Address is not.

Quoted 2 lines

Since 'n' can be speculated, the writes/reads can be moved up

Quoted 1 line

before the load of the tail value.

For my curiosity: ok, I understand that 'n' value can be speculated, and speculative stores could start before n is calculated properly... But are you saying that such speculative store results might be visible to the other observers (different cpu)?

Quoted 8 lines

Good explanation. The address calculations does not depend on tail/n, only the limit/last one depends on it, while it can be speculated.

Quoted 4 lines
Quoted 3 lines

Konstantin

<snip>

#12 lore raw
Quoted 85 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is

observed.

Quoted 32 lines
Quoted 32 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

This issue was reported by NXP on 8-A72 DPAA2 board. The

problem

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

is

most

Quoted 3 lines

likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to

read

Quoted 5 lines
Quoted 5 lines

a

Quoted 4 lines

stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides

the

Quoted 17 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

required

Quoted 3 lines

ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC)

case.

Quoted 12 lines
Quoted 12 lines
Quoted 9 lines
Quoted 6 lines

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered before reading the tail. However, to prevent *writing* the ring get reordered *before reading* the tail you need a full memory barrier, i.e. rte_smp_mb().

ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST

Option) orders ST/ST.

Quoted 2 lines

For more details, please refer to: Table B2-1 Encoding of the DMB and DSB

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

<option> parameter in

Quoted 2 lines

https://developer.arm.com/docs/ddi0487/latest/arm-architectu
re-

reference-manual-armv8-for-armv8-a-architecture-profile

I see. But you have to change the rte_smp_rmb() function definition in lib/librte_eal/common/include/generic/rte_atomic.h and assure that

all other architectures follows same rules.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with

Store.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read

barrier(load/store and load/load) is enough.

For x86, I don't think you need any barrier here, as with IA memory

mode:

Quoted 24 lines
Quoted 3 lines
Quoted 2 lines

- Reads are not reordered with other reads.
- Writes are not reordered with older reads.

Agree

I understand herein no instruction level barriers are required for IA, but how about the compiler barrier: rte_compiler_barrier?

Quoted 17 lines
Quoted 4 lines

BTW, could you explain a bit more why barrier is necessary even on arm

here?

Quoted 3 lines

As I can see, there is a data dependency between the tail value and subsequent address calculations for ring writes/reads. Isn't that sufficient to prevent re-ordering even for weak memory model?

The tail value affects 'n'. But, the value of 'n' can be speculated because of the following 'if' statement:

if (unlikely(n > *free_entries))

                        n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 :

*free_entries;

The address calculations for actual ring writes/reads do not depend on the tail value.

Ok, agree I formulated it wrongly, only limit value is dependent on cons.tail. Address is not.

Quoted 8 lines
Quoted 2 lines

Since 'n' can be speculated, the writes/reads can be moved up

Quoted 1 line

before the load of the tail value.

For my curiosity: ok, I understand that 'n' value can be speculated, and speculative stores could start before n is calculated properly... But are you saying that such speculative store results might be visible to the other observers (different cpu)?

You are correct. The speculative stores will NOT be visible to other observers till the value of 'n' is fixed. Speculative stores might have to be discarded depending on the value of 'n' (which will affect cache performance). There is also a control dependency between the load of cons.tail and the stores to the ring. That should also keep the load and stores from getting reordered (though I am not sure if it still allows for speculative stores). So, IMO, the barrier in enqueue is not needed. Is this what you wanted to drive at?

#13 lore raw
Quoted 93 lines
Quoted 85 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is

observed.

Quoted 32 lines
Quoted 32 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

This issue was reported by NXP on 8-A72 DPAA2 board. The

problem

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

is

most

Quoted 3 lines

likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to

read

Quoted 5 lines
Quoted 5 lines

a

Quoted 4 lines

stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides

the

Quoted 17 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

required

Quoted 3 lines

ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC)

case.

Quoted 12 lines
Quoted 12 lines
Quoted 9 lines
Quoted 6 lines

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered before reading the tail. However, to prevent *writing* the ring get reordered *before reading* the tail you need a full memory barrier, i.e. rte_smp_mb().

ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST

Option) orders ST/ST.

Quoted 2 lines

For more details, please refer to: Table B2-1 Encoding of the DMB and DSB

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

<option> parameter in

Quoted 2 lines

https://developer.arm.com/docs/ddi0487/latest/arm-architectu
re-

reference-manual-armv8-for-armv8-a-architecture-profile

I see. But you have to change the rte_smp_rmb() function definition in lib/librte_eal/common/include/generic/rte_atomic.h and assure that

all other architectures follows same rules.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with

Store.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read

barrier(load/store and load/load) is enough.

For x86, I don't think you need any barrier here, as with IA memory

mode:

Quoted 24 lines
Quoted 3 lines
Quoted 2 lines

- Reads are not reordered with other reads.
- Writes are not reordered with older reads.

Agree

I understand herein no instruction level barriers are required for IA, but how about the compiler barrier: rte_compiler_barrier?

Quoted 17 lines
Quoted 4 lines

BTW, could you explain a bit more why barrier is necessary even on arm

here?

Quoted 3 lines

As I can see, there is a data dependency between the tail value and subsequent address calculations for ring writes/reads. Isn't that sufficient to prevent re-ordering even for weak memory model?

The tail value affects 'n'. But, the value of 'n' can be speculated because of the following 'if' statement:

if (unlikely(n > *free_entries))

                        n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 :

*free_entries;

The address calculations for actual ring writes/reads do not depend on the tail value.

Ok, agree I formulated it wrongly, only limit value is dependent on cons.tail. Address is not.

Quoted 8 lines
Quoted 2 lines

Since 'n' can be speculated, the writes/reads can be moved up

Quoted 1 line

before the load of the tail value.

For my curiosity: ok, I understand that 'n' value can be speculated, and speculative stores could start before n is calculated properly... But are you saying that such speculative store results might be visible to the other observers (different cpu)?

Quoted 5 lines

You are correct. The speculative stores will NOT be visible to other observers till the value of 'n' is fixed. Speculative stores might have to be discarded depending on the value of 'n' (which will affect cache performance).
There is also a control dependency between the load of cons.tail and the stores to the ring. That should also keep the load and stores from getting reordered (though I am not sure if it still allows for speculative stores).
So, IMO, the barrier in enqueue is not needed. Is this what you wanted to drive at?

Yes, that was my thought.
Thanks
Konstantin

#14 lore raw
In reply to Gavin Hu (Arm Technology China)
Quoted 86 lines

-----Original Message-----
From: Gavin Hu (Arm Technology China)
Sent: Thursday, March 7, 2019 6:45 PM
To: Ilya Maximets <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 4 lines

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 5:48 PM
To: Gavin Hu (Arm Technology China) <[email protected]>;

[email protected]

Quoted 6 lines

Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 10 lines
Quoted 9 lines · Gavin Hu (Arm Technology China)
Quoted 9 lines

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 4:52 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa

Nagarahalli

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

<[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 1 line · gavin hu

In weak memory models, like arm64, reading the {prod,cons}.tail may

get

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

reordered after reading or writing the ring slots, which corrupts the

ring

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The problem

is

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

most

Quoted 1 line

likely caused by missing the acquire semantics when reading cons.tail

(in

Quoted 18 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

SP enqueue) or prod.tail (in SC dequeue) which makes it possible to

read

Quoted 16 lines
Quoted 8 lines

a

Quoted 3 lines

stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides the

required

Quoted 2 lines

ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.

Quoted 6 lines

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e. rte_smp_mb().

ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while

WMB(ST

Quoted 2 lines

Option) orders ST/ST.

Quoted 1 line

For more details, please refer to: Table B2-1 Encoding of the DMB and

DSB

Quoted 10 lines

<option> parameter in

Quoted 1 line

https://developer.arm.com/docs/ddi0487/latest/arm-architecture-

reference-manual-armv8-for-armv8-a-architecture-profile

I see. But you have to change the rte_smp_rmb() function definition in lib/librte_eal/common/include/generic/rte_atomic.h and assure that all other architectures follows same rules.
Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.

Quoted 3 lines

Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read barrier(load/store and load/load) is enough.

Hi Ilya,

I would expand the rmb definition to cover load/store, in addition to load/load.
For X86, as a strong memory order model, rmb is actually equivalent to mb, as implemented as a compiler barrier: rte_compiler_barrier, arm32 is also this case. For PPC, both 32 and 64-bit, rmb=wmb=mb, lwsync/sync orders load/store, load/load, store/load, store/store, looking at the table on this page:
https://www.ibm.com/developerworks/systems/articles/powerpc.html

In summary, we are safe to expand this definition for all the architectures DPDK support? Any comments are welcome!

BR. Gavin

Quoted 52 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
---
 lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Review

Quoted 1 line
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_generic.h

lib/librte_ring/rte_ring_generic.h __rte_ring_move_prod_head(struct rte_ring

Quoted 4 lines

index ea7dbe5..1bd3dfd 100644

lib/librte_ring/rte_ring_generic.h +0 -0
@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring

*r,

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

unsigned int is_sp,

lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct

Quoted 15 lines
lib/librte_ring/rte_ring_generic.h +5 -3
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sp)
-			r->prod.head = *new_head, success = 1;
-		else
+		if (is_sp) {
+			r->prod.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->prod.head,
 					*old_head, *new_head);
 	} while (unlikely(success == 0));
@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct

rte_ring

Quoted 18 lines

*r,

Quoted 17 lines
Quoted 17 lines

unsigned int is_sc,

lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,

Quoted 12 lines
lib/librte_ring/rte_ring_generic.h +5 -3
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sc)
-			r->cons.head = *new_head, success = 1;
-		else
+		if (is_sc) {
+			r->cons.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else
 			success = rte_atomic32_cmpset(&r->cons.head,

*old_head,

Quoted 3 lines
lib/librte_ring/rte_ring_generic.h +0 -0
 					*new_head);
 	} while (unlikely(success == 0));
#15 lore raw
Quoted 65 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · gavin hu
Quoted 2 lines

In weak memory models, like arm64, reading the {prod,cons}.tail may

get

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

reordered after reading or writing the ring slots, which corrupts the

ring

Quoted 18 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The problem

is

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

most

Quoted 2 lines

likely caused by missing the acquire semantics when reading cons.tail

(in

Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

SP enqueue) or prod.tail (in SC dequeue) which makes it possible to

read

Quoted 7 lines
Quoted 7 lines

a

Quoted 3 lines

stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides the

required

Quoted 2 lines

ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC)

case.

Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered before reading the tail. However, to prevent *writing* the ring get reordered *before reading* the tail you need a full memory barrier, i.e.
rte_smp_mb().

ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while

WMB(ST

Quoted 3 lines

Option) orders ST/ST.

Quoted 2 lines

For more details, please refer to: Table B2-1 Encoding of the DMB and

DSB

Quoted 10 lines

<option> parameter in

Quoted 1 line

https://developer.arm.com/docs/ddi0487/latest/arm-architecture-

reference-manual-armv8-for-armv8-a-architecture-profile

I see. But you have to change the rte_smp_rmb() function definition in lib/librte_eal/common/include/generic/rte_atomic.h and assure that all other architectures follows same rules. Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.

Quoted 3 lines

Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read barrier(load/store and load/load) is enough.

Hi Ilya,

I would expand the rmb definition to cover load/store, in addition to load/load.
For X86, as a strong memory order model, rmb is actually equivalent to mb, as implemented as a compiler barrier: rte_compiler_barrier, arm32 is also this case.
For PPC, both 32 and 64-bit, rmb=wmb=mb, lwsync/sync orders load/store, load/load, store/load, store/store, looking at the table on this page:
https://www.ibm.com/developerworks/systems/articles/powerpc.html

Quoted 2 lines

In summary, we are safe to expand this definition for all the architectures DPDK support?

Essentially, it is a documentation bug. i.e. the current implementation of rte_smp_rmb() already behaves as load/load and load/store barrier.

Quoted 4 lines

Any comments are welcome!

BR. Gavin

<snip>

#16 lore raw
In reply to Gavin Hu (Arm Technology China)
Quoted 97 lines

-----Original Message-----
From: Gavin Hu (Arm Technology China) [mailto:[email protected]]
Sent: Friday, March 8, 2019 4:23 AM
To: Ilya Maximets <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]
Subject: RE: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 84 lines

-----Original Message-----
From: Gavin Hu (Arm Technology China)
Sent: Thursday, March 7, 2019 6:45 PM
To: Ilya Maximets <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 4 lines

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 5:48 PM
To: Gavin Hu (Arm Technology China) <[email protected]>;

[email protected]

Quoted 6 lines

Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 10 lines
Quoted 9 lines · Gavin Hu (Arm Technology China)
Quoted 9 lines

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 4:52 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa

Nagarahalli

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

<[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 1 line · gavin hu

In weak memory models, like arm64, reading the {prod,cons}.tail may

get

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

reordered after reading or writing the ring slots, which corrupts the

ring

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The problem

is

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

most

Quoted 1 line

likely caused by missing the acquire semantics when reading cons.tail

(in

Quoted 18 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

SP enqueue) or prod.tail (in SC dequeue) which makes it possible to

read

Quoted 16 lines
Quoted 8 lines

a

Quoted 3 lines

stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides the

required

Quoted 2 lines

ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.

Quoted 6 lines

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e. rte_smp_mb().

ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while

WMB(ST

Quoted 2 lines

Option) orders ST/ST.

Quoted 1 line

For more details, please refer to: Table B2-1 Encoding of the DMB and

DSB

Quoted 10 lines

<option> parameter in

Quoted 1 line

https://developer.arm.com/docs/ddi0487/latest/arm-architecture-

reference-manual-armv8-for-armv8-a-architecture-profile

I see. But you have to change the rte_smp_rmb() function definition in lib/librte_eal/common/include/generic/rte_atomic.h and assure that all other architectures follows same rules.
Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.

Quoted 8 lines
Quoted 3 lines

Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read barrier(load/store and load/load) is enough.

Hi Ilya,

I would expand the rmb definition to cover load/store, in addition to load/load. For X86, as a strong memory order model, rmb is actually equivalent to mb,

That's not exactly the case, on x86 we have:
smp_rmb == compiler_barrier
smp_mb is a proper memory barrier.

Konstantin

Quoted 11 lines

as implemented as a compiler barrier: rte_compiler_barrier,
arm32 is also this case.
For PPC, both 32 and 64-bit, rmb=wmb=mb, lwsync/sync orders load/store, load/load, store/load, store/store, looking at the table on this page:
https://www.ibm.com/developerworks/systems/articles/powerpc.html

In summary, we are safe to expand this definition for all the architectures DPDK support? Any comments are welcome!

BR. Gavin

#17 lore raw

Hi Konstantin,

Quoted 138 lines

-----Original Message-----
From: Ananyev, Konstantin <[email protected]>
Sent: Friday, March 8, 2019 8:13 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; Ilya Maximets <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]
Subject: RE: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 5 lines

-----Original Message-----
From: Gavin Hu (Arm Technology China) [mailto:[email protected]]
Sent: Friday, March 8, 2019 4:23 AM
To: Ilya Maximets <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected];

[email protected]; [email protected]; Honnappa

Quoted 1 line

Nagarahalli <[email protected]>;

[email protected]; Richardson, Bruce
<[email protected]>; Ananyev,

Quoted 1 line

Konstantin <[email protected]>;

[email protected]

Quoted 1 line

Subject: RE: [PATCH v2] ring: enforce reading the tails before ring

operations

Quoted 9 lines
Quoted 9 lines

-----Original Message-----
From: Gavin Hu (Arm Technology China)
Sent: Thursday, March 7, 2019 6:45 PM
To: Ilya Maximets <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa

Nagarahalli

Quoted 4 lines
Quoted 4 lines

<[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH v2] ring: enforce reading the tails before ring

operations

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 5:48 PM
To: Gavin Hu (Arm Technology China) <[email protected]>;

[email protected]

Quoted 2 lines

Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa

Nagarahalli

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

<[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 2 lines · Gavin Hu (Arm Technology China)
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 4:52 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected];

[email protected];

Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

[email protected]; [email protected]; Honnappa

Nagarahalli

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

<[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 1 line · gavin hu

In weak memory models, like arm64, reading the {prod,cons}.tail

may

Quoted 2 lines
Quoted 2 lines

get

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

reordered after reading or writing the ring slots, which corrupts

the

Quoted 4 lines
Quoted 4 lines

ring

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The

problem

Quoted 3 lines
Quoted 3 lines

is

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

most

Quoted 1 line

likely caused by missing the acquire semantics when reading

cons.tail

Quoted 2 lines
Quoted 2 lines

(in

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

SP enqueue) or prod.tail (in SC dequeue) which makes it possible

to

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

read

Quoted 4 lines
Quoted 4 lines

a

Quoted 3 lines

stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

required

Quoted 1 line

ordering. This patch is to prevent reading and writing the ring

slots get

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line

reordered before reading {prod,cons}.tail for SP (and SC) case.

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e. rte_smp_mb().

Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while

WMB(ST

Quoted 2 lines

Option) orders ST/ST.

Quoted 1 line

For more details, please refer to: Table B2-1 Encoding of the DMB

and

Quoted 6 lines
Quoted 6 lines

DSB

Quoted 5 lines

<option> parameter in

Quoted 1 line

https://developer.arm.com/docs/ddi0487/latest/arm-architecture-

reference-manual-armv8-for-armv8-a-architecture-profile

I see. But you have to change the rte_smp_rmb() function definition

in

Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_eal/common/include/generic/rte_atomic.h and assure that

all

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

other architectures follows same rules.
Otherwise, this change is logically wrong, because read barrier in

current

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

definition could not be used to order Load with Store.

Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm,

read

Quoted 5 lines
Quoted 1 line

barrier(load/store and load/load) is enough.

Hi Ilya,

I would expand the rmb definition to cover load/store, in addition to

load/load.

Quoted 8 lines
Quoted 1 line

For X86, as a strong memory order model, rmb is actually equivalent to

mb,

That's not exactly the case, on x86 we have:
smp_rmb == compiler_barrier
smp_mb is a proper memory barrier.

Konstantin

Sorry I did not make it clear.
Anyway, on x86, smp_rmb, as a compiler barrier, applies to load/store, not only load/load. This is the case also for arm, arm64, ppc32, ppc64.
I will submit a patch to expand the definition of this API.

Quoted 14 lines
Quoted 4 lines

as implemented as a compiler barrier: rte_compiler_barrier, arm32 is also this case.
For PPC, both 32 and 64-bit, rmb=wmb=mb, lwsync/sync orders

load/store, load/load, store/load, store/store, looking at the table on this

Quoted 4 lines

page:
https://www.ibm.com/developerworks/systems/articles/powerpc.html

In summary, we are safe to expand this definition for all the

architectures DPDK support?

Quoted 4 lines

Any comments are welcome!

BR. Gavin

#18 lore raw
Quoted 69 lines
Quoted 69 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

In weak memory models, like arm64, reading the {prod,cons}.tail

may

Quoted 2 lines
Quoted 2 lines

get

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

reordered after reading or writing the ring slots, which corrupts

the

Quoted 4 lines
Quoted 4 lines

ring

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The

problem

Quoted 3 lines
Quoted 3 lines

is

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

most

Quoted 1 line

likely caused by missing the acquire semantics when reading

cons.tail

Quoted 2 lines
Quoted 2 lines

(in

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

SP enqueue) or prod.tail (in SC dequeue) which makes it possible

to

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

read

Quoted 4 lines
Quoted 4 lines

a

Quoted 3 lines

stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

required

Quoted 1 line

ordering. This patch is to prevent reading and writing the ring

slots get

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line

reordered before reading {prod,cons}.tail for SP (and SC) case.

Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e. rte_smp_mb().

Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while

WMB(ST

Quoted 2 lines

Option) orders ST/ST.

Quoted 1 line

For more details, please refer to: Table B2-1 Encoding of the DMB

and

Quoted 6 lines
Quoted 6 lines

DSB

Quoted 5 lines

<option> parameter in

Quoted 1 line

https://developer.arm.com/docs/ddi0487/latest/arm-architecture-

reference-manual-armv8-for-armv8-a-architecture-profile

I see. But you have to change the rte_smp_rmb() function definition

in

Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_eal/common/include/generic/rte_atomic.h and assure that

all

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

other architectures follows same rules.
Otherwise, this change is logically wrong, because read barrier in

current

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

definition could not be used to order Load with Store.

Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm,

read

Quoted 5 lines
Quoted 1 line

barrier(load/store and load/load) is enough.

Hi Ilya,

I would expand the rmb definition to cover load/store, in addition to

load/load.

Quoted 1 line

For X86, as a strong memory order model, rmb is actually equivalent to

mb,

Quoted 8 lines
Quoted 5 lines

That's not exactly the case, on x86 we have:
smp_rmb == compiler_barrier
smp_mb is a proper memory barrier.

Konstantin

Sorry I did not make it clear.
Anyway, on x86, smp_rmb, as a compiler barrier, applies to load/store, not only load/load.

Yes, that's true, but I think that's happened by coincidence, not intentionally.

Quoted 2 lines

This is the case also for arm, arm64, ppc32, ppc64. I will submit a patch to expand the definition of this API.

I understand your intention, but that does mean we would also need to change not only rte_smp_rmb() but rte_rmb() too (to keep things consistent)? That sounds worring.
Might be better to keep smp_rmb() definition as it is, and introduce new function that fits your purposes (smp_rwmb or smp_load_store_barrier)?

Konstantin

Quoted 15 lines
Quoted 15 lines
Quoted 5 lines

as implemented as a compiler barrier: rte_compiler_barrier, arm32 is also this case.
For PPC, both 32 and 64-bit, rmb=wmb=mb, lwsync/sync orders

load/store, load/load, store/load, store/store, looking at the table on this

Quoted 4 lines

page:
https://www.ibm.com/developerworks/systems/articles/powerpc.html

In summary, we are safe to expand this definition for all the

architectures DPDK support?

Quoted 4 lines

Any comments are welcome!

BR. Gavin

#19 lore raw

08/03/2019 16:50, Ananyev, Konstantin:

Quoted 6 lines

08/03/2019 16:05, Gavin Hu (Arm Technology China):

Quoted 1 line

Anyway, on x86, smp_rmb, as a compiler barrier, applies to load/store, not only load/load.

Yes, that's true, but I think that's happened by coincidence, not intentionally.

Quoted 8 lines
Quoted 2 lines

This is the case also for arm, arm64, ppc32, ppc64. I will submit a patch to expand the definition of this API.

I understand your intention, but that does mean we would also need to change not only rte_smp_rmb() but rte_rmb() too (to keep things consistent)? That sounds worring.
Might be better to keep smp_rmb() definition as it is, and introduce new function that fits your purposes (smp_rwmb or smp_load_store_barrier)?

How is it managed in other projects?

#20 lore raw
Quoted 16 lines

08/03/2019 16:50, Ananyev, Konstantin:

Quoted 2 lines

08/03/2019 16:05, Gavin Hu (Arm Technology China):

Quoted 1 line

Anyway, on x86, smp_rmb, as a compiler barrier, applies to load/store, not

only load/load.

Quoted 12 lines

Yes, that's true, but I think that's happened by coincidence, not intentionally.

Quoted 2 lines

This is the case also for arm, arm64, ppc32, ppc64. I will submit a patch to expand the definition of this API.

I understand your intention, but that does mean we would also need to change not only rte_smp_rmb() but rte_rmb() too (to keep things consistent)? That sounds worring.
Might be better to keep smp_rmb() definition as it is, and introduce new function that fits your purposes (smp_rwmb or smp_load_store_barrier)?

Looking at rte_rmb, rte_io_rmb, rte_cio_rmb implementations for Arm, they all provide load/store barrier as well. If other architectures also provide load/store barrier with rte_xxx_rmb, then we could extend the meaning of the existing APIs.

Even if a new API is provided, we need to do provide the same APIs for IO and CIO variants.

Quoted 2 lines

How is it managed in other projects?

In my experience, I usually have been changing the algorithms to use C11 memory model. So, I have not come across this issue yet. Others can comment.

Quoted 1 line
#21 lore raw
Quoted 31 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, March 9, 2019 7:48 AM
To: [email protected]; Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Cc: Ilya Maximets <[email protected]>; [email protected]; nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v2] ring: enforce reading the tails before ring operations

Quoted 3 lines

08/03/2019 16:50, Ananyev, Konstantin:

Quoted 2 lines

08/03/2019 16:05, Gavin Hu (Arm Technology China):

Quoted 1 line

Anyway, on x86, smp_rmb, as a compiler barrier, applies to

load/store, not

Quoted 9 lines

only load/load.

Quoted 8 lines

Yes, that's true, but I think that's happened by coincidence, not intentionally.

Quoted 2 lines

This is the case also for arm, arm64, ppc32, ppc64. I will submit a patch to expand the definition of this API.

I understand your intention, but that does mean we would also need

to

Quoted 1 line
Quoted 1 line

change not only rte_smp_rmb() but rte_rmb() too (to keep things

consistent)?

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

That sounds worring.
Might be better to keep smp_rmb() definition as it is, and introduce new function that fits your purposes (smp_rwmb or

smp_load_store_barrier)?
Looking at rte_rmb, rte_io_rmb, rte_cio_rmb implementations for Arm, they all provide load/store barrier as well. If other architectures also provide load/store barrier with rte_xxx_rmb, then we could extend the meaning of the existing APIs.

Further looking at rte_rmb, rte_io_rmb, rte_cio_rmb implementations for PPC64 and x86, They also provide load/store barrier. It is safe to extend the meaning of the existing rte_XXX_rmb API.

Quoted 3 lines

Even if a new API is provided, we need to do provide the same APIs for IO and CIO variants.

Since rte_XXX_rmbs API for all architectures already provide the desired load/store ordering, a new API is redundant and not needed.

Quoted 7 lines
Quoted 2 lines

How is it managed in other projects?

In my experience, I usually have been changing the algorithms to use C11 memory model. So, I have not come across this issue yet. Others can comment.

Quoted 1 line
author
#22 lore raw

v3 cover ring: enforce reading the tail before reading ring slots

In reply to gavin hu

Change log:
-V1: First version of the patch.

-V2: Update the commit message to describe the ordering of detailed memory
accesses.

-V3: As suggested by Konstantin Ananyev, remove the rte_smp_rmb for enqueue
case, as there is a control dependency in place, making rmb unnecessary. For dequeue case, it is still required as the control dependency does not order loads with loads.

gavin hu (1):
ring: enforce reading the tail before reading ring slots

 lib/librte_ring/rte_ring_generic.h | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
Signature · 1 line
2.7.4
author
#23 lore raw

PATCH v3 1/1 ring: enforce reading the tail before reading ring slots

From: gavin hu <[email protected]>

In weak memory models, like arm64, reading the prod.tail may get reordered after reading the ring slots, which corrupts the ring and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides the required ordering. For SP case, the control depependency between if-statement(which depends on the read of r->cons.tail) and the later stores to the ring slots make RMB unnecessary. About the control dependency, read more at:
https://www.cl.cam.ac.uk/~pes20/ppc-supplemental/test7.pdf

This patch is adding the required read barrier to prevent reading the ring slots get reordered before reading prod.tail for SC case.

Fixes c9fb3c62896f ("ring: move code in a new header file")
---
 lib/librte_ring/rte_ring_generic.h | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
lib/librte_ring/rte_ring_generic.h +6 -3
index ea7dbe5..953cdbb 100644
@@ -158,11 +158,14 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sc)
-			r->cons.head = *new_head, success = 1;
-		else
+		if (is_sc) {
+			r->cons.head = *new_head;
+			rte_smp_rmb();
+			success = 1;
+		} else {
 			success = rte_atomic32_cmpset(&r->cons.head, *old_head,
 					*new_head);
+		}
 	} while (unlikely(success == 0));
 	return n;
 }
Signature · 1 line
2.7.4
Acked-by on v3 1/1
#24 lore raw

Re: [PATCH v3 1/1] ring: enforce reading the tail before reading ring slots

Quoted 39 lines

-----Original Message-----
From: Gavin Hu [mailto:[email protected]]
Sent: Tuesday, March 12, 2019 10:29 PM
To: [email protected]
Cc: [email protected]; gavin hu <[email protected]>; [email protected]; [email protected]; [email protected]; Hemant Agrawal <[email protected]>; Nipun Gupta <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 1/1] ring: enforce reading the tail before reading ring slots

From: gavin hu <[email protected]>

In weak memory models, like arm64, reading the prod.tail may get reordered after reading the ring slots, which corrupts the ring and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides the required ordering. For SP case, the control depependency between if- statement(which
depends on the read of r->cons.tail) and the later stores to the ring slots make RMB unnecessary. About the control dependency, read more at:
https://emea01.safelinks.protection.outlook.com/?url=https:%2F%2Fwww.c
l.cam.ac.uk%2F~pes20%2Fppc- supplemental%2Ftest7.pdf&amp;data=02%7C01%7Cnipun.gupta%40nxp.co m%7Cc3df030ec49449bbaf0508d6a70c0b9f%7C686ea1d3bc2b4c6fa92cd99c5c 301635%7C0%7C0%7C636880067514602937&amp;sdata=ogPdd%2F4UGYWE8 nAwY1lkXPCB9MpUIFY0VOQr2N1lwe4%3D&amp;reserved=0

This patch is adding the required read barrier to prevent reading the ring slots get reordered before reading prod.tail for SC case.

Quoted 6 lines
Fixes c9fb3c62896f ("ring: move code in a new header file")

Also, I have revalidated this updated patch.

Acked-by on v3 1/1
#25 lore raw

Re: [PATCH v3 1/1] ring: enforce reading the tail before reading ring slots

In reply to Gavin Hu
Quoted 54 lines

-----Original Message-----
From: Gavin Hu [mailto:[email protected]]
Sent: Tuesday, March 12, 2019 4:59 PM
To: [email protected]
Cc: [email protected]; gavin hu <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 1/1] ring: enforce reading the tail before reading ring slots

From: gavin hu <[email protected]>

In weak memory models, like arm64, reading the prod.tail may get reordered after reading the ring slots, which corrupts the ring and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides the required ordering. For SP case, the control depependency between if-statement(which depends on the read of r->cons.tail) and the later stores to the ring slots make RMB unnecessary. About the control dependency, read more at:
https://www.cl.cam.ac.uk/~pes20/ppc-supplemental/test7.pdf

This patch is adding the required read barrier to prevent reading the ring slots get reordered before reading prod.tail for SC case.

Fixes c9fb3c62896f ("ring: move code in a new header file")
---
 lib/librte_ring/rte_ring_generic.h | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
lib/librte_ring/rte_ring_generic.h +2 -3
index ea7dbe5..953cdbb 100644
@@ -158,11 +158,14 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
 			return 0;
 
 		*new_head = *old_head + n;
-		if (is_sc)
-			r->cons.head = *new_head, success = 1;
-		else
+		if (is_sc) {
+			r->cons.head = *new_head;

lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,

Quoted 10 lines
lib/librte_ring/rte_ring_generic.h +4 -1
+			rte_smp_rmb();
+			success = 1;
+		} else {
 			success = rte_atomic32_cmpset(&r->cons.head, *old_head,
 					*new_head);
+		}
 	} while (unlikely(success == 0));
 	return n;
 }
--
Quoted 1 line

2.7.4

#26 lore raw

Re: [PATCH v3 1/1] ring: enforce reading the tail before reading ring slots

Quoted 18 lines
Quoted 18 lines

In weak memory models, like arm64, reading the prod.tail may get reordered after reading the ring slots, which corrupts the ring and stale data is observed.

This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots.

For MP (and MC) case, rte_atomic32_cmpset() already provides the required ordering. For SP case, the control depependency between if-statement(which depends on the read of r->cons.tail) and the later stores to the ring slots make RMB unnecessary. About the control dependency, read more at:
https://www.cl.cam.ac.uk/~pes20/ppc-supplemental/test7.pdf

This patch is adding the required read barrier to prevent reading the ring slots get reordered before reading prod.tail for SC case.

Quoted 8 lines
Quoted 6 lines
Fixes c9fb3c62896f ("ring: move code in a new header file")

Applied, thanks