ring: enforce reading the tails before ring operations
PATCH 1/1 ring: enforce reading the tails before ring operations
In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots. There must be a read fence before writing or reading the ring slots, rte_atomic32_cmpset() provides the same ordering for MP (and MC) case. This patch is to enforce this ordering for SP (and SC) case.
--- lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
lib/librte_ring/rte_ring_generic.h +10 -6
@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,return 0 ; *new_head = *old_head + n; -if (is_sp) - r->prod.head = *new_head, success =1 ; -else +if (is_sp) { + r->prod.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success =rte_atomic32_cmpset (&r->prod.head, *old_head, *new_head); }while (unlikely (success ==0 ));
@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,return 0 ; *new_head = *old_head + n; -if (is_sc) - r->cons.head = *new_head, success =1 ; -else +if (is_sc) { + r->cons.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success =rte_atomic32_cmpset (&r->cons.head, *old_head, *new_head); }while (unlikely (success ==0 ));
Signature · 1 line
2.7.4
Quoted 7 lines · gavin hu
In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots. There must be a read fence before writing
Sorry, but the phrase "There must be a read fence before writing" makes no sense. Could you please rephrase or describe in details which reads you're trying to keep in exact order?
Quoted 46 lines
or reading the ring slots,
rte_atomic32_cmpset()provides the same ordering for MP (and MC) case. This patch is to enforce this ordering for SP (and SC) case.Signed-off-by gavin hu <[email protected]>Reviewed-by Ola Liljedahl <[email protected]>Tested-by Nipun Gupta <[email protected]>--- lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)lib/librte_ring/rte_ring_generic.h +10
-6@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,return 0 ; *new_head = *old_head + n; -if (is_sp) - r->prod.head = *new_head, success =1 ; -else +if (is_sp) { + r->prod.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success =rte_atomic32_cmpset (&r->prod.head, *old_head, *new_head); }while (unlikely (success ==0 ));@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,return 0 ; *new_head = *old_head + n; -if (is_sc) - r->cons.head = *new_head, success =1 ; -else +if (is_sc) { + r->cons.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success =rte_atomic32_cmpset (&r->cons.head, *old_head, *new_head); }while (unlikely (success ==0 ));
Hi ilya,
Thanks for your comments, inline comments and new commit message in V2.
/Gavin
Quoted 19 lines
-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Wednesday, March 6, 2019 7:49 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: Re: [v1] ring: enforce reading the tails before ring operationsQuoted 4 lines · gavin hu
In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem ismost
Quoted 2 lines
likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to read
a
Quoted 8 lines
Quoted 1 line
stale value from the ring slots. There must be a read fence before
writing
Sorry, but the phrase "There must be a read fence before writing" makes no sense.
Could you please rephrase or describe in details which reads you're trying to
keep in exact order?
This patch is to keep in exact order for reading the tails(to calculate the available /free slots of ring when moving the heads) before reading or writing the ring slots.
I rephrased the commit message in V2, could you have a look?
Quoted 51 lines
Quoted 1 line
or reading the ring slots,
rte_atomic32_cmpset()provides the sameordering
Review
Quoted 11 lines
for MP (and MC) case. This patch is to enforce this ordering for SP (and SC) case.
Signed-off-by gavin hu <[email protected]>Reviewed-by Ola Liljedahl <[email protected]>Tested-by Nipun Gupta <[email protected]>--- lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)(quoted diff) +0
-0
b/lib/librte_ring/rte_ring_generic.hlib/librte_ring/rte_ring_generic.h __rte_ring_move_prod_head(struct rte_ring *r,
Quoted 4 lines
index ea7dbe5..1bd3dfd 100644
lib/librte_ring/rte_ring_generic.h +0
-0@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r,unsigned int
is_sp,lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r,
Quoted 15 lines
lib/librte_ring/rte_ring_generic.h +5
-3return 0 ; *new_head = *old_head + n; -if (is_sp) - r->prod.head = *new_head, success =1 ; -else +if (is_sp) { + r->prod.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success =rte_atomic32_cmpset (&r->prod.head, *old_head, *new_head); }while (unlikely (success ==0 ));@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring *r,unsigned int
is_sc,lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
Quoted 12 lines
lib/librte_ring/rte_ring_generic.h +5
-3return 0 ; *new_head = *old_head + n; -if (is_sc) - r->cons.head = *new_head, success =1 ; -else +if (is_sc) { + r->cons.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success = rte_atomic32_cmpset(&r->cons.head,
*old_head,Quoted 3 lines
lib/librte_ring/rte_ring_generic.h +0
-0*new_head); }while (unlikely (success ==0 ));
PATCH v2 1/1 ring: enforce reading the tails before ring operations
In reply to gavin huIn weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots.
For MP (and MC) case, rte_atomic32_cmpset() already provides the required ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.
--- lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
lib/librte_ring/rte_ring_generic.h +10 -6
@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,return 0 ; *new_head = *old_head + n; -if (is_sp) - r->prod.head = *new_head, success =1 ; -else +if (is_sp) { + r->prod.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success =rte_atomic32_cmpset (&r->prod.head, *old_head, *new_head); }while (unlikely (success ==0 ));
@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,return 0 ; *new_head = *old_head + n; -if (is_sc) - r->cons.head = *new_head, success =1 ; -else +if (is_sc) { + r->cons.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success =rte_atomic32_cmpset (&r->cons.head, *old_head, *new_head); }while (unlikely (success ==0 ));
Signature · 1 line
2.7.4
Quoted 4 lines · gavin hu
In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
Quoted 8 lines
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already provides the required ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.
Read barrier rte_smp_rmb() is OK to prevent reading the ring get reordered before reading the tail. However, to prevent *writing* the ring get reordered *before reading* the tail you need a full memory barrier, i.e. rte_smp_mb().
Quoted 43 lines
Signed-off-by gavin hu <[email protected]>Reviewed-by Ola Liljedahl <[email protected]>Tested-by Nipun Gupta <[email protected]>--- lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)lib/librte_ring/rte_ring_generic.h +10
-6@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,return 0 ; *new_head = *old_head + n; -if (is_sp) - r->prod.head = *new_head, success =1 ; -else +if (is_sp) { + r->prod.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success =rte_atomic32_cmpset (&r->prod.head, *old_head, *new_head); }while (unlikely (success ==0 ));@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,return 0 ; *new_head = *old_head + n; -if (is_sc) - r->cons.head = *new_head, success =1 ; -else +if (is_sc) { + r->cons.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success =rte_atomic32_cmpset (&r->cons.head, *old_head, *new_head); }while (unlikely (success ==0 ));
Quoted 30 lines
-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 4:52 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 5 lines · gavin hu
In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is
most
Quoted 2 lines
likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to read
a
Quoted 3 lines
stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already provides therequired
Quoted 2 lines
ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.
Quoted 6 lines
Read barrier
rte_smp_rmb()is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e.rte_smp_mb().
ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST Option) orders ST/ST. For more details, please refer to: Table B2-1 Encoding of the DMB and DSB <option> parameter in
https://developer.arm.com/docs/ddi0487/latest/arm-architecture-reference-manual-armv8-for-armv8-a-architecture-profile
Quoted 48 lines
Review
Quoted 10 lines
Signed-off-by gavin hu <[email protected]>Reviewed-by Ola Liljedahl <[email protected]>Tested-by Nipun Gupta <[email protected]>--- lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)(quoted diff) +0
-0
b/lib/librte_ring/rte_ring_generic.hlib/librte_ring/rte_ring_generic.h __rte_ring_move_prod_head(struct rte_ring *r,
Quoted 4 lines
index ea7dbe5..1bd3dfd 100644
lib/librte_ring/rte_ring_generic.h +0
-0@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r,unsigned int
is_sp,lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r,
Quoted 15 lines
lib/librte_ring/rte_ring_generic.h +5
-3return 0 ; *new_head = *old_head + n; -if (is_sp) - r->prod.head = *new_head, success =1 ; -else +if (is_sp) { + r->prod.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success =rte_atomic32_cmpset (&r->prod.head, *old_head, *new_head); }while (unlikely (success ==0 ));@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring *r,unsigned int
is_sc,lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
Quoted 12 lines
lib/librte_ring/rte_ring_generic.h +5
-3return 0 ; *new_head = *old_head + n; -if (is_sc) - r->cons.head = *new_head, success =1 ; -else +if (is_sc) { + r->cons.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success = rte_atomic32_cmpset(&r->cons.head,
*old_head,Quoted 3 lines
lib/librte_ring/rte_ring_generic.h +0
-0*new_head); }while (unlikely (success ==0 ));
Quoted 37 lines · Gavin Hu (Arm Technology China)
Quoted 30 lines
-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 4:52 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 5 lines · gavin hu
In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is
most
Quoted 2 lines
likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to read
a
Quoted 3 lines
stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already provides therequired
Quoted 2 lines
ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.
Quoted 6 lines
Read barrier
rte_smp_rmb()is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e.rte_smp_mb().
Quoted 3 lines
ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST Option) orders ST/ST. For more details, please refer to: Table B2-1 Encoding of the DMB and DSB <option> parameter in
https://developer.arm.com/docs/ddi0487/latest/arm-architecture-reference-manual-armv8-for-armv8-a-architecture-profile
I see. But you have to change the rte_smp_rmb() function definition in lib/librte_eal/common/include/generic/rte_atomic.h and assure that all other architectures follows same rules.
Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.
Quoted 49 lines
Quoted 49 lines
Review
Quoted 11 lines
Signed-off-by gavin hu <[email protected]>Reviewed-by Ola Liljedahl <[email protected]>Tested-by Nipun Gupta <[email protected]>--- lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)(quoted diff) +0
-0
b/lib/librte_ring/rte_ring_generic.hlib/librte_ring/rte_ring_generic.h __rte_ring_move_prod_head(struct rte_ring *r,
Quoted 4 lines
index ea7dbe5..1bd3dfd 100644
lib/librte_ring/rte_ring_generic.h +0
-0@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r,unsigned int
is_sp,lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r,
Quoted 15 lines
lib/librte_ring/rte_ring_generic.h +5
-3return 0 ; *new_head = *old_head + n; -if (is_sp) - r->prod.head = *new_head, success =1 ; -else +if (is_sp) { + r->prod.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success =rte_atomic32_cmpset (&r->prod.head, *old_head, *new_head); }while (unlikely (success ==0 ));@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring *r,unsigned int
is_sc,lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
Quoted 12 lines
lib/librte_ring/rte_ring_generic.h +5
-3return 0 ; *new_head = *old_head + n; -if (is_sc) - r->cons.head = *new_head, success =1 ; -else +if (is_sc) { + r->cons.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success = rte_atomic32_cmpset(&r->cons.head,
*old_head,Quoted 3 lines
lib/librte_ring/rte_ring_generic.h +0
-0*new_head); }while (unlikely (success ==0 ));
Quoted 55 lines
-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 5:48 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 14 lines · Gavin Hu (Arm Technology China)
Quoted 14 lines
-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 4:52 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operationsOn 07.03.2019 9:45, gavin hu wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is
most
Quoted 2 lines
likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to
read
Quoted 8 lines
Quoted 8 lines
a
Quoted 3 lines
stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already provides therequired
Quoted 2 lines
ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.
Quoted 8 lines
Quoted 6 lines
Read barrier
rte_smp_rmb()is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e.rte_smp_mb().ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST
Option) orders ST/ST.
Quoted 1 line
For more details, please refer to: Table B2-1 Encoding of the DMB and DSB
<option> parameter in
Quoted 9 lines
reference-manual-armv8-for-armv8-a-architecture-profile
I see. But you have to change the
rte_smp_rmb()function definition inlib/librte_eal/common/include/generic/rte_atomic.hand assure that all other architectures follows same rules.
Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.
Good points, let me re-think how to handle for other architectures.
Full MB is required for other architectures(x86? Ppc?), but for arm, read barrier(load/store and load/load) is enough.
Quoted 50 lines
Quoted 19 lines
Quoted 19 lines
Review
Quoted 11 lines
Signed-off-by gavin hu <[email protected]>Reviewed-by Ola Liljedahl <[email protected]>Tested-by Nipun Gupta <[email protected]>--- lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)(quoted diff) +0
-0
b/lib/librte_ring/rte_ring_generic.hlib/librte_ring/rte_ring_generic.h __rte_ring_move_prod_head(struct rte_ring *r,
Quoted 4 lines
index ea7dbe5..1bd3dfd 100644
lib/librte_ring/rte_ring_generic.h +0
-0@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r,unsigned int
is_sp,Quoted 2 lines
lib/librte_ring/rte_ring_generic.h +0
-0return 0 ;Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
lib/librte_ring/rte_ring_generic.h +1
-3*new_head = *old_head + n; -if (is_sp) - r->prod.head = *new_head, success =1 ; -else +if (is_sp) {Quoted 8 lines
lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring
Quoted 8 lines
lib/librte_ring/rte_ring_generic.h +4
-0+ r->prod.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success =rte_atomic32_cmpset (&r->prod.head, *old_head, *new_head); }while (unlikely (success ==0 ));@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring*r,
Quoted 17 lines
Quoted 17 lines
unsigned int
is_sc,lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
Quoted 12 lines
lib/librte_ring/rte_ring_generic.h +5
-3return 0 ; *new_head = *old_head + n; -if (is_sc) - r->cons.head = *new_head, success =1 ; -else +if (is_sc) { + r->cons.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success = rte_atomic32_cmpset(&r->cons.head,
*old_head,Quoted 3 lines
lib/librte_ring/rte_ring_generic.h +0
-0*new_head); }while (unlikely (success ==0 ));
Hi Gavin,
Quoted 38 lines
Quoted 28 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is
most
Quoted 2 lines
likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to
read
Quoted 8 lines
Quoted 8 lines
a
Quoted 3 lines
stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already provides therequired
Quoted 2 lines
ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.
Quoted 8 lines
Quoted 6 lines
Read barrier
rte_smp_rmb()is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e.rte_smp_mb().ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST
Option) orders ST/ST.
Quoted 1 line
For more details, please refer to: Table B2-1 Encoding of the DMB and DSB
<option> parameter in
Quoted 9 lines
reference-manual-armv8-for-armv8-a-architecture-profile
I see. But you have to change the
rte_smp_rmb()function definition inlib/librte_eal/common/include/generic/rte_atomic.hand assure that all other architectures follows same rules.
Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.
Quoted 2 lines
Good points, let me re-think how to handle for other architectures.
Full MB is required for other architectures(x86? Ppc?), but for arm, read barrier(load/store and load/load) is enough.
For x86, I don't think you need any barrier here, as with IA memory mode:
- Reads are not reordered with other reads.
- Writes are not reordered with older reads.
BTW, could you explain a bit more why barrier is necessary even on arm here? As I can see, there is a data dependency between the tail value and subsequent address calculations for ring writes/reads.
Isn't that sufficient to prevent re-ordering even for weak memory model? Konstantin
Quoted 51 lines
Quoted 51 lines
Quoted 20 lines
Quoted 20 lines
Quoted 4 lines
Review
Quoted 8 lines
Signed-off-by gavin hu <[email protected]>Reviewed-by Ola Liljedahl <[email protected]>Tested-by Nipun Gupta <[email protected]>--- lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)(quoted diff) +0
-0
b/lib/librte_ring/rte_ring_generic.hlib/librte_ring/rte_ring_generic.h __rte_ring_move_prod_head(struct rte_ring *r,
Quoted 4 lines
index ea7dbe5..1bd3dfd 100644
lib/librte_ring/rte_ring_generic.h +0
-0@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring *r,unsigned int
is_sp,Quoted 2 lines
lib/librte_ring/rte_ring_generic.h +0
-0return 0 ;Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
lib/librte_ring/rte_ring_generic.h +1
-3*new_head = *old_head + n; -if (is_sp) - r->prod.head = *new_head, success =1 ; -else +if (is_sp) {Quoted 8 lines
lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring
Quoted 8 lines
lib/librte_ring/rte_ring_generic.h +4
-0+ r->prod.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success =rte_atomic32_cmpset (&r->prod.head, *old_head, *new_head); }while (unlikely (success ==0 ));@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct rte_ring*r,
Quoted 17 lines
Quoted 17 lines
unsigned int
is_sc,lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
Quoted 12 lines
lib/librte_ring/rte_ring_generic.h +5
-3return 0 ; *new_head = *old_head + n; -if (is_sc) - r->cons.head = *new_head, success =1 ; -else +if (is_sc) { + r->cons.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success = rte_atomic32_cmpset(&r->cons.head,
*old_head,Quoted 3 lines
lib/librte_ring/rte_ring_generic.h +0
-0*new_head); }while (unlikely (success ==0 ));
Quoted 44 lines
Hi Gavin,
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is
most
Quoted 3 lines
likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to
Quoted 8 lines
Quoted 8 lines
read
Quoted 7 lines
Quoted 7 lines
a
Quoted 3 lines
stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already provides therequired
Quoted 2 lines
ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC)
case.
Quoted 23 lines
Quoted 22 lines
Quoted 9 lines
Quoted 6 lines
Read barrier
rte_smp_rmb()is OK to prevent reading the ring get reordered before reading the tail. However, to prevent *writing* the ring get reordered *before reading* the tail you need a full memory barrier, i.e.rte_smp_mb().ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST
Option) orders ST/ST.
Quoted 2 lines
For more details, please refer to: Table B2-1 Encoding of the DMB and DSB
<option> parameter in
reference-manual-armv8-for-armv8-a-architecture-profile
I see. But you have to change the
rte_smp_rmb()function definition inlib/librte_eal/common/include/generic/rte_atomic.hand assure that all other architectures follows same rules. Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.
Quoted 7 lines
Quoted 2 lines
Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read
barrier(load/store and load/load) is enough.
For x86, I don't think you need any barrier here, as with IA memory mode:
- Reads are not reordered with other reads.
- Writes are not reordered with older reads.
Agree
Quoted 5 lines
BTW, could you explain a bit more why barrier is necessary even on arm here? As I can see, there is a data dependency between the tail value and subsequent address calculations for ring writes/reads.
Isn't that sufficient to prevent re-ordering even for weak memory model?
The tail value affects 'n'. But, the value of 'n' can be speculated because of the following 'if' statement:
if (unlikely(n > *free_entries))
n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 : *free_entries;
The address calculations for actual ring writes/reads do not depend on the tail value. Since 'n' can be speculated, the writes/reads can be moved up before the load of the tail value.
Quoted 3 lines
Konstantin
<snip>
Hi Konstantin,
Quoted 58 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Friday, March 8, 2019 11:21 AM
To: Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; Ilya Maximets <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 7 lines
Hi Gavin,
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The
problem
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
is
most
Quoted 3 lines
likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to
read
Quoted 4 lines
Quoted 4 lines
a
Quoted 3 lines
stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already providesthe
Quoted 27 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
required
Quoted 2 lines
ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC)
case.
Quoted 23 lines
Quoted 22 lines
Quoted 9 lines
Quoted 6 lines
Read barrier
rte_smp_rmb()is OK to prevent reading the ring get reordered before reading the tail. However, to prevent *writing* the ring get reordered *before reading* the tail you need a full memory barrier, i.e.rte_smp_mb().ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST
Option) orders ST/ST.
Quoted 2 lines
For more details, please refer to: Table B2-1 Encoding of the DMB and DSB
<option> parameter in
reference-manual-armv8-for-armv8-a-architecture-profile
I see. But you have to change the
rte_smp_rmb()function definition inlib/librte_eal/common/include/generic/rte_atomic.hand assure that all other architectures follows same rules. Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read
barrier(load/store and load/load) is enough.
For x86, I don't think you need any barrier here, as with IA memory mode:
- Reads are not reordered with other reads.
- Writes are not reordered with older reads.Agree
I understand herein no instruction level barriers are required for IA, but how about the compiler barrier: rte_compiler_barrier?
Quoted 10 lines
Quoted 3 lines
BTW, could you explain a bit more why barrier is necessary even on arm
here?
Quoted 3 lines
As I can see, there is a data dependency between the tail value and subsequent address calculations for ring writes/reads.
Isn't that sufficient to prevent re-ordering even for weak memory model?The tail value affects 'n'. But, the value of 'n' can be speculated because of the following 'if' statement:
Quoted 6 lines
if (
unlikely(n > *free_entries))n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 : *free_entries;The address calculations for actual ring writes/reads do not depend on the tail value. Since 'n' can be speculated, the writes/reads can be moved up before the load of the tail value.
Good explanation. The address calculations does not depend on tail/n, only the limit/last one depends on it, while it can be speculated.
Quoted 4 lines
Quoted 3 lines
Konstantin
<snip>
Quoted 49 lines
Quoted 49 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The
problem
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
is
most
Quoted 3 lines
likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to
read
Quoted 4 lines
Quoted 4 lines
a
Quoted 3 lines
stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already providesthe
Quoted 31 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
required
Quoted 2 lines
ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC)
case.
Quoted 23 lines
Quoted 13 lines
Quoted 9 lines
Quoted 6 lines
Read barrier
rte_smp_rmb()is OK to prevent reading the ring get reordered before reading the tail. However, to prevent *writing* the ring get reordered *before reading* the tail you need a full memory barrier, i.e.rte_smp_mb().ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST
Option) orders ST/ST.
Quoted 2 lines
For more details, please refer to: Table B2-1 Encoding of the DMB and DSB
<option> parameter in
Quoted 9 lines
reference-manual-armv8-for-armv8-a-architecture-profile
I see. But you have to change the
rte_smp_rmb()function definition inlib/librte_eal/common/include/generic/rte_atomic.hand assure that all other architectures follows same rules. Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.Quoted 2 lines
Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read
barrier(load/store and load/load) is enough.
Quoted 23 lines
Quoted 4 lines
Quoted 3 lines
For x86, I don't think you need any barrier here, as with IA memory mode:
- Reads are not reordered with other reads.
- Writes are not reordered with older reads.Agree
I understand herein no instruction level barriers are required for IA, but how about the compiler barrier:
rte_compiler_barrier?Quoted 15 lines
Quoted 3 lines
BTW, could you explain a bit more why barrier is necessary even on arm
here?
Quoted 3 lines
As I can see, there is a data dependency between the tail value and subsequent address calculations for ring writes/reads.
Isn't that sufficient to prevent re-ordering even for weak memory model?The tail value affects 'n'. But, the value of 'n' can be speculated because of the following 'if' statement:
if (
unlikely(n > *free_entries))n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 : *free_entries;The address calculations for actual ring writes/reads do not depend on the tail value.
Ok, agree I formulated it wrongly, only limit value is dependent on cons.tail. Address is not.
Quoted 2 lines
Since 'n' can be speculated, the writes/reads can be moved up
Quoted 1 line
before the load of the tail value.
For my curiosity: ok, I understand that 'n' value can be speculated, and speculative stores could start before n is calculated properly... But are you saying that such speculative store results might be visible to the other observers (different cpu)?
Quoted 8 lines
Good explanation. The address calculations does not depend on tail/n, only the limit/last one depends on it, while it can be speculated.
Quoted 4 lines
Quoted 3 lines
Konstantin
<snip>
Quoted 85 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is
observed.
Quoted 32 lines
Quoted 32 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This issue was reported by NXP on 8-A72 DPAA2 board. The
problem
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
is
most
Quoted 3 lines
likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to
read
Quoted 5 lines
Quoted 5 lines
a
Quoted 4 lines
stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already providesthe
Quoted 17 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
required
Quoted 3 lines
ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC)
case.
Quoted 12 lines
Quoted 12 lines
Quoted 9 lines
Quoted 6 lines
Read barrier
rte_smp_rmb()is OK to prevent reading the ring get reordered before reading the tail. However, to prevent *writing* the ring get reordered *before reading* the tail you need a full memory barrier, i.e.rte_smp_mb().ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST
Option) orders ST/ST.
Quoted 2 lines
For more details, please refer to: Table B2-1 Encoding of the DMB and DSB
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
<option> parameter in
Quoted 2 lines
https://developer.arm.com/docs/ddi0487/latest/arm-architectu
re-reference-manual-armv8-for-armv8-a-architecture-profile
I see. But you have to change the
rte_smp_rmb()function definition inlib/librte_eal/common/include/generic/rte_atomic.hand assure thatall other architectures follows same rules.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with
Store.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read
barrier(load/store and load/load) is enough.
For x86, I don't think you need any barrier here, as with IA memory
mode:
Quoted 24 lines
Quoted 3 lines
Quoted 2 lines
- Reads are not reordered with other reads.
- Writes are not reordered with older reads.Agree
I understand herein no instruction level barriers are required for IA, but how about the compiler barrier:
rte_compiler_barrier?Quoted 17 lines
Quoted 4 lines
BTW, could you explain a bit more why barrier is necessary even on arm
here?
Quoted 3 lines
As I can see, there is a data dependency between the tail value and subsequent address calculations for ring writes/reads. Isn't that sufficient to prevent re-ordering even for weak memory model?
The tail value affects 'n'. But, the value of 'n' can be speculated because of the following 'if' statement:
if (
unlikely(n > *free_entries))n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 :
*free_entries;The address calculations for actual ring writes/reads do not depend on the tail value.
Ok, agree I formulated it wrongly, only limit value is dependent on cons.tail. Address is not.
Quoted 8 lines
Quoted 2 lines
Since 'n' can be speculated, the writes/reads can be moved up
Quoted 1 line
before the load of the tail value.
For my curiosity: ok, I understand that 'n' value can be speculated, and speculative stores could start before n is calculated properly... But are you saying that such speculative store results might be visible to the other observers (different cpu)?
You are correct. The speculative stores will NOT be visible to other observers till the value of 'n' is fixed. Speculative stores might have to be discarded depending on the value of 'n' (which will affect cache performance). There is also a control dependency between the load of cons.tail and the stores to the ring. That should also keep the load and stores from getting reordered (though I am not sure if it still allows for speculative stores). So, IMO, the barrier in enqueue is not needed. Is this what you wanted to drive at?
Quoted 93 lines
Quoted 85 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
In weak memory models, like arm64, reading the {prod,cons}.tail may get reordered after reading or writing the ring slots, which corrupts the ring and stale data is
observed.
Quoted 32 lines
Quoted 32 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This issue was reported by NXP on 8-A72 DPAA2 board. The
problem
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
is
most
Quoted 3 lines
likely caused by missing the acquire semantics when reading cons.tail (in SP enqueue) or prod.tail (in SC dequeue) which makes it possible to
read
Quoted 5 lines
Quoted 5 lines
a
Quoted 4 lines
stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already providesthe
Quoted 17 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
required
Quoted 3 lines
ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC)
case.
Quoted 12 lines
Quoted 12 lines
Quoted 9 lines
Quoted 6 lines
Read barrier
rte_smp_rmb()is OK to prevent reading the ring get reordered before reading the tail. However, to prevent *writing* the ring get reordered *before reading* the tail you need a full memory barrier, i.e.rte_smp_mb().ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while WMB(ST
Option) orders ST/ST.
Quoted 2 lines
For more details, please refer to: Table B2-1 Encoding of the DMB and DSB
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
<option> parameter in
Quoted 2 lines
https://developer.arm.com/docs/ddi0487/latest/arm-architectu
re-reference-manual-armv8-for-armv8-a-architecture-profile
I see. But you have to change the
rte_smp_rmb()function definition inlib/librte_eal/common/include/generic/rte_atomic.hand assure thatall other architectures follows same rules.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with
Store.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read
barrier(load/store and load/load) is enough.
For x86, I don't think you need any barrier here, as with IA memory
mode:
Quoted 24 lines
Quoted 3 lines
Quoted 2 lines
- Reads are not reordered with other reads.
- Writes are not reordered with older reads.Agree
I understand herein no instruction level barriers are required for IA, but how about the compiler barrier:
rte_compiler_barrier?Quoted 17 lines
Quoted 4 lines
BTW, could you explain a bit more why barrier is necessary even on arm
here?
Quoted 3 lines
As I can see, there is a data dependency between the tail value and subsequent address calculations for ring writes/reads. Isn't that sufficient to prevent re-ordering even for weak memory model?
The tail value affects 'n'. But, the value of 'n' can be speculated because of the following 'if' statement:
if (
unlikely(n > *free_entries))n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 :
*free_entries;The address calculations for actual ring writes/reads do not depend on the tail value.
Ok, agree I formulated it wrongly, only limit value is dependent on cons.tail. Address is not.
Quoted 8 lines
Quoted 2 lines
Since 'n' can be speculated, the writes/reads can be moved up
Quoted 1 line
before the load of the tail value.
For my curiosity: ok, I understand that 'n' value can be speculated, and speculative stores could start before n is calculated properly... But are you saying that such speculative store results might be visible to the other observers (different cpu)?
Quoted 5 lines
You are correct. The speculative stores will NOT be visible to other observers till the value of 'n' is fixed. Speculative stores might have to be discarded depending on the value of 'n' (which will affect cache performance).
There is also a control dependency between the load of cons.tail and the stores to the ring. That should also keep the load and stores from getting reordered (though I am not sure if it still allows for speculative stores).
So, IMO, the barrier in enqueue is not needed. Is this what you wanted to drive at?
Yes, that was my thought.
Thanks
Konstantin
Quoted 86 lines
-----Original Message-----
From: Gavin Hu (Arm Technology China)
Sent: Thursday, March 7, 2019 6:45 PM
To: Ilya Maximets <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 4 lines
-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 5:48 PM
To: Gavin Hu (Arm Technology China) <[email protected]>;Quoted 6 lines
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 10 lines
Quoted 9 lines · Gavin Hu (Arm Technology China)
Quoted 9 lines
-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 4:52 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; HonnappaNagarahalli
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
<[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 1 line · gavin hu
In weak memory models, like arm64, reading the {prod,cons}.tail may
get
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
reordered after reading or writing the ring slots, which corrupts the
ring
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem
is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
most
Quoted 1 line
likely caused by missing the acquire semantics when reading cons.tail
(in
Quoted 18 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
SP enqueue) or prod.tail (in SC dequeue) which makes it possible to
read
Quoted 16 lines
Quoted 8 lines
a
Quoted 3 lines
stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already provides therequired
Quoted 2 lines
ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.
Quoted 6 lines
Read barrier
rte_smp_rmb()is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e.rte_smp_mb().ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while
WMB(ST
Quoted 2 lines
Option) orders ST/ST.
Quoted 1 line
For more details, please refer to: Table B2-1 Encoding of the DMB and
DSB
Quoted 10 lines
<option> parameter in
reference-manual-armv8-for-armv8-a-architecture-profile
I see. But you have to change the
rte_smp_rmb()function definition inlib/librte_eal/common/include/generic/rte_atomic.hand assure that all other architectures follows same rules.
Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.
Quoted 3 lines
Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read barrier(load/store and load/load) is enough.
Hi Ilya,
I would expand the rmb definition to cover load/store, in addition to load/load.
For X86, as a strong memory order model, rmb is actually equivalent to mb, as implemented as a compiler barrier: rte_compiler_barrier, arm32 is also this case. For PPC, both 32 and 64-bit, rmb=wmb=mb, lwsync/sync orders load/store, load/load, store/load, store/store, looking at the table on this page:
https://www.ibm.com/developerworks/systems/articles/powerpc.html
In summary, we are safe to expand this definition for all the architectures DPDK support? Any comments are welcome!
BR. Gavin
Quoted 52 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Signed-off-by gavin hu <[email protected]>Reviewed-by Ola Liljedahl <[email protected]>Tested-by Nipun Gupta <[email protected]>--- lib/librte_ring/rte_ring_generic.h | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Review
Quoted 1 line
(quoted diff) +0
-0
b/lib/librte_ring/rte_ring_generic.hlib/librte_ring/rte_ring_generic.h __rte_ring_move_prod_head(struct rte_ring
Quoted 4 lines
index ea7dbe5..1bd3dfd 100644
lib/librte_ring/rte_ring_generic.h +0
-0@@ -90,9 +90,11 @@ __rte_ring_move_prod_head(struct rte_ring*r,
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
unsigned int
is_sp,lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct
Quoted 15 lines
lib/librte_ring/rte_ring_generic.h +5
-3return 0 ; *new_head = *old_head + n; -if (is_sp) - r->prod.head = *new_head, success =1 ; -else +if (is_sp) { + r->prod.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success =rte_atomic32_cmpset (&r->prod.head, *old_head, *new_head); }while (unlikely (success ==0 ));@@ -158,9 +160,11 @@ __rte_ring_move_cons_head(struct
rte_ringQuoted 18 lines
*r,
Quoted 17 lines
Quoted 17 lines
unsigned int
is_sc,lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
Quoted 12 lines
lib/librte_ring/rte_ring_generic.h +5
-3return 0 ; *new_head = *old_head + n; -if (is_sc) - r->cons.head = *new_head, success =1 ; -else +if (is_sc) { + r->cons.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else success = rte_atomic32_cmpset(&r->cons.head,
*old_head,Quoted 3 lines
lib/librte_ring/rte_ring_generic.h +0
-0*new_head); }while (unlikely (success ==0 ));
Quoted 65 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · gavin hu
Quoted 2 lines
In weak memory models, like arm64, reading the {prod,cons}.tail may
get
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
reordered after reading or writing the ring slots, which corrupts the
ring
Quoted 18 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem
is
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
most
Quoted 2 lines
likely caused by missing the acquire semantics when reading cons.tail
(in
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
SP enqueue) or prod.tail (in SC dequeue) which makes it possible to
read
Quoted 7 lines
Quoted 7 lines
a
Quoted 3 lines
stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already provides therequired
Quoted 2 lines
ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC)
case.
Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Read barrier
rte_smp_rmb()is OK to prevent reading the ring get reordered before reading the tail. However, to prevent *writing* the ring get reordered *before reading* the tail you need a full memory barrier, i.e.rte_smp_mb().ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while
WMB(ST
Quoted 3 lines
Option) orders ST/ST.
Quoted 2 lines
For more details, please refer to: Table B2-1 Encoding of the DMB and
DSB
Quoted 10 lines
<option> parameter in
reference-manual-armv8-for-armv8-a-architecture-profile
I see. But you have to change the
rte_smp_rmb()function definition inlib/librte_eal/common/include/generic/rte_atomic.hand assure that all other architectures follows same rules. Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.Quoted 3 lines
Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read barrier(load/store and load/load) is enough.
Hi Ilya,
I would expand the rmb definition to cover load/store, in addition to load/load.
For X86, as a strong memory order model, rmb is actually equivalent to mb, as implemented as a compiler barrier:rte_compiler_barrier, arm32 is also this case.
For PPC, both 32 and 64-bit, rmb=wmb=mb, lwsync/sync orders load/store, load/load, store/load, store/store, looking at the table on this page:
https://www.ibm.com/developerworks/systems/articles/powerpc.html
Quoted 2 lines
In summary, we are safe to expand this definition for all the architectures DPDK support?
Essentially, it is a documentation bug. i.e. the current implementation of rte_smp_rmb() already behaves as load/load and load/store barrier.
Quoted 4 lines
Any comments are welcome!
BR. Gavin
<snip>
Quoted 97 lines
-----Original Message-----
From: Gavin Hu (Arm Technology China) [mailto:[email protected]]
Sent: Friday, March 8, 2019 4:23 AM
To: Ilya Maximets <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]
Subject: RE: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 84 lines
-----Original Message-----
From: Gavin Hu (Arm Technology China)
Sent: Thursday, March 7, 2019 6:45 PM
To: Ilya Maximets <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 4 lines
-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 5:48 PM
To: Gavin Hu (Arm Technology China) <[email protected]>;Quoted 6 lines
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 10 lines
Quoted 9 lines · Gavin Hu (Arm Technology China)
Quoted 9 lines
-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 4:52 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; HonnappaNagarahalli
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
<[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 1 line · gavin hu
In weak memory models, like arm64, reading the {prod,cons}.tail may
get
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
reordered after reading or writing the ring slots, which corrupts the
ring
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem
is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
most
Quoted 1 line
likely caused by missing the acquire semantics when reading cons.tail
(in
Quoted 18 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
SP enqueue) or prod.tail (in SC dequeue) which makes it possible to
read
Quoted 16 lines
Quoted 8 lines
a
Quoted 3 lines
stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already provides therequired
Quoted 2 lines
ordering. This patch is to prevent reading and writing the ring slots get reordered before reading {prod,cons}.tail for SP (and SC) case.
Quoted 6 lines
Read barrier
rte_smp_rmb()is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e.rte_smp_mb().ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while
WMB(ST
Quoted 2 lines
Option) orders ST/ST.
Quoted 1 line
For more details, please refer to: Table B2-1 Encoding of the DMB and
DSB
Quoted 10 lines
<option> parameter in
reference-manual-armv8-for-armv8-a-architecture-profile
I see. But you have to change the
rte_smp_rmb()function definition inlib/librte_eal/common/include/generic/rte_atomic.hand assure that all other architectures follows same rules.
Otherwise, this change is logically wrong, because read barrier in current definition could not be used to order Load with Store.
Quoted 8 lines
Quoted 3 lines
Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm, read barrier(load/store and load/load) is enough.
Hi Ilya,
I would expand the rmb definition to cover load/store, in addition to load/load. For X86, as a strong memory order model, rmb is actually equivalent to mb,
That's not exactly the case, on x86 we have:smp_rmb == compiler_barriersmp_mb is a proper memory barrier.
Konstantin
Quoted 11 lines
as implemented as a compiler barrier:
rte_compiler_barrier,
arm32 is also this case.
For PPC, both 32 and 64-bit, rmb=wmb=mb, lwsync/sync orders load/store, load/load, store/load, store/store, looking at the table on this page:
https://www.ibm.com/developerworks/systems/articles/powerpc.htmlIn summary, we are safe to expand this definition for all the architectures DPDK support? Any comments are welcome!
BR. Gavin
Hi Konstantin,
Quoted 138 lines
-----Original Message-----
From: Ananyev, Konstantin <[email protected]>
Sent: Friday, March 8, 2019 8:13 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; Ilya Maximets <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]
Subject: RE: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 5 lines
-----Original Message-----
From: Gavin Hu (Arm Technology China) [mailto:[email protected]]
Sent: Friday, March 8, 2019 4:23 AM
To: Ilya Maximets <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected];[email protected]; [email protected]; Honnappa
Quoted 1 line
Nagarahalli <[email protected]>;
[email protected]; Richardson, Bruce
<[email protected]>; Ananyev,Quoted 1 line
Konstantin <[email protected]>;
Quoted 1 line
Subject: RE: [PATCH v2] ring: enforce reading the tails before ring
operations
Quoted 9 lines
Quoted 9 lines
-----Original Message-----
From: Gavin Hu (Arm Technology China)
Sent: Thursday, March 7, 2019 6:45 PM
To: Ilya Maximets <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; HonnappaNagarahalli
Quoted 4 lines
Quoted 4 lines
<[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH v2] ring: enforce reading the tails before ringoperations
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 5:48 PM
To: Gavin Hu (Arm Technology China) <[email protected]>;Quoted 2 lines
Cc: nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa
Nagarahalli
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
<[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 2 lines · Gavin Hu (Arm Technology China)
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
-----Original Message-----
From: Ilya Maximets <[email protected]>
Sent: Thursday, March 7, 2019 4:52 PM
To: Gavin Hu (Arm Technology China) <[email protected]>; [email protected]
Cc: nd <[email protected]>; [email protected];Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
[email protected]; [email protected]; Honnappa
Nagarahalli
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
<[email protected]>; [email protected]
Subject: Re: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 1 line · gavin hu
In weak memory models, like arm64, reading the {prod,cons}.tail
may
Quoted 2 lines
Quoted 2 lines
get
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
reordered after reading or writing the ring slots, which corrupts
the
Quoted 4 lines
Quoted 4 lines
ring
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The
problem
Quoted 3 lines
Quoted 3 lines
is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
most
Quoted 1 line
likely caused by missing the acquire semantics when reading
cons.tail
Quoted 2 lines
Quoted 2 lines
(in
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
SP enqueue) or prod.tail (in SC dequeue) which makes it possible
to
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
read
Quoted 4 lines
Quoted 4 lines
a
Quoted 3 lines
stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already providesthe
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
required
Quoted 1 line
ordering. This patch is to prevent reading and writing the ring
slots get
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
reordered before reading {prod,cons}.tail for SP (and SC) case.
Read barrier
rte_smp_rmb()is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e.rte_smp_mb().Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while
WMB(ST
Quoted 2 lines
Option) orders ST/ST.
Quoted 1 line
For more details, please refer to: Table B2-1 Encoding of the DMB
and
Quoted 6 lines
Quoted 6 lines
DSB
Quoted 5 lines
<option> parameter in
reference-manual-armv8-for-armv8-a-architecture-profile
I see. But you have to change the
rte_smp_rmb()function definitionin
Quoted 1 line
Quoted 1 line
Quoted 1 line
lib/librte_eal/common/include/generic/rte_atomic.hand assure thatall
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
other architectures follows same rules.
Otherwise, this change is logically wrong, because read barrier incurrent
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
definition could not be used to order Load with Store.
Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm,
read
Quoted 5 lines
Quoted 1 line
barrier(load/store and load/load) is enough.
Hi Ilya,
I would expand the rmb definition to cover load/store, in addition to
load/load.
Quoted 8 lines
Quoted 1 line
For X86, as a strong memory order model, rmb is actually equivalent to
mb,
That's not exactly the case, on x86 we have:
smp_rmb==compiler_barriersmp_mbis a proper memory barrier.Konstantin
Sorry I did not make it clear.
Anyway, on x86, smp_rmb, as a compiler barrier, applies to load/store, not only load/load. This is the case also for arm, arm64, ppc32, ppc64.
I will submit a patch to expand the definition of this API.
Quoted 14 lines
Quoted 4 lines
as implemented as a compiler barrier:
rte_compiler_barrier, arm32 is also this case.
For PPC, both 32 and 64-bit, rmb=wmb=mb, lwsync/sync ordersload/store, load/load, store/load, store/store, looking at the table on this
Quoted 4 lines
page:
https://www.ibm.com/developerworks/systems/articles/powerpc.htmlIn summary, we are safe to expand this definition for all the
architectures DPDK support?
Quoted 4 lines
Any comments are welcome!
BR. Gavin
Quoted 69 lines
Quoted 69 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
In weak memory models, like arm64, reading the {prod,cons}.tail
may
Quoted 2 lines
Quoted 2 lines
get
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
reordered after reading or writing the ring slots, which corrupts
the
Quoted 4 lines
Quoted 4 lines
ring
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The
problem
Quoted 3 lines
Quoted 3 lines
is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
most
Quoted 1 line
likely caused by missing the acquire semantics when reading
cons.tail
Quoted 2 lines
Quoted 2 lines
(in
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
SP enqueue) or prod.tail (in SC dequeue) which makes it possible
to
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
read
Quoted 4 lines
Quoted 4 lines
a
Quoted 3 lines
stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already providesthe
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
required
Quoted 1 line
ordering. This patch is to prevent reading and writing the ring
slots get
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
reordered before reading {prod,cons}.tail for SP (and SC) case.
Read barrier
rte_smp_rmb()is OK to prevent reading the ring get reordered
before reading the tail. However, to prevent *writing* the ring get reordered
*before reading* the tail you need a full memory barrier, i.e.rte_smp_mb().Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
ISHLD(rte_smp_rmb is DMB(ishld) orders LD/LD and LD/ST, while
WMB(ST
Quoted 2 lines
Option) orders ST/ST.
Quoted 1 line
For more details, please refer to: Table B2-1 Encoding of the DMB
and
Quoted 6 lines
Quoted 6 lines
DSB
Quoted 5 lines
<option> parameter in
reference-manual-armv8-for-armv8-a-architecture-profile
I see. But you have to change the
rte_smp_rmb()function definitionin
Quoted 1 line
Quoted 1 line
Quoted 1 line
lib/librte_eal/common/include/generic/rte_atomic.hand assure thatall
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
other architectures follows same rules.
Otherwise, this change is logically wrong, because read barrier incurrent
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
definition could not be used to order Load with Store.
Good points, let me re-think how to handle for other architectures. Full MB is required for other architectures(x86? Ppc?), but for arm,
read
Quoted 5 lines
Quoted 1 line
barrier(load/store and load/load) is enough.
Hi Ilya,
I would expand the rmb definition to cover load/store, in addition to
load/load.
Quoted 1 line
For X86, as a strong memory order model, rmb is actually equivalent to
mb,
Quoted 8 lines
Quoted 5 lines
That's not exactly the case, on x86 we have:
smp_rmb==compiler_barriersmp_mbis a proper memory barrier.Konstantin
Sorry I did not make it clear.
Anyway, on x86,smp_rmb, as a compiler barrier, applies to load/store, not only load/load.
Yes, that's true, but I think that's happened by coincidence, not intentionally.
Quoted 2 lines
This is the case also for arm, arm64, ppc32, ppc64. I will submit a patch to expand the definition of this API.
I understand your intention, but that does mean we would also need to change not only rte_smp_rmb() but rte_rmb() too (to keep things consistent)? That sounds worring.
Might be better to keep smp_rmb() definition as it is, and introduce new function that fits your purposes (smp_rwmb or smp_load_store_barrier)?
Konstantin
Quoted 15 lines
Quoted 15 lines
Quoted 5 lines
as implemented as a compiler barrier:
rte_compiler_barrier, arm32 is also this case.
For PPC, both 32 and 64-bit, rmb=wmb=mb, lwsync/sync ordersload/store, load/load, store/load, store/store, looking at the table on this
Quoted 4 lines
page:
https://www.ibm.com/developerworks/systems/articles/powerpc.htmlIn summary, we are safe to expand this definition for all the
architectures DPDK support?
Quoted 4 lines
Any comments are welcome!
BR. Gavin
08/03/2019 16:50, Ananyev, Konstantin:
Quoted 6 lines
08/03/2019 16:05, Gavin Hu (Arm Technology China):
Quoted 1 line
Anyway, on x86,
smp_rmb, as a compiler barrier, applies to load/store, not only load/load.Yes, that's true, but I think that's happened by coincidence, not intentionally.
Quoted 8 lines
Quoted 2 lines
This is the case also for arm, arm64, ppc32, ppc64. I will submit a patch to expand the definition of this API.
I understand your intention, but that does mean we would also need to change not only
rte_smp_rmb()butrte_rmb()too (to keep things consistent)? That sounds worring.
Might be better to keepsmp_rmb()definition as it is, and introduce new function that fits your purposes (smp_rwmborsmp_load_store_barrier)?
How is it managed in other projects?
Quoted 16 lines
08/03/2019 16:50, Ananyev, Konstantin:
Quoted 2 lines
08/03/2019 16:05, Gavin Hu (Arm Technology China):
Quoted 1 line
Anyway, on x86,
smp_rmb, as a compiler barrier, applies to load/store, notonly load/load.
Quoted 12 lines
Yes, that's true, but I think that's happened by coincidence, not intentionally.
Quoted 2 lines
This is the case also for arm, arm64, ppc32, ppc64. I will submit a patch to expand the definition of this API.
I understand your intention, but that does mean we would also need to change not only
rte_smp_rmb()butrte_rmb()too (to keep things consistent)? That sounds worring.
Might be better to keepsmp_rmb()definition as it is, and introduce new function that fits your purposes (smp_rwmborsmp_load_store_barrier)?
Looking at rte_rmb, rte_io_rmb, rte_cio_rmb implementations for Arm, they all provide load/store barrier as well. If other architectures also provide load/store barrier with rte_xxx_rmb, then we could extend the meaning of the existing APIs.
Even if a new API is provided, we need to do provide the same APIs for IO and CIO variants.
Quoted 2 lines
How is it managed in other projects?
In my experience, I usually have been changing the algorithms to use C11 memory model. So, I have not come across this issue yet. Others can comment.
Quoted 1 line
Quoted 31 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, March 9, 2019 7:48 AM
To: [email protected]; Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Cc: Ilya Maximets <[email protected]>; [email protected]; nd <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v2] ring: enforce reading the tails before ring operationsQuoted 3 lines
08/03/2019 16:50, Ananyev, Konstantin:
Quoted 2 lines
08/03/2019 16:05, Gavin Hu (Arm Technology China):
Quoted 1 line
Anyway, on x86,
smp_rmb, as a compiler barrier, applies toload/store, not
Quoted 9 lines
only load/load.
Quoted 8 lines
Yes, that's true, but I think that's happened by coincidence, not intentionally.
Quoted 2 lines
This is the case also for arm, arm64, ppc32, ppc64. I will submit a patch to expand the definition of this API.
I understand your intention, but that does mean we would also need
to
Quoted 1 line
Quoted 1 line
change not only
rte_smp_rmb()butrte_rmb()too (to keep thingsconsistent)?
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
That sounds worring.
Might be better to keepsmp_rmb()definition as it is, and introduce new function that fits your purposes (smp_rwmbor
smp_load_store_barrier)?
Looking atrte_rmb,rte_io_rmb,rte_cio_rmbimplementations for Arm, they all provide load/store barrier as well. If other architectures also provide load/store barrier withrte_xxx_rmb, then we could extend the meaning of the existing APIs.
Further looking at rte_rmb, rte_io_rmb, rte_cio_rmb implementations for PPC64 and x86, They also provide load/store barrier. It is safe to extend the meaning of the existing rte_XXX_rmb API.
Quoted 3 lines
Even if a new API is provided, we need to do provide the same APIs for IO and CIO variants.
Since rte_XXX_rmbs API for all architectures already provide the desired load/store ordering, a new API is redundant and not needed.
Quoted 7 lines
Quoted 2 lines
How is it managed in other projects?
In my experience, I usually have been changing the algorithms to use C11 memory model. So, I have not come across this issue yet. Others can comment.
Quoted 1 line
v3 cover ring: enforce reading the tail before reading ring slots
In reply to gavin huChange log:
-V1: First version of the patch.
-V2: Update the commit message to describe the ordering of detailed memory
accesses.
-V3: As suggested by Konstantin Ananyev, remove the rte_smp_rmb for enqueue
case, as there is a control dependency in place, making rmb unnecessary. For dequeue case, it is still required as the control dependency does not order loads with loads.
gavin hu (1):
ring: enforce reading the tail before reading ring slots
lib/librte_ring/rte_ring_generic.h | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
Signature · 1 line
2.7.4
PATCH v3 1/1 ring: enforce reading the tail before reading ring slots
From: gavin hu <[email protected]>
In weak memory models, like arm64, reading the prod.tail may get reordered after reading the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots.
For MP (and MC) case, rte_atomic32_cmpset() already provides the required ordering. For SP case, the control depependency between if-statement(which depends on the read of r->cons.tail) and the later stores to the ring slots make RMB unnecessary. About the control dependency, read more at:
https://www.cl.cam.ac.uk/~pes20/ppc-supplemental/test7.pdf
This patch is adding the required read barrier to prevent reading the ring slots get reordered before reading prod.tail for SC case.
--- lib/librte_ring/rte_ring_generic.h | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
lib/librte_ring/rte_ring_generic.h +6 -3
@@ -158,11 +158,14 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,return 0 ; *new_head = *old_head + n; -if (is_sc) - r->cons.head = *new_head, success =1 ; -else +if (is_sc) { + r->cons.head = *new_head; +rte_smp_rmb (); + success =1 ; + }else { success =rte_atomic32_cmpset (&r->cons.head, *old_head, *new_head); + } }while (unlikely (success ==0 ));return n; }
Signature · 1 line
2.7.4
Re: [PATCH v3 1/1] ring: enforce reading the tail before reading ring slots
Quoted 39 lines
-----Original Message-----
From: Gavin Hu [mailto:[email protected]]
Sent: Tuesday, March 12, 2019 10:29 PM
To: [email protected]
Cc: [email protected]; gavin hu <[email protected]>; [email protected]; [email protected]; [email protected]; Hemant Agrawal <[email protected]>; Nipun Gupta <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 1/1] ring: enforce reading the tail before reading ring slotsFrom: gavin hu <[email protected]>
In weak memory models, like arm64, reading the prod.tail may get reordered after reading the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already provides the required ordering. For SP case, the control depependency between if- statement(which
depends on the read ofr->cons.tail) and the later stores to the ring slots make RMB unnecessary. About the control dependency, read more at:
https://emea01.safelinks.protection.outlook.com/?url=https:%2F%2Fwww.c
l.cam.ac.uk%2F~pes20%2Fppc- supplemental%2Ftest7.pdf&data=02%7C01%7Cnipun.gupta%40nxp.co m%7Cc3df030ec49449bbaf0508d6a70c0b9f%7C686ea1d3bc2b4c6fa92cd99c5c 301635%7C0%7C0%7C636880067514602937&sdata=ogPdd%2F4UGYWE8 nAwY1lkXPCB9MpUIFY0VOQr2N1lwe4%3D&reserved=0This patch is adding the required read barrier to prevent reading the ring slots get reordered before reading prod.tail for SC case.
Quoted 6 lines
Fixes c9fb3c62896f ("ring: move code in a new header file")Signed-off-by gavin hu <[email protected]>Reviewed-by Ola Liljedahl <[email protected]>Tested-by Nipun Gupta <[email protected]>
Also, I have revalidated this updated patch.
Re: [PATCH v3 1/1] ring: enforce reading the tail before reading ring slots
In reply to Gavin HuQuoted 54 lines
-----Original Message-----
From: Gavin Hu [mailto:[email protected]]
Sent: Tuesday, March 12, 2019 4:59 PM
To: [email protected]
Cc: [email protected]; gavin hu <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 1/1] ring: enforce reading the tail before reading ring slotsFrom: gavin hu <[email protected]>
In weak memory models, like arm64, reading the prod.tail may get reordered after reading the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already provides the required ordering. For SP case, the control depependency between if-statement(which depends on the read ofr->cons.tail) and the later stores to the ring slots make RMB unnecessary. About the control dependency, read more at:
https://www.cl.cam.ac.uk/~pes20/ppc-supplemental/test7.pdfThis patch is adding the required read barrier to prevent reading the ring slots get reordered before reading prod.tail for SC case.
Fixes c9fb3c62896f ("ring: move code in a new header file")Signed-off-by gavin hu <[email protected]>Reviewed-by Ola Liljedahl <[email protected]>Tested-by Nipun Gupta <[email protected]>--- lib/librte_ring/rte_ring_generic.h | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)lib/librte_ring/rte_ring_generic.h +2
-3@@ -158,11 +158,14 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,return 0 ; *new_head = *old_head + n; -if (is_sc) - r->cons.head = *new_head, success =1 ; -else +if (is_sc) { + r->cons.head = *new_head;
lib/librte_ring/rte_ring_generic.h __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
Quoted 10 lines
lib/librte_ring/rte_ring_generic.h +4
-1+rte_smp_rmb (); + success =1 ; + }else { success =rte_atomic32_cmpset (&r->cons.head, *old_head, *new_head); + } }while (unlikely (success ==0 ));return n; } --
Quoted 1 line
2.7.4
Thomas Monjalon
AppliedRe: [PATCH v3 1/1] ring: enforce reading the tail before reading ring slots
Quoted 18 lines
Quoted 18 lines
In weak memory models, like arm64, reading the prod.tail may get reordered after reading the ring slots, which corrupts the ring and stale data is observed.
This issue was reported by NXP on 8-A72 DPAA2 board. The problem is most likely caused by missing the acquire semantics when reading prod.tail (in SC dequeue) which makes it possible to read a stale value from the ring slots.
For MP (and MC) case,
rte_atomic32_cmpset()already provides the required ordering. For SP case, the control depependency between if-statement(which depends on the read ofr->cons.tail) and the later stores to the ring slots make RMB unnecessary. About the control dependency, read more at:
https://www.cl.cam.ac.uk/~pes20/ppc-supplemental/test7.pdfThis patch is adding the required read barrier to prevent reading the ring slots get reordered before reading prod.tail for SC case.
Quoted 8 lines
Quoted 6 lines
Fixes c9fb3c62896f ("ring: move code in a new header file")Signed-off-by gavin hu <[email protected]>Reviewed-by Ola Liljedahl <[email protected]>Tested-by Nipun Gupta <[email protected]>
Applied, thanks