Skip to contentSkip to conversation

Add 128-bit compare and set

PATCH Superseded Gage Eads 5 participants 36 messages Jan 28, 2019 – Apr 4, 2019

lore
Share & tools
lore

Gage Eads

author
#0 lore raw

cover Add 128-bit compare and set

This patch addresses x86-64 only; other architectures can/will be supported in the future. The __atomic intrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the __atomic_compare_exchange_16 (which itself is based on the C++11 memory model) to best support weak consistency architectures.

This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not the entire non-blocking stack patchset.

[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[1] http://mails.dpdk.org/archives/dev/2019-January/123653.html

Gage Eads (1):
eal: add 128-bit cmpset (x86-64 only)

 .../common/include/arch/x86/rte_atomic_64.h        | 31 +++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 65 ++++++++++++++++++++++
 2 files changed, 96 insertions(+)
Signature · 1 line
2.13.6
author
#1 lore raw

PATCH 1/1 eal: add 128-bit cmpset (x86-64 only)

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 31 +++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 65 ++++++++++++++++++++++
 2 files changed, 96 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +31 -0
index fd2ec9c53..b7b90b83e 100644
@@ -34,6 +34,7 @@
 /*
  * Inspired from FreeBSD src/sys/amd64/include/atomic.h
  * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
  * All rights reserved.
  */
 
@@ -46,6 +47,7 @@
 
 #include <stdint.h>
 #include <rte_common.h>
+#include <rte_compat.h>
 #include <rte_atomic.h>
 
 /*------------------------- 64 bit atomic operations -------------------------*/
@@ -208,4 +210,33 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v)
 }
 #endif
 
+static inline int __rte_experimental
+rte_atomic128_cmpset(volatile rte_int128_t *dst,
+		     rte_int128_t *exp, rte_int128_t *src,
+		     unsigned int weak,
+		     enum rte_atomic_memmodel_t success,
+		     enum rte_atomic_memmodel_t failure)
+{
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=A" (exp->val[0]),
+			[res] "=r" (res)
+		      : "c" (src->val[1]),
+			"b" (src->val[0]),
+			"m" (dst->val[0]),
+			"d" (exp->val[1]),
+			"a" (exp->val[0])
+		      : "memory");
+
+	return res;
+}
+
 #endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +65 -0

75 lines not on this page · raw message

Signature · 1 line
2.13.6
#2 lore raw

Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 35 lines · Gage Eads

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 31 +++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 65
++++++++++++++++++++++

 2 files changed, 96 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1 -0
index fd2ec9c53..b7b90b83e 100644
@@ -34,6 +34,7 @@
  /*
   * Inspired from FreeBSD src/sys/amd64/include/atomic.h
   * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
   * All rights reserved.
   */

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +0 -0
@@ -46,6 +47,7 @@


 #include <stdint.h>
 #include <rte_common.h>
+#include <rte_compat.h>
 #include <rte_atomic.h>

 /*------------------------- 64 bit atomic operations ------------------------

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +0 -1
-*/
@@ -208,4 +210,33 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v)
  }
  #endif
Quoted 2 lines


+static inline int __rte_experimental

__rte_always_inline?

Quoted 1 line

+rte_atomic128_cmpset(volatile rte_int128_t *dst,

No need to declare the location volatile. Volatile doesn't do what you think it does.
https://youtu.be/lkgszkPnV8g?t=1027

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h

Quoted 1 line
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1 -0
+		     rte_int128_t *exp,

I would declare 'exp' const as well and document that 'exp' is not updated (with the old value) for a failure. The reason being that ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of the critical section) using e.g. non-atomic 64-bit reads so there isn't actually any requirement for an atomic 128-bit read of the location.

Quoted 1 line

rte_int128_t *src,

const rte_int128_t *src?

But why are we not passing 'exp' and 'src' by value? That works great, even with structs. Passing by value simplifies the compiler's life, especially if the call is inlined. Ask a compiler developer.

Quoted 45 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +26 -0
+		     unsigned int weak,
+		     enum rte_atomic_memmodel_t success,
+		     enum rte_atomic_memmodel_t failure)
+{
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=A" (exp->val[0]),
+			[res] "=r" (res)
+		      : "c" (src->val[1]),
+			"b" (src->val[0]),
+			"m" (dst->val[0]),
+			"d" (exp->val[1]),
+			"a" (exp->val[0])
+		      : "memory");
+
+	return res;
+}
+

 #endif /* _RTE_ATOMIC_X86_64_H_ */

lib/librte_eal/common/include/generic/rte_atomic.h +0 -0
index b99ba4688..8d612d566 100644
@@ -14,6 +14,7 @@


 #include <stdint.h>
 #include <rte_common.h>
+#include <rte_compat.h>

 #ifdef __DOXYGEN__

lib/librte_eal/common/include/generic/rte_atomic.h +0 -0
@@ -1082,4 +1083,68 @@ static inline void rte_atomic64_clear(rte_atomic64_t
 *v)
  }
  #endif

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093

Quoted 9 lines
lib/librte_eal/common/include/generic/rte_atomic.h +8 -1
+/*------------------------ 128 bit atomic operations ------------------------
-*/
+
+/**
+ * 128-bit integer structure.
+ */
+typedef struct {
+	uint64_t val[2];
+} __rte_aligned(16) rte_int128_t;

So we can't use __int128?

Quoted 58 lines
lib/librte_eal/common/include/generic/rte_atomic.h +37 -0
+
+/**
+ * Memory consistency models used in atomic operations. These control the
+ * behavior of the operation with respect to memory barriers and
+ * thread synchronization.
+ *
+ * These directly match those in the C++11 standard; for details on their
+ * behavior, refer to the standard.
+ */
+enum rte_atomic_memmodel_t {
+	RTE_ATOMIC_RELAXED,
+	RTE_ATOMIC_CONSUME,
+	RTE_ATOMIC_ACQUIRE,
+	RTE_ATOMIC_RELEASE,
+	RTE_ATOMIC_ACQ_REL,
+	RTE_ATOMIC_SEQ_CST,
+};
+
+/* Only implemented on x86-64 currently. The ifdef prevents compilation from
+ * failing for architectures without a definition of this function.
+ */
+#if defined(RTE_ARCH_X86_64)
+/**
+ * An atomic compare and set function used by the mutex functions.
+ * (atomic) equivalent to:
+ *   if (*dst == exp)
+ *     *dst = src (all 128-bit words)
+ *
+ * @param dst
+ *   The destination into which the value will be written.
+ * @param exp
+ *   Pointer to the expected value. If the operation fails, this memory is
+ *   updated with the actual value.
+ * @param src
+ *   Pointer to the new value.
+ * @param weak
+ *   A value of true allows the comparison to spuriously fail.

Implementations
+ *   may ignore this argument and only implement the strong variant.
+ * @param success
+ *   If successful, the operation's memory behavior conforms to this (or a
+ *   stronger) model.
+ * @param failure
+ *   If unsuccessful, the operation's memory behavior conforms to this (or a
+ *   stronger) model. This argument cannot be RTE_ATOMIC_RELEASE,
+ *   RTE_ATOMIC_ACQ_REL, or a stronger model than success.
+ * @return
+ *   Non-zero on success; 0 on failure.
+ */

lib/librte_eal/common/include/generic/rte_atomic.h +8 -0
+static inline int __rte_experimental
+rte_atomic128_cmpset(volatile rte_int128_t *dst,
+		     rte_int128_t *exp, rte_int128_t *src,
+		     unsigned int weak,
+		     enum rte_atomic_memmodel_t success,
+		     enum rte_atomic_memmodel_t failure);
+#endif
+

 #endif /* _RTE_ATOMIC_H_ */

Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
author
#3 lore raw

Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 52 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Monday, January 28, 2019 5:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 36 lines · Gage Eads

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 31 +++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 65
++++++++++++++++++++++

 2 files changed, 96 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2 -1
index fd2ec9c53..b7b90b83e 100644
@@ -34,6 +34,7 @@
  /*
   * Inspired from FreeBSD src/sys/amd64/include/atomic.h
   * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
   * All rights reserved.
   */
 
@@ -46,6 +47,7 @@
 
  #include <stdint.h>
  #include <rte_common.h>
+#include <rte_compat.h>
  #include <rte_atomic.h>
 
  /*------------------------- 64 bit atomic operations
------------------------ -*/ @@ -208,4 +210,33 @@ static inline void

rte_atomic64_clear(rte_atomic64_t *v)
 }
 #endif

Quoted 8 lines
Quoted 1 line

+static inline int __rte_experimental

__rte_always_inline?

Quoted 1 line

+rte_atomic128_cmpset(volatile rte_int128_t *dst,

No need to declare the location volatile. Volatile doesn't do what you think it does.
https://youtu.be/lkgszkPnV8g?t=1027

I made this volatile to match the existing rte_atomicN_cmpset definitions, which presumably have a good reason for using the keyword. Maintainers, any input here?

Quoted 10 lines

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h

Quoted 2 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1 -0
+		     rte_int128_t *exp,

I would declare 'exp' const as well and document that 'exp' is not updated (with the old value) for a failure. The reason being that ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of the critical section) using e.g. non-atomic 64-bit reads so there isn't actually any requirement for an atomic 128-bit read of the location.

Will change in v2.

Quoted 2 lines
Quoted 1 line

rte_int128_t *src,

const rte_int128_t *src?

Sure, I don't see any harm in using const.

Quoted 4 lines

But why are we not passing 'exp' and 'src' by value? That works great, even with structs. Passing by value simplifies the compiler's life, especially if the call is inlined. Ask a compiler developer.

I ran objdump on the nb_stack code with both approaches, and pass-by-reference resulted in fewer overall x86_64 assembly ops.
PBV: 100 ops for push, 97 ops for pop
PBR: 92 ops for push, 84 ops for pop

(Using the in-progress v5 nb_stack code)

Another factor -- though much less compelling -- is that with pass-by-reference, the user can create a 16B structure and cast it to rte_int128_t when they call rte_atomic128_cmpset, whereas with pass-by-value they need to put that struct in a union with rte_int128_t.

Quoted 46 lines
Quoted 46 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +25 -0
+		     unsigned int weak,
+		     enum rte_atomic_memmodel_t success,
+		     enum rte_atomic_memmodel_t failure) {
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=A" (exp->val[0]),
+			[res] "=r" (res)
+		      : "c" (src->val[1]),
+			"b" (src->val[0]),
+			"m" (dst->val[0]),
+			"d" (exp->val[1]),
+			"a" (exp->val[0])
+		      : "memory");
+
+	return res;
+}
+

 #endif /* _RTE_ATOMIC_X86_64_H_ */

lib/librte_eal/common/include/generic/rte_atomic.h +1 -0
index b99ba4688..8d612d566 100644
@@ -14,6 +14,7 @@
 
  #include <stdint.h>
  #include <rte_common.h>
+#include <rte_compat.h>
 
  #ifdef __DOXYGEN__
 
@@ -1082,4 +1083,68 @@ static inline void

rte_atomic64_clear(rte_atomic64_t
*v)
 }
 #endif

Quoted 12 lines

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093

Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9 -1
+/*------------------------ 128 bit atomic operations
+------------------------
-*/
+
+/**
+ * 128-bit integer structure.
+ */
+typedef struct {
+	uint64_t val[2];
+} __rte_aligned(16) rte_int128_t;

So we can't use __int128?

I'll put it in a union with val[2], in case any implementations want to use it.

Thanks,
Gage

[snip]

#4 lore raw

Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 98 lines · Gage
Quoted 57 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Monday, January 28, 2019 5:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 37 lines · Gage Eads

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 31 +++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 65
++++++++++++++++++++++

 2 files changed, 96 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2 -1
index fd2ec9c53..b7b90b83e 100644
@@ -34,6 +34,7 @@
  /*
   * Inspired from FreeBSD src/sys/amd64/include/atomic.h
   * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
   * All rights reserved.
   */
 
@@ -46,6 +47,7 @@
 
  #include <stdint.h>
  #include <rte_common.h>
+#include <rte_compat.h>
  #include <rte_atomic.h>
 
  /*------------------------- 64 bit atomic operations
------------------------ -*/ @@ -208,4 +210,33 @@ static inline void

rte_atomic64_clear(rte_atomic64_t *v)
 }
 #endif

Quoted 1 line

+static inline int __rte_experimental

__rte_always_inline?

Quoted 1 line
Quoted 6 lines
Quoted 1 line

+rte_atomic128_cmpset(volatile rte_int128_t *dst,

No need to declare the location volatile. Volatile doesn't do what you think it
does.
https://youtu.be/lkgszkPnV8g?t=1027

I made this volatile to match the existing rte_atomicN_cmpset definitions, which presumably have a good reason for using the keyword. Maintainers, any input here?

Quoted 16 lines

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h

Quoted 4 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1 -0
+		     rte_int128_t *exp,

I would declare 'exp' const as well and document that 'exp' is not updated (with
the old value) for a failure. The reason being that ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad
for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of
the critical section) using e.g. non-atomic 64-bit reads so there isn't actually any
requirement for an atomic 128-bit read of the location.

Will change in v2.

Quoted 4 lines
Quoted 3 lines

 rte_int128_t *src,

const rte_int128_t *src?

Sure, I don't see any harm in using const.

Quoted 7 lines

But why are we not passing 'exp' and 'src' by value? That works great, even with
structs. Passing by value simplifies the compiler's life, especially if the call is
inlined. Ask a compiler developer.

Quoted 4 lines

I ran objdump on the nb_stack code with both approaches, and pass-by-reference resulted in fewer overall x86_64 assembly ops.
PBV: 100 ops for push, 97 ops for pop
PBR: 92 ops for push, 84 ops for pop

OK I have never checked x86_64 code generation... I have good experiences with ARM/AArch64, everything seems to be done using registers. I am surprised there is a difference.

Did a quick check with lfring, passing 'src' (third param) by reference and by value. No difference in code generation on x86_64.

But if you insist let's go with PBR.

Quoted 7 lines

(Using the in-progress v5 nb_stack code)

Another factor -- though much less compelling -- is that with pass-by- reference, the user can create a 16B structure and cast it to rte_int128_t when they call rte_atomic128_cmpset, whereas with pass-by-value they need to put that struct in a union with rte_int128_t.

Which is what I always do nowadays... Trying to use as few casts as possible and lie to the compiler as seldom as possible. But I can see the freedom provided by taking a pointer to something and cast it it rte_int128_t ptr in the call to rte_atomic128_cmpset().

Would prefer a name that is more similar to __atomic_compare_exchange(). E.g. rte_atomic128_compare_exchange() (or perhaps just rte_atomic128_cmpxchg)? All the rte_atomicXX_cmpset() functions do not take any memory order parameters. From an Arm perspective, we are not happy with that.

Quoted 61 lines
Quoted 49 lines
Quoted 49 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +25 -0
+		     unsigned int weak,
+		     enum rte_atomic_memmodel_t success,
+		     enum rte_atomic_memmodel_t failure) {
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=A" (exp->val[0]),
+			[res] "=r" (res)
+		      : "c" (src->val[1]),
+			"b" (src->val[0]),
+			"m" (dst->val[0]),
+			"d" (exp->val[1]),
+			"a" (exp->val[0])
+		      : "memory");
+
+	return res;
+}
+

 #endif /* _RTE_ATOMIC_X86_64_H_ */

lib/librte_eal/common/include/generic/rte_atomic.h +1 -0
index b99ba4688..8d612d566 100644
@@ -14,6 +14,7 @@
 
  #include <stdint.h>
  #include <rte_common.h>
+#include <rte_compat.h>
 
  #ifdef __DOXYGEN__
 
@@ -1082,4 +1083,68 @@ static inline void

rte_atomic64_clear(rte_atomic64_t
*v)
 }
 #endif

Quoted 12 lines

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093

Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9 -1
+/*------------------------ 128 bit atomic operations
+------------------------
-*/
+
+/**
+ * 128-bit integer structure.
+ */
+typedef struct {
+	uint64_t val[2];
+} __rte_aligned(16) rte_int128_t;

So we can't use __int128?

Quoted 2 lines

I'll put it in a union with val[2], in case any implementations want to use it.

Thinking on this one more time, since the inline asm functions (e.g. for x86_64 cmpxchg16b and for AArch64 LDXP/STXP) anyway will use 64-bit registers, it makes most sense to make rte_int128_t a struct of 2x64b. The question is whether to use an array like above or a struct with two elements (which I normally do internally). Can you compare code generation with the following definition? typedef struct {
        uint64_t lo, hi;
} __rte_aligned(16) rte_int128_t;

Quoted 5 lines

Thanks,
Gage

[snip]

Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
author
#5 lore raw

Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 135 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Friday, February 1, 2019 1:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; Gavin Hu (Arm Technology China) <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 92 lines · Gage
Quoted 56 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Monday, January 28, 2019 5:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 39 lines · Gage Eads

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 31
+++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 65
++++++++++++++++++++++

 2 files changed, 96 insertions(+)

diff --git a/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h b/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h index fd2ec9c53..b7b90b83e 100644

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2 -1
@@ -34,6 +34,7 @@
  /*
   * Inspired from FreeBSD src/sys/amd64/include/atomic.h
   * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
   * All rights reserved.
   */
 
@@ -46,6 +47,7 @@
 
  #include <stdint.h>
  #include <rte_common.h>
+#include <rte_compat.h>
  #include <rte_atomic.h>
 
  /*------------------------- 64 bit atomic operations
------------------------ -*/ @@ -208,4 +210,33 @@ static inline

void rte_atomic64_clear(rte_atomic64_t *v)
 }
 #endif

Quoted 9 lines
Quoted 1 line

+static inline int __rte_experimental

__rte_always_inline?

Quoted 2 lines

+rte_atomic128_cmpset(volatile rte_int128_t *dst,

No need to declare the location volatile. Volatile doesn't do what you think it does.
https://youtu.be/lkgszkPnV8g?t=1027

I made this volatile to match the existing rte_atomicN_cmpset definitions, which presumably have a good reason for using the keyword. Maintainers, any input here?

Quoted 13 lines

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h

Quoted 4 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1 -0
+		     rte_int128_t *exp,

I would declare 'exp' const as well and document that 'exp' is not updated (with the old value) for a failure. The reason being that ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of the critical section) using e.g. non-atomic 64-bit reads so there isn't actually any requirement for an atomic 128-bit read of the location.

Will change in v2.

Quoted 4 lines
Quoted 3 lines

 rte_int128_t *src,

const rte_int128_t *src?

Sure, I don't see any harm in using const.

Quoted 2 lines
Quoted 7 lines
Quoted 3 lines

But why are we not passing 'exp' and 'src' by value? That works great, even with structs. Passing by value simplifies the compiler's life, especially if the call is inlined. Ask a compiler developer.

I ran objdump on the nb_stack code with both approaches, and pass-by-reference resulted in fewer overall x86_64 assembly ops.
PBV: 100 ops for push, 97 ops for pop
PBR: 92 ops for push, 84 ops for pop

OK I have never checked x86_64 code generation... I have good experiences with ARM/AArch64, everything seems to be done using registers. I am surprised there is a difference.

Did a quick check with lfring, passing 'src' (third param) by reference and by value. No difference in code generation on x86_64.

But if you insist let's go with PBR.

Quoted 7 lines

(Using the in-progress v5 nb_stack code)

Another factor -- though much less compelling -- is that with pass-by- reference, the user can create a 16B structure and cast it to rte_int128_t when they call rte_atomic128_cmpset, whereas with pass-by-value they need to put that struct in a union with rte_int128_t.

Which is what I always do nowadays... Trying to use as few casts as possible and lie to the compiler as seldom as possible. But I can see the freedom provided by taking a pointer to something and cast it it rte_int128_t ptr in the call to rte_atomic128_cmpset().

Quoted 6 lines

Would prefer a name that is more similar to __atomic_compare_exchange(). E.g.
rte_atomic128_compare_exchange() (or perhaps just rte_atomic128_cmpxchg)? All the rte_atomicXX_cmpset() functions do not take any memory order parameters.
From an Arm perspective, we are not happy with that.

Since the function returns a boolean success value, isn't compare-and-set the appropriate term?

Quoted 64 lines
Quoted 62 lines
Quoted 50 lines
Quoted 50 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +25 -0
+		     unsigned int weak,
+		     enum rte_atomic_memmodel_t success,
+		     enum rte_atomic_memmodel_t failure) {
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=A" (exp->val[0]),
+			[res] "=r" (res)
+		      : "c" (src->val[1]),
+			"b" (src->val[0]),
+			"m" (dst->val[0]),
+			"d" (exp->val[1]),
+			"a" (exp->val[0])
+		      : "memory");
+
+	return res;
+}
+

 #endif /* _RTE_ATOMIC_X86_64_H_ */ diff --git a/lib/librte_eal/common/include/generic/rte_atomic.h b/lib/librte_eal/common/include/generic/rte_atomic.h index b99ba4688..8d612d566 100644

lib/librte_eal/common/include/generic/rte_atomic.h +1 -0
@@ -14,6 +14,7 @@
 
  #include <stdint.h>
  #include <rte_common.h>
+#include <rte_compat.h>
 
  #ifdef __DOXYGEN__
 
@@ -1082,4 +1083,68 @@ static inline void

rte_atomic64_clear(rte_atomic64_t
*v)
 }
 #endif

Quoted 12 lines

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093

Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9 -1
+/*------------------------ 128 bit atomic operations
+------------------------
-*/
+
+/**
+ * 128-bit integer structure.
+ */
+typedef struct {
+	uint64_t val[2];
+} __rte_aligned(16) rte_int128_t;

So we can't use __int128?

Quoted 2 lines

I'll put it in a union with val[2], in case any implementations want to use it.

Quoted 10 lines

Thinking on this one more time, since the inline asm functions (e.g. for x86_64 cmpxchg16b and for AArch64 LDXP/STXP) anyway will use 64-bit registers, it makes most sense to make rte_int128_t a struct of 2x64b. The question is whether to use an array like above or a struct with two elements (which I normally do internally). Can you compare code generation with the following definition?
typedef struct {
        uint64_t lo, hi;
} __rte_aligned(16) rte_int128_t;

Interestingly, that made no difference in the PBV code but added more instructions overall to PBR:
PBV: 100 insts for push, 97 insts for pop
PBR: 100 insts for push, 83 insts for pop

Quoted 8 lines
Quoted 5 lines

Thanks,
Gage

[snip]

--
Ola Liljedahl, Networking System Architect, Arm Phone +46706866373, Skype ola.liljedahl

#6 lore raw

Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 149 lines · Gage
Quoted 149 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Friday, February 1, 2019 1:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; Gavin Hu (Arm Technology China) <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 101 lines · Gage
Quoted 64 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Monday, January 28, 2019 5:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 40 lines · Gage Eads

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 31
+++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 65
++++++++++++++++++++++

 2 files changed, 96 insertions(+)

diff --git a/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h b/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h index fd2ec9c53..b7b90b83e 100644

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2 -1
@@ -34,6 +34,7 @@
  /*
   * Inspired from FreeBSD src/sys/amd64/include/atomic.h
   * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
   * All rights reserved.
   */
 
@@ -46,6 +47,7 @@
 
  #include <stdint.h>
  #include <rte_common.h>
+#include <rte_compat.h>
  #include <rte_atomic.h>
 
  /*------------------------- 64 bit atomic operations
------------------------ -*/ @@ -208,4 +210,33 @@ static inline

void rte_atomic64_clear(rte_atomic64_t *v)
 }
 #endif

Quoted 1 line

+static inline int __rte_experimental

__rte_always_inline?

Quoted 2 lines
Quoted 5 lines
Quoted 1 line

+rte_atomic128_cmpset(volatile rte_int128_t *dst,

No need to declare the location volatile. Volatile doesn't do what you think it does.
https://youtu.be/lkgszkPnV8g?t=1027

I made this volatile to match the existing rte_atomicN_cmpset definitions, which presumably have a good reason for using the keyword. Maintainers, any input here?

Quoted 5 lines
Quoted 5 lines
Quoted 10 lines

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h

Quoted 1 line
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1 -0
+		     rte_int128_t *exp,

I would declare 'exp' const as well and document that 'exp' is not updated (with the old value) for a failure. The reason being that ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of the critical section) using e.g. non-atomic 64-bit reads so there isn't actually any requirement for an atomic 128-bit read of the location.

Will change in v2.

Quoted 6 lines
Quoted 5 lines

 rte_int128_t *src,

const rte_int128_t *src?

Sure, I don't see any harm in using const.

Quoted 3 lines
Quoted 7 lines
Quoted 3 lines

But why are we not passing 'exp' and 'src' by value? That works great, even with structs. Passing by value simplifies the compiler's life, especially if the call is inlined. Ask a compiler developer.

I ran objdump on the nb_stack code with both approaches, and pass-by-reference resulted in fewer overall x86_64 assembly ops.
PBV: 100 ops for push, 97 ops for pop
PBR: 92 ops for push, 84 ops for pop

OK I have never checked x86_64 code generation... I have good experiences with ARM/AArch64, everything seems to be done using registers. I am surprised
there is a difference.

Did a quick check with lfring, passing 'src' (third param) by reference and by
value. No difference in code generation on x86_64.

But if you insist let's go with PBR.

Quoted 8 lines

(Using the in-progress v5 nb_stack code)

Another factor -- though much less compelling -- is that with pass-by- reference, the user can create a 16B structure and cast it to rte_int128_t when they call rte_atomic128_cmpset, whereas with pass-by-value they need to put that struct in a union with rte_int128_t.

Which is what I always do nowadays... Trying to use as few casts as possible and
lie to the compiler as seldom as possible. But I can see the freedom provided by
taking a pointer to something and cast it it rte_int128_t ptr in the call to rte_atomic128_cmpset().

Quoted 8 lines
Quoted 6 lines

Would prefer a name that is more similar to __atomic_compare_exchange(). E.g.
rte_atomic128_compare_exchange() (or perhaps just rte_atomic128_cmpxchg)? All the rte_atomicXX_cmpset() functions do not take any memory order parameters.
From an Arm perspective, we are not happy with that.

Since the function returns a boolean success value, isn't compare-and-set the appropriate term?

I was thinking of the memory ordering parameters that __atomic_xxx builtins have and we want (from an Arm perspective).

GCC __atomic_compare_exchange also returns a boolean.
https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html
bool __atomic_compare_exchange_n (type *ptr, type *expected, type desired, bool weak, int success_memorder, int failure_memorder)
bool __atomic_compare_exchange (type *ptr, type *expected, type *desired, bool weak, int success_memorder, int failure_memorder)

rte_atomic128_compare_exchange(rte_int128_t *dst, const rte_int128_t *exp, const rte_int182_t *src, bool weak, int mo_success, int mo_failure);

Quoted 80 lines
Quoted 69 lines
Quoted 67 lines
Quoted 55 lines
Quoted 55 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +25 -0
+		     unsigned int weak,
+		     enum rte_atomic_memmodel_t success,
+		     enum rte_atomic_memmodel_t failure) {
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=A" (exp->val[0]),
+			[res] "=r" (res)
+		      : "c" (src->val[1]),
+			"b" (src->val[0]),
+			"m" (dst->val[0]),
+			"d" (exp->val[1]),
+			"a" (exp->val[0])
+		      : "memory");
+
+	return res;
+}
+

 #endif /* _RTE_ATOMIC_X86_64_H_ */ diff --git a/lib/librte_eal/common/include/generic/rte_atomic.h b/lib/librte_eal/common/include/generic/rte_atomic.h index b99ba4688..8d612d566 100644

lib/librte_eal/common/include/generic/rte_atomic.h +1 -0
@@ -14,6 +14,7 @@
 
  #include <stdint.h>
  #include <rte_common.h>
+#include <rte_compat.h>
 
  #ifdef __DOXYGEN__
 
@@ -1082,4 +1083,68 @@ static inline void

rte_atomic64_clear(rte_atomic64_t
*v)
 }
 #endif

Quoted 12 lines

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093

Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9 -1
+/*------------------------ 128 bit atomic operations
+------------------------
-*/
+
+/**
+ * 128-bit integer structure.
+ */
+typedef struct {
+	uint64_t val[2];
+} __rte_aligned(16) rte_int128_t;

So we can't use __int128?

Quoted 2 lines

I'll put it in a union with val[2], in case any implementations want to use it.

Quoted 11 lines

Thinking on this one more time, since the inline asm functions (e.g. for x86_64
cmpxchg16b and for AArch64 LDXP/STXP) anyway will use 64-bit registers, it makes most sense to make rte_int128_t a struct of 2x64b. The question is whether to use an array like above or a struct with two elements (which I normally do internally). Can you compare code generation with the following definition?
typedef struct {
        uint64_t lo, hi;
} __rte_aligned(16) rte_int128_t;

Quoted 4 lines

Interestingly, that made no difference in the PBV code but added more instructions overall to PBR:
PBV: 100 insts for push, 97 insts for pop
PBR: 100 insts for push, 83 insts for pop

I think we learned something here... Trying to understand exactly what. But I think this result settles it.

I should test the different alternatives on Arm, does code generation behave the same as for x86_64.

Quoted 11 lines
Quoted 11 lines
Quoted 8 lines

Thanks,
Gage

[snip]

--
Ola Liljedahl, Networking System Architect, Arm Phone +46706866373, Skype ola.liljedahl

Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
author
#7 lore raw

Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 184 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Friday, February 1, 2019 1:44 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; Gavin Hu (Arm Technology China) <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 138 lines · Gage
Quoted 138 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Friday, February 1, 2019 1:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; Gavin Hu (Arm Technology China) <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 104 lines · Gage
Quoted 65 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Monday, January 28, 2019 5:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64
only)

Quoted 40 lines · Gage Eads

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 31
+++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 65
++++++++++++++++++++++

 2 files changed, 96 insertions(+)

diff --git a/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h b/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h index fd2ec9c53..b7b90b83e 100644

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2 -1
@@ -34,6 +34,7 @@
  /*
   * Inspired from FreeBSD src/sys/amd64/include/atomic.h
   * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
   * All rights reserved.
   */
 
@@ -46,6 +47,7 @@
 
  #include <stdint.h>
  #include <rte_common.h>
+#include <rte_compat.h>
  #include <rte_atomic.h>
 
  /*------------------------- 64 bit atomic operations
------------------------ -*/ @@ -208,4 +210,33 @@ static

inline void rte_atomic64_clear(rte_atomic64_t *v)
 }
 #endif

Quoted 1 line

+static inline int __rte_experimental

__rte_always_inline?

Quoted 2 lines
Quoted 5 lines
Quoted 1 line

+rte_atomic128_cmpset(volatile rte_int128_t *dst,

No need to declare the location volatile. Volatile doesn't do what you think it does.
https://youtu.be/lkgszkPnV8g?t=1027

I made this volatile to match the existing rte_atomicN_cmpset definitions, which presumably have a good reason for using the keyword. Maintainers, any input here?

Quoted 5 lines
Quoted 5 lines
Quoted 12 lines

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h

Quoted 1 line
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1 -0
+		     rte_int128_t *exp,

I would declare 'exp' const as well and document that 'exp' is not updated (with the old value) for a failure. The reason being that
ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of the critical
section) using e.g. non-atomic 64-bit reads so there isn't actually any requirement for an atomic 128-bit read of the location.

Will change in v2.

Quoted 6 lines
Quoted 5 lines

 rte_int128_t *src,

const rte_int128_t *src?

Sure, I don't see any harm in using const.

Quoted 3 lines
Quoted 7 lines
Quoted 3 lines

But why are we not passing 'exp' and 'src' by value? That works great, even with structs. Passing by value simplifies the compiler's life, especially if the call is inlined. Ask a compiler developer.

I ran objdump on the nb_stack code with both approaches, and pass-by-reference resulted in fewer overall x86_64 assembly ops.
PBV: 100 ops for push, 97 ops for pop
PBR: 92 ops for push, 84 ops for pop

OK I have never checked x86_64 code generation... I have good experiences with ARM/AArch64, everything seems to be done using registers. I am surprised there is a difference.

Did a quick check with lfring, passing 'src' (third param) by reference and by value. No difference in code generation on x86_64.

But if you insist let's go with PBR.

Quoted 2 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

(Using the in-progress v5 nb_stack code)

Another factor -- though much less compelling -- is that with pass-by- reference, the user can create a 16B structure and cast it to rte_int128_t when they call rte_atomic128_cmpset, whereas with pass-by-value they need to put that struct in a union with

rte_int128_t.

Quoted 8 lines
Quoted 8 lines

Which is what I always do nowadays... Trying to use as few casts as possible and lie to the compiler as seldom as possible. But I can see the freedom provided by taking a pointer to something and cast it it rte_int128_t ptr in the call to rte_atomic128_cmpset().

Would prefer a name that is more similar to __atomic_compare_exchange(). E.g.
rte_atomic128_compare_exchange() (or perhaps just

rte_atomic128_cmpxchg)?

Quoted 5 lines
Quoted 3 lines

All the rte_atomicXX_cmpset() functions do not take any memory order parameters.
From an Arm perspective, we are not happy with that.

Since the function returns a boolean success value, isn't compare-and-set the appropriate term?

I was thinking of the memory ordering parameters that __atomic_xxx builtins have and we want (from an Arm perspective).

GCC __atomic_compare_exchange also returns a boolean.
https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html
bool __atomic_compare_exchange_n (type *ptr, type *expected, type desired, bool weak, int success_memorder, int failure_memorder) bool __atomic_compare_exchange (type *ptr, type *expected, type *desired, bool weak, int success_memorder, int failure_memorder)

Quoted 3 lines

rte_atomic128_compare_exchange(rte_int128_t *dst, const rte_int128_t *exp, const rte_int182_t *src, bool weak, int mo_success, int mo_failure);

I assumed GCC chose the name "exchange" because the builtin modifies 'exp' on failure, but I could be wrong. I'm fine with either name. Anyone else have a preference?

<snip>

#8 lore raw

Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 201 lines · Gage
Quoted 201 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Friday, February 1, 2019 1:44 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; Gavin Hu (Arm Technology China) <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 151 lines · Gage
Quoted 151 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Friday, February 1, 2019 1:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; Gavin Hu (Arm Technology China) <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 113 lines · Gage
Quoted 69 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Monday, January 28, 2019 5:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64
only)

Quoted 41 lines · Gage Eads

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 31
+++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 65
++++++++++++++++++++++

 2 files changed, 96 insertions(+)

diff --git a/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h b/lib/librte_eal/common/include/arch/x86/rte_atomic_64.h index fd2ec9c53..b7b90b83e 100644

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2 -1
@@ -34,6 +34,7 @@
  /*
   * Inspired from FreeBSD src/sys/amd64/include/atomic.h
   * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
   * All rights reserved.
   */
 
@@ -46,6 +47,7 @@
 
  #include <stdint.h>
  #include <rte_common.h>
+#include <rte_compat.h>
  #include <rte_atomic.h>
 
  /*------------------------- 64 bit atomic operations
------------------------ -*/ @@ -208,4 +210,33 @@ static

inline void rte_atomic64_clear(rte_atomic64_t *v)
 }
 #endif

Quoted 1 line

+static inline int __rte_experimental

__rte_always_inline?

Quoted 3 lines
Quoted 5 lines
Quoted 1 line

+rte_atomic128_cmpset(volatile rte_int128_t *dst,

No need to declare the location volatile. Volatile doesn't do what you think it does.
https://youtu.be/lkgszkPnV8g?t=1027

I made this volatile to match the existing rte_atomicN_cmpset definitions, which presumably have a good reason for using the keyword. Maintainers, any input here?

Quoted 7 lines
Quoted 7 lines
Quoted 12 lines

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h

Quoted 1 line
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1 -0
+		     rte_int128_t *exp,

I would declare 'exp' const as well and document that 'exp' is not updated (with the old value) for a failure. The reason being that
ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of the critical
section) using e.g. non-atomic 64-bit reads so there isn't actually any requirement for an atomic 128-bit read of the location.

Will change in v2.

Quoted 8 lines
Quoted 7 lines

 rte_int128_t *src,

const rte_int128_t *src?

Sure, I don't see any harm in using const.

Quoted 4 lines
Quoted 8 lines
Quoted 4 lines

But why are we not passing 'exp' and 'src' by value? That works great, even with structs. Passing by value simplifies the compiler's life, especially if the call is inlined. Ask a compiler developer.

I ran objdump on the nb_stack code with both approaches, and pass-by-reference resulted in fewer overall x86_64 assembly ops.
PBV: 100 ops for push, 97 ops for pop
PBR: 92 ops for push, 84 ops for pop

OK I have never checked x86_64 code generation... I have good experiences with ARM/AArch64, everything seems to be done using registers. I am surprised there is a difference.

Did a quick check with lfring, passing 'src' (third param) by reference and by value. No difference in code generation on x86_64.

But if you insist let's go with PBR.

Quoted 3 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

(Using the in-progress v5 nb_stack code)

Another factor -- though much less compelling -- is that with pass-by- reference, the user can create a 16B structure and cast it to rte_int128_t when they call rte_atomic128_cmpset, whereas with pass-by-value they need to put that struct in a union with

rte_int128_t.

Quoted 10 lines
Quoted 10 lines

Which is what I always do nowadays... Trying to use as few casts as possible and lie to the compiler as seldom as possible. But I can see the freedom provided by taking a pointer to something and cast it it rte_int128_t ptr in the call to rte_atomic128_cmpset().

Would prefer a name that is more similar to __atomic_compare_exchange(). E.g.
rte_atomic128_compare_exchange() (or perhaps just

rte_atomic128_cmpxchg)?

Quoted 7 lines
Quoted 5 lines

All the rte_atomicXX_cmpset() functions do not take any memory order parameters.
From an Arm perspective, we are not happy with that.

Since the function returns a boolean success value, isn't compare-and-set the appropriate term?

I was thinking of the memory ordering parameters that __atomic_xxx builtins have and we want (from an Arm perspective).

GCC __atomic_compare_exchange also returns a boolean.
https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html
bool __atomic_compare_exchange_n (type *ptr, type *expected, type desired, bool weak, int success_memorder, int failure_memorder) bool __atomic_compare_exchange (type *ptr, type *expected, type *desired, bool weak, int success_memorder, int failure_memorder)

Quoted 5 lines
Quoted 3 lines

rte_atomic128_compare_exchange(rte_int128_t *dst, const rte_int128_t *exp, const rte_int182_t *src, bool weak, int mo_success, int mo_failure);

I assumed GCC chose the name "exchange" because the builtin modifies 'exp' on failure, but I could be wrong.

You are probably right. But exchange doesn't necessary mean that the old value is returned, just that it is replaced with a different value? However __atomic_exchange() (unconditionally) returns the old value so there is some precedent for retunrning the old value. And it can be done, just not guaranteed to be atomic. Perhaps there is a way to both eat and have the cake?

Re-use the prototype of __atomic_compare_exchange_16() so must return a value if the comparison fails. But depending on the 'weak' flag, we for weak=false require an atomic read which on ARMv8.0 must be accomplished by writing back the old value (if the comparison fails). Or for weak=true we allow both spurious LL/SC failures (this is what the weak parameter is for) and non-atomic read of the old value (this is my 'frail' atomic compare and exchange in lfring).

The GCC documentation says this:
"If they are not equal, the operation is a read and the current contents of ptr are written into expected."
We follow this but just don't guarantee an atomic read (for weak=true).

bool
rte_atomic128_compare_exchange(rte_int128_t *dst, rte_int128_t *exp, //Note no const now const rte_int128_t *src, bool weak,
                               int mo_success,
                               int mo_failure);

Quoted 4 lines

I'm fine with either name. Anyone else have a preference?

<snip>

Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
#9 lore raw

Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

In reply to Eads, Gage
Quoted 60 lines
Quoted 38 lines · Gage Eads
Quoted 36 lines

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 31 +++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 65
++++++++++++++++++++++

 2 files changed, 96 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2 -1
index fd2ec9c53..b7b90b83e 100644
@@ -34,6 +34,7 @@
  /*
   * Inspired from FreeBSD src/sys/amd64/include/atomic.h
   * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
   * All rights reserved.
   */
 
@@ -46,6 +47,7 @@
 
  #include <stdint.h>
  #include <rte_common.h>
+#include <rte_compat.h>
  #include <rte_atomic.h>
 
  /*------------------------- 64 bit atomic operations
------------------------ -*/ @@ -208,4 +210,33 @@ static inline void

rte_atomic64_clear(rte_atomic64_t *v)
 }
 #endif

Quoted 8 lines
Quoted 1 line

+static inline int __rte_experimental

__rte_always_inline?

Quoted 1 line

+rte_atomic128_cmpset(volatile rte_int128_t *dst,

No need to declare the location volatile. Volatile doesn't do what you think it does.
https://youtu.be/lkgszkPnV8g?t=1027

I made this volatile to match the existing rte_atomicN_cmpset definitions, which presumably have a good reason for using the keyword. Maintainers, any input here?

Quoted 9 lines

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h

Quoted 2 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1 -0
+		     rte_int128_t *exp,

I would declare 'exp' const as well and document that 'exp' is not updated (with the old value) for a failure. The reason being that ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of the critical section) using e.g. non-atomic 64-bit reads so there isn't actually any

Quoted 5 lines

requirement for an atomic 128-bit read of the location.

Quoted 1 line

Will change in v2.

IMO, we should not change the definition of this API, because
1) This API will differ from __atomic_compare_exchange_n API. It will be a new API to learn for the users.
2) The definition in this patch will make it easy to replace this API call with __atomic_xxx API (whenever it supports 128b natively on all the platforms)
3) I do not see any documentation in [1] indicating that the 'read on failure' will be an atomic read.

[1] https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html

Quoted 87 lines
Quoted 2 lines
Quoted 1 line

rte_int128_t *src,

const rte_int128_t *src?

Sure, I don't see any harm in using const.

Quoted 4 lines

But why are we not passing 'exp' and 'src' by value? That works great, even with structs. Passing by value simplifies the compiler's life, especially if the call is inlined. Ask a compiler developer.

I ran objdump on the nb_stack code with both approaches, and pass-by- reference resulted in fewer overall x86_64 assembly ops.
PBV: 100 ops for push, 97 ops for pop
PBR: 92 ops for push, 84 ops for pop

(Using the in-progress v5 nb_stack code)

Another factor -- though much less compelling -- is that with pass-by-reference, the user can create a 16B structure and cast it to rte_int128_t when they call rte_atomic128_cmpset, whereas with pass-by-value they need to put that struct in a union with rte_int128_t.

Quoted 46 lines
Quoted 46 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +25 -0
+		     unsigned int weak,
+		     enum rte_atomic_memmodel_t success,
+		     enum rte_atomic_memmodel_t failure) {
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=A" (exp->val[0]),
+			[res] "=r" (res)
+		      : "c" (src->val[1]),
+			"b" (src->val[0]),
+			"m" (dst->val[0]),
+			"d" (exp->val[1]),
+			"a" (exp->val[0])
+		      : "memory");
+
+	return res;
+}
+

 #endif /* _RTE_ATOMIC_X86_64_H_ */

lib/librte_eal/common/include/generic/rte_atomic.h +1 -0
index b99ba4688..8d612d566 100644
@@ -14,6 +14,7 @@
 
  #include <stdint.h>
  #include <rte_common.h>
+#include <rte_compat.h>
 
  #ifdef __DOXYGEN__
 
@@ -1082,4 +1083,68 @@ static inline void

rte_atomic64_clear(rte_atomic64_t
*v)
 }
 #endif

Quoted 12 lines

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093

Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9 -1
+/*------------------------ 128 bit atomic operations
+------------------------
-*/
+
+/**
+ * 128-bit integer structure.
+ */
+typedef struct {
+	uint64_t val[2];
+} __rte_aligned(16) rte_int128_t;

So we can't use __int128?

I'll put it in a union with val[2], in case any implementations want to use it.

Thanks,
Gage

[snip]

#10 lore raw

Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

In reply to Gage Eads
Quoted 35 lines

This operation can be used for non-blocking algorithms, such as a non- blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 31 +++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 65
++++++++++++++++++++++
 2 files changed, 96 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2 -0
index fd2ec9c53..b7b90b83e 100644
@@ -34,6 +34,7 @@
 /*
  * Inspired from FreeBSD src/sys/amd64/include/atomic.h
  * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
  * All rights reserved.
  */
 
@@ -46,6 +47,7 @@
 
 #include <stdint.h>
 #include <rte_common.h>
+#include <rte_compat.h>
 #include <rte_atomic.h>
 
 /*------------------------- 64 bit atomic operations -------------------------*/ @@ -

208,4 +210,33 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v) } #endif

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h

Quoted 2 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2 -0
+static inline int __rte_experimental
+rte_atomic128_cmpset(volatile rte_int128_t *dst,

Does it make sense to call is rte_atomic128_compare_exchange (or ..._cmp_xchg) to indicate it is a compare-exchange operation?

Quoted 43 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +26 -0
+		     rte_int128_t *exp, rte_int128_t *src,
+		     unsigned int weak,
+		     enum rte_atomic_memmodel_t success,
+		     enum rte_atomic_memmodel_t failure) {
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=A" (exp->val[0]),
+			[res] "=r" (res)
+		      : "c" (src->val[1]),
+			"b" (src->val[0]),
+			"m" (dst->val[0]),
+			"d" (exp->val[1]),
+			"a" (exp->val[0])
+		      : "memory");
+
+	return res;
+}
+
 #endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +1 -0
index b99ba4688..8d612d566 100644
@@ -14,6 +14,7 @@
 
 #include <stdint.h>
 #include <rte_common.h>
+#include <rte_compat.h>
 
 #ifdef __DOXYGEN__
 
@@ -1082,4 +1083,68 @@ static inline void

rte_atomic64_clear(rte_atomic64_t *v) } #endif

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093

Quoted 9 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9 -0
+/*------------------------ 128 bit atomic operations
+-------------------------*/
+
+/**
+ * 128-bit integer structure.
+ */
+typedef struct {
+	uint64_t val[2];
+} __rte_aligned(16) rte_int128_t;

It looks like '__int128' is available from gcc 4.6. I think we should use '__int128'. We can have it as an internal structure for ease of programming.

Quoted 9 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9 -0
+
+/**
+ * Memory consistency models used in atomic operations. These control
+the
+ * behavior of the operation with respect to memory barriers and
+ * thread synchronization.
+ *
+ * These directly match those in the C++11 standard; for details on
+their

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1110

Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +10 -0
+ * behavior, refer to the standard.
+ */
+enum rte_atomic_memmodel_t {
+	RTE_ATOMIC_RELAXED,
+	RTE_ATOMIC_CONSUME,
+	RTE_ATOMIC_ACQUIRE,
+	RTE_ATOMIC_RELEASE,
+	RTE_ATOMIC_ACQ_REL,
+	RTE_ATOMIC_SEQ_CST,
+};

IMO, we can use the GCC provided names. I do not see any advantage to defining our own.

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1114

Quoted 5 lines
lib/librte_eal/common/include/generic/rte_atomic.h +5 -0
+
+/* Only implemented on x86-64 currently. The ifdef prevents compilation
+from
+ * failing for architectures without a definition of this function.
+ */

Minor comment. We can skip the above comments, the #if below is pretty obvious.

Quoted 38 lines
lib/librte_eal/common/include/generic/rte_atomic.h +16 -0
+#if defined(RTE_ARCH_X86_64)
+/**
+ * An atomic compare and set function used by the mutex functions.
+ * (atomic) equivalent to:
+ *   if (*dst == exp)
+ *     *dst = src (all 128-bit words)
+ *
+ * @param dst
+ *   The destination into which the value will be written.
+ * @param exp
+ *   Pointer to the expected value. If the operation fails, this memory is
+ *   updated with the actual value.
+ * @param src
+ *   Pointer to the new value.
+ * @param weak
+ *   A value of true allows the comparison to spuriously fail.

Implementations
+ * may ignore this argument and only implement the strong variant.
+ * @param success
+ * If successful, the operation's memory behavior conforms to this (or a
+ * stronger) model.
+ * @param failure

+ *   If unsuccessful, the operation's memory behavior conforms to this (or a
+ *   stronger) model. This argument cannot be RTE_ATOMIC_RELEASE,
+ *   RTE_ATOMIC_ACQ_REL, or a stronger model than success.

+ * @return
+ * Non-zero on success; 0 on failure.
+ */

lib/librte_eal/common/include/generic/rte_atomic.h +7 -1
+static inline int __rte_experimental
+rte_atomic128_cmpset(volatile rte_int128_t *dst,
+		     rte_int128_t *exp, rte_int128_t *src,
+		     unsigned int weak,
+		     enum rte_atomic_memmodel_t success,
+		     enum rte_atomic_memmodel_t failure); #endif
+
 #endif /* _RTE_ATOMIC_H_ */
--

2.13.6

author
#11 lore raw

Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 49 lines

-----Original Message-----
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, January 30, 2019 11:48 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; [email protected]; [email protected]; nd <[email protected]>
Subject: RE: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)

Quoted 36 lines

This operation can be used for non-blocking algorithms, such as a non- blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 31 +++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 65
++++++++++++++++++++++
 2 files changed, 96 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2 -1
index fd2ec9c53..b7b90b83e 100644
@@ -34,6 +34,7 @@
 /*
  * Inspired from FreeBSD src/sys/amd64/include/atomic.h
  * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
  * All rights reserved.
  */
 
@@ -46,6 +47,7 @@
 
 #include <stdint.h>
 #include <rte_common.h>
+#include <rte_compat.h>
 #include <rte_atomic.h>
 
 /*------------------------- 64 bit atomic operations
-------------------------*/ @@ -

208,4 +210,33 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v) } #endif

Quoted 5 lines

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h

Quoted 2 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2 -0
+static inline int __rte_experimental
+rte_atomic128_cmpset(volatile rte_int128_t *dst,

Does it make sense to call is rte_atomic128_compare_exchange (or ..._cmp_xchg) to indicate it is a compare-exchange operation?

Good point, though for v2 I'm planning to change this to true cmpset semantics (no update to exp on failure). See Ola's reply for the justification.

Quoted 43 lines
Quoted 43 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +26 -0
+		     rte_int128_t *exp, rte_int128_t *src,
+		     unsigned int weak,
+		     enum rte_atomic_memmodel_t success,
+		     enum rte_atomic_memmodel_t failure) {
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=A" (exp->val[0]),
+			[res] "=r" (res)
+		      : "c" (src->val[1]),
+			"b" (src->val[0]),
+			"m" (dst->val[0]),
+			"d" (exp->val[1]),
+			"a" (exp->val[0])
+		      : "memory");
+
+	return res;
+}
+
 #endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +1 -0
index b99ba4688..8d612d566 100644
@@ -14,6 +14,7 @@
 
 #include <stdint.h>
 #include <rte_common.h>
+#include <rte_compat.h>
 
 #ifdef __DOXYGEN__
 
@@ -1082,4 +1083,68 @@ static inline void

rte_atomic64_clear(rte_atomic64_t *v) } #endif

Quoted 12 lines

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093

Quoted 9 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9 -0
+/*------------------------ 128 bit atomic operations
+-------------------------*/
+
+/**
+ * 128-bit integer structure.
+ */
+typedef struct {
+	uint64_t val[2];
+} __rte_aligned(16) rte_int128_t;

It looks like '__int128' is available from gcc 4.6. I think we should use '__int128'. We can have it as an internal structure for ease of programming.

Will add in v2.

Quoted 13 lines
Quoted 13 lines
lib/librte_eal/common/include/generic/rte_atomic.h +13 -0
+
+/**
+ * Memory consistency models used in atomic operations. These control
+the
+ * behavior of the operation with respect to memory barriers and
+ * thread synchronization.
+ *
+ * These directly match those in the C++11 standard; for details on
+their
+ * behavior, refer to the standard.
+ */
+enum rte_atomic_memmodel_t {
+	RTE_ATOMIC_RELAXED,
Quoted 9 lines

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1110

Quoted 6 lines
lib/librte_eal/common/include/generic/rte_atomic.h +6 -0
+	RTE_ATOMIC_CONSUME,
+	RTE_ATOMIC_ACQUIRE,
+	RTE_ATOMIC_RELEASE,
+	RTE_ATOMIC_ACQ_REL,
+	RTE_ATOMIC_SEQ_CST,
+};

IMO, we can use the GCC provided names. I do not see any advantage to defining our own.

Will change in v2. I was trying to avoid issues with GCC versions that don't have C++11 support, but DPDK's recommended minimum version (4.9) is later than the version that added __atomic builtins (4.7).

Quoted 8 lines

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1114

Quoted 5 lines
lib/librte_eal/common/include/generic/rte_atomic.h +5 -0
+
+/* Only implemented on x86-64 currently. The ifdef prevents
+compilation from
+ * failing for architectures without a definition of this function.
+ */

Minor comment. We can skip the above comments, the #if below is pretty obvious.

Sure.

Thanks,
Gage

<snip>

author
#12 lore raw

v2 cover Add 128-bit compare and set

In reply to Gage Eads

This patch addresses x86-64 only; other architectures can/will be supported in the future. The __atomic intrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the __atomic_compare_exchange_16 (which itself is based on the C++11 memory model) to best support weak consistency architectures.

This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not on the entire non-blocking stack patchset.

[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123653.html

v2:
- Rename function rte_atomic128_cmpxchg()
- Replace "=A" output constraint with "=a" and "=d" to prevent GCC from using the al register as the sete destination
- Extend 'weak' definition to explicitly allow non-atomic 'exp' updates.
- Add const keyword to 'src' and remove volatile keyword from 'dst'
- Put __int128 in a union in rte_int128_t and move the structure definition inside the RTE_ARCH_x86_64 ifdef
- Drop enum rte_atomic_memmodel_t in favor of compiler-defined __ATOMIC_*
- Drop unnecessary comment relating to X86_64
- Tweak the pseudocode to reflect the 'exp' update on failure.
Gage Eads (1):
eal: add 128-bit cmpxchg (x86-64 only)

 .../common/include/arch/x86/rte_atomic_64.h        | 33 ++++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 59 ++++++++++++++++++++++
 2 files changed, 92 insertions(+)
Signature · 1 line
2.13.6
author
#13 lore raw

PATCH v2 1/1 eal: add 128-bit cmpxchg (x86-64 only)

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 33 ++++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 59 ++++++++++++++++++++++
 2 files changed, 92 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +33 -0
index fd2ec9c53..413e5361b 100644
@@ -34,6 +34,7 @@
 /*
  * Inspired from FreeBSD src/sys/amd64/include/atomic.h
  * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
  * All rights reserved.
  */
 
@@ -46,6 +47,7 @@
 
 #include <stdint.h>
 #include <rte_common.h>
+#include <rte_compat.h>
 #include <rte_atomic.h>
 
 /*------------------------- 64 bit atomic operations -------------------------*/
@@ -208,4 +210,35 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v)
 }
 #endif
 
+static inline int __rte_experimental
+rte_atomic128_cmpxchg(rte_int128_t *dst,
+		      rte_int128_t *exp,
+		      const rte_int128_t *src,
+		      unsigned int weak,
+		      int success,
+		      int failure)
+{
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=a" (exp->val[0]),
+			"=d" (exp->val[1]),
+			[res] "=r" (res)
+		      : "b" (src->val[0]),
+			"c" (src->val[1]),
+			"a" (exp->val[0]),
+			"d" (exp->val[1]),
+			"m" (dst->val[0])
+		      : "memory");
+
+	return res;
+}
+
 #endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +59 -0

69 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v2 1/1
#14 lore raw

Re: [PATCH v2 1/1] eal: add 128-bit cmpxchg (x86-64 only)

Quoted 92 lines

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 33 ++++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 59
++++++++++++++++++++++
 2 files changed, 92 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2 -0
index fd2ec9c53..413e5361b 100644
@@ -34,6 +34,7 @@
 /*
  * Inspired from FreeBSD src/sys/amd64/include/atomic.h
  * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
  * All rights reserved.
  */
 
@@ -46,6 +47,7 @@
 
 #include <stdint.h>
 #include <rte_common.h>
+#include <rte_compat.h>
 #include <rte_atomic.h>
 
 /*------------------------- 64 bit atomic operations -------------------------*/ @@ -

208,4 +210,35 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v) } #endif

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +31 -0
+static inline int __rte_experimental
+rte_atomic128_cmpxchg(rte_int128_t *dst,
+		      rte_int128_t *exp,
+		      const rte_int128_t *src,
+		      unsigned int weak,
+		      int success,
+		      int failure)
+{
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=a" (exp->val[0]),
+			"=d" (exp->val[1]),
+			[res] "=r" (res)
+		      : "b" (src->val[0]),
+			"c" (src->val[1]),
+			"a" (exp->val[0]),
+			"d" (exp->val[1]),
+			"m" (dst->val[0])
+		      : "memory");
+
+	return res;
+}
+
 #endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +1 -0
index 4afd1acc3..7c43896e9 100644
@@ -14,6 +14,7 @@
 
 #include <stdint.h>
 #include <rte_common.h>
+#include <rte_compat.h>
 
 #ifdef __DOXYGEN__
 
@@ -1082,4 +1083,62 @@ static inline void

rte_atomic64_clear(rte_atomic64_t *v) } #endif

lib/librte_eal/common/include/generic/rte_atomic.h +10 -0
+/*------------------------ 128 bit atomic operations
+-------------------------*/
+
+#if defined(RTE_ARCH_X86_64)
+/**
+ * 128-bit integer structure.
+ */
+RTE_STD_C11
+typedef struct {
+	RTE_STD_C11

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1104

Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +10 -0
+	union {
+		uint64_t val[2];
+		__int128 int128;
+	};
+} __rte_aligned(16) rte_int128_t;
+
+/**
+ * An atomic compare and set function used by the mutex functions.
+ * (Atomically) Equivalent to:
+ *   if (*dst == exp)

Should be, "if (*dst == *exp)"

Quoted 1 line

+ * *dst = src

Should be "*dst = *src"

Quoted 23 lines

+ * else
+ * *exp = *dst
+ *
+ * @note The success and failure arguments must be one of the

lib/librte_eal/common/include/generic/rte_atomic.h +19 -0
+__ATOMIC_* values
+ * defined in the C++11 standard. For details on their behavior, refer
+to the
+ * standard.
+ *
+ * @param dst
+ *   The destination into which the value will be written.
+ * @param exp
+ *   Pointer to the expected value. If the operation fails, this memory is
+ *   updated with the actual value.
+ * @param src
+ *   Pointer to the new value.
+ * @param weak
+ *   A value of true allows the comparison to spuriously fail and allows the
+ *   'exp' update to occur non-atomically (i.e. a torn read may occur).
+ *   Implementations may ignore this argument and only implement the strong
+ *   variant.
+ * @param success
+ *   If successful, the operation's memory behavior conforms to this (or a

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1136

Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +10 -0
+ *   stronger) model.
+ * @param failure
+ *   If unsuccessful, the operation's memory behavior conforms to this (or a
+ *   stronger) model. This argument cannot be __ATOMIC_RELEASE,
+ *   __ATOMIC_ACQ_REL, or a stronger model than success.
+ * @return
+ *   Non-zero on success; 0 on failure.
+ */
+static inline int __rte_experimental
+rte_atomic128_cmpxchg(rte_int128_t *dst,

Name could be more neutral. May be rte_atomic128_compare/cmp_exchange?

Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +7 -1
+		      rte_int128_t *exp,
+		      const rte_int128_t *src,
+		      unsigned int weak,
+		      int success,
+		      int failure);
+#endif
+
 #endif /* _RTE_ATOMIC_H_ */
--

2.13.6

Few minor comments.
I have not reviewed the x86 implementation.
Otherwise,

author
#15 lore raw

Re: [PATCH v2 1/1] eal: add 128-bit cmpxchg (x86-64 only)

[snip]

Quoted 9 lines
Quoted 4 lines

+/**
+ * An atomic compare and set function used by the mutex functions.
+ * (Atomically) Equivalent to:
+ * if (*dst == exp)

Should be, "if (*dst == *exp)"

Quoted 1 line

+ * *dst = src

Should be "*dst = *src"

Good catches, will fix both.

Quoted 36 lines
Quoted 10 lines

+ * else
+ * *exp = *dst
+ *
+ * @note The success and failure arguments must be one of the

lib/librte_eal/common/include/generic/rte_atomic.h +6 -0
+__ATOMIC_* values
+ * defined in the C++11 standard. For details on their behavior,
+refer to the
+ * standard.
+ *
+ * @param dst

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v)

Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +10 -0
+ *   The destination into which the value will be written.
+ * @param exp
+ *   Pointer to the expected value. If the operation fails, this memory is
+ *   updated with the actual value.
+ * @param src
+ *   Pointer to the new value.
+ * @param weak
+ *   A value of true allows the comparison to spuriously fail and allows the
+ *   'exp' update to occur non-atomically (i.e. a torn read may occur).
+ *   Implementations may ignore this argument and only implement the

strong

Review

Quoted 13 lines

+ * variant.
+ * @param success
+ * If successful, the operation's memory behavior conforms to this (or a
+ * stronger) model.
+ * @param failure

+ *   If unsuccessful, the operation's memory behavior conforms to this (or a
+ *   stronger) model. This argument cannot be __ATOMIC_RELEASE,
+ *   __ATOMIC_ACQ_REL, or a stronger model than success.

+ * @return
+ * Non-zero on success; 0 on failure.
+ */

(quoted diff) +2 -0
+static inline int __rte_experimental
+rte_atomic128_cmpxchg(rte_int128_t *dst,

Name could be more neutral. May be rte_atomic128_compare/cmp_exchange?

Quoted 11 lines
Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +7 -1
+		      rte_int128_t *exp,
+		      const rte_int128_t *src,
+		      unsigned int weak,
+		      int success,
+		      int failure);
+#endif
+
 #endif /* _RTE_ATOMIC_H_ */
--

2.13.6

Sure -- I'll rename it rte_atomic128_cmp_exchange in the next version.

Quoted 4 lines

Few minor comments.
I have not reviewed the x86 implementation.
Otherwise,

Thanks,
Gage

author
#16 lore raw

v3 cover Add 128-bit compare and set

In reply to Gage Eads

This patch addresses x86-64 only; other architectures can/will be supported in the future. The __atomic intrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the __atomic_compare_exchange_16 (which itself is based on the C++11 memory model) to best support weak consistency architectures.

This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not on the entire non-blocking stack patchset.

[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123653.html

v3:
- Rename function to ISA-neutral rte_atomic128_cmp_exchange()
- Fix two pseudocode bugs in function documentation

v2:
- Rename function to rte_atomic128_cmpxchg()
- Replace "=A" output constraint with "=a" and "=d" to prevent GCC from using the al register for the sete destination
- Extend 'weak' definition to allow non-atomic 'exp' updates.
- Add const keyword to 'src' and remove volatile keyword from 'dst'
- Put __int128 in a union in rte_int128_t and move the structure definition inside the RTE_ARCH_x86_64 ifdef
- Drop enum rte_atomic_memmodel_t in favor of compiler-defined __ATOMIC_*
- Drop unnecessary comment relating to X86_64
- Tweak the pseudocode to reflect the 'exp' update on failure.

Gage Eads (1):
eal: add 128-bit compare exchange (x86-64 only)

 .../common/include/arch/x86/rte_atomic_64.h        | 33 ++++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 59 ++++++++++++++++++++++
 2 files changed, 92 insertions(+)
Signature · 1 line
2.13.6
author
#17 lore raw

PATCH v3 1/1 eal: add 128-bit compare exchange (x86-64 only)

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 33 ++++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 59 ++++++++++++++++++++++
 2 files changed, 92 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +33 -0
index fd2ec9c53..3b061ff7c 100644
@@ -34,6 +34,7 @@
 /*
  * Inspired from FreeBSD src/sys/amd64/include/atomic.h
  * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
  * All rights reserved.
  */
 
@@ -46,6 +47,7 @@
 
 #include <stdint.h>
 #include <rte_common.h>
+#include <rte_compat.h>
 #include <rte_atomic.h>
 
 /*------------------------- 64 bit atomic operations -------------------------*/
@@ -208,4 +210,35 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v)
 }
 #endif
 
+static inline int __rte_experimental
+rte_atomic128_cmp_exchange(rte_int128_t *dst,
+			   rte_int128_t *exp,
+			   const rte_int128_t *src,
+			   unsigned int weak,
+			   int success,
+			   int failure)
+{
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=a" (exp->val[0]),
+			"=d" (exp->val[1]),
+			[res] "=r" (res)
+		      : "b" (src->val[0]),
+			"c" (src->val[1]),
+			"a" (exp->val[0]),
+			"d" (exp->val[1]),
+			"m" (dst->val[0])
+		      : "memory");
+
+	return res;
+}
+
 #endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +59 -0

69 lines not on this page · raw message

Signature · 1 line
2.13.6
#18 lore raw

Re: [PATCH v3 1/1] eal: add 128-bit compare exchange (x86-64 only)

04/03/2019 21:51, Gage Eads:

Quoted 4 lines
lib/librte_eal/common/include/generic/rte_atomic.h +2 -0
+#if defined(RTE_ARCH_X86_64)
+/**

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1099

Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +10 -0
+ * 128-bit integer structure.
+ */
+RTE_STD_C11
+typedef struct {
+       RTE_STD_C11
+       union {
+               uint64_t val[2];
+               __int128 int128;
+       };
+} __rte_aligned(16) rte_int128_t;

Why adding an arch-specific definition in a generic file? Can we move it to the x86_64 file?

author
#19 lore raw

Re: [PATCH v3 1/1] eal: add 128-bit compare exchange (x86-64 only)

Quoted 10 lines

04/03/2019 21:51, Gage Eads:

Quoted 9 lines
lib/librte_eal/common/include/generic/rte_atomic.h +7 -0
+#if defined(RTE_ARCH_X86_64)
+/**
+ * 128-bit integer structure.
+ */
+RTE_STD_C11
+typedef struct {
+       RTE_STD_C11
Quoted 9 lines

lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1099

Quoted 5 lines
lib/librte_eal/common/include/generic/rte_atomic.h +5 -0
+       union {
+               uint64_t val[2];
+               __int128 int128;
+       };
+} __rte_aligned(16) rte_int128_t;

Why adding an arch-specific definition in a generic file? Can we move it to the x86_64 file?

We can. I put it in the generic header in anticipation of other ISA implementations coming later, and since the atomic operations are documented in the generic header (although that ifdef appears to prevent it from being included in doxygen output).

I'll move it to x86/rte_atomic_64.h if that's preferred, and we can consider moving it back to generic as other implementations are added.

Thanks,
Gage

author
#20 lore raw

v4 cover Add 128-bit compare and set

In reply to Gage Eads

This patch addresses x86-64 only; other architectures can/will be supported in the future. The __atomic intrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the __atomic_compare_exchange_16 (which itself is based on the C++11 memory model) to best support weak consistency architectures.

This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not on the entire non-blocking stack patchset.

[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123653.html

v4:
- Move function declaration from generic/rte_atomic.h to x86-64 header file

v3:
- Rename function to ISA-neutral rte_atomic128_cmp_exchange()
- Fix two pseudocode bugs in function documentation

v2:
- Rename function to rte_atomic128_cmpxchg()
- Replace "=A" output constraint with "=a" and "=d" to prevent GCC from using the al register for the sete destination
- Extend 'weak' definition to allow non-atomic 'exp' updates.
- Add const keyword to 'src' and remove volatile keyword from 'dst'
- Put __int128 in a union in rte_int128_t and move the structure definition inside the RTE_ARCH_x86_64 ifdef
- Drop enum rte_atomic_memmodel_t in favor of compiler-defined __ATOMIC_*
- Drop unnecessary comment relating to X86_64
- Tweak the pseudocode to reflect the 'exp' update on failure.

Gage Eads (1):
eal: add 128-bit compare exchange (x86-64 only)

 .../common/include/arch/x86/rte_atomic_64.h        | 81 ++++++++++++++++++++++
 1 file changed, 81 insertions(+)
Signature · 1 line
2.13.6
author
#21 lore raw

PATCH v4 1/1 eal: add 128-bit compare exchange (x86-64 only)

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---
 .../common/include/arch/x86/rte_atomic_64.h        | 81 ++++++++++++++++++++++
 1 file changed, 81 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +81 -0
index fd2ec9c53..49bce32c7 100644
@@ -34,6 +34,7 @@
 /*
  * Inspired from FreeBSD src/sys/amd64/include/atomic.h
  * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
  * All rights reserved.
  */
 
@@ -46,6 +47,7 @@
 
 #include <stdint.h>
 #include <rte_common.h>
+#include <rte_compat.h>
 #include <rte_atomic.h>
 
 /*------------------------- 64 bit atomic operations -------------------------*/
@@ -208,4 +210,83 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v)
 }
 #endif
 
+/*------------------------ 128 bit atomic operations -------------------------*/
+
+/**
+ * 128-bit integer structure.
+ */
+RTE_STD_C11
+typedef struct {
+	RTE_STD_C11
+	union {
+		uint64_t val[2];
+		__int128 int128;
+	};
+} __rte_aligned(16) rte_int128_t;
+
+/**
+ * An atomic compare and set function used by the mutex functions.
+ * (Atomically) Equivalent to:
+ *   if (*dst == *exp)
+ *     *dst = *src
+ *   else
+ *     *exp = *dst
+ *
+ * @note The success and failure arguments must be one of the __ATOMIC_* values
+ * defined in the C++11 standard. For details on their behavior, refer to the
+ * standard.
+ *
+ * @param dst
+ *   The destination into which the value will be written.
+ * @param exp
+ *   Pointer to the expected value. If the operation fails, this memory is
+ *   updated with the actual value.
+ * @param src
+ *   Pointer to the new value.
+ * @param weak
+ *   A value of true allows the comparison to spuriously fail and allows the
+ *   'exp' update to occur non-atomically (i.e. a torn read may occur).
+ *   Implementations may ignore this argument and only implement the strong
+ *   variant.
+ * @param success
+ *   If successful, the operation's memory behavior conforms to this (or a
+ *   stronger) model.
+ * @param failure
+ *   If unsuccessful, the operation's memory behavior conforms to this (or a
+ *   stronger) model. This argument cannot be __ATOMIC_RELEASE,
+ *   __ATOMIC_ACQ_REL, or a stronger model than success.
+ * @return
+ *   Non-zero on success; 0 on failure.
+ */
+static inline int __rte_experimental
+rte_atomic128_cmp_exchange(rte_int128_t *dst,
+			   rte_int128_t *exp,
+			   const rte_int128_t *src,
+			   unsigned int weak,
+			   int success,
+			   int failure)
+{
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=a" (exp->val[0]),
+			"=d" (exp->val[1]),
+			[res] "=r" (res)
+		      : "b" (src->val[0]),
+			"c" (src->val[1]),
+			"a" (exp->val[0]),
+			"d" (exp->val[1]),
+			"m" (dst->val[0])
+		      : "memory");
+
+	return res;
+}
+
 #endif /* _RTE_ATOMIC_X86_64_H_ */
Signature · 1 line
2.13.6
#22 lore raw

Re: [PATCH v4 1/1] eal: add 128-bit compare exchange (x86-64 only)

03/04/2019 19:34, Gage Eads:

Quoted 39 lines

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +31 -0
+/**
+ * An atomic compare and set function used by the mutex functions.
+ * (Atomically) Equivalent to:
+ *   if (*dst == *exp)
+ *     *dst = *src
+ *   else
+ *     *exp = *dst
+ *
+ * @note The success and failure arguments must be one of the __ATOMIC_* values
+ * defined in the C++11 standard. For details on their behavior, refer to the
+ * standard.
+ *
+ * @param dst
+ *   The destination into which the value will be written.
+ * @param exp
+ *   Pointer to the expected value. If the operation fails, this memory is
+ *   updated with the actual value.
+ * @param src
+ *   Pointer to the new value.
+ * @param weak
+ *   A value of true allows the comparison to spuriously fail and allows the
+ *   'exp' update to occur non-atomically (i.e. a torn read may occur).
+ *   Implementations may ignore this argument and only implement the strong
+ *   variant.
+ * @param success
+ *   If successful, the operation's memory behavior conforms to this (or a
+ *   stronger) model.
+ * @param failure
+ *   If unsuccessful, the operation's memory behavior conforms to this (or a
+ *   stronger) model. This argument cannot be __ATOMIC_RELEASE,
+ *   __ATOMIC_ACQ_REL, or a stronger model than success.

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 267

Quoted 10 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +10 -0
+ * @return
+ *   Non-zero on success; 0 on failure.
+ */
+static inline int __rte_experimental
+rte_atomic128_cmp_exchange(rte_int128_t *dst,
+			   rte_int128_t *exp,
+			   const rte_int128_t *src,
+			   unsigned int weak,
+			   int success,
+			   int failure)

I was thinking about keeping the doxygen in the generic file. Is it possible?

author
#23 lore raw

Re: [PATCH v4 1/1] eal: add 128-bit compare exchange (x86-64 only)

Quoted 42 lines

03/04/2019 19:34, Gage Eads:

Quoted 23 lines

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +15 -0
+/**
+ * An atomic compare and set function used by the mutex functions.
+ * (Atomically) Equivalent to:
+ *   if (*dst == *exp)
+ *     *dst = *src
+ *   else
+ *     *exp = *dst
+ *
+ * @note The success and failure arguments must be one of the
+__ATOMIC_* values
+ * defined in the C++11 standard. For details on their behavior,
+refer to the
+ * standard.
+ *
+ * @param dst

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h static inline void rte_atomic64_clear(rte_atomic64_t *v)

Quoted 10 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +10 -0
+ *   The destination into which the value will be written.
+ * @param exp
+ *   Pointer to the expected value. If the operation fails, this memory is
+ *   updated with the actual value.
+ * @param src
+ *   Pointer to the new value.
+ * @param weak
+ *   A value of true allows the comparison to spuriously fail and allows the
+ *   'exp' update to occur non-atomically (i.e. a torn read may occur).
+ *   Implementations may ignore this argument and only implement the

strong

Quoted 6 lines

+ * variant.
+ * @param success
+ * If successful, the operation's memory behavior conforms to this (or a
+ * stronger) model.
+ * @param failure
+ * If unsuccessful, the operation's memory behavior conforms to this (or

a

Quoted 16 lines

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 267

Quoted 12 lines

+ * stronger) model. This argument cannot be __ATOMIC_RELEASE,
+ * __ATOMIC_ACQ_REL, or a stronger model than success.
+ * @return
+ * Non-zero on success; 0 on failure.
+ */

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +7 -0
+static inline int __rte_experimental
+rte_atomic128_cmp_exchange(rte_int128_t *dst,
+			   rte_int128_t *exp,
+			   const rte_int128_t *src,
+			   unsigned int weak,
+			   int success,
+			   int failure)

I was thinking about keeping the doxygen in the generic file. Is it possible?

We'd need to include the definition of rte_int128_t, so we'd also need either an ifdef on RTE_ARCH_64 or RTE_ARCH_X86_64 to protect 32-bit builds. That macro would prevent doxygen from parsing that section, unless we add a workaround like, for example:

#if defined(RTE_ARCH_64) || defined(__DOXYGEN__)

So the patch would look like the v3, with the declaration in generic/rte_atomic.h, but with that preprocessor change. If we change RTE_ARCH_X86_64 to RTE_ARCH_64, I'd add a note clarifying that it's only implemented for x86-64. What do you think?

#24 lore raw

Re: [PATCH v4 1/1] eal: add 128-bit compare exchange (x86-64 only)

03/04/2019 21:21, Eads, Gage:

Quoted 59 lines
Quoted 42 lines

03/04/2019 19:34, Gage Eads:

Quoted 23 lines

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

---

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +15 -0
+/**
+ * An atomic compare and set function used by the mutex functions.
+ * (Atomically) Equivalent to:
+ *   if (*dst == *exp)
+ *     *dst = *src
+ *   else
+ *     *exp = *dst
+ *
+ * @note The success and failure arguments must be one of the
+__ATOMIC_* values
+ * defined in the C++11 standard. For details on their behavior,
+refer to the
+ * standard.
+ *
+ * @param dst

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h static inline void rte_atomic64_clear(rte_atomic64_t *v)

Quoted 10 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +10 -0
+ *   The destination into which the value will be written.
+ * @param exp
+ *   Pointer to the expected value. If the operation fails, this memory is
+ *   updated with the actual value.
+ * @param src
+ *   Pointer to the new value.
+ * @param weak
+ *   A value of true allows the comparison to spuriously fail and allows the
+ *   'exp' update to occur non-atomically (i.e. a torn read may occur).
+ *   Implementations may ignore this argument and only implement the

strong

Quoted 6 lines

+ * variant.
+ * @param success
+ * If successful, the operation's memory behavior conforms to this (or a
+ * stronger) model.
+ * @param failure
+ * If unsuccessful, the operation's memory behavior conforms to this (or

a

Quoted 16 lines

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 267

Quoted 12 lines

+ * stronger) model. This argument cannot be __ATOMIC_RELEASE,
+ * __ATOMIC_ACQ_REL, or a stronger model than success.
+ * @return
+ * Non-zero on success; 0 on failure.
+ */

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +7 -0
+static inline int __rte_experimental
+rte_atomic128_cmp_exchange(rte_int128_t *dst,
+			   rte_int128_t *exp,
+			   const rte_int128_t *src,
+			   unsigned int weak,
+			   int success,
+			   int failure)

I was thinking about keeping the doxygen in the generic file. Is it possible?

Quoted 5 lines

We'd need to include the definition of rte_int128_t, so we'd also need either an ifdef on RTE_ARCH_64 or RTE_ARCH_X86_64 to protect 32-bit builds. That macro would prevent doxygen from parsing that section, unless we add a workaround like, for example:

#if defined(RTE_ARCH_64) || defined(__DOXYGEN__)

So the patch would look like the v3, with the declaration in generic/rte_atomic.h, but with that preprocessor change. If we change RTE_ARCH_X86_64 to RTE_ARCH_64, I'd add a note clarifying that it's only implemented for x86-64. What do you think?

I would like to see the doc in the generic file
and the implementation in the x86 file.

I tried forward declaration of the typedef:

	struct rte_int128;
	typedef struct rte_int128 rte_int128_t;
I don't why it does not work.
So I'm trying to protect the declaration with
	#ifdef __DOXYGEN__
#25 lore raw

PATCH v5 1/1 eal/x86: add 128-bit atomic compare exchange

In reply to Gage Eads

From: Gage Eads <[email protected]>

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

It is available only for x86_64.

---

This patch addresses x86-64 only; other architectures can/will be supported in the future. The __atomic intrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the __atomic_compare_exchange_16 (which itself is based on the C++11 memory model) to best support weak consistency architectures.

This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not on the entire non-blocking stack patchset.

[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123653.html

v5:
- Move declaration in the generic file for doxygen only (Thomas)

v4:
- Move function declaration from generic/rte_atomic.h to x86-64 header file

v3:
- Rename function to ISA-neutral rte_atomic128_cmp_exchange()
- Fix two pseudocode bugs in function documentation

v2:
- Rename function to rte_atomic128_cmpxchg()
- Replace "=A" output constraint with "=a" and "=d" to prevent GCC from using the al register for the sete destination
- Extend 'weak' definition to allow non-atomic 'exp' updates.
- Add const keyword to 'src' and remove volatile keyword from 'dst'
- Put __int128 in a union in rte_int128_t and move the structure definition inside the RTE_ARCH_x86_64 ifdef
- Drop enum rte_atomic_memmodel_t in favor of compiler-defined __ATOMIC_*
- Drop unnecessary comment relating to X86_64
- Tweak the pseudocode to reflect the 'exp' update on failure.

---
 .../common/include/arch/x86/rte_atomic_64.h   | 47 ++++++++++++++++++
 .../common/include/generic/rte_atomic.h       | 48 +++++++++++++++++++
 2 files changed, 95 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +47 -0
index fd2ec9c53..4b8315926 100644
@@ -34,6 +34,7 @@
 /*
  * Inspired from FreeBSD src/sys/amd64/include/atomic.h
  * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
  * All rights reserved.
  */
 
@@ -46,6 +47,7 @@
 
 #include <stdint.h>
 #include <rte_common.h>
+#include <rte_compat.h>
 #include <rte_atomic.h>
 
 /*------------------------- 64 bit atomic operations -------------------------*/
@@ -208,4 +210,49 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v)
 }
 #endif
 
+/*------------------------ 128 bit atomic operations -------------------------*/
+
+/**
+ * 128-bit integer structure.
+ */
+RTE_STD_C11
+typedef struct {
+	RTE_STD_C11
+	union {
+		uint64_t val[2];
+		__int128 int128;
+	};
+} __rte_aligned(16) rte_int128_t;
+
+static inline int __rte_experimental
+rte_atomic128_cmp_exchange(rte_int128_t *dst,
+			   rte_int128_t *exp,
+			   const rte_int128_t *src,
+			   unsigned int weak,
+			   int success,
+			   int failure)
+{
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=a" (exp->val[0]),
+			"=d" (exp->val[1]),
+			[res] "=r" (res)
+		      : "b" (src->val[0]),
+			"c" (src->val[1]),
+			"a" (exp->val[0]),
+			"d" (exp->val[1]),
+			"m" (dst->val[0])
+		      : "memory");
+
+	return res;
+}
+
 #endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +48 -0

52 lines not on this page · raw message

Signature · 1 line
2.21.0
author
#26 lore raw

PATCH v6 1/1 eal/x86: add 128-bit atomic compare exchange

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

It is available only for x86_64.

---
This patch addresses x86-64 only; other architectures can/will be supported in the future. The __atomic intrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the __atomic_compare_exchange_16 (which itself is based on the C++11 memory model) to best support weak consistency architectures.

This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not on the entire non-blocking stack patchset.

[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123653.html

v6:
- Use @code and @endcode to clean up pseudo-code generation
- Add note that the function is currently limited to x86-64

v5:
- Move declaration in the generic file for doxygen only (Thomas)

v4:
- Move function declaration from generic/rte_atomic.h to x86-64 header file

v3:
- Rename function to ISA-neutral rte_atomic128_cmp_exchange()
- Fix two pseudocode bugs in function documentation

v2:
- Rename function to rte_atomic128_cmpxchg()
- Replace "=A" output constraint with "=a" and "=d" to prevent GCC from using the al register for the sete destination
- Extend 'weak' definition to allow non-atomic 'exp' updates.
- Add const keyword to 'src' and remove volatile keyword from 'dst'
- Put __int128 in a union in rte_int128_t and move the structure definition inside the RTE_ARCH_x86_64 ifdef
- Drop enum rte_atomic_memmodel_t in favor of compiler-defined __ATOMIC_*
- Drop unnecessary comment relating to X86_64
- Tweak the pseudocode to reflect the 'exp' update on failure.

 .../common/include/arch/x86/rte_atomic_64.h        | 47 +++++++++++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 52 ++++++++++++++++++++++
 2 files changed, 99 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +47 -0
index fd2ec9c53..4b8315926 100644
@@ -34,6 +34,7 @@
 /*
  * Inspired from FreeBSD src/sys/amd64/include/atomic.h
  * Copyright (c) 1998 Doug Rabson
+ * Copyright (c) 2019 Intel Corporation
  * All rights reserved.
  */
 
@@ -46,6 +47,7 @@
 
 #include <stdint.h>
 #include <rte_common.h>
+#include <rte_compat.h>
 #include <rte_atomic.h>
 
 /*------------------------- 64 bit atomic operations -------------------------*/
@@ -208,4 +210,49 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v)
 }
 #endif
 
+/*------------------------ 128 bit atomic operations -------------------------*/
+
+/**
+ * 128-bit integer structure.
+ */
+RTE_STD_C11
+typedef struct {
+	RTE_STD_C11
+	union {
+		uint64_t val[2];
+		__int128 int128;
+	};
+} __rte_aligned(16) rte_int128_t;
+
+static inline int __rte_experimental
+rte_atomic128_cmp_exchange(rte_int128_t *dst,
+			   rte_int128_t *exp,
+			   const rte_int128_t *src,
+			   unsigned int weak,
+			   int success,
+			   int failure)
+{
+	RTE_SET_USED(weak);
+	RTE_SET_USED(success);
+	RTE_SET_USED(failure);
+	uint8_t res;
+
+	asm volatile (
+		      MPLOCKED
+		      "cmpxchg16b %[dst];"
+		      " sete %[res]"
+		      : [dst] "=m" (dst->val[0]),
+			"=a" (exp->val[0]),
+			"=d" (exp->val[1]),
+			[res] "=r" (res)
+		      : "b" (src->val[0]),
+			"c" (src->val[1]),
+			"a" (exp->val[0]),
+			"d" (exp->val[1]),
+			"m" (dst->val[0])
+		      : "memory");
+
+	return res;
+}
+
 #endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +52 -0

56 lines not on this page · raw message

Signature · 1 line
2.13.6
#27 lore raw

Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange

03/04/2019 21:44, Gage Eads:

Quoted 49 lines

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

It is available only for x86_64.

---
This patch addresses x86-64 only; other architectures can/will be supported in the future. The __atomic intrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the __atomic_compare_exchange_16 (which itself is based on the C++11 memory model) to best support weak consistency architectures.

This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not on the entire non-blocking stack patchset.

[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123653.html

v6:
- Use @code and @endcode to clean up pseudo-code generation
- Add note that the function is currently limited to x86-64

v5:
- Move declaration in the generic file for doxygen only (Thomas)

v4:
- Move function declaration from generic/rte_atomic.h to x86-64 header file

v3:
- Rename function to ISA-neutral rte_atomic128_cmp_exchange()
- Fix two pseudocode bugs in function documentation

v2:
- Rename function to rte_atomic128_cmpxchg()
- Replace "=A" output constraint with "=a" and "=d" to prevent GCC from using the al register for the sete destination
- Extend 'weak' definition to allow non-atomic 'exp' updates.
- Add const keyword to 'src' and remove volatile keyword from 'dst'
- Put __int128 in a union in rte_int128_t and move the structure definition inside the RTE_ARCH_x86_64 ifdef
- Drop enum rte_atomic_memmodel_t in favor of compiler-defined __ATOMIC_*
- Drop unnecessary comment relating to X86_64
- Tweak the pseudocode to reflect the 'exp' update on failure.

Quoted 3 lines
 .../common/include/arch/x86/rte_atomic_64.h        | 47 +++++++++++++++++++
 lib/librte_eal/common/include/generic/rte_atomic.h | 52 ++++++++++++++++++++++
 2 files changed, 99 insertions(+)

Applied, thanks

#28 lore raw

Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange

In reply to Gage Eads
Quoted 37 lines · Gage Eads

This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.

It is available only for x86_64.

---
This patch addresses x86-64 only; other architectures can/will be supported in the future. The __atomic intrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the __atomic_compare_exchange_16 (which itself is based on the C++11 memory model) to best support weak consistency architectures.

This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not on the entire non-blocking stack patchset.

[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123653.html

v6:
- Use @code and @endcode to clean up pseudo-code generation
- Add note that the function is currently limited to x86-64

v5:
- Move declaration in the generic file for doxygen only (Thomas)

v4:
- Move function declaration from generic/rte_atomic.h to x86-64 header file

v3:
- Rename function to ISA-neutral rte_atomic128_cmp_exchange()
- Fix two pseudocode bugs in function documentation

Quoted 12 lines

v2:
- Rename function to rte_atomic128_cmpxchg()
- Replace "=A" output constraint with "=a" and "=d" to prevent GCC from using the al register for the sete destination
- Extend 'weak' definition to allow non-atomic 'exp' updates.
- Add const keyword to 'src' and remove volatile keyword from 'dst'
- Put __int128 in a union in rte_int128_t and move the structure definition inside the RTE_ARCH_x86_64 ifdef
- Drop enum rte_atomic_memmodel_t in favor of compiler-defined __ATOMIC_*
- Drop unnecessary comment relating to X86_64
- Tweak the pseudocode to reflect the 'exp' update on failure.

<...>

I am getting a build error, I guess generated from this patch, while building mlx drivers, when MLX._DEBUG enabled [1].

This looks because of the "-pedantic" parameter mlx drivers have [2].

Gage, Shahaf, Matan, Yongseok,

Can you please check this?

[1] .../dpdk/x86_64-native-linuxapp-gcc/include/rte_atomic_64.h:223:3: error: ISO C does not support ‘__int128’ types [-Werror=pedantic]

__int128 int128;

^~~~~~~~

cc1: all warnings being treated as errors

[2]
Indeed I would like to discuss this distinct behavior of the mlx drivers, it is many time "-pedantic" caused problem, I will bring the discussion on separate thread.

#30 lore raw

Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange

04/04/2019 14:08, Thomas Monjalon:

Quoted 6 lines

04/04/2019 13:47, Ferruh Yigit:

Quoted 2 lines

.../dpdk/x86_64-native-linuxapp-gcc/include/rte_atomic_64.h:223:3: error: ISO C does not support ‘__int128’ types [-Werror=pedantic]

We can try this kind of workaround (disable pedantic locally):
https://github.com/HowardHinnant/date/pull/38/commits/177032852d5b4614112ca1ab3ef42d6b41824816

Or better:
__extension__ typedef __int128 int128;

author
#31 lore raw

Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange

Quoted 23 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Thursday, April 4, 2019 7:13 AM
To: Yigit, Ferruh <[email protected]>; Eads, Gage <[email protected]>
Cc: Shahaf Shuler <[email protected]>; Matan Azrad <[email protected]>; Yongseok Koh <[email protected]>; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v6] eal/x86: add 128-bit atomic compare exchange

04/04/2019 14:08, Thomas Monjalon:

Quoted 4 lines

04/04/2019 13:47, Ferruh Yigit:

Quoted 2 lines

.../dpdk/x86_64-native-linuxapp-gcc/include/rte_atomic_64.h:223:3:
error: ISO C does not support ‘__int128’ types [-Werror=pedantic]

Quoted 9 lines
Quoted 2 lines

We can try this kind of workaround (disable pedantic locally):

https://github.com/HowardHinnant/date/pull/38/commits/177032852d5b46
14

Quoted 1 line

112ca1ab3ef42d6b41824816

Or better:
__extension__ typedef __int128 int128;

Taking that one step further -- RTE_STD_C11 evaluates to __extension__ (when the STD C version is sufficiently old).

#32 lore raw

Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange

04/04/2019 14:14, Eads, Gage:

Quoted 10 lines

From: Thomas Monjalon [mailto:[email protected]]

Quoted 9 lines

04/04/2019 14:08, Thomas Monjalon:

Quoted 6 lines

04/04/2019 13:47, Ferruh Yigit:

Quoted 2 lines

.../dpdk/x86_64-native-linuxapp-gcc/include/rte_atomic_64.h:223:3:
error: ISO C does not support ‘__int128’ types [-Werror=pedantic]

We can try this kind of workaround (disable pedantic locally):

https://github.com/HowardHinnant/date/pull/38/commits/177032852d5b46
14

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

112ca1ab3ef42d6b41824816

Or better:
__extension__ typedef __int128 int128;

Taking that one step further -- RTE_STD_C11 evaluates to __extension__ (when the STD C version is sufficiently old).

I don't think __int128 is part of C11. Is it?

Ferruh, I cannot reproduce the compiler error. What is your compiler? Please, could you test this patch?

lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1 -1
@@ -220,7 +220,7 @@ typedef struct {
        RTE_STD_C11
        union {
                uint64_t val[2];
-               __int128 int128;
+               __extension__ __int128 int128;
        };
 } __rte_aligned(16) rte_int128_t;
author
#33 lore raw

Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange

Quoted 30 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Thursday, April 4, 2019 7:18 AM
To: Eads, Gage <[email protected]>; Yigit, Ferruh <[email protected]>
Cc: Shahaf Shuler <[email protected]>; Matan Azrad <[email protected]>; Yongseok Koh <[email protected]>; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v6] eal/x86: add 128-bit atomic compare exchange

04/04/2019 14:14, Eads, Gage:

Quoted 9 lines

From: Thomas Monjalon [mailto:[email protected]]

Quoted 8 lines

04/04/2019 14:08, Thomas Monjalon:

Quoted 6 lines

04/04/2019 13:47, Ferruh Yigit:

Quoted 2 lines

.../dpdk/x86_64-native-linuxapp-gcc/include/rte_atomic_64.h:223:3:
error: ISO C does not support ‘__int128’ types [-Werror=pedantic]

We can try this kind of workaround (disable pedantic locally):

https://github.com/HowardHinnant/date/pull/38/commits/177032852d5b46

Quoted 3 lines
Quoted 3 lines

14

Quoted 1 line

112ca1ab3ef42d6b41824816

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Or better:
__extension__ typedef __int128 int128;

Taking that one step further -- RTE_STD_C11 evaluates to __extension__

(when the STD C version is sufficiently old).

I don't think __int128 is part of C11. Is it?

You're right, this is a compiler extension. Using '__extension__' makes more sense.

author
#34 lore raw

Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange

In reply to Thomas Monjalon
Quoted 56 lines

-----Original Message-----
From: Eads, Gage
Sent: Thursday, April 4, 2019 7:23 AM
To: 'Thomas Monjalon' <[email protected]>; Yigit, Ferruh <[email protected]>
Cc: Shahaf Shuler <[email protected]>; Matan Azrad <[email protected]>; Yongseok Koh <[email protected]>; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [dpdk-dev] [PATCH v6] eal/x86: add 128-bit atomic compare exchange

Quoted 7 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Thursday, April 4, 2019 7:18 AM
To: Eads, Gage <[email protected]>; Yigit, Ferruh <[email protected]>
Cc: Shahaf Shuler <[email protected]>; Matan Azrad <[email protected]>; Yongseok Koh <[email protected]>;

[email protected];

Quoted 4 lines

[email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 9 lines

[email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v6] eal/x86: add 128-bit atomic compare exchange

04/04/2019 14:14, Eads, Gage:

Quoted 4 lines

From: Thomas Monjalon [mailto:[email protected]]

Quoted 3 lines

04/04/2019 14:08, Thomas Monjalon:

Quoted 2 lines

04/04/2019 13:47, Ferruh Yigit:

Quoted 1 line

.../dpdk/x86_64-native-linuxapp-

gcc/include/rte_atomic_64.h:223:3:

Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

error: ISO C does not support ‘__int128’ types
[-Werror=pedantic]

We can try this kind of workaround (disable pedantic locally):

https://github.com/HowardHinnant/date/pull/38/commits/177032852d5b46

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

14

Quoted 1 line

112ca1ab3ef42d6b41824816

Or better:
__extension__ typedef __int128 int128;

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Taking that one step further -- RTE_STD_C11 evaluates to __extension__

(when the STD C version is sufficiently old).

I don't think __int128 is part of C11. Is it?

You're right, this is a compiler extension. Using '__extension__' makes more sense.

FYI, __int128 was added in GCC 4.6 (and DPDK's minimum version is currently 4.9).

#35 lore raw

Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange

In reply to Thomas Monjalon
Quoted 17 lines · Thomas Monjalon

04/04/2019 14:14, Eads, Gage:

Quoted 16 lines

From: Thomas Monjalon [mailto:[email protected]]

Quoted 14 lines

04/04/2019 14:08, Thomas Monjalon:

Quoted 6 lines

04/04/2019 13:47, Ferruh Yigit:

Quoted 2 lines

.../dpdk/x86_64-native-linuxapp-gcc/include/rte_atomic_64.h:223:3:
error: ISO C does not support ‘__int128’ types [-Werror=pedantic]

We can try this kind of workaround (disable pedantic locally):

https://github.com/HowardHinnant/date/pull/38/commits/177032852d5b46
14

Quoted 1 line

112ca1ab3ef42d6b41824816

Or better:
__extension__ typedef __int128 int128;

Quoted 6 lines
Quoted 1 line

Taking that one step further -- RTE_STD_C11 evaluates to __extension__ (when the STD C version is sufficiently old).

I don't think __int128 is part of C11. Is it?

Ferruh, I cannot reproduce the compiler error. What is your compiler? Please, could you test this patch?

It is gcc [1]. Sure will test it.

[1]
gcc (GCC) 8.3.1 20190223 (Red Hat 8.3.1-2)

Quoted 14 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1 -1
@@ -220,7 +220,7 @@ typedef struct {
        RTE_STD_C11
        union {
                uint64_t val[2];
-               __int128 int128;
+               __extension__ __int128 int128;
        };
 } __rte_aligned(16) rte_int128_t;