Add 128-bit compare and set
cover Add 128-bit compare and set
This patch addresses x86-64 only; other architectures can/will be supported in the future. The __atomic intrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the __atomic_compare_exchange_16 (which itself is based on the C++11 memory model) to best support weak consistency architectures.
This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not the entire non-blocking stack patchset.
[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[1] http://mails.dpdk.org/archives/dev/2019-January/123653.html
Gage Eads (1):
eal: add 128-bit cmpset (x86-64 only)
.../common/include/arch/x86/rte_atomic_64.h | 31 +++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 65 ++++++++++++++++++++++ 2 files changed, 96 insertions(+)
Signature · 1 line
2.13.6
PATCH 1/1 eal: add 128-bit cmpset (x86-64 only)
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
--- .../common/include/arch/x86/rte_atomic_64.h | 31 +++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 65 ++++++++++++++++++++++ 2 files changed, 96 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +31 -0
@@ -34,6 +34,7 @@/* * Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */
@@ -46,6 +47,7 @@#include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations -------------------------*/
@@ -208,4 +210,33 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v) }#endif +static inline int __rte_experimental +rte_atomic128_cmpset (volatile rte_int128_t *dst, +rte_int128_t *exp,rte_int128_t *src, +unsigned int weak, +enum rte_atomic_memmodel_t success, +enum rte_atomic_memmodel_t failure) +{ +RTE_SET_USED (weak); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( +MPLOCKED +"cmpxchg16b %[dst];" +" sete %[res]" + : [dst]"=m" (dst->val[0 ]), +"=A" (exp->val[0 ]), + [res]"=r" (res) + :"c" (src->val[1 ]), +"b" (src->val[0 ]), +"m" (dst->val[0 ]), +"d" (exp->val[1 ]), +"a" (exp->val[0 ]) + :"memory" ); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +65 -0
75 lines not on this page · raw message
Signature · 1 line
2.13.6
Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)
Quoted 35 lines · Gage Eads
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
Signed-off-by Gage Eads <[email protected]>---
.../common/include/arch/x86/rte_atomic_64.h | 31 +++++++++++lib/librte_eal/common/include/generic/rte_atomic.h| 65
++++++++++++++++++++++2 files changed, 96 insertions(+)lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1
-0@@ -34,6 +34,7 @@ /** Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */ lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +0
-0@@ -46,6 +47,7 @@
#include <stdint.h>
#include <rte_common.h>
+#include <rte_compat.h>
#include <rte_atomic.h>
/*------------------------- 64 bit atomic operations ------------------------lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +0
-1-*/ @@ -208,4 +210,33 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v) } #endif
Quoted 2 lines
+static inline int__rte_experimental
__rte_always_inline?
Quoted 1 line
+rte_atomic128_cmpset(volatile
rte_int128_t*dst,
No need to declare the location volatile. Volatile doesn't do what you think it does.
https://youtu.be/lkgszkPnV8g?t=1027
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
Quoted 1 line
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1
-0+ rte_int128_t *exp,
I would declare 'exp' const as well and document that 'exp' is not updated (with the old value) for a failure. The reason being that ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of the critical section) using e.g. non-atomic 64-bit reads so there isn't actually any requirement for an atomic 128-bit read of the location.
Quoted 1 line
rte_int128_t*src,
const rte_int128_t *src?
But why are we not passing 'exp' and 'src' by value? That works great, even with structs. Passing by value simplifies the compiler's life, especially if the call is inlined. Ask a compiler developer.
Quoted 45 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +26
-0+ unsigned int weak, + enum rte_atomic_memmodel_t success, + enum rte_atomic_memmodel_t failure) +{ +RTE_SET_USED (weak); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( + MPLOCKED + "cmpxchg16b %[dst];" + " sete %[res]" + : [dst] "=m" (dst->val[0]), +"=A" (exp->val[0 ]), + [res]"=r" (res) + : "c" (src->val[1]), +"b" (src->val[0 ]), +"m" (dst->val[0 ]), +"d" (exp->val[1 ]), +"a" (exp->val[0 ]) + : "memory"); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +0
-0@@ -14,6 +14,7 @@
#include <stdint.h>
#include <rte_common.h>
+#include <rte_compat.h>
#ifdef__DOXYGEN__lib/librte_eal/common/include/generic/rte_atomic.h +0
-0@@ -1082,4 +1083,68 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v) } #endif
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093
Quoted 9 lines
lib/librte_eal/common/include/generic/rte_atomic.h +8
-1+/*------------------------ 128 bit atomic operations ------------------------ -*/ + +/** +* 128-bit integer structure. +*/ +typedef struct { +uint64_t val[2 ]; +} __rte_aligned(16) rte_int128_t;
So we can't use __int128?
Quoted 58 lines
lib/librte_eal/common/include/generic/rte_atomic.h +37
-0+ +/** +* Memory consistency models used in atomic operations. These control the +* behavior of the operation with respect to memory barriers and +* thread synchronization. +* +* These directly match those in the C++11 standard; for details on their +* behavior, refer to the standard. +*/ +enum rte_atomic_memmodel_t { +RTE_ATOMIC_RELAXED , +RTE_ATOMIC_CONSUME , +RTE_ATOMIC_ACQUIRE , +RTE_ATOMIC_RELEASE , +RTE_ATOMIC_ACQ_REL , +RTE_ATOMIC_SEQ_CST , +}; + +/* Only implemented on x86-64 currently. The ifdef prevents compilation from +* failing for architectures without a definition of this function. +*/ +#if defined(RTE_ARCH_X86_64 ) +/** +* An atomic compare and set function used by the mutex functions. +* (atomic) equivalent to: +* if (*dst == exp) +* *dst = src (all 128-bit words) +* +* @param dst +* The destination into which the value will be written. +* @param exp +* Pointer to the expected value. If the operation fails, this memory is +* updated with the actual value. +* @param src +* Pointer to the new value. +* @param weak +* A value of true allows the comparison to spuriously fail. Implementations
+ * may ignore this argument and only implement the strong variant.
+ * @param success
+ * If successful, the operation's memory behavior conforms to this (or a
+ * stronger) model.
+ * @param failure
+ * If unsuccessful, the operation's memory behavior conforms to this (or a
+ * stronger) model. This argument cannot beRTE_ATOMIC_RELEASE,
+ * RTE_ATOMIC_ACQ_REL, or a stronger model than success.
+ * @return
+ * Non-zero on success; 0 on failure.
+ */lib/librte_eal/common/include/generic/rte_atomic.h +8
-0+static inline int __rte_experimental +rte_atomic128_cmpset (volatile rte_int128_t *dst, + rte_int128_t *exp, rte_int128_t *src, + unsigned int weak, + enum rte_atomic_memmodel_t success, + enum rte_atomic_memmodel_t failure); +#endif +#endif /* _RTE_ATOMIC_H_ */
Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm Phone +46706866373, Skype ola.liljedahl
Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)
Quoted 52 lines
-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Monday, January 28, 2019 5:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)Quoted 36 lines · Gage Eads
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
Signed-off-by Gage Eads <[email protected]>---
.../common/include/arch/x86/rte_atomic_64.h | 31 +++++++++++lib/librte_eal/common/include/generic/rte_atomic.h| 65
++++++++++++++++++++++2 files changed, 96 insertions(+)lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2
-1@@ -34,6 +34,7 @@ /** Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */ @@ -46,6 +47,7 @@ #include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations ------------------------ -*/ @@ -208,4 +210,33 @@ static inline void
rte_atomic64_clear(rte_atomic64_t *v)
}
#endif
Quoted 8 lines
Quoted 1 line
+static inline int
__rte_experimental
__rte_always_inline?Quoted 1 line
+rte_atomic128_cmpset(volatile
rte_int128_t*dst,No need to declare the location volatile. Volatile doesn't do what you think it does.
https://youtu.be/lkgszkPnV8g?t=1027
I made this volatile to match the existing rte_atomicN_cmpset definitions, which presumably have a good reason for using the keyword. Maintainers, any input here?
Quoted 10 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
Quoted 2 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1
-0+ rte_int128_t *exp,I would declare 'exp' const as well and document that 'exp' is not updated (with the old value) for a failure. The reason being that ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of the critical section) using e.g. non-atomic 64-bit reads so there isn't actually any requirement for an atomic 128-bit read of the location.
Will change in v2.
Quoted 2 lines
Quoted 1 line
rte_int128_t*src,const
rte_int128_t*src?
Sure, I don't see any harm in using const.
Quoted 4 lines
But why are we not passing 'exp' and 'src' by value? That works great, even with structs. Passing by value simplifies the compiler's life, especially if the call is inlined. Ask a compiler developer.
I ran objdump on the nb_stack code with both approaches, and pass-by-reference resulted in fewer overall x86_64 assembly ops.
PBV: 100 ops for push, 97 ops for pop
PBR: 92 ops for push, 84 ops for pop
(Using the in-progress v5 nb_stack code)
Another factor -- though much less compelling -- is that with pass-by-reference, the user can create a 16B structure and cast it to rte_int128_t when they call rte_atomic128_cmpset, whereas with pass-by-value they need to put that struct in a union with rte_int128_t.
Quoted 46 lines
Quoted 46 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +25
-0+ unsigned int weak, + enum rte_atomic_memmodel_t success, + enum rte_atomic_memmodel_t failure) { +RTE_SET_USED (weak); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( + MPLOCKED + "cmpxchg16b %[dst];" + " sete %[res]" + : [dst] "=m" (dst->val[0]), +"=A" (exp->val[0 ]), + [res]"=r" (res) + : "c" (src->val[1]), +"b" (src->val[0 ]), +"m" (dst->val[0 ]), +"d" (exp->val[1 ]), +"a" (exp->val[0 ]) + : "memory"); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +1
-0@@ -14,6 +14,7 @@ #include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #ifdef __DOXYGEN__@@ -1082,4 +1083,68 @@ static inline voidrte_atomic64_clear(rte_atomic64_t
*v)
}
#endif
Quoted 12 lines
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093
Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9
-1+/*------------------------ 128 bit atomic operations +------------------------ -*/ + +/** +* 128-bit integer structure. +*/ +typedef struct { +uint64_t val[2 ]; +} __rte_aligned(16) rte_int128_t;So we can't use
__int128?
I'll put it in a union with val[2], in case any implementations want to use it.
Thanks,
Gage
[snip]
Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)
Quoted 98 lines · Gage
Quoted 57 lines
-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Monday, January 28, 2019 5:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)Quoted 37 lines · Gage Eads
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
Signed-off-by Gage Eads <[email protected]>---
.../common/include/arch/x86/rte_atomic_64.h | 31 +++++++++++lib/librte_eal/common/include/generic/rte_atomic.h| 65
++++++++++++++++++++++2 files changed, 96 insertions(+)lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2
-1@@ -34,6 +34,7 @@ /** Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */ @@ -46,6 +47,7 @@ #include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations ------------------------ -*/ @@ -208,4 +210,33 @@ static inline void
rte_atomic64_clear(rte_atomic64_t *v)
}
#endifQuoted 1 line
+static inline int
__rte_experimental
__rte_always_inline?Quoted 1 line
Quoted 6 lines
Quoted 1 line
+rte_atomic128_cmpset(volatile
rte_int128_t*dst,No need to declare the location volatile. Volatile doesn't do what you think it
does.
https://youtu.be/lkgszkPnV8g?t=1027I made this volatile to match the existing
rte_atomicN_cmpsetdefinitions, which presumably have a good reason for using the keyword. Maintainers, any input here?Quoted 16 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
Quoted 4 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1
-0+ rte_int128_t *exp,I would declare 'exp' const as well and document that 'exp' is not updated (with
the old value) for a failure. The reason being that ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad
for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of
the critical section) using e.g. non-atomic 64-bit reads so there isn't actually any
requirement for an atomic 128-bit read of the location.Will change in v2.
Quoted 4 lines
Quoted 3 lines
rte_int128_t *src,
const
rte_int128_t*src?Sure, I don't see any harm in using const.
Quoted 7 lines
But why are we not passing 'exp' and 'src' by value? That works great, even with
structs. Passing by value simplifies the compiler's life, especially if the call is
inlined. Ask a compiler developer.
Quoted 4 lines
I ran objdump on the
nb_stackcode with both approaches, and pass-by-reference resulted in fewer overallx86_64assembly ops.
PBV: 100 ops for push, 97 ops for pop
PBR: 92 ops for push, 84 ops for pop
OK I have never checked x86_64 code generation... I have good experiences with ARM/AArch64, everything seems to be done using registers. I am surprised there is a difference.
Did a quick check with lfring, passing 'src' (third param) by reference and by value. No difference in code generation on x86_64.
But if you insist let's go with PBR.
Quoted 7 lines
(Using the in-progress v5
nb_stackcode)Another factor -- though much less compelling -- is that with pass-by- reference, the user can create a 16B structure and cast it to
rte_int128_twhen they callrte_atomic128_cmpset, whereas with pass-by-value they need to put that struct in a union withrte_int128_t.
Which is what I always do nowadays... Trying to use as few casts as possible and lie to the compiler as seldom as possible. But I can see the freedom provided by taking a pointer to something and cast it it rte_int128_t ptr in the call to rte_atomic128_cmpset().
Would prefer a name that is more similar to __atomic_compare_exchange(). E.g. rte_atomic128_compare_exchange() (or perhaps just rte_atomic128_cmpxchg)? All the rte_atomicXX_cmpset() functions do not take any memory order parameters. From an Arm perspective, we are not happy with that.
Quoted 61 lines
Quoted 49 lines
Quoted 49 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +25
-0+ unsigned int weak, + enum rte_atomic_memmodel_t success, + enum rte_atomic_memmodel_t failure) { +RTE_SET_USED (weak); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( + MPLOCKED + "cmpxchg16b %[dst];" + " sete %[res]" + : [dst] "=m" (dst->val[0]), +"=A" (exp->val[0 ]), + [res]"=r" (res) + : "c" (src->val[1]), +"b" (src->val[0 ]), +"m" (dst->val[0 ]), +"d" (exp->val[1 ]), +"a" (exp->val[0 ]) + : "memory"); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +1
-0@@ -14,6 +14,7 @@ #include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #ifdef __DOXYGEN__@@ -1082,4 +1083,68 @@ static inline voidrte_atomic64_clear(rte_atomic64_t
*v)
}
#endifQuoted 12 lines
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093
Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9
-1+/*------------------------ 128 bit atomic operations +------------------------ -*/ + +/** +* 128-bit integer structure. +*/ +typedef struct { +uint64_t val[2 ]; +} __rte_aligned(16) rte_int128_t;So we can't use
__int128?
Quoted 2 lines
I'll put it in a union with val[2], in case any implementations want to use it.
Thinking on this one more time, since the inline asm functions (e.g. for x86_64 cmpxchg16b and for AArch64 LDXP/STXP) anyway will use 64-bit registers, it makes most sense to make rte_int128_t a struct of 2x64b. The question is whether to use an array like above or a struct with two elements (which I normally do internally). Can you compare code generation with the following definition? typedef struct {
uint64_t lo, hi;
} __rte_aligned(16) rte_int128_t;
Quoted 5 lines
Thanks,
Gage[snip]
Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm Phone +46706866373, Skype ola.liljedahl
Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)
Quoted 135 lines
-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Friday, February 1, 2019 1:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; Gavin Hu (Arm Technology China) <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)Quoted 92 lines · Gage
Quoted 56 lines
-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Monday, January 28, 2019 5:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)Quoted 39 lines · Gage Eads
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
Signed-off-by Gage Eads <[email protected]>---
.../common/include/arch/x86/rte_atomic_64.h | 31
+++++++++++
lib/librte_eal/common/include/generic/rte_atomic.h | 65
++++++++++++++++++++++2 files changed, 96 insertions(+)diff --git
a/lib/librte_eal/common/include/arch/x86/rte_atomic_64.hb/lib/librte_eal/common/include/arch/x86/rte_atomic_64.hindexfd2ec9c53..b7b90b83e100644lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2
-1@@ -34,6 +34,7 @@ /** Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */ @@ -46,6 +47,7 @@ #include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations ------------------------ -*/ @@ -208,4 +210,33 @@ static inlinevoid
rte_atomic64_clear(rte_atomic64_t *v)
}
#endifQuoted 9 lines
Quoted 1 line
+static inline int
__rte_experimental
__rte_always_inline?Quoted 2 lines
+rte_atomic128_cmpset(volatile
rte_int128_t*dst,No need to declare the location volatile. Volatile doesn't do what you think it does.
https://youtu.be/lkgszkPnV8g?t=1027I made this volatile to match the existing
rte_atomicN_cmpsetdefinitions, which presumably have a good reason for using the keyword. Maintainers, any input here?Quoted 13 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
Quoted 4 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1
-0+ rte_int128_t *exp,I would declare 'exp' const as well and document that 'exp' is not updated (with the old value) for a failure. The reason being that ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of the critical section) using e.g. non-atomic 64-bit reads so there isn't actually any requirement for an atomic 128-bit read of the location.
Will change in v2.
Quoted 4 lines
Quoted 3 lines
rte_int128_t *src,
const
rte_int128_t*src?Sure, I don't see any harm in using const.
Quoted 2 lines
Quoted 7 lines
Quoted 3 lines
But why are we not passing 'exp' and 'src' by value? That works great, even with structs. Passing by value simplifies the compiler's life, especially if the call is inlined. Ask a compiler developer.
I ran objdump on the
nb_stackcode with both approaches, and pass-by-reference resulted in fewer overallx86_64assembly ops.
PBV: 100 ops for push, 97 ops for pop
PBR: 92 ops for push, 84 ops for popOK I have never checked
x86_64code generation... I have good experiences with ARM/AArch64, everything seems to be done using registers. I am surprised there is a difference.Did a quick check with lfring, passing 'src' (third param) by reference and by value. No difference in code generation on
x86_64.But if you insist let's go with PBR.
Quoted 7 lines
(Using the in-progress v5
nb_stackcode)Another factor -- though much less compelling -- is that with pass-by- reference, the user can create a 16B structure and cast it to
rte_int128_twhen they callrte_atomic128_cmpset, whereas with pass-by-value they need to put that struct in a union withrte_int128_t.Which is what I always do nowadays... Trying to use as few casts as possible and lie to the compiler as seldom as possible. But I can see the freedom provided by taking a pointer to something and cast it it
rte_int128_tptr in the call to rte_atomic128_cmpset().
Quoted 6 lines
Would prefer a name that is more similar to
__atomic_compare_exchange(). E.g.rte_atomic128_compare_exchange()(or perhaps justrte_atomic128_cmpxchg)? All therte_atomicXX_cmpset()functions do not take any memory order parameters.
From an Arm perspective, we are not happy with that.
Since the function returns a boolean success value, isn't compare-and-set the appropriate term?
Quoted 64 lines
Quoted 62 lines
Quoted 50 lines
Quoted 50 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +25
-0+ unsigned int weak, + enum rte_atomic_memmodel_t success, + enum rte_atomic_memmodel_t failure) { +RTE_SET_USED (weak); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( + MPLOCKED + "cmpxchg16b %[dst];" + " sete %[res]" + : [dst] "=m" (dst->val[0]), +"=A" (exp->val[0 ]), + [res]"=r" (res) + : "c" (src->val[1]), +"b" (src->val[0 ]), +"m" (dst->val[0 ]), +"d" (exp->val[1 ]), +"a" (exp->val[0 ]) + : "memory"); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */ diff --git
a/lib/librte_eal/common/include/generic/rte_atomic.hb/lib/librte_eal/common/include/generic/rte_atomic.hindexb99ba4688..8d612d566100644lib/librte_eal/common/include/generic/rte_atomic.h +1
-0@@ -14,6 +14,7 @@ #include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #ifdef __DOXYGEN__@@ -1082,4 +1083,68 @@ static inline voidrte_atomic64_clear(rte_atomic64_t
*v)
}
#endifQuoted 12 lines
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093
Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9
-1+/*------------------------ 128 bit atomic operations +------------------------ -*/ + +/** +* 128-bit integer structure. +*/ +typedef struct { +uint64_t val[2 ]; +} __rte_aligned(16) rte_int128_t;So we can't use
__int128?Quoted 2 lines
I'll put it in a union with val[2], in case any implementations want to use it.
Quoted 10 lines
Thinking on this one more time, since the inline asm functions (e.g. for
x86_64cmpxchg16b and for AArch64 LDXP/STXP) anyway will use 64-bit registers, it makes most sense to makerte_int128_ta struct of 2x64b. The question is whether to use an array like above or a struct with two elements (which I normally do internally). Can you compare code generation with the following definition?
typedef struct {
uint64_tlo, hi;
}__rte_aligned(16)rte_int128_t;
Interestingly, that made no difference in the PBV code but added more instructions overall to PBR:
PBV: 100 insts for push, 97 insts for pop
PBR: 100 insts for push, 83 insts for pop
Quoted 8 lines
Quoted 5 lines
Thanks,
Gage[snip]
--
Ola Liljedahl, Networking System Architect, Arm Phone +46706866373, Skype ola.liljedahl
Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)
Quoted 149 lines · Gage
Quoted 149 lines
-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Friday, February 1, 2019 1:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; Gavin Hu (Arm Technology China) <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)Quoted 101 lines · Gage
Quoted 64 lines
-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Monday, January 28, 2019 5:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)Quoted 40 lines · Gage Eads
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
Signed-off-by Gage Eads <[email protected]>---
.../common/include/arch/x86/rte_atomic_64.h | 31
+++++++++++
lib/librte_eal/common/include/generic/rte_atomic.h | 65
++++++++++++++++++++++2 files changed, 96 insertions(+)diff --git
a/lib/librte_eal/common/include/arch/x86/rte_atomic_64.hb/lib/librte_eal/common/include/arch/x86/rte_atomic_64.hindexfd2ec9c53..b7b90b83e100644lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2
-1@@ -34,6 +34,7 @@ /** Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */ @@ -46,6 +47,7 @@ #include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations ------------------------ -*/ @@ -208,4 +210,33 @@ static inlinevoid
rte_atomic64_clear(rte_atomic64_t *v)
}
#endifQuoted 1 line
+static inline int
__rte_experimental
__rte_always_inline?Quoted 2 lines
Quoted 5 lines
Quoted 1 line
+rte_atomic128_cmpset(volatile
rte_int128_t*dst,No need to declare the location volatile. Volatile doesn't do what you think it does.
https://youtu.be/lkgszkPnV8g?t=1027I made this volatile to match the existing
rte_atomicN_cmpsetdefinitions, which presumably have a good reason for using the keyword. Maintainers, any input here?Quoted 5 lines
Quoted 5 lines
Quoted 10 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
Quoted 1 line
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1
-0+ rte_int128_t *exp,I would declare 'exp' const as well and document that 'exp' is not updated (with the old value) for a failure. The reason being that ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of the critical section) using e.g. non-atomic 64-bit reads so there isn't actually any requirement for an atomic 128-bit read of the location.
Will change in v2.
Quoted 6 lines
Quoted 5 lines
rte_int128_t *src,
const
rte_int128_t*src?Sure, I don't see any harm in using const.
Quoted 3 lines
Quoted 7 lines
Quoted 3 lines
But why are we not passing 'exp' and 'src' by value? That works great, even with structs. Passing by value simplifies the compiler's life, especially if the call is inlined. Ask a compiler developer.
I ran objdump on the
nb_stackcode with both approaches, and pass-by-reference resulted in fewer overallx86_64assembly ops.
PBV: 100 ops for push, 97 ops for pop
PBR: 92 ops for push, 84 ops for popOK I have never checked
x86_64code generation... I have good experiences with ARM/AArch64, everything seems to be done using registers. I am surprised
there is a difference.Did a quick check with lfring, passing 'src' (third param) by reference and by
value. No difference in code generation onx86_64.But if you insist let's go with PBR.
Quoted 8 lines
(Using the in-progress v5
nb_stackcode)Another factor -- though much less compelling -- is that with pass-by- reference, the user can create a 16B structure and cast it to
rte_int128_twhen they callrte_atomic128_cmpset, whereas with pass-by-value they need to put that struct in a union withrte_int128_t.Which is what I always do nowadays... Trying to use as few casts as possible and
lie to the compiler as seldom as possible. But I can see the freedom provided by
taking a pointer to something and cast it itrte_int128_tptr in the call to rte_atomic128_cmpset().
Quoted 8 lines
Quoted 6 lines
Would prefer a name that is more similar to
__atomic_compare_exchange(). E.g.rte_atomic128_compare_exchange()(or perhaps justrte_atomic128_cmpxchg)? All therte_atomicXX_cmpset()functions do not take any memory order parameters.
From an Arm perspective, we are not happy with that.Since the function returns a boolean success value, isn't compare-and-set the appropriate term?
I was thinking of the memory ordering parameters that __atomic_xxx builtins have and we want (from an Arm perspective).
GCC __atomic_compare_exchange also returns a boolean.
https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html
bool __atomic_compare_exchange_n (type *ptr, type *expected, type desired, bool weak, int success_memorder, int failure_memorder)
bool __atomic_compare_exchange (type *ptr, type *expected, type *desired, bool weak, int success_memorder, int failure_memorder)
rte_atomic128_compare_exchange(rte_int128_t *dst, const rte_int128_t *exp, const rte_int182_t *src, bool weak, int mo_success, int mo_failure);
Quoted 80 lines
Quoted 69 lines
Quoted 67 lines
Quoted 55 lines
Quoted 55 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +25
-0+ unsigned int weak, + enum rte_atomic_memmodel_t success, + enum rte_atomic_memmodel_t failure) { +RTE_SET_USED (weak); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( + MPLOCKED + "cmpxchg16b %[dst];" + " sete %[res]" + : [dst] "=m" (dst->val[0]), +"=A" (exp->val[0 ]), + [res]"=r" (res) + : "c" (src->val[1]), +"b" (src->val[0 ]), +"m" (dst->val[0 ]), +"d" (exp->val[1 ]), +"a" (exp->val[0 ]) + : "memory"); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */ diff --git
a/lib/librte_eal/common/include/generic/rte_atomic.hb/lib/librte_eal/common/include/generic/rte_atomic.hindexb99ba4688..8d612d566100644lib/librte_eal/common/include/generic/rte_atomic.h +1
-0@@ -14,6 +14,7 @@ #include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #ifdef __DOXYGEN__@@ -1082,4 +1083,68 @@ static inline voidrte_atomic64_clear(rte_atomic64_t
*v)
}
#endifQuoted 12 lines
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093
Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9
-1+/*------------------------ 128 bit atomic operations +------------------------ -*/ + +/** +* 128-bit integer structure. +*/ +typedef struct { +uint64_t val[2 ]; +} __rte_aligned(16) rte_int128_t;So we can't use
__int128?Quoted 2 lines
I'll put it in a union with val[2], in case any implementations want to use it.
Quoted 11 lines
Thinking on this one more time, since the inline asm functions (e.g. for
x86_64
cmpxchg16b and for AArch64 LDXP/STXP) anyway will use 64-bit registers, it makes most sense to makerte_int128_ta struct of 2x64b. The question is whether to use an array like above or a struct with two elements (which I normally do internally). Can you compare code generation with the following definition?
typedef struct {
uint64_tlo, hi;
}__rte_aligned(16)rte_int128_t;
Quoted 4 lines
Interestingly, that made no difference in the PBV code but added more instructions overall to PBR:
PBV: 100 insts for push, 97 insts for pop
PBR: 100 insts for push, 83 insts for pop
I think we learned something here... Trying to understand exactly what. But I think this result settles it.
I should test the different alternatives on Arm, does code generation behave the same as for x86_64.
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
Thanks,
Gage[snip]
--
Ola Liljedahl, Networking System Architect, Arm Phone +46706866373, Skype ola.liljedahl
Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm Phone +46706866373, Skype ola.liljedahl
Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)
Quoted 184 lines
-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Friday, February 1, 2019 1:44 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; Gavin Hu (Arm Technology China) <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)Quoted 138 lines · Gage
Quoted 138 lines
-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Friday, February 1, 2019 1:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; Gavin Hu (Arm Technology China) <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)Quoted 104 lines · Gage
Quoted 65 lines
-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Monday, January 28, 2019 5:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64
only)Quoted 40 lines · Gage Eads
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
Signed-off-by Gage Eads <[email protected]>---
.../common/include/arch/x86/rte_atomic_64.h | 31
+++++++++++
lib/librte_eal/common/include/generic/rte_atomic.h | 65
++++++++++++++++++++++2 files changed, 96 insertions(+)diff --git
a/lib/librte_eal/common/include/arch/x86/rte_atomic_64.hb/lib/librte_eal/common/include/arch/x86/rte_atomic_64.hindexfd2ec9c53..b7b90b83e100644lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2
-1@@ -34,6 +34,7 @@ /** Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */ @@ -46,6 +47,7 @@ #include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations ------------------------ -*/ @@ -208,4 +210,33 @@ staticinline void
rte_atomic64_clear(rte_atomic64_t *v)
}
#endifQuoted 1 line
+static inline int
__rte_experimental
__rte_always_inline?Quoted 2 lines
Quoted 5 lines
Quoted 1 line
+rte_atomic128_cmpset(volatile
rte_int128_t*dst,No need to declare the location volatile. Volatile doesn't do what you think it does.
https://youtu.be/lkgszkPnV8g?t=1027I made this volatile to match the existing
rte_atomicN_cmpsetdefinitions, which presumably have a good reason for using the keyword. Maintainers, any input here?Quoted 5 lines
Quoted 5 lines
Quoted 12 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
Quoted 1 line
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1
-0+ rte_int128_t *exp,I would declare 'exp' const as well and document that 'exp' is not updated (with the old value) for a failure. The reason being that
ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of the critical
section) using e.g. non-atomic 64-bit reads so there isn't actually any requirement for an atomic 128-bit read of the location.Will change in v2.
Quoted 6 lines
Quoted 5 lines
rte_int128_t *src,
const
rte_int128_t*src?Sure, I don't see any harm in using const.
Quoted 3 lines
Quoted 7 lines
Quoted 3 lines
But why are we not passing 'exp' and 'src' by value? That works great, even with structs. Passing by value simplifies the compiler's life, especially if the call is inlined. Ask a compiler developer.
I ran objdump on the
nb_stackcode with both approaches, and pass-by-reference resulted in fewer overallx86_64assembly ops.
PBV: 100 ops for push, 97 ops for pop
PBR: 92 ops for push, 84 ops for popOK I have never checked
x86_64code generation... I have good experiences with ARM/AArch64, everything seems to be done using registers. I am surprised there is a difference.Did a quick check with lfring, passing 'src' (third param) by reference and by value. No difference in code generation on
x86_64.But if you insist let's go with PBR.
Quoted 2 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
(Using the in-progress v5
nb_stackcode)Another factor -- though much less compelling -- is that with pass-by- reference, the user can create a 16B structure and cast it to
rte_int128_twhen they callrte_atomic128_cmpset, whereas with pass-by-value they need to put that struct in a union with
rte_int128_t.Quoted 8 lines
Quoted 8 lines
Which is what I always do nowadays... Trying to use as few casts as possible and lie to the compiler as seldom as possible. But I can see the freedom provided by taking a pointer to something and cast it it
rte_int128_tptr in the call to rte_atomic128_cmpset().Would prefer a name that is more similar to
__atomic_compare_exchange(). E.g.rte_atomic128_compare_exchange()(or perhaps just
rte_atomic128_cmpxchg)?Quoted 5 lines
Quoted 3 lines
All the
rte_atomicXX_cmpset()functions do not take any memory order parameters.
From an Arm perspective, we are not happy with that.Since the function returns a boolean success value, isn't compare-and-set the appropriate term?
I was thinking of the memory ordering parameters that
__atomic_xxxbuiltins have and we want (from an Arm perspective).GCC
__atomic_compare_exchangealso returns a boolean.
https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html
bool__atomic_compare_exchange_n(type *ptr, type *expected, type desired, bool weak, intsuccess_memorder, intfailure_memorder) bool__atomic_compare_exchange(type *ptr, type *expected, type *desired, bool weak, intsuccess_memorder, intfailure_memorder)
Quoted 3 lines
rte_atomic128_compare_exchange(rte_int128_t *dst, const
rte_int128_t*exp, constrte_int182_t*src, bool weak, intmo_success, intmo_failure);
I assumed GCC chose the name "exchange" because the builtin modifies 'exp' on failure, but I could be wrong. I'm fine with either name. Anyone else have a preference?
<snip>
Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)
Quoted 201 lines · Gage
Quoted 201 lines
-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Friday, February 1, 2019 1:44 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; Gavin Hu (Arm Technology China) <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)Quoted 151 lines · Gage
Quoted 151 lines
-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Friday, February 1, 2019 1:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; Gavin Hu (Arm Technology China) <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)Quoted 113 lines · Gage
Quoted 69 lines
-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Monday, January 28, 2019 5:02 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/1] eal: add 128-bit cmpset (x86-64
only)Quoted 41 lines · Gage Eads
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
Signed-off-by Gage Eads <[email protected]>---
.../common/include/arch/x86/rte_atomic_64.h | 31
+++++++++++
lib/librte_eal/common/include/generic/rte_atomic.h | 65
++++++++++++++++++++++2 files changed, 96 insertions(+)diff --git
a/lib/librte_eal/common/include/arch/x86/rte_atomic_64.hb/lib/librte_eal/common/include/arch/x86/rte_atomic_64.hindexfd2ec9c53..b7b90b83e100644lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2
-1@@ -34,6 +34,7 @@ /** Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */ @@ -46,6 +47,7 @@ #include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations ------------------------ -*/ @@ -208,4 +210,33 @@ staticinline void
rte_atomic64_clear(rte_atomic64_t *v)
}
#endifQuoted 1 line
+static inline int
__rte_experimental
__rte_always_inline?Quoted 3 lines
Quoted 5 lines
Quoted 1 line
+rte_atomic128_cmpset(volatile
rte_int128_t*dst,No need to declare the location volatile. Volatile doesn't do what you think it does.
https://youtu.be/lkgszkPnV8g?t=1027I made this volatile to match the existing
rte_atomicN_cmpsetdefinitions, which presumably have a good reason for using the keyword. Maintainers, any input here?Quoted 7 lines
Quoted 7 lines
Quoted 12 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
Quoted 1 line
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1
-0+ rte_int128_t *exp,I would declare 'exp' const as well and document that 'exp' is not updated (with the old value) for a failure. The reason being that
ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of the critical
section) using e.g. non-atomic 64-bit reads so there isn't actually any requirement for an atomic 128-bit read of the location.Will change in v2.
Quoted 8 lines
Quoted 7 lines
rte_int128_t *src,
const
rte_int128_t*src?Sure, I don't see any harm in using const.
Quoted 4 lines
Quoted 8 lines
Quoted 4 lines
But why are we not passing 'exp' and 'src' by value? That works great, even with structs. Passing by value simplifies the compiler's life, especially if the call is inlined. Ask a compiler developer.
I ran objdump on the
nb_stackcode with both approaches, and pass-by-reference resulted in fewer overallx86_64assembly ops.
PBV: 100 ops for push, 97 ops for pop
PBR: 92 ops for push, 84 ops for popOK I have never checked
x86_64code generation... I have good experiences with ARM/AArch64, everything seems to be done using registers. I am surprised there is a difference.Did a quick check with lfring, passing 'src' (third param) by reference and by value. No difference in code generation on
x86_64.But if you insist let's go with PBR.
Quoted 3 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
(Using the in-progress v5
nb_stackcode)Another factor -- though much less compelling -- is that with pass-by- reference, the user can create a 16B structure and cast it to
rte_int128_twhen they callrte_atomic128_cmpset, whereas with pass-by-value they need to put that struct in a union with
rte_int128_t.Quoted 10 lines
Quoted 10 lines
Which is what I always do nowadays... Trying to use as few casts as possible and lie to the compiler as seldom as possible. But I can see the freedom provided by taking a pointer to something and cast it it
rte_int128_tptr in the call to rte_atomic128_cmpset().Would prefer a name that is more similar to
__atomic_compare_exchange(). E.g.rte_atomic128_compare_exchange()(or perhaps just
rte_atomic128_cmpxchg)?Quoted 7 lines
Quoted 5 lines
All the
rte_atomicXX_cmpset()functions do not take any memory order parameters.
From an Arm perspective, we are not happy with that.Since the function returns a boolean success value, isn't compare-and-set the appropriate term?
I was thinking of the memory ordering parameters that
__atomic_xxxbuiltins have and we want (from an Arm perspective).GCC
__atomic_compare_exchangealso returns a boolean.
https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html
bool__atomic_compare_exchange_n(type *ptr, type *expected, type desired, bool weak, intsuccess_memorder, intfailure_memorder) bool__atomic_compare_exchange(type *ptr, type *expected, type *desired, bool weak, intsuccess_memorder, intfailure_memorder)
Quoted 5 lines
Quoted 3 lines
rte_atomic128_compare_exchange(rte_int128_t *dst, const
rte_int128_t*exp, constrte_int182_t*src, bool weak, intmo_success, intmo_failure);I assumed GCC chose the name "exchange" because the builtin modifies 'exp' on failure, but I could be wrong.
You are probably right. But exchange doesn't necessary mean that the old value is returned, just that it is replaced with a different value? However __atomic_exchange() (unconditionally) returns the old value so there is some precedent for retunrning the old value. And it can be done, just not guaranteed to be atomic. Perhaps there is a way to both eat and have the cake?
Re-use the prototype of __atomic_compare_exchange_16() so must return a value if the comparison fails. But depending on the 'weak' flag, we for weak=false require an atomic read which on ARMv8.0 must be accomplished by writing back the old value (if the comparison fails). Or for weak=true we allow both spurious LL/SC failures (this is what the weak parameter is for) and non-atomic read of the old value (this is my 'frail' atomic compare and exchange in lfring).
The GCC documentation says this:
"If they are not equal, the operation is a read and the current contents of ptr are written into expected."
We follow this but just don't guarantee an atomic read (for weak=true).
bool
rte_atomic128_compare_exchange(rte_int128_t *dst, rte_int128_t *exp, //Note no const now const rte_int128_t *src, bool weak,
int mo_success,
int mo_failure);
Quoted 4 lines
I'm fine with either name. Anyone else have a preference?
<snip>
Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm Phone +46706866373, Skype ola.liljedahl
Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)
In reply to Eads, GageQuoted 60 lines
Quoted 38 lines · Gage Eads
Quoted 36 lines
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
Signed-off-by Gage Eads <[email protected]>---
.../common/include/arch/x86/rte_atomic_64.h | 31 +++++++++++lib/librte_eal/common/include/generic/rte_atomic.h| 65
++++++++++++++++++++++2 files changed, 96 insertions(+)lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2
-1@@ -34,6 +34,7 @@ /** Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */ @@ -46,6 +47,7 @@ #include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations ------------------------ -*/ @@ -208,4 +210,33 @@ static inline void
rte_atomic64_clear(rte_atomic64_t *v)
}
#endifQuoted 8 lines
Quoted 1 line
+static inline int
__rte_experimental
__rte_always_inline?Quoted 1 line
+rte_atomic128_cmpset(volatile
rte_int128_t*dst,No need to declare the location volatile. Volatile doesn't do what you think it does.
https://youtu.be/lkgszkPnV8g?t=1027I made this volatile to match the existing
rte_atomicN_cmpsetdefinitions, which presumably have a good reason for using the keyword. Maintainers, any input here?Quoted 9 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
Quoted 2 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1
-0+ rte_int128_t *exp,I would declare 'exp' const as well and document that 'exp' is not updated (with the old value) for a failure. The reason being that ARMv8.0/AArch64 cannot atomically read the old value without also writing the location and that is bad for performance (unnecessary writes leads to unnecessary contention and worse scalability). And the user must anyway read the location (in the start of the critical section) using e.g. non-atomic 64-bit reads so there isn't actually any
Quoted 5 lines
requirement for an atomic 128-bit read of the location.
Quoted 1 line
Will change in v2.
IMO, we should not change the definition of this API, because
1) This API will differ from __atomic_compare_exchange_n API. It will be a new API to learn for the users.
2) The definition in this patch will make it easy to replace this API call with __atomic_xxx API (whenever it supports 128b natively on all the platforms)
3) I do not see any documentation in [1] indicating that the 'read on failure' will be an atomic read.
[1] https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html
Quoted 87 lines
Quoted 2 lines
Quoted 1 line
rte_int128_t*src,const
rte_int128_t*src?Sure, I don't see any harm in using const.
Quoted 4 lines
But why are we not passing 'exp' and 'src' by value? That works great, even with structs. Passing by value simplifies the compiler's life, especially if the call is inlined. Ask a compiler developer.
I ran objdump on the
nb_stackcode with both approaches, and pass-by- reference resulted in fewer overallx86_64assembly ops.
PBV: 100 ops for push, 97 ops for pop
PBR: 92 ops for push, 84 ops for pop(Using the in-progress v5
nb_stackcode)Another factor -- though much less compelling -- is that with pass-by-reference, the user can create a 16B structure and cast it to
rte_int128_twhen they callrte_atomic128_cmpset, whereas with pass-by-value they need to put that struct in a union withrte_int128_t.Quoted 46 lines
Quoted 46 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +25
-0+ unsigned int weak, + enum rte_atomic_memmodel_t success, + enum rte_atomic_memmodel_t failure) { +RTE_SET_USED (weak); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( + MPLOCKED + "cmpxchg16b %[dst];" + " sete %[res]" + : [dst] "=m" (dst->val[0]), +"=A" (exp->val[0 ]), + [res]"=r" (res) + : "c" (src->val[1]), +"b" (src->val[0 ]), +"m" (dst->val[0 ]), +"d" (exp->val[1 ]), +"a" (exp->val[0 ]) + : "memory"); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +1
-0@@ -14,6 +14,7 @@ #include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #ifdef __DOXYGEN__@@ -1082,4 +1083,68 @@ static inline voidrte_atomic64_clear(rte_atomic64_t
*v)
}
#endifQuoted 12 lines
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093
Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9
-1+/*------------------------ 128 bit atomic operations +------------------------ -*/ + +/** +* 128-bit integer structure. +*/ +typedef struct { +uint64_t val[2 ]; +} __rte_aligned(16) rte_int128_t;So we can't use
__int128?I'll put it in a union with val[2], in case any implementations want to use it.
Thanks,
Gage[snip]
Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)
In reply to Gage EadsQuoted 35 lines
This operation can be used for non-blocking algorithms, such as a non- blocking stack or ring.
Signed-off-by Gage Eads <[email protected]>--- .../common/include/arch/x86/rte_atomic_64.h | 31 +++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 65 ++++++++++++++++++++++ 2 files changed, 96 insertions(+)lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2
-0@@ -34,6 +34,7 @@/* * Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */ @@ -46,6 +47,7 @@#include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations -------------------------*/ @@ -208,4 +210,33 @@ static inline void
rte_atomic64_clear(rte_atomic64_t *v)} #endif
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
Quoted 2 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2
-0+static inline int __rte_experimental +rte_atomic128_cmpset(volatile rte_int128_t *dst,
Does it make sense to call is rte_atomic128_compare_exchange (or ..._cmp_xchg) to indicate it is a compare-exchange operation?
Quoted 43 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +26
-0+ rte_int128_t *exp, rte_int128_t *src, + unsigned int weak, + enum rte_atomic_memmodel_t success, + enum rte_atomic_memmodel_t failure) { +RTE_SET_USED (weak ); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( +MPLOCKED +"cmpxchg16b %[dst];" +" sete %[res]" + : [dst]"=m" (dst->val[0 ]), +"=A" (exp->val[0 ]), + [res]"=r" (res) + :"c" (src->val[1 ]), +"b" (src->val[0 ]), +"m" (dst->val[0 ]), +"d" (exp->val[1 ]), +"a" (exp->val[0 ]) + :"memory" ); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */ lib/librte_eal/common/include/generic/rte_atomic.h +1
-0@@ -14,6 +14,7 @@#include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #ifdef __DOXYGEN__@@ -1082,4 +1083,68 @@ static inline void
rte_atomic64_clear(rte_atomic64_t *v)} #endif
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093
Quoted 9 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9
-0+/*------------------------ 128 bit atomic operations +-------------------------*/ + +/** +* 128-bit integer structure. +*/ +typedef struct { +uint64_t val[2 ]; +} __rte_aligned(16) rte_int128_t;
It looks like '__int128' is available from gcc 4.6. I think we should use '__int128'. We can have it as an internal structure for ease of programming.
Quoted 9 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9
-0+ +/** +* Memory consistency models used in atomic operations. These control +the +* behavior of the operation with respect to memory barriers and +* thread synchronization. +* +* These directly match those in the C++11 standard; for details on +their
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1110
Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +10
-0+* behavior, refer to the standard. +*/ +enum rte_atomic_memmodel_t { +RTE_ATOMIC_RELAXED , +RTE_ATOMIC_CONSUME , +RTE_ATOMIC_ACQUIRE , +RTE_ATOMIC_RELEASE , +RTE_ATOMIC_ACQ_REL , +RTE_ATOMIC_SEQ_CST , +};
IMO, we can use the GCC provided names. I do not see any advantage to defining our own.
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1114
Quoted 5 lines
lib/librte_eal/common/include/generic/rte_atomic.h +5
-0+ +/* Only implemented on x86-64 currently. The ifdef prevents compilation +from +* failing for architectures without a definition of this function. +*/
Minor comment. We can skip the above comments, the #if below is pretty obvious.
Quoted 38 lines
lib/librte_eal/common/include/generic/rte_atomic.h +16
-0+#if defined(RTE_ARCH_X86_64 ) +/** +* An atomic compare and set function used by the mutex functions. +* (atomic) equivalent to: +* if (*dst == exp) +* *dst = src (all 128-bit words) +* +* @param dst +* The destination into which the value will be written. +* @param exp +* Pointer to the expected value. If the operation fails, this memory is +* updated with the actual value. +* @param src +* Pointer to the new value. +* @param weak +* A value of true allows the comparison to spuriously fail. Implementations
+ * may ignore this argument and only implement the strong variant.
+ * @param success
+ * If successful, the operation's memory behavior conforms to this (or a
+ * stronger) model.
+ * @param failure+ * If unsuccessful, the operation's memory behavior conforms to this (or a + * stronger) model. This argument cannot be RTE_ATOMIC_RELEASE, + * RTE_ATOMIC_ACQ_REL, or a stronger model than success.+ * @return
+ * Non-zero on success; 0 on failure.
+ */lib/librte_eal/common/include/generic/rte_atomic.h +7
-1+static inline int __rte_experimental +rte_atomic128_cmpset (volatile rte_int128_t *dst, +rte_int128_t *exp,rte_int128_t *src, +unsigned int weak, +enum rte_atomic_memmodel_t success, +enum rte_atomic_memmodel_t failure);#endif +#endif /* _RTE_ATOMIC_H_ */ --2.13.6
Re: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)
Quoted 49 lines
-----Original Message-----
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, January 30, 2019 11:48 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; [email protected]; [email protected]; nd <[email protected]>
Subject: RE: [PATCH 1/1] eal: add 128-bit cmpset (x86-64 only)Quoted 36 lines
This operation can be used for non-blocking algorithms, such as a non- blocking stack or ring.
Signed-off-by Gage Eads <[email protected]>--- .../common/include/arch/x86/rte_atomic_64.h | 31 +++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 65 ++++++++++++++++++++++ 2 files changed, 96 insertions(+)lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2
-1@@ -34,6 +34,7 @@/* * Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */ @@ -46,6 +47,7 @@#include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations -------------------------*/ @@ -208,4 +210,33 @@ static inline void
rte_atomic64_clear(rte_atomic64_t *v)} #endif
Quoted 5 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h
Quoted 2 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2
-0+static inline int __rte_experimental +rte_atomic128_cmpset(volatile rte_int128_t *dst,Does it make sense to call is
rte_atomic128_compare_exchange(or ..._cmp_xchg) to indicate it is a compare-exchange operation?
Good point, though for v2 I'm planning to change this to true cmpset semantics (no update to exp on failure). See Ola's reply for the justification.
Quoted 43 lines
Quoted 43 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +26
-0+ rte_int128_t *exp, rte_int128_t *src, + unsigned int weak, + enum rte_atomic_memmodel_t success, + enum rte_atomic_memmodel_t failure) { +RTE_SET_USED (weak ); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( +MPLOCKED +"cmpxchg16b %[dst];" +" sete %[res]" + : [dst]"=m" (dst->val[0 ]), +"=A" (exp->val[0 ]), + [res]"=r" (res) + :"c" (src->val[1 ]), +"b" (src->val[0 ]), +"m" (dst->val[0 ]), +"d" (exp->val[1 ]), +"a" (exp->val[0 ]) + :"memory" ); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */ lib/librte_eal/common/include/generic/rte_atomic.h +1
-0@@ -14,6 +14,7 @@#include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #ifdef __DOXYGEN__@@ -1082,4 +1083,68 @@ static inline void
rte_atomic64_clear(rte_atomic64_t *v)} #endif
Quoted 12 lines
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1093
Quoted 9 lines
lib/librte_eal/common/include/generic/rte_atomic.h +9
-0+/*------------------------ 128 bit atomic operations +-------------------------*/ + +/** +* 128-bit integer structure. +*/ +typedef struct { +uint64_t val[2 ]; +} __rte_aligned(16) rte_int128_t;It looks like '
__int128' is available from gcc 4.6. I think we should use '__int128'. We can have it as an internal structure for ease of programming.
Will add in v2.
Quoted 13 lines
Quoted 13 lines
lib/librte_eal/common/include/generic/rte_atomic.h +13
-0+ +/** +* Memory consistency models used in atomic operations. These control +the +* behavior of the operation with respect to memory barriers and +* thread synchronization. +* +* These directly match those in the C++11 standard; for details on +their +* behavior, refer to the standard. +*/ +enum rte_atomic_memmodel_t { +RTE_ATOMIC_RELAXED ,
Quoted 9 lines
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1110
Quoted 6 lines
lib/librte_eal/common/include/generic/rte_atomic.h +6
-0+RTE_ATOMIC_CONSUME , +RTE_ATOMIC_ACQUIRE , +RTE_ATOMIC_RELEASE , +RTE_ATOMIC_ACQ_REL , + RTE_ATOMIC_SEQ_CST, +};IMO, we can use the GCC provided names. I do not see any advantage to defining our own.
Will change in v2. I was trying to avoid issues with GCC versions that don't have C++11 support, but DPDK's recommended minimum version (4.9) is later than the version that added __atomic builtins (4.7).
Quoted 8 lines
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1114
Quoted 5 lines
lib/librte_eal/common/include/generic/rte_atomic.h +5
-0+ +/* Only implemented on x86-64 currently. The ifdef prevents +compilation from +* failing for architectures without a definition of this function. +*/ Minor comment. We can skip the above comments, the #if below is pretty obvious.
Sure.
Thanks,
Gage
<snip>
v2 cover Add 128-bit compare and set
In reply to Gage EadsThis patch addresses x86-64 only; other architectures can/will be supported in the future. The __atomic intrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the __atomic_compare_exchange_16 (which itself is based on the C++11 memory model) to best support weak consistency architectures.
This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not on the entire non-blocking stack patchset.
[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123653.html
v2:
- Rename function rte_atomic128_cmpxchg()
- Replace "=A" output constraint with "=a" and "=d" to prevent GCC from using the al register as the sete destination
- Extend 'weak' definition to explicitly allow non-atomic 'exp' updates.
- Add const keyword to 'src' and remove volatile keyword from 'dst'
- Put __int128 in a union in rte_int128_t and move the structure definition inside the RTE_ARCH_x86_64 ifdef
- Drop enum rte_atomic_memmodel_t in favor of compiler-defined __ATOMIC_*
- Drop unnecessary comment relating to X86_64
- Tweak the pseudocode to reflect the 'exp' update on failure.
Gage Eads (1):
eal: add 128-bit cmpxchg (x86-64 only)
.../common/include/arch/x86/rte_atomic_64.h | 33 ++++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 59 ++++++++++++++++++++++ 2 files changed, 92 insertions(+)
Signature · 1 line
2.13.6
PATCH v2 1/1 eal: add 128-bit cmpxchg (x86-64 only)
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
--- .../common/include/arch/x86/rte_atomic_64.h | 33 ++++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 59 ++++++++++++++++++++++ 2 files changed, 92 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +33 -0
@@ -34,6 +34,7 @@/* * Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */
@@ -46,6 +47,7 @@#include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations -------------------------*/
@@ -208,4 +210,35 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v) }#endif +static inline int __rte_experimental +rte_atomic128_cmpxchg (rte_int128_t *dst, +rte_int128_t *exp, +const rte_int128_t *src, +unsigned int weak, +int success, +int failure) +{ +RTE_SET_USED (weak); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( +MPLOCKED +"cmpxchg16b %[dst];" +" sete %[res]" + : [dst]"=m" (dst->val[0 ]), +"=a" (exp->val[0 ]), +"=d" (exp->val[1 ]), + [res]"=r" (res) + :"b" (src->val[0 ]), +"c" (src->val[1 ]), +"a" (exp->val[0 ]), +"d" (exp->val[1 ]), +"m" (dst->val[0 ]) + :"memory" ); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +59 -0
69 lines not on this page · raw message
Signature · 1 line
2.13.6
Re: [PATCH v2 1/1] eal: add 128-bit cmpxchg (x86-64 only)
Quoted 92 lines
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
Signed-off-by Gage Eads <[email protected]>--- .../common/include/arch/x86/rte_atomic_64.h | 33 ++++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 59 ++++++++++++++++++++++ 2 files changed, 92 insertions(+)lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +2
-0@@ -34,6 +34,7 @@/* * Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */ @@ -46,6 +47,7 @@#include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations -------------------------*/ @@ -208,4 +210,35 @@ static inline void
rte_atomic64_clear(rte_atomic64_t *v)} #endiflib/librte_eal/common/include/arch/x86/rte_atomic_64.h +31
-0+static inline int __rte_experimental +rte_atomic128_cmpxchg (rte_int128_t *dst, +rte_int128_t *exp, +const rte_int128_t *src, +unsigned int weak, +int success, +int failure) +{ +RTE_SET_USED (weak); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( +MPLOCKED +"cmpxchg16b %[dst];" +" sete %[res]" + : [dst]"=m" (dst->val[0 ]), +"=a" (exp->val[0 ]), +"=d" (exp->val[1 ]), + [res]"=r" (res) + :"b" (src->val[0 ]), +"c" (src->val[1 ]), +"a" (exp->val[0 ]), +"d" (exp->val[1 ]), +"m" (dst->val[0 ]) + :"memory" ); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */ lib/librte_eal/common/include/generic/rte_atomic.h +1
-0@@ -14,6 +14,7 @@#include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #ifdef __DOXYGEN__@@ -1082,4 +1083,62 @@ static inline void
rte_atomic64_clear(rte_atomic64_t *v)} #endiflib/librte_eal/common/include/generic/rte_atomic.h +10
-0+/*------------------------ 128 bit atomic operations +-------------------------*/ + +#if defined(RTE_ARCH_X86_64 ) +/** +* 128-bit integer structure. +*/ +RTE_STD_C11 +typedef struct { +RTE_STD_C11
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1104
Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +10
-0+union { +uint64_t val[2 ]; +__int128 int128; + }; +} __rte_aligned(16) rte_int128_t; + +/** +* An atomic compare and set function used by the mutex functions. +* (Atomically) Equivalent to: +* if (*dst == exp)
Should be, "if (*dst == *exp)"
Quoted 1 line
+ * *dst = src
Should be "*dst = *src"
Quoted 23 lines
+ * else
+ * *exp = *dst
+ *
+ * @note The success and failure arguments must be one of thelib/librte_eal/common/include/generic/rte_atomic.h +19
-0+__ATOMIC_* values + * defined in the C++11 standard. For details on their behavior, refer +to the + * standard. + * + * @param dst + * The destination into which the value will be written. + * @param exp + * Pointer to the expected value. If the operation fails, this memory is + * updated with the actual value. + * @param src + * Pointer to the new value. + * @param weak + * A value of true allows the comparison to spuriously fail and allows the + * 'exp' update to occur non-atomically (i.e. a torn read may occur). + * Implementations may ignore this argument and only implement the strong + * variant. + * @param success + * If successful, the operation's memory behavior conforms to this (or a
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1136
Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +10
-0+* stronger) model. +* @param failure +* If unsuccessful, the operation's memory behavior conforms to this (or a +* stronger) model. This argument cannot be __ATOMIC_RELEASE, +* __ATOMIC_ACQ_REL, or a stronger model than success. +* @return +* Non-zero on success; 0 on failure. +*/ +static inline int __rte_experimental +rte_atomic128_cmpxchg(rte_int128_t *dst,
Name could be more neutral. May be rte_atomic128_compare/cmp_exchange?
Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +7
-1+rte_int128_t *exp, + const rte_int128_t *src, + unsigned int weak, + int success, + int failure); +#endif +#endif /* _RTE_ATOMIC_H_ */ --2.13.6
Few minor comments.
I have not reviewed the x86 implementation.
Otherwise,
Re: [PATCH v2 1/1] eal: add 128-bit cmpxchg (x86-64 only)
[snip]
Quoted 9 lines
Quoted 4 lines
+/**
+ * An atomic compare and set function used by the mutex functions.
+ * (Atomically) Equivalent to:
+ * if (*dst == exp)Should be, "if (*dst == *exp)"
Quoted 1 line
+ * *dst = src
Should be "*dst = *src"
Good catches, will fix both.
Quoted 36 lines
Quoted 10 lines
+ * else
+ * *exp = *dst
+ *
+ * @note The success and failure arguments must be one of thelib/librte_eal/common/include/generic/rte_atomic.h +6
-0+__ATOMIC_* values + * defined in the C++11 standard. For details on their behavior, +refer to the + * standard. + * + * @param dstlib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v)
Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +10
-0+* The destination into which the value will be written. +* @param exp +* Pointer to the expected value. If the operation fails, this memory is +* updated with the actual value. +* @param src +* Pointer to the new value. +* @param weak +* A value of true allows the comparison to spuriously fail and allows the +* 'exp' update to occur non-atomically (i.e. a torn read may occur). +* Implementations may ignore this argument and only implement the strong
Review
Quoted 13 lines
+ * variant.
+ * @param success
+ * If successful, the operation's memory behavior conforms to this (or a
+ * stronger) model.
+ * @param failure+ * If unsuccessful, the operation's memory behavior conforms to this (or a + * stronger) model. This argument cannot be __ATOMIC_RELEASE, + * __ATOMIC_ACQ_REL, or a stronger model than success.+ * @return
+ * Non-zero on success; 0 on failure.
+ */(quoted diff) +2
-0+static inline int __rte_experimental +rte_atomic128_cmpxchg(rte_int128_t *dst,Name could be more neutral. May be rte_atomic128_compare/cmp_exchange?
Quoted 11 lines
Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +7
-1+rte_int128_t *exp, + const rte_int128_t *src, + unsigned int weak, + int success, + int failure); +#endif +#endif /* _RTE_ATOMIC_H_ */ --2.13.6
Sure -- I'll rename it rte_atomic128_cmp_exchange in the next version.
Quoted 4 lines
Few minor comments.
I have not reviewed the x86 implementation.
Otherwise,Reviewed-by Honnappa Nagarahalli <[email protected]>
Thanks,
Gage
v3 cover Add 128-bit compare and set
In reply to Gage EadsThis patch addresses x86-64 only; other architectures can/will be supported in the future. The __atomic intrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the __atomic_compare_exchange_16 (which itself is based on the C++11 memory model) to best support weak consistency architectures.
This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not on the entire non-blocking stack patchset.
[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123653.html
v3:
- Rename function to ISA-neutral rte_atomic128_cmp_exchange()
- Fix two pseudocode bugs in function documentation
v2:
- Rename function to rte_atomic128_cmpxchg()
- Replace "=A" output constraint with "=a" and "=d" to prevent GCC from using the al register for the sete destination
- Extend 'weak' definition to allow non-atomic 'exp' updates.
- Add const keyword to 'src' and remove volatile keyword from 'dst'
- Put __int128 in a union in rte_int128_t and move the structure definition inside the RTE_ARCH_x86_64 ifdef
- Drop enum rte_atomic_memmodel_t in favor of compiler-defined __ATOMIC_*
- Drop unnecessary comment relating to X86_64
- Tweak the pseudocode to reflect the 'exp' update on failure.
Gage Eads (1):
eal: add 128-bit compare exchange (x86-64 only)
.../common/include/arch/x86/rte_atomic_64.h | 33 ++++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 59 ++++++++++++++++++++++ 2 files changed, 92 insertions(+)
Signature · 1 line
2.13.6
PATCH v3 1/1 eal: add 128-bit compare exchange (x86-64 only)
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
--- .../common/include/arch/x86/rte_atomic_64.h | 33 ++++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 59 ++++++++++++++++++++++ 2 files changed, 92 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +33 -0
@@ -34,6 +34,7 @@/* * Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */
@@ -46,6 +47,7 @@#include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations -------------------------*/
@@ -208,4 +210,35 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v) }#endif +static inline int __rte_experimental +rte_atomic128_cmp_exchange (rte_int128_t *dst, +rte_int128_t *exp, +const rte_int128_t *src, +unsigned int weak, +int success, +int failure) +{ +RTE_SET_USED (weak); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( +MPLOCKED +"cmpxchg16b %[dst];" +" sete %[res]" + : [dst]"=m" (dst->val[0 ]), +"=a" (exp->val[0 ]), +"=d" (exp->val[1 ]), + [res]"=r" (res) + :"b" (src->val[0 ]), +"c" (src->val[1 ]), +"a" (exp->val[0 ]), +"d" (exp->val[1 ]), +"m" (dst->val[0 ]) + :"memory" ); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +59 -0
69 lines not on this page · raw message
Signature · 1 line
2.13.6
Re: [PATCH v3 1/1] eal: add 128-bit compare exchange (x86-64 only)
04/03/2019 21:51, Gage Eads:
Quoted 4 lines
lib/librte_eal/common/include/generic/rte_atomic.h +2
-0+#if defined(RTE_ARCH_X86_64 ) +/**
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1099
Quoted 10 lines
lib/librte_eal/common/include/generic/rte_atomic.h +10
-0+* 128-bit integer structure. +*/ +RTE_STD_C11 +typedef struct { +RTE_STD_C11 + union { +uint64_t val[2 ]; +__int128 int128; + }; +} __rte_aligned(16) rte_int128_t;
Why adding an arch-specific definition in a generic file? Can we move it to the x86_64 file?
Re: [PATCH v3 1/1] eal: add 128-bit compare exchange (x86-64 only)
Quoted 10 lines
04/03/2019 21:51, Gage Eads:
Quoted 9 lines
lib/librte_eal/common/include/generic/rte_atomic.h +7
-0+#if defined(RTE_ARCH_X86_64 ) +/** +* 128-bit integer structure. +*/ +RTE_STD_C11 +typedef struct { +RTE_STD_C11
Quoted 9 lines
lib/librte_eal/common/include/generic/rte_atomic.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 1099
Quoted 5 lines
lib/librte_eal/common/include/generic/rte_atomic.h +5
-0+union { +uint64_t val[2 ]; +__int128 int128; + }; +} __rte_aligned(16) rte_int128_t;Why adding an arch-specific definition in a generic file? Can we move it to the
x86_64file?
We can. I put it in the generic header in anticipation of other ISA implementations coming later, and since the atomic operations are documented in the generic header (although that ifdef appears to prevent it from being included in doxygen output).
I'll move it to x86/rte_atomic_64.h if that's preferred, and we can consider moving it back to generic as other implementations are added.
Thanks,
Gage
v4 cover Add 128-bit compare and set
In reply to Gage EadsThis patch addresses x86-64 only; other architectures can/will be supported in the future. The __atomic intrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the __atomic_compare_exchange_16 (which itself is based on the C++11 memory model) to best support weak consistency architectures.
This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not on the entire non-blocking stack patchset.
[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123653.html
v4:
- Move function declaration from generic/rte_atomic.h to x86-64 header file
v3:
- Rename function to ISA-neutral rte_atomic128_cmp_exchange()
- Fix two pseudocode bugs in function documentation
v2:
- Rename function to rte_atomic128_cmpxchg()
- Replace "=A" output constraint with "=a" and "=d" to prevent GCC from using the al register for the sete destination
- Extend 'weak' definition to allow non-atomic 'exp' updates.
- Add const keyword to 'src' and remove volatile keyword from 'dst'
- Put __int128 in a union in rte_int128_t and move the structure definition inside the RTE_ARCH_x86_64 ifdef
- Drop enum rte_atomic_memmodel_t in favor of compiler-defined __ATOMIC_*
- Drop unnecessary comment relating to X86_64
- Tweak the pseudocode to reflect the 'exp' update on failure.
Gage Eads (1):
eal: add 128-bit compare exchange (x86-64 only)
.../common/include/arch/x86/rte_atomic_64.h | 81 ++++++++++++++++++++++ 1 file changed, 81 insertions(+)
Signature · 1 line
2.13.6
PATCH v4 1/1 eal: add 128-bit compare exchange (x86-64 only)
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
--- .../common/include/arch/x86/rte_atomic_64.h | 81 ++++++++++++++++++++++ 1 file changed, 81 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +81 -0
@@ -34,6 +34,7 @@/* * Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */
@@ -46,6 +47,7 @@#include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations -------------------------*/
@@ -208,4 +210,83 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v) }#endif +/*------------------------ 128 bit atomic operations -------------------------*/ + +/** +* 128-bit integer structure. +*/ +RTE_STD_C11 +typedef struct { +RTE_STD_C11 + union { +uint64_t val[2 ]; +__int128 int128; + }; +} __rte_aligned(16) rte_int128_t; + +/** +* An atomic compare and set function used by the mutex functions. +* (Atomically) Equivalent to: +* if (*dst == *exp) +* *dst = *src +* else +* *exp = *dst +* +* @note The success and failure arguments must be one of the __ATOMIC_* values +* defined in the C++11 standard. For details on their behavior, refer to the +* standard. +* +* @param dst +* The destination into which the value will be written. +* @param exp +* Pointer to the expected value. If the operation fails, this memory is +* updated with the actual value. +* @param src +* Pointer to the new value. +* @param weak +* A value of true allows the comparison to spuriously fail and allows the +* 'exp' update to occur non-atomically (i.e. a torn read may occur). +* Implementations may ignore this argument and only implement the strong +* variant. +* @param success +* If successful, the operation's memory behavior conforms to this (or a +* stronger) model. +* @param failure +* If unsuccessful, the operation's memory behavior conforms to this (or a +* stronger) model. This argument cannot be __ATOMIC_RELEASE, +* __ATOMIC_ACQ_REL, or a stronger model than success. +* @return +* Non-zero on success; 0 on failure. +*/ +static inline int __rte_experimental +rte_atomic128_cmp_exchange (rte_int128_t *dst, +rte_int128_t *exp, +const rte_int128_t *src, +unsigned int weak, +int success, +int failure) +{ +RTE_SET_USED (weak); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( +MPLOCKED +"cmpxchg16b %[dst];" +" sete %[res]" + : [dst]"=m" (dst->val[0 ]), +"=a" (exp->val[0 ]), +"=d" (exp->val[1 ]), + [res]"=r" (res) + :"b" (src->val[0 ]), +"c" (src->val[1 ]), +"a" (exp->val[0 ]), +"d" (exp->val[1 ]), +"m" (dst->val[0 ]) + :"memory" ); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */
Signature · 1 line
2.13.6
Re: [PATCH v4 1/1] eal: add 128-bit compare exchange (x86-64 only)
03/04/2019 19:34, Gage Eads:
Quoted 39 lines
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
Signed-off-by Gage Eads <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>---
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +31
-0+/** +* An atomic compare and set function used by the mutex functions. +* (Atomically) Equivalent to: +* if (*dst == *exp) +* *dst = *src +* else +* *exp = *dst +* +* @note The success and failure arguments must be one of the __ATOMIC_* values +* defined in the C++11 standard. For details on their behavior, refer to the +* standard. +* +* @param dst +* The destination into which the value will be written. +* @param exp +* Pointer to the expected value. If the operation fails, this memory is +* updated with the actual value. +* @param src +* Pointer to the new value. +* @param weak +* A value of true allows the comparison to spuriously fail and allows the +* 'exp' update to occur non-atomically (i.e. a torn read may occur). +* Implementations may ignore this argument and only implement the strong +* variant. +* @param success +* If successful, the operation's memory behavior conforms to this (or a +* stronger) model. +* @param failure +* If unsuccessful, the operation's memory behavior conforms to this (or a +* stronger) model. This argument cannot be __ATOMIC_RELEASE, +* __ATOMIC_ACQ_REL, or a stronger model than success.
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 267
Quoted 10 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +10
-0+* @return +* Non-zero on success; 0 on failure. +*/ +static inline int __rte_experimental +rte_atomic128_cmp_exchange (rte_int128_t *dst, +rte_int128_t *exp, +const rte_int128_t *src, +unsigned int weak, +int success, +int failure)
I was thinking about keeping the doxygen in the generic file. Is it possible?
Re: [PATCH v4 1/1] eal: add 128-bit compare exchange (x86-64 only)
Quoted 42 lines
03/04/2019 19:34, Gage Eads:
Quoted 23 lines
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
Signed-off-by Gage Eads <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>---
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +15
-0+/** +* An atomic compare and set function used by the mutex functions. +* (Atomically) Equivalent to: +* if (*dst == *exp) +* *dst = *src +* else +* *exp = *dst +* +* @note The success and failure arguments must be one of the +__ATOMIC_* values +* defined in the C++11 standard. For details on their behavior, +refer to the +* standard. +* +* @param dst lib/librte_eal/common/include/arch/x86/rte_atomic_64.h static inline void rte_atomic64_clear(rte_atomic64_t *v)
Quoted 10 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +10
-0+* The destination into which the value will be written. +* @param exp +* Pointer to the expected value. If the operation fails, this memory is +* updated with the actual value. +* @param src +* Pointer to the new value. +* @param weak +* A value of true allows the comparison to spuriously fail and allows the +* 'exp' update to occur non-atomically (i.e. a torn read may occur). +* Implementations may ignore this argument and only implement the strong
Quoted 6 lines
+ * variant.
+ * @param success
+ * If successful, the operation's memory behavior conforms to this (or a
+ * stronger) model.
+ * @param failure
+ * If unsuccessful, the operation's memory behavior conforms to this (ora
Quoted 16 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 267
Quoted 12 lines
+ * stronger) model. This argument cannot be
__ATOMIC_RELEASE,
+ *__ATOMIC_ACQ_REL, or a stronger model than success.
+ * @return
+ * Non-zero on success; 0 on failure.
+ */lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +7
-0+static inline int __rte_experimental +rte_atomic128_cmp_exchange (rte_int128_t *dst, +rte_int128_t *exp, +const rte_int128_t *src, +unsigned int weak, +int success, +int failure)I was thinking about keeping the doxygen in the generic file. Is it possible?
We'd need to include the definition of rte_int128_t, so we'd also need either an ifdef on RTE_ARCH_64 or RTE_ARCH_X86_64 to protect 32-bit builds. That macro would prevent doxygen from parsing that section, unless we add a workaround like, for example:
#if defined(RTE_ARCH_64) || defined(__DOXYGEN__)
So the patch would look like the v3, with the declaration in generic/rte_atomic.h, but with that preprocessor change. If we change RTE_ARCH_X86_64 to RTE_ARCH_64, I'd add a note clarifying that it's only implemented for x86-64. What do you think?
Re: [PATCH v4 1/1] eal: add 128-bit compare exchange (x86-64 only)
03/04/2019 21:21, Eads, Gage:
Quoted 59 lines
Quoted 42 lines
03/04/2019 19:34, Gage Eads:
Quoted 23 lines
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
Signed-off-by Gage Eads <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>---
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +15
-0+/** +* An atomic compare and set function used by the mutex functions. +* (Atomically) Equivalent to: +* if (*dst == *exp) +* *dst = *src +* else +* *exp = *dst +* +* @note The success and failure arguments must be one of the +__ATOMIC_* values +* defined in the C++11 standard. For details on their behavior, +refer to the +* standard. +* +* @param dst lib/librte_eal/common/include/arch/x86/rte_atomic_64.h static inline void rte_atomic64_clear(rte_atomic64_t *v)
Quoted 10 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +10
-0+* The destination into which the value will be written. +* @param exp +* Pointer to the expected value. If the operation fails, this memory is +* updated with the actual value. +* @param src +* Pointer to the new value. +* @param weak +* A value of true allows the comparison to spuriously fail and allows the +* 'exp' update to occur non-atomically (i.e. a torn read may occur). +* Implementations may ignore this argument and only implement the strong
Quoted 6 lines
+ * variant.
+ * @param success
+ * If successful, the operation's memory behavior conforms to this (or a
+ * stronger) model.
+ * @param failure
+ * If unsuccessful, the operation's memory behavior conforms to this (ora
Quoted 16 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h static inline void rte_atomic64_clear(rte_atomic64_t *v) · line 267
Quoted 12 lines
+ * stronger) model. This argument cannot be
__ATOMIC_RELEASE,
+ *__ATOMIC_ACQ_REL, or a stronger model than success.
+ * @return
+ * Non-zero on success; 0 on failure.
+ */lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +7
-0+static inline int __rte_experimental +rte_atomic128_cmp_exchange (rte_int128_t *dst, +rte_int128_t *exp, +const rte_int128_t *src, +unsigned int weak, +int success, +int failure)I was thinking about keeping the doxygen in the generic file. Is it possible?
Quoted 5 lines
We'd need to include the definition of
rte_int128_t, so we'd also need either an ifdef onRTE_ARCH_64orRTE_ARCH_X86_64to protect 32-bit builds. That macro would prevent doxygen from parsing that section, unless we add a workaround like, for example:#if
defined(RTE_ARCH_64)||defined(__DOXYGEN__)So the patch would look like the v3, with the declaration in
generic/rte_atomic.h, but with that preprocessor change. If we changeRTE_ARCH_X86_64toRTE_ARCH_64, I'd add a note clarifying that it's only implemented for x86-64. What do you think?
I would like to see the doc in the generic file
and the implementation in the x86 file.
I tried forward declaration of the typedef:
struct rte_int128; typedef struct rte_int128 rte_int128_t; I don't why it does not work. So I'm trying to protect the declaration with #ifdef __DOXYGEN__
PATCH v5 1/1 eal/x86: add 128-bit atomic compare exchange
In reply to Gage EadsFrom: Gage Eads <[email protected]>
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
It is available only for x86_64.
---
This patch addresses x86-64 only; other architectures can/will be supported in the future. The __atomic intrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the __atomic_compare_exchange_16 (which itself is based on the C++11 memory model) to best support weak consistency architectures.
This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not on the entire non-blocking stack patchset.
[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123653.html
v5:
- Move declaration in the generic file for doxygen only (Thomas)
v4:
- Move function declaration from generic/rte_atomic.h to x86-64 header file
v3:
- Rename function to ISA-neutral rte_atomic128_cmp_exchange()
- Fix two pseudocode bugs in function documentation
v2:
- Rename function to rte_atomic128_cmpxchg()
- Replace "=A" output constraint with "=a" and "=d" to prevent GCC from using the al register for the sete destination
- Extend 'weak' definition to allow non-atomic 'exp' updates.
- Add const keyword to 'src' and remove volatile keyword from 'dst'
- Put __int128 in a union in rte_int128_t and move the structure definition inside the RTE_ARCH_x86_64 ifdef
- Drop enum rte_atomic_memmodel_t in favor of compiler-defined __ATOMIC_*
- Drop unnecessary comment relating to X86_64
- Tweak the pseudocode to reflect the 'exp' update on failure.
--- .../common/include/arch/x86/rte_atomic_64.h | 47 ++++++++++++++++++ .../common/include/generic/rte_atomic.h | 48 +++++++++++++++++++ 2 files changed, 95 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +47 -0
@@ -34,6 +34,7 @@/* * Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */
@@ -46,6 +47,7 @@#include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations -------------------------*/
@@ -208,4 +210,49 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v) }#endif +/*------------------------ 128 bit atomic operations -------------------------*/ + +/** +* 128-bit integer structure. +*/ +RTE_STD_C11 +typedef struct { +RTE_STD_C11 + union { +uint64_t val[2 ]; +__int128 int128; + }; +} __rte_aligned(16) rte_int128_t; + +static inline int __rte_experimental +rte_atomic128_cmp_exchange (rte_int128_t *dst, +rte_int128_t *exp, +const rte_int128_t *src, +unsigned int weak, +int success, +int failure) +{ +RTE_SET_USED (weak); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( +MPLOCKED +"cmpxchg16b %[dst];" +" sete %[res]" + : [dst]"=m" (dst->val[0 ]), +"=a" (exp->val[0 ]), +"=d" (exp->val[1 ]), + [res]"=r" (res) + :"b" (src->val[0 ]), +"c" (src->val[1 ]), +"a" (exp->val[0 ]), +"d" (exp->val[1 ]), +"m" (dst->val[0 ]) + :"memory" ); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +48 -0
52 lines not on this page · raw message
Signature · 1 line
2.21.0
PATCH v6 1/1 eal/x86: add 128-bit atomic compare exchange
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
It is available only for x86_64.
---
This patch addresses x86-64 only; other architectures can/will be supported in the future. The __atomic intrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the __atomic_compare_exchange_16 (which itself is based on the C++11 memory model) to best support weak consistency architectures.
This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not on the entire non-blocking stack patchset.
[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123653.html
v6:
- Use @code and @endcode to clean up pseudo-code generation
- Add note that the function is currently limited to x86-64
v5:
- Move declaration in the generic file for doxygen only (Thomas)
v4:
- Move function declaration from generic/rte_atomic.h to x86-64 header file
v3:
- Rename function to ISA-neutral rte_atomic128_cmp_exchange()
- Fix two pseudocode bugs in function documentation
v2:
- Rename function to rte_atomic128_cmpxchg()
- Replace "=A" output constraint with "=a" and "=d" to prevent GCC from using the al register for the sete destination
- Extend 'weak' definition to allow non-atomic 'exp' updates.
- Add const keyword to 'src' and remove volatile keyword from 'dst'
- Put __int128 in a union in rte_int128_t and move the structure definition inside the RTE_ARCH_x86_64 ifdef
- Drop enum rte_atomic_memmodel_t in favor of compiler-defined __ATOMIC_*
- Drop unnecessary comment relating to X86_64
- Tweak the pseudocode to reflect the 'exp' update on failure.
.../common/include/arch/x86/rte_atomic_64.h | 47 +++++++++++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 52 ++++++++++++++++++++++ 2 files changed, 99 insertions(+)
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +47 -0
@@ -34,6 +34,7 @@/* * Inspired from FreeBSD src/sys/amd64/include/atomic.h * Copyright (c) 1998 Doug Rabson +* Copyright (c) 2019 Intel Corporation * All rights reserved. */
@@ -46,6 +47,7 @@#include <stdint.h> #include <rte_common.h> +#include <rte_compat.h> #include <rte_atomic.h> /*------------------------- 64 bit atomic operations -------------------------*/
@@ -208,4 +210,49 @@ static inline void rte_atomic64_clear(rte_atomic64_t *v) }#endif +/*------------------------ 128 bit atomic operations -------------------------*/ + +/** +* 128-bit integer structure. +*/ +RTE_STD_C11 +typedef struct { +RTE_STD_C11 + union { +uint64_t val[2 ]; +__int128 int128; + }; +} __rte_aligned(16) rte_int128_t; + +static inline int __rte_experimental +rte_atomic128_cmp_exchange (rte_int128_t *dst, +rte_int128_t *exp, +const rte_int128_t *src, +unsigned int weak, +int success, +int failure) +{ +RTE_SET_USED (weak); +RTE_SET_USED (success); +RTE_SET_USED (failure); +uint8_t res; + + asmvolatile ( +MPLOCKED +"cmpxchg16b %[dst];" +" sete %[res]" + : [dst]"=m" (dst->val[0 ]), +"=a" (exp->val[0 ]), +"=d" (exp->val[1 ]), + [res]"=r" (res) + :"b" (src->val[0 ]), +"c" (src->val[1 ]), +"a" (exp->val[0 ]), +"d" (exp->val[1 ]), +"m" (dst->val[0 ]) + :"memory" ); + +return res; +} +#endif /* _RTE_ATOMIC_X86_64_H_ */
lib/librte_eal/common/include/generic/rte_atomic.h +52 -0
56 lines not on this page · raw message
Signature · 1 line
2.13.6
Thomas Monjalon
AppliedRe: [PATCH v6] eal/x86: add 128-bit atomic compare exchange
03/04/2019 21:44, Gage Eads:
Quoted 49 lines
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
It is available only for
x86_64.Signed-off-by Gage Eads <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>---
This patch addresses x86-64 only; other architectures can/will be supported in the future. The__atomicintrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the__atomic_compare_exchange_16(which itself is based on the C++11 memory model) to best support weak consistency architectures.This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not on the entire non-blocking stack patchset.
[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123653.htmlv6:
- Use @code and @endcode to clean up pseudo-code generation
- Add note that the function is currently limited to x86-64v5:
- Move declaration in the generic file for doxygen only (Thomas)v4:
- Move function declaration fromgeneric/rte_atomic.hto x86-64 header filev3:
- Rename function to ISA-neutralrte_atomic128_cmp_exchange()
- Fix two pseudocode bugs in function documentationv2:
- Rename function torte_atomic128_cmpxchg()
- Replace "=A" output constraint with "=a" and "=d" to prevent GCC from using the al register for the sete destination
- Extend 'weak' definition to allow non-atomic 'exp' updates.
- Add const keyword to 'src' and remove volatile keyword from 'dst'
- Put__int128in a union inrte_int128_tand move the structure definition inside theRTE_ARCH_x86_64ifdef
- Drop enumrte_atomic_memmodel_tin favor of compiler-defined __ATOMIC_*
- Drop unnecessary comment relating toX86_64
- Tweak the pseudocode to reflect the 'exp' update on failure.
Quoted 3 lines
.../common/include/arch/x86/rte_atomic_64.h | 47 +++++++++++++++++++ lib/librte_eal/common/include/generic/rte_atomic.h | 52 ++++++++++++++++++++++ 2 files changed, 99 insertions(+)
Applied, thanks
Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange
In reply to Gage EadsQuoted 37 lines · Gage Eads
This operation can be used for non-blocking algorithms, such as a non-blocking stack or ring.
It is available only for
x86_64.Signed-off-by Gage Eads <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>---
This patch addresses x86-64 only; other architectures can/will be supported in the future. The__atomicintrinsic was considered for the implementation, however libatomic was found[1] to use locks to implement the 128-bit CAS on at least one architecture and so is eschewed here. The interface is modeled after the__atomic_compare_exchange_16(which itself is based on the C++11 memory model) to best support weak consistency architectures.This patch was originally part of a series that introduces a non-blocking stack mempool handler[2], and is required by a non-blocking ring patchset. This patch was spun off so that the the NB ring depends only on this patch and not on the entire non-blocking stack patchset.
[1] http://mails.dpdk.org/archives/dev/2019-January/124002.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123653.htmlv6:
- Use @code and @endcode to clean up pseudo-code generation
- Add note that the function is currently limited to x86-64v5:
- Move declaration in the generic file for doxygen only (Thomas)v4:
- Move function declaration fromgeneric/rte_atomic.hto x86-64 header filev3:
- Rename function to ISA-neutralrte_atomic128_cmp_exchange()
- Fix two pseudocode bugs in function documentation
Quoted 12 lines
v2:
- Rename function torte_atomic128_cmpxchg()
- Replace "=A" output constraint with "=a" and "=d" to prevent GCC from using the al register for the sete destination
- Extend 'weak' definition to allow non-atomic 'exp' updates.
- Add const keyword to 'src' and remove volatile keyword from 'dst'
- Put__int128in a union inrte_int128_tand move the structure definition inside theRTE_ARCH_x86_64ifdef
- Drop enumrte_atomic_memmodel_tin favor of compiler-defined __ATOMIC_*
- Drop unnecessary comment relating toX86_64
- Tweak the pseudocode to reflect the 'exp' update on failure.
<...>
I am getting a build error, I guess generated from this patch, while building mlx drivers, when MLX._DEBUG enabled [1].
This looks because of the "-pedantic" parameter mlx drivers have [2].
Gage, Shahaf, Matan, Yongseok,
Can you please check this?
[1] .../dpdk/x86_64-native-linuxapp-gcc/include/rte_atomic_64.h:223:3: error: ISO C does not support ‘__int128’ types [-Werror=pedantic]
__int128 int128;
^~~~~~~~
cc1: all warnings being treated as errors
[2]
Indeed I would like to discuss this distinct behavior of the mlx drivers, it is many time "-pedantic" caused problem, I will bring the discussion on separate thread.
Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange
04/04/2019 13:47, Ferruh Yigit:
Quoted 2 lines
.../dpdk/x86_64-native-linuxapp-gcc/include/rte_atomic_64.h:223:3: error: ISO C does not support ‘__int128’ types [-Werror=pedantic]
We can try this kind of workaround (disable pedantic locally):
https://github.com/HowardHinnant/date/pull/38/commits/177032852d5b4614112ca1ab3ef42d6b41824816
Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange
04/04/2019 14:08, Thomas Monjalon:
Quoted 6 lines
04/04/2019 13:47, Ferruh Yigit:
Quoted 2 lines
.../dpdk/x86_64-native-linuxapp-gcc/include/rte_atomic_64.h:223:3: error: ISO C does not support ‘__int128’ types [-Werror=pedantic]We can try this kind of workaround (disable pedantic locally):
https://github.com/HowardHinnant/date/pull/38/commits/177032852d5b4614112ca1ab3ef42d6b41824816
Or better:__extension__ typedef __int128 int128;
Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange
Quoted 23 lines
-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Thursday, April 4, 2019 7:13 AM
To: Yigit, Ferruh <[email protected]>; Eads, Gage <[email protected]>
Cc: Shahaf Shuler <[email protected]>; Matan Azrad <[email protected]>; Yongseok Koh <[email protected]>; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v6] eal/x86: add 128-bit atomic compare exchange04/04/2019 14:08, Thomas Monjalon:
Quoted 4 lines
04/04/2019 13:47, Ferruh Yigit:
Quoted 2 lines
.../dpdk/x86_64-native-linuxapp-gcc/include/rte_atomic_64.h:223:3:
error: ISO C does not support ‘__int128’ types [-Werror=pedantic]
Quoted 9 lines
Quoted 2 lines
We can try this kind of workaround (disable pedantic locally):
https://github.com/HowardHinnant/date/pull/38/commits/177032852d5b46
14Quoted 1 line
112ca1ab3ef42d6b41824816Or better:
__extension__typedef__int128int128;
Taking that one step further -- RTE_STD_C11 evaluates to __extension__ (when the STD C version is sufficiently old).
Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange
04/04/2019 14:14, Eads, Gage:
Quoted 10 lines
From: Thomas Monjalon [mailto:[email protected]]
Quoted 9 lines
04/04/2019 14:08, Thomas Monjalon:
Quoted 6 lines
04/04/2019 13:47, Ferruh Yigit:
Quoted 2 lines
.../dpdk/x86_64-native-linuxapp-gcc/include/rte_atomic_64.h:223:3:
error: ISO C does not support ‘__int128’ types [-Werror=pedantic]We can try this kind of workaround (disable pedantic locally):
https://github.com/HowardHinnant/date/pull/38/commits/177032852d5b46
14
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
112ca1ab3ef42d6b41824816Or better:
__extension__typedef__int128int128;Taking that one step further --
RTE_STD_C11evaluates to__extension__(when the STD C version is sufficiently old).
I don't think __int128 is part of C11. Is it?
Ferruh, I cannot reproduce the compiler error. What is your compiler? Please, could you test this patch?
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1 -1
@@ -220,7 +220,7 @@ typedef struct {RTE_STD_C11 union {uint64_t val[2 ]; -__int128 int128; + __extension____int128 int128; }; } __rte_aligned(16) rte_int128_t;
Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange
Quoted 30 lines
-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Thursday, April 4, 2019 7:18 AM
To: Eads, Gage <[email protected]>; Yigit, Ferruh <[email protected]>
Cc: Shahaf Shuler <[email protected]>; Matan Azrad <[email protected]>; Yongseok Koh <[email protected]>; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v6] eal/x86: add 128-bit atomic compare exchange04/04/2019 14:14, Eads, Gage:
Quoted 9 lines
From: Thomas Monjalon [mailto:[email protected]]
Quoted 8 lines
04/04/2019 14:08, Thomas Monjalon:
Quoted 6 lines
04/04/2019 13:47, Ferruh Yigit:
Quoted 2 lines
.../dpdk/x86_64-native-linuxapp-gcc/include/rte_atomic_64.h:223:3:
error: ISO C does not support ‘__int128’ types [-Werror=pedantic]We can try this kind of workaround (disable pedantic locally):
https://github.com/HowardHinnant/date/pull/38/commits/177032852d5b46
Quoted 3 lines
Quoted 3 lines
14
Quoted 1 line
112ca1ab3ef42d6b41824816
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Or better:
__extension__typedef__int128int128;Taking that one step further --
RTE_STD_C11evaluates to__extension__(when the STD C version is sufficiently old).
I don't think
__int128is part of C11. Is it?
You're right, this is a compiler extension. Using '__extension__' makes more sense.
Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange
In reply to Thomas MonjalonQuoted 56 lines
-----Original Message-----
From: Eads, Gage
Sent: Thursday, April 4, 2019 7:23 AM
To: 'Thomas Monjalon' <[email protected]>; Yigit, Ferruh <[email protected]>
Cc: Shahaf Shuler <[email protected]>; Matan Azrad <[email protected]>; Yongseok Koh <[email protected]>; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [dpdk-dev] [PATCH v6] eal/x86: add 128-bit atomic compare exchangeQuoted 7 lines
-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Thursday, April 4, 2019 7:18 AM
To: Eads, Gage <[email protected]>; Yigit, Ferruh <[email protected]>
Cc: Shahaf Shuler <[email protected]>; Matan Azrad <[email protected]>; Yongseok Koh <[email protected]>;Quoted 4 lines
[email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected];
Quoted 9 lines
[email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v6] eal/x86: add 128-bit atomic compare exchange04/04/2019 14:14, Eads, Gage:
Quoted 4 lines
From: Thomas Monjalon [mailto:[email protected]]
Quoted 3 lines
04/04/2019 14:08, Thomas Monjalon:
Quoted 2 lines
04/04/2019 13:47, Ferruh Yigit:
Quoted 1 line
.../dpdk/x86_64-native-linuxapp-
gcc/include/rte_atomic_64.h:223:3:Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
error: ISO C does not support ‘__int128’ types
[-Werror=pedantic]We can try this kind of workaround (disable pedantic locally):
https://github.com/HowardHinnant/date/pull/38/commits/177032852d5b46
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
14
Quoted 1 line
112ca1ab3ef42d6b41824816Or better:
__extension__typedef__int128int128;
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Taking that one step further --
RTE_STD_C11evaluates to__extension__(when the STD C version is sufficiently old).
I don't think
__int128is part of C11. Is it?You're right, this is a compiler extension. Using '
__extension__' makes more sense.
FYI, __int128 was added in GCC 4.6 (and DPDK's minimum version is currently 4.9).
Re: [PATCH v6] eal/x86: add 128-bit atomic compare exchange
In reply to Thomas MonjalonQuoted 17 lines · Thomas Monjalon
04/04/2019 14:14, Eads, Gage:
Quoted 16 lines
From: Thomas Monjalon [mailto:[email protected]]
Quoted 14 lines
04/04/2019 14:08, Thomas Monjalon:
Quoted 6 lines
04/04/2019 13:47, Ferruh Yigit:
Quoted 2 lines
.../dpdk/x86_64-native-linuxapp-gcc/include/rte_atomic_64.h:223:3:
error: ISO C does not support ‘__int128’ types [-Werror=pedantic]We can try this kind of workaround (disable pedantic locally):
https://github.com/HowardHinnant/date/pull/38/commits/177032852d5b46
14Quoted 1 line
112ca1ab3ef42d6b41824816Or better:
__extension__typedef__int128int128;
Quoted 6 lines
Quoted 1 line
Taking that one step further --
RTE_STD_C11evaluates to__extension__(when the STD C version is sufficiently old).I don't think
__int128is part of C11. Is it?Ferruh, I cannot reproduce the compiler error. What is your compiler? Please, could you test this patch?
It is gcc [1]. Sure will test it.
[1]
gcc (GCC) 8.3.1 20190223 (Red Hat 8.3.1-2)
Quoted 14 lines
lib/librte_eal/common/include/arch/x86/rte_atomic_64.h +1
-1@@ -220,7 +220,7 @@ typedef struct {RTE_STD_C11 union {uint64_t val[2 ]; -__int128 int128; + __extension____int128 int128; }; } __rte_aligned(16) rte_int128_t;