graph: expose node context as pointers
PATCH v3 1/1 graph: expose node context as pointers
In some cases, the node context data is used to store two pointers because the data is larger than the reserved 16 bytes. Having to define intermediate structures just to be able to cast is tedious. Add two pointers that take the same space than ctx.
---
Notes:
v3:
* Added __extension__ to the unnamed struct inside the union.
* Fixed C++ header checks.
* Replaced alignas() with an explicit static_assert.
v2:
* Added __extension__ (not sure where it is needed, I don't have access to windows).
* It still fails the header check for C++. It seems not possible to align an unnamed union... Tyler, do you have an idea about how to fix that?
* Added static_assert to ensure the anonymous union is not larger than RTE_NODE_CTX_SZ.
lib/graph/rte_graph_worker_common.h | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
lib/graph/rte_graph_worker_common.h +20 -1
@@ -12,7 +12,9 @@* process, enqueue and move streams of objects to the next nodes. */ +#include <assert.h> #include <stdalign.h> +#include <stddef.h> #include <rte_common.h> #include <rte_cycles.h>
@@ -112,7 +114,19 @@ struct __rte_cache_aligned rte_node { };/* Fast path area */ #define RTE_NODE_CTX_SZ 16 - alignas(RTE_CACHE_LINE_SIZE )uint8_t ctx[RTE_NODE_CTX_SZ ];/**< Node Context. */ +/* +* alignas(RTE_CACHE_LINE_SIZE) cannot be used for ctx since it is part of an unnamed union. +* The compiler shifts the next field on the next cache line which is not what we want. +* The alignment is enforced via a explcicit static asserts below. +*/ +union { +uint8_t ctx[RTE_NODE_CTX_SZ ]; +/* Convenience aliases to store pointers without complex casting. */ + __extension__struct { +void *ctx_ptr; +void *ctx_ptr2; + }; + };/**< Node Context. */ uint16_t size;/**< Total number of objects available. */ uint16_t idx;/**< Number of objects used. */ rte_graph_off_t off;/**< Offset of node in the graph reel. */
@@ -130,6 +144,11 @@ struct __rte_cache_aligned rte_node { alignas(RTE_CACHE_LINE_MIN_SIZE )struct rte_node *nodes[];/**< Next nodes. */ }; +static_assert (offsetof(struct rte_node , ctx) %RTE_CACHE_LINE_SIZE ==0 , +"rte_node ctx must be aligned on a cache line" ); +static_assert (offsetof(struct rte_node , size) - offsetof(struct rte_node , ctx) ==RTE_NODE_CTX_SZ , +"rte_node context union cannot be larger than RTE_NODE_CTX_SZ" ); +/** * @internal *
Signature · 1 line
2.44.0
Quoted 62 lines · Robin Jarry
In some cases, the node context data is used to store two pointers because the data is larger than the reserved 16 bytes. Having to define intermediate structures just to be able to cast is tedious. Add two pointers that take the same space than ctx.
Signed-off-by Robin Jarry <[email protected]>---
Notes:
v3:* Added
__extension__to the unnamed struct inside the union.
* Fixed C++ header checks.
* Replacedalignas()with an explicitstatic_assert.v2:
* Added
__extension__(not sure where it is needed, I don't have access to windows).
* It still fails the header check for C++. It seems not possible to align an unnamed union... Tyler, do you have an idea about how to fix that?
* Addedstatic_assertto ensure the anonymous union is not larger thanRTE_NODE_CTX_SZ.lib/graph/rte_graph_worker_common.h | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)lib/graph/rte_graph_worker_common.h +15
-1@@ -12,7 +12,9 @@* process, enqueue and move streams of objects to the next nodes. */ +#include <assert.h> #include <stdalign.h> +#include <stddef.h> #include <rte_common.h> #include <rte_cycles.h> @@ -112,7 +114,19 @@ struct __rte_cache_aligned rte_node { };/* Fast path area */ #define RTE_NODE_CTX_SZ 16 - alignas(RTE_CACHE_LINE_SIZE )uint8_t ctx[RTE_NODE_CTX_SZ ];/**< Node Context. */ +/* +* alignas(RTE_CACHE_LINE_SIZE) cannot be used for ctx since it is part of an unnamed union. +* The compiler shifts the next field on the next cache line which is not what we want. +* The alignment is enforced via a explcicit static asserts below. +*/ +union { +uint8_t ctx[RTE_NODE_CTX_SZ ]; +/* Convenience aliases to store pointers without complex casting. */ + __extension__struct { +void *ctx_ptr; +void *ctx_ptr2; + }; + };/**< Node Context. */ uint16_t size;/**< Total number of objects available. */ uint16_t idx;/**< Number of objects used. */ rte_graph_off_t off;/**< Offset of node in the graph reel. */
lib/graph/rte_graph_worker_common.h struct __rte_cache_aligned rte_node { · line 148
Quoted 5 lines
lib/graph/rte_graph_worker_common.h +2
-0@@ -130,6 +144,11 @@ struct __rte_cache_aligned rte_node { alignas(RTE_CACHE_LINE_MIN_SIZE )struct rte_node *nodes[];/**< Next nodes. */ }; +static_assert (offsetof(struct rte_node , ctx) %RTE_CACHE_LINE_SIZE ==0 , +"rte_node ctx must be aligned on a cache line" );
This will fail in 32bit machine.
https://mails.dpdk.org/archives/test-report/2024-March/623806.html
I can think of following solution to add before ctx. RTE_MARKER fastpath __rte_cache__aligned;
Quoted 9 lines
lib/graph/rte_graph_worker_common.h +3
-1+static_assert (offsetof(struct rte_node , size) - offsetof(struct rte_node , ctx) ==RTE_NODE_CTX_SZ , +"rte_node context union cannot be larger than RTE_NODE_CTX_SZ" ); +/** * @internal * -- 2.44.0
Jerin Jacob, Mar 25, 2024 at 11:59:
Quoted 6 lines
lib/graph/rte_graph_worker_common.h struct __rte_cache_aligned rte_node { · line 148
Quoted 2 lines
lib/graph/rte_graph_worker_common.h +2
-0+static_assert (offsetof(struct rte_node , ctx) %RTE_CACHE_LINE_SIZE ==0 , +"rte_node ctx must be aligned on a cache line" );This will fail in 32bit machine.
https://mails.dpdk.org/archives/test-report/2024-March/623806.html
Hi Jerin, yes I saw that :(
Quoted 2 lines
I can think of following solution to add before ctx.
RTE_MARKERfastpath__rte_cache__aligned;
It will not be taken into account for MSVC. Is that OK?
Quoted 9 lines · Robin Jarry
Jerin Jacob, Mar 25, 2024 at 11:59:
Quoted 6 lines
lib/graph/rte_graph_worker_common.h struct __rte_cache_aligned rte_node { · line 148
Quoted 2 lines
lib/graph/rte_graph_worker_common.h +2
-0+static_assert (offsetof(struct rte_node , ctx) %RTE_CACHE_LINE_SIZE ==0 , +"rte_node ctx must be aligned on a cache line" );This will fail in 32bit machine.
https://mails.dpdk.org/archives/test-report/2024-March/623806.html
Quoted 6 lines
Hi Jerin, yes I saw that :(
Quoted 2 lines
I can think of following solution to add before ctx.
RTE_MARKERfastpath__rte_cache__aligned;It will not be taken into account for MSVC. Is that OK?
Why?. rte_mbuf has a similar scheme.RTE_MARKER cacheline1 __rte_cache_min_aligned;
Quoted 1 line
Jerin Jacob, Mar 25, 2024 at 12:08:
Quoted 4 lines
Quoted 1 line
It will not be taken into account for MSVC. Is that OK?
Why?.
rte_mbufhas a similar scheme.RTE_MARKERcacheline1__rte_cache_min_aligned;
RTE_MARKER* types seem not defined for the MSVC toolchain.
https://github.com/DPDK/dpdk/blob/v24.03-rc4/lib/eal/include/rte_common.h#L589-L602
Maybe I am missing something.
Quoted 10 lines · Robin Jarry
Jerin Jacob, Mar 25, 2024 at 12:08:
Quoted 4 lines
Quoted 1 line
It will not be taken into account for MSVC. Is that OK?
Why?.
rte_mbufhas a similar scheme.RTE_MARKERcacheline1__rte_cache_min_aligned;RTE_MARKER* types seem not defined for the MSVC toolchain.
https://github.com/DPDK/dpdk/blob/v24.03-rc4/lib/eal/include/rte_common.h#L589-L602
Hmm. Not sure, how mbuf is building for MSCV tool chain then.
Another option could be to have a helper inline function/macro to take care of casting to make app code clean of casting.
Quoted 2 lines
Maybe I am missing something.
Quoted 8 lines · Jerin Jacob
Quoted 7 lines · Robin Jarry
Jerin Jacob, Mar 25, 2024 at 12:08:
Quoted 4 lines
Quoted 1 line
It will not be taken into account for MSVC. Is that OK?
Why?.
rte_mbufhas a similar scheme.RTE_MARKERcacheline1__rte_cache_min_aligned;
Quoted 8 lines
Quoted 3 lines
RTE_MARKER* types seem not defined for the MSVC toolchain.
https://github.com/DPDK/dpdk/blob/v24.03-rc4/lib/eal/include/rte_common.h#L589-L602
Hmm. Not sure, how mbuf is building for MSCV tool chain then.
Another option could be to have a helper inline function/macro to take care of casting to make app code clean of casting.
The markers are being removed from DPDK and being replaced by more portable, and more standards-conforming constructs. We should not be adding more markers to existing structures. See [1]
/Bruce
[1] https://patches.dpdk.org/project/dpdk/list/?series=31579
Quoted 10 lines · Jerin Jacob
Quoted 8 lines · Robin Jarry
Jerin Jacob, Mar 25, 2024 at 12:08:
Quoted 4 lines
Quoted 1 line
It will not be taken into account for MSVC. Is that OK?
Why?.
rte_mbufhas a similar scheme.RTE_MARKERcacheline1__rte_cache_min_aligned;RTE_MARKER* types seem not defined for the MSVC toolchain.
There is some work in progress to stop using those markers.
https://patchwork.dpdk.org/project/dpdk/list/?series=31579&state=*
Quoted 4 lines
Hmm. Not sure, how mbuf is building for MSCV tool chain then.
Atm, MSVC builds a really small list of libraries.
http://git.dpdk.org/dpdk/tree/lib/meson.build#n71?id=v24.03-rc4
if is_ms_compiler
libraries = [
'log',
'kvargs',
'telemetry',
'eal',
'ring',
]endif
Quoted 3 lines
Another option could be to have a helper inline function/macro to take care of casting to make app code clean of casting.
That could be an option.
Signature · 1 line
David Marchand
Jerin Jacob, Mar 25, 2024 at 12:35:
Quoted 2 lines
Another option could be to have a helper inline function/macro to take care of casting to make app code clean of casting.
Would something like this be suitable?
#define RTE_NODE_CTX_PTR1(n) ((void **)(n)->ctx)[0] #define RTE_NODE_CTX_PTR2(n) ((void **)(n)->ctx)[1]
Quoted 9 lines · Robin Jarry
Jerin Jacob, Mar 25, 2024 at 12:35:
Quoted 2 lines
Another option could be to have a helper inline function/macro to take care of casting to make app code clean of casting.
Would something like this be suitable?
#define
RTE_NODE_CTX_PTR1(n)((void **)(n)->ctx)[0] #defineRTE_NODE_CTX_PTR2(n)((void **)(n)->ctx)[1]
Works for me. No strong opinion about the name, RTE_NODE_CTX_AS_PTR1 may be more reflecting the intent.
Quoted 1 line
Jerin Jacob, Mar 25, 2024 at 16:47:
Quoted 5 lines
Quoted 2 lines
#define
RTE_NODE_CTX_PTR1(n)((void **)(n)->ctx)[0] #defineRTE_NODE_CTX_PTR2(n)((void **)(n)->ctx)[1]Works for me. No strong opinion about the name,
RTE_NODE_CTX_AS_PTR1may be more reflecting the intent.
I also thought about adding inline getter/setter functions but that's more code. It may be cleaner:
static inline void *rte_node_ctx_ptr1_get(struct rte_node *n) {
return ((void **)node->ctx)[0];
}
static inline void *rte_node_ctx_ptr2_get(struct rte_node *n) {
return ((void **)node->ctx)[1];
}
static inline void rte_node_ctx_ptr1_set(struct rte_node *n, void *p) {
((void **)node->ctx)[0] = p;
}
static inline void rte_node_ctx_ptr2_set(struct rte_node *n, void *p) {
((void **)node->ctx)[1] = p;
}I don't have a strong opinion. I'll go either way.
Quoted 22 lines · Robin Jarry
Jerin Jacob, Mar 25, 2024 at 16:47:
Quoted 5 lines
Quoted 2 lines
#define
RTE_NODE_CTX_PTR1(n)((void **)(n)->ctx)[0] #defineRTE_NODE_CTX_PTR2(n)((void **)(n)->ctx)[1]Works for me. No strong opinion about the name,
RTE_NODE_CTX_AS_PTR1may be more reflecting the intent.I also thought about adding inline getter/setter functions but that's more code. It may be cleaner:
static inline void *rte_node_ctx_ptr1_get(struct rte_node *n) { return ((void **)node->ctx)[0]; } static inline void *rte_node_ctx_ptr2_get(struct rte_node *n) { return ((void **)node->ctx)[1]; } static inline void rte_node_ctx_ptr1_set(struct rte_node *n, void *p) { ((void **)node->ctx)[0] = p; } static inline void rte_node_ctx_ptr2_set(struct rte_node *n, void *p) { ((void **)node->ctx)[1] = p;
Quoted 3 lines
}
I don't have a strong opinion. I'll go either way.
Inline is better.
Quoted 1 line
PATCH v4 1/1 graph: expose node context as pointers
In reply to Robin JarryIn some cases, the node context data is used to store two pointers because the data is larger than the reserved 16 bytes. Having to define intermediate structures just to be able to cast is tedious. Add helper inline functions to cast the node context to opaque pointers.
---
Notes:
v4:
* Replaced the unnamed union with helper inline functions.
v3:
* Added __extension__ to the unnamed struct inside the union.
* Fixed C++ header checks.
* Replaced alignas() with an explicit static_assert.
lib/graph/rte_graph_worker_common.h | 54 +++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+)
lib/graph/rte_graph_worker_common.h +54 -0
@@ -130,6 +130,60 @@ struct __rte_cache_aligned rte_node { alignas(RTE_CACHE_LINE_MIN_SIZE )struct rte_node *nodes[];/**< Next nodes. */ }; +/** +* Cast the first 8 bytes of node context as an opaque pointer. +* +* @param node +* Pointer to the node object. +* +* @return +* The opaque pointer value. +*/ +static inline void *rte_node_ctx_ptr1_get (struct rte_node *node) +{ +return ((void **)node->ctx)[0 ]; +} + +/** +* Cast the last 8 bytes of node context as an opaque pointer. +* +* @param node +* Pointer to the node object. +* +* @return +* The opaque pointer value. +*/ +static inline void *rte_node_ctx_ptr2_get (struct rte_node *node) +{ +return ((void **)node->ctx)[1 ]; +} + +/** +* Set the first 8 bytes of node context to an opaque pointer value. +* +* @param node +* Pointer to the node object. +* @param ptr +* The opaque pointer value. +*/ +static inline void rte_node_ctx_ptr1_set (struct rte_node *node,void *ptr) +{ + ((void **)node->ctx)[0 ] = ptr; +} + +/** +* Set the last 8 bytes of node context to an opaque pointer value. +* +* @param node +* Pointer to the node object. +* @param ptr +* The opaque pointer value. +*/ +static inline void rte_node_ctx_ptr2_set (struct rte_node *node,void *ptr) +{ + ((void **)node->ctx)[1 ] = ptr; +} +/** * @internal *
Signature · 1 line
2.44.0
Quoted 31 lines · Robin Jarry
In some cases, the node context data is used to store two pointers because the data is larger than the reserved 16 bytes. Having to define intermediate structures just to be able to cast is tedious. Add helper inline functions to cast the node context to opaque pointers.
Signed-off-by Robin Jarry <[email protected]>---
Notes:
v4:* Replaced the unnamed union with helper inline functions.
v3:
* Added
__extension__to the unnamed struct inside the union.
* Fixed C++ header checks.
* Replacedalignas()with an explicitstatic_assert.lib/graph/rte_graph_worker_common.h | 54 +++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+)lib/graph/rte_graph_worker_common.h +0
-0@@ -130,6 +130,60 @@ struct __rte_cache_aligned rte_node { alignas(RTE_CACHE_LINE_MIN_SIZE )struct rte_node *nodes[];/**< Next nodes. */ };
lib/graph/rte_graph_worker_common.h struct __rte_cache_aligned rte_node { · line 142
Quoted 10 lines
lib/graph/rte_graph_worker_common.h +10
-0+/** +* Cast the first 8 bytes of node context as an opaque pointer. +* +* @param node +* Pointer to the node object. +* +* @return +* The opaque pointer value. +*/ +static inline void *rte_node_ctx_ptr1_get (struct rte_node *node)
New line after void*. Same for all new functions.
With that change:
Quoted 50 lines
lib/graph/rte_graph_worker_common.h +44
-1+{ +return ((void **)node->ctx)[0 ]; +} + +/** +* Cast the last 8 bytes of node context as an opaque pointer. +* +* @param node +* Pointer to the node object. +* +* @return +* The opaque pointer value. +*/ +static inline void *rte_node_ctx_ptr2_get (struct rte_node *node) +{ +return ((void **)node->ctx)[1 ]; +} + +/** +* Set the first 8 bytes of node context to an opaque pointer value. +* +* @param node +* Pointer to the node object. +* @param ptr +* The opaque pointer value. +*/ +static inline void rte_node_ctx_ptr1_set (struct rte_node *node,void *ptr) +{ + ((void **)node->ctx)[0 ] = ptr; +} + +/** +* Set the last 8 bytes of node context to an opaque pointer value. +* +* @param node +* Pointer to the node object. +* @param ptr +* The opaque pointer value. +*/ +static inline void rte_node_ctx_ptr2_set (struct rte_node *node,void *ptr) +{ + ((void **)node->ctx)[1 ] = ptr; +} +/** * @internal * -- 2.44.0
PATCH v5 1/1 graph: expose node context as pointers
In reply to Robin JarryIn some cases, the node context data is used to store two pointers because the data is larger than the reserved 16 bytes. Having to define intermediate structures just to be able to cast is tedious. And without intermediate structures, casting to opaque pointers is hard without violating strict aliasing rules.
Add an unnamed union to allow storing opaque pointers in the node context. Unfortunately, aligning an unnamed union that contains an array produces inconsistent results between C and C++. To preserve ABI/API compatibility in both C and C++, move all fast-path area fields into an unnamed struct which is cache aligned. Use __rte_cache_min_aligned to preserve existing alignment on architectures where cache lines are 128 bytes.
Add a static assert to ensure that the unnamed union is not larger than the context array (RTE_NODE_CTX_SZ).
---
Notes:
v5:
* Helper functions to hide casting proved to be harder than expected. Naive casting may even be impossible without breaking strict aliasing rules. The only other option would be to use explicit memcpy calls.
* Unnamed union tentative again. As suggested by Tyler (thank you!), using an intermediate unnamed struct to carry the alignment produces consistent ABI in C and C++.
* Also, Tyler (thank you!) suggested that the fast path area alignment size may be incorrect for architectures where the cache line is not 64 bytes. There will be a 64 bytes hole in the structure at the end of the unnamed struct before the zero length next nodes array. Use __rte_cache_min_aligned to preserve existing alignment.
v4:
* Replaced the unnamed union with helper inline functions.
v3:
* Added __extension__ to the unnamed struct inside the union.
* Fixed C++ header checks.
* Replaced alignas() with an explicit static_assert.
lib/graph/rte_graph_worker_common.h | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-)
lib/graph/rte_graph_worker_common.h +20 -7
@@ -12,7 +12,9 @@* process, enqueue and move streams of objects to the next nodes. */ +#include <assert.h> #include <stdalign.h> +#include <stddef.h> #include <rte_common.h> #include <rte_cycles.h>
@@ -111,14 +113,21 @@ struct __rte_cache_aligned rte_node { } dispatch; };/* Fast path area */ + __extension__ struct __rte_cache_min_aligned { #define RTE_NODE_CTX_SZ 16 - alignas(RTE_CACHE_LINE_SIZE )uint8_t ctx[RTE_NODE_CTX_SZ ];/**< Node Context. */ -uint16_t size;/**< Total number of objects available. */ -uint16_t idx;/**< Number of objects used. */ -rte_graph_off_t off;/**< Offset of node in the graph reel. */ -uint64_t total_cycles;/**< Cycles spent in this node. */ -uint64_t total_calls;/**< Calls done to this node. */ -uint64_t total_objs;/**< Objects processed by this node. */ + union { + uint8_t ctx[RTE_NODE_CTX_SZ]; + __extension__ struct { + void *ctx_ptr; + void *ctx_ptr2; + }; + }; /**< Node Context. */ + uint16_t size; /**< Total number of objects available. */ + uint16_t idx; /**< Number of objects used. */ + rte_graph_off_t off; /**< Offset of node in the graph reel. */ + uint64_t total_cycles; /**< Cycles spent in this node. */ + uint64_t total_calls; /**< Calls done to this node. */ + uint64_t total_objs; /**< Objects processed by this node. */ union { void **objs; /**< Array of object pointers. */ uint64_t objs_u64;
@@ -127,9 +136,13 @@ struct __rte_cache_aligned rte_node {rte_node_process_t process;/**< Process function. */ uint64_t process_u64; }; + }; alignas(RTE_CACHE_LINE_MIN_SIZE )struct rte_node *nodes[];/**< Next nodes. */ }; +static_assert (offsetof(struct rte_node , size) - offsetof(struct rte_node , ctx) ==RTE_NODE_CTX_SZ , +"rte_node context must be RTE_NODE_CTX_SZ bytes exactly" ); +/** * @internal *
Signature · 1 line
2.44.0
Quoted 109 lines · Robin Jarry
In some cases, the node context data is used to store two pointers because the data is larger than the reserved 16 bytes. Having to define intermediate structures just to be able to cast is tedious. And without intermediate structures, casting to opaque pointers is hard without violating strict aliasing rules.
Add an unnamed union to allow storing opaque pointers in the node context. Unfortunately, aligning an unnamed union that contains an array produces inconsistent results between C and C++. To preserve ABI/API compatibility in both C and C++, move all fast-path area fields into an unnamed struct which is cache aligned. Use
__rte_cache_min_alignedto preserve existing alignment on architectures where cache lines are 128 bytes.Add a static assert to ensure that the unnamed union is not larger than the context array (
RTE_NODE_CTX_SZ).Signed-off-by Robin Jarry <[email protected]>---
Notes:
v5:* Helper functions to hide casting proved to be harder than expected. Naive casting may even be impossible without breaking strict aliasing rules. The only other option would be to use explicit memcpy calls.
* Unnamed union tentative again. As suggested by Tyler (thank you!), using an intermediate unnamed struct to carry the alignment produces consistent ABI in C and C++.
* Also, Tyler (thank you!) suggested that the fast path area alignment size may be incorrect for architectures where the cache line is not 64 bytes. There will be a 64 bytes hole in the structure at the end of the unnamed struct before the zero length next nodes array. Use__rte_cache_min_alignedto preserve existing alignment.v4:
* Replaced the unnamed union with helper inline functions.
v3:
* Added
__extension__to the unnamed struct inside the union.
* Fixed C++ header checks.
* Replacedalignas()with an explicitstatic_assert.lib/graph/rte_graph_worker_common.h | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-)lib/graph/rte_graph_worker_common.h +20
-8@@ -12,7 +12,9 @@* process, enqueue and move streams of objects to the next nodes. */ +#include <assert.h> #include <stdalign.h> +#include <stddef.h> #include <rte_common.h> #include <rte_cycles.h> @@ -111,14 +113,21 @@ struct __rte_cache_aligned rte_node { } dispatch; };/* Fast path area */ + __extension__ struct __rte_cache_min_aligned { #define RTE_NODE_CTX_SZ 16 - alignas(RTE_CACHE_LINE_SIZE )uint8_t ctx[RTE_NODE_CTX_SZ ];/**< Node Context. */ -uint16_t size;/**< Total number of objects available. */ -uint16_t idx;/**< Number of objects used. */ -rte_graph_off_t off;/**< Offset of node in the graph reel. */ -uint64_t total_cycles;/**< Cycles spent in this node. */ -uint64_t total_calls;/**< Calls done to this node. */ -uint64_t total_objs;/**< Objects processed by this node. */ + union { + uint8_t ctx[RTE_NODE_CTX_SZ]; + __extension__ struct { + void *ctx_ptr; + void *ctx_ptr2; + }; + }; /**< Node Context. */ + uint16_t size; /**< Total number of objects available. */ + uint16_t idx; /**< Number of objects used. */ + rte_graph_off_t off; /**< Offset of node in the graph reel. */ + uint64_t total_cycles; /**< Cycles spent in this node. */ + uint64_t total_calls; /**< Calls done to this node. */ + uint64_t total_objs; /**< Objects processed by this node. */ union { void **objs; /**< Array of object pointers. */ uint64_t objs_u64;@@ -127,9 +136,13 @@ struct __rte_cache_aligned rte_node {rte_node_process_t process;/**< Process function. */ uint64_t process_u64; }; + }; alignas(RTE_CACHE_LINE_MIN_SIZE )struct rte_node *nodes[];/**< Next nodes. */ }; +static_assert (offsetof(struct rte_node , size) - offsetof(struct rte_node , ctx) ==RTE_NODE_CTX_SZ , +"rte_node context must be RTE_NODE_CTX_SZ bytes exactly" ); +/** * @internal * -- 2.44.0
Re Robin,
Quoted 24 lines · Robin Jarry
In some cases, the node context data is used to store two pointers because the data is larger than the reserved 16 bytes. Having to define intermediate structures just to be able to cast is tedious. And without intermediate structures, casting to opaque pointers is hard without violating strict aliasing rules.
Add an unnamed union to allow storing opaque pointers in the node context. Unfortunately, aligning an unnamed union that contains an array produces inconsistent results between C and C++. To preserve ABI/API compatibility in both C and C++, move all fast-path area fields into an unnamed struct which is cache aligned. Use
__rte_cache_min_alignedto preserve existing alignment on architectures where cache lines are 128 bytes.Add a static assert to ensure that the unnamed union is not larger than the context array (
RTE_NODE_CTX_SZ).Signed-off-by Robin Jarry <[email protected]>---
Notes:
v5:
Quoted 11 lines
* Helper functions to hide casting proved to be harder than expected. Naive casting may even be impossible without breaking strict aliasing rules. The only other option would be to use explicit memcpy calls.
* Unnamed union tentative again. As suggested by Tyler (thank you!), using an intermediate unnamed struct to carry the alignment produces consistent ABI in C and C++.
* Also, Tyler (thank you!) suggested that the fast path area alignment size may be incorrect for architectures where the cache line is not 64 bytes. There will be a 64 bytes hole in the structure at the end of the unnamed struct before the zero length next nodes array. Use__rte_cache_min_alignedto preserve existing alignment.
- There is still an issue with that approach on 128 bytes cache line arches, like ARM.
This results in a ABI breakage:
Functions changes summary: 0 Removed, 1 Changed (9 filtered out), 0 Added functions
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable
1 function with some indirect sub-type change:
[C] 'function bool__rte_graph_mcore_dispatch_sched_node_enqueue(rte_node*, rte_graph_rq_head*)' at rte_graph_model_mcore_dispatch.c:117:1 has some indirect sub-type changes:
parameter 1 of type 'rte_node*' has sub-type changes:
in pointed to type 'struct rte_node' at rte_graph_worker_common.h:92:1:
type size changed from 3072 to 2048 (in bits)
7 data member deletions:
'uint8_t ctx[16]', at offset 2048 (in bits) at
rte_graph_worker_common.h:115:1
'uint16_t size', at offset 2176 (in bits) at
rte_graph_worker_common.h:116:1
'uint16_t idx', at offset 2192 (in bits) atrte_graph_worker_common.h:117:1
'rte_graph_off_t off', at offset 2208 (in bits) at
rte_graph_worker_common.h:118:1
'uint64_t total_cycles', at offset 2240 (in bits) at
rte_graph_worker_common.h:119:1
'uint64_t total_calls', at offset 2304 (in bits) at
rte_graph_worker_common.h:120:1
'uint64_t total_objs', at offset 2368 (in bits) atrte_graph_worker_common.h:121:1
1 data member insertion:
'struct {union {uint8_t ctx[16]; struct {void* ctx_ptr;
void* ctx_ptr2;};}; uint16_t size; uint16_t idx; rte_graph_off_t off; uint64_t total_cycles; uint64_t total_calls; uint64_t total_objs; union {void** objs; uint64_t objs_u64;}; union {rte_node_process_t process; uint64_t process_u64;};}', at offset 1536 (in bits)
1 data member changes (1 filtered):
'rte_node* nodes[]' offset changed from 2560 to 2048 (in
bits) (by -512 bits)
Before the patch, the rte_node object layout was:
struct rte_node {
...
/* XXX 64 bytes hole, try to pack */
/* --- cacheline 4 boundary (256 bytes) --- */
uint8_t ctx[16]
__attribute__((__aligned__(128))); /* 256 16 */
uint16_t size; /* 272 2 */
uint16_t idx; /* 274 2 */
rte_graph_off_t off; /* 276 4 */
uint64_t total_cycles; /* 280 8 */
uint64_t total_calls; /* 288 8 */
uint64_t total_objs; /* 296 8 */
union {
void * * objs; /* 304 8 */
uint64_t objs_u64; /* 304 8 */
}; /* 304 8 */
union {
rte_node_process_t process; /* 312 8 */
uint64_t process_u64; /* 312 8 */
}; /* 312 8 */
/* --- cacheline 5 boundary (320 bytes) --- */
struct rte_node * nodes[]__attribute__((__aligned__(64))); /* 320 0 */
/* size: 384, cachelines: 6, members: 20 */
/* sum members: 250, holes: 3, sum holes: 70 */
/* padding: 64 */
/* forced alignments: 2, forced holes: 1, sum forced holes: 64 */
} __attribute__((__aligned__(128)));After this patch:
struct rte_node {
...
/* --- cacheline 3 boundary (192 bytes) --- */
struct {
union {
uint8_t ctx[16]; /* 192 16 */
struct {
void * ctx_ptr; /* 192 8 */
void * ctx_ptr2; /* 200 8 */
}; /* 192 16 */
}; /* 192 16 */
uint16_t size; /* 208 2 */
uint16_t idx; /* 210 2 */
rte_graph_off_t off; /* 212 4 */
uint64_t total_cycles; /* 216 8 */
uint64_t total_calls; /* 224 8 */
uint64_t total_objs; /* 232 8 */
union {
void * * objs; /* 240 8 */
uint64_t objs_u64; /* 240 8 */
}; /* 240 8 */
union {
rte_node_process_t process; /* 248 8 */
uint64_t process_u64; /* 248 8 */
}; /* 248 8 */
} __attribute__((__aligned__(64)))
__attribute__((__aligned__(64))); /* 192 64 */
/* --- cacheline 4 boundary (256 bytes) --- */
struct rte_node * nodes[]__attribute__((__aligned__(64))); /* 256 0 */
/* size: 256, cachelines: 4, members: 12 */
/* sum members: 250, holes: 2, sum holes: 6 */
/* forced alignments: 2 */
} __attribute__((__aligned__(128)));The introduced anonymous structure gets aligned on the minimum cache line size (64 bytes): with this change, ctx[] move from offset 256, to offset 192.
Similarly, nodes[] moves from offset 320 to offset 256.
As we discussed offlist, there are a few options to workaround this issue (like moving nodes[] inside the anonymous struct though it still results in an increased rte_node struct, or like adding an explicit padding field right before the newly introduced anonymous struct, ...).
- Additionally, anonymous structures are not correctly handled with libabigail 2.4 which is the version used in the CI.
At the moment, the ABI check in GHA and UNH will fail on x86 with:
1 function with some indirect sub-type change:
[C] 'function bool__rte_graph_mcore_dispatch_sched_node_enqueue(rte_node*, rte_graph_rq_head*)' at rte_graph_model_mcore_dispatch.c:117:1 has some indirect sub-type changes:
parameter 1 of type 'rte_node*' has sub-type changes:
in pointed to type 'struct rte_node' at rte_graph_worker_common.h:92:1:
type size hasn't changed
2 data member deletions:
'union {void** objs; uint64_t objs_u64;}', at offset 1920 (in bits) 'union {rte_node_process_t process; uint64_t process_u64;}',
at offset 1984 (in bits)
no data member changes (2 filtered);
On this topic, we have to either put a suppression rule on the rte_node structure, or bump the libabigail version in UNH, GHA, and the maintainers build env (though the latter won't happen overnight, and we are really close to rc1).
For those two reasons, it is better to revisit this patch and have it ready for the next release.
While at it, it may be worth cleaning up the rte_node structure in v24.11, if so, please announce in a deprecation notice for this planned ABI breakage.
Signature · 1 line
David Marchand
Sad :(
Quoted 10 lines
The introduced anonymous structure gets aligned on the minimum cache line size (64 bytes): with this change, ctx[] move from offset 256, to offset 192.
Similarly, nodes[] moves from offset 320 to offset 256.As we discussed offlist, there are a few options to workaround this issue (like moving nodes[] inside the anonymous struct though it still results in an increased
rte_nodestruct, or like adding an explicit padding field right before the newly introduced anonymous struct, ...).
[snip]
Quoted 5 lines
For those two reasons, it is better to revisit this patch and have it ready for the next release.
While at it, it may be worth cleaning up therte_nodestructure in v24.11, if so, please announce in a deprecation notice for this planned ABI breakage.
Jerin, wouldn't it be better if we managed to fill in that 64 bytes hole?
I don't know what to announce precisely about the breakage nature.
Quoted 14 lines · Robin Jarry
Sad :(
Quoted 10 lines
The introduced anonymous structure gets aligned on the minimum cache line size (64 bytes): with this change, ctx[] move from offset 256, to offset 192.
Similarly, nodes[] moves from offset 320 to offset 256.As we discussed offlist, there are a few options to workaround this issue (like moving nodes[] inside the anonymous struct though it still results in an increased
rte_nodestruct, or like adding an explicit padding field right before the newly introduced anonymous struct, ...).[snip]
Quoted 8 lines
Quoted 5 lines
For those two reasons, it is better to revisit this patch and have it ready for the next release.
While at it, it may be worth cleaning up therte_nodestructure in v24.11, if so, please announce in a deprecation notice for this planned ABI breakage.Jerin, wouldn't it be better if we managed to fill in that 64 bytes hole?
It will be available only for 128B cache line system. So may not make sense. I think, following change will resolve the issue in your patch.
From
__extension__ struct __rte_cache_min_aligned {
#define RTE_NODE_CTX_SZ 16
To
__extension__ struct __rte_cache__aligned {
#define RTE_NODE_CTX_SZ 16
Quoted 3 lines
I don't know what to announce precisely about the breakage nature.