generic rte atomic APIs deprecate proposal
cover generic rte atomic APIs deprecate proposal
DPDK provides generic rte_atomic APIs to do several atomic operations. These APIs are using the deprecated __sync built-ins and enforce full memory barriers on aarch64. However, full barriers are not necessary in many use cases. In order to address such use cases, C language offers C11 atomic APIs. The C11 atomic APIs provide finer memory barrier control by making use of the memory ordering parameter provided by the user. Various patches submitted in the past [1] and the patches in this series indicate significant performance gains on multiple aarch64 CPUs and no performance loss on x86.
But the existing rte_atomic API implementations cannot be changed as the APIs do not take the memory ordering parameter. The only choice available is replacing the usage of the rte_atomic APIs with C11 atomic APIs. In order to make this change, the following steps are proposed:
[1] deprecate rte_atomic APIs so that future patches do not use rte_atomic APIs (a script is added to flag the usages).
[2] refactor the code that uses rte_atomic APIs to use c11 atomic APIs.
This patchset contains:
1) the checkpatch script changes to flag rte_atomic API usage in patches.
2) changes to programmer guide describing writing efficient code for aarch64.
3) changes to various libraries to make use of c11 atomic APIs.
We are planning to replicate this idea across all the other libraries, drivers, examples, test applications. In the next phase, we will add changes to the mbuf, the EAL interrupts and the event timer adapter libraries.
Honnappa Nagarahalli (2):
service: avoid race condition for MT unsafe service
service: identify service running on another core correctly
Phil Yang (8):
doc: add generic atomic deprecation section
devtools: prevent use of rte atomic APIs in future patches
vhost: optimize broadcast rarp sync with c11 atomic
ipsec: optimize with c11 atomic for sa outbound sqn update
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 one-way barrier
service: relax barriers with C11 atomic operations
devtools/checkpatches.sh | 9 ++ doc/guides/prog_guide/writing_efficient_code.rst | 60 +++++++- lib/librte_eal/common/rte_service.c | 175 ++++++++++++----------- lib/librte_ipsec/ipsec_sqn.h | 3 +- lib/librte_ipsec/sa.h | 2 +- lib/librte_vhost/vhost.h | 2 +- lib/librte_vhost/vhost_user.c | 7 +- lib/librte_vhost/virtio_net.c | 16 ++- 8 files changed, 175 insertions(+), 99 deletions(-)
Signature · 1 line
2.7.4
PATCH 1/10 doc: add generic atomic deprecation section
Add deprecating the generic rte_atomic_xx APIs to c11 atomic built-ins guide and examples.
--- doc/guides/prog_guide/writing_efficient_code.rst | 60 +++++++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-)
doc/guides/prog_guide/writing_efficient_code.rst +59 -1
@@ -167,7 +167,13 @@ but with the added cost of lower throughput. Locks and Atomic Operations --------------------------- -Atomic operations imply a lock prefix before the instruction, +This section describes some key considerations when using locks and atomic +operations in the DPDK environment. + +Locks +~~~~~ + +On x86, atomic operations imply a lock prefix before the instruction, causing the processor's LOCK# signal to be asserted during execution of the following instruction. This has a big impact on performance in a multicore environment.
@@ -176,6 +182,58 @@ It can often be replaced by other solutions like per-lcore variables. Also, some locking techniques are more efficient than others. For instance, the Read-Copy-Update (RCU) algorithm can frequently replace simple rwlocks. +Atomic Operations: Use C11 Atomic Built-ins +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +DPDK `generic rte_atomic <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_atomic.h>`_ operations are +implemented by `__sync built-ins <https://gcc.gnu.org/onlinedocs/gcc/_005f_005fsync-Builtins.html>`_. +These __sync built-ins result in full barriers on aarch64, which are unnecessary +in many use cases. They can be replaced by `__atomic built-ins <https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html>`_ that +conform to the C11 memory model and provide finer memory order control. + +So replacing the rte_atomic operations with __atomic built-ins might improve +performance for aarch64 machines. `More details <https://www.dpdk.org/wp-content/uploads/sites/35/2019/10/StateofC11Code.pdf>`_. + +Some typical optimization cases are listed below: + +Atomicity +^^^^^^^^^ + +Some use cases require atomicity alone, the ordering of the memory operations +does not matter. For example the packets statistics in the `vhost <https://github.com/DPDK/dpdk/blob/v20.02/examples/vhost/main.c#L796>`_ example application. + +It just updates the number of transmitted packets, no subsequent logic depends +on these counters. So the RELAXED memory ordering is sufficient: + +.. code-block:: c + + static __rte_always_inline void + virtio_xmit(struct vhost_dev *dst_vdev, struct vhost_dev *src_vdev, + struct rte_mbuf *m) + { + ... + ... + if (enable_stats) { + __atomic_add_fetch(&dst_vdev->stats.rx_total_atomic, 1, __ATOMIC_RELAXED); + __atomic_add_fetch(&dst_vdev->stats.rx_atomic, ret, __ATOMIC_RELAXED); + ... + } + } + +One-way Barrier +^^^^^^^^^^^^^^^ + +Some use cases allow for memory reordering in one way while requiring memory +ordering in the other direction. + +For example, the memory operations before the `lock <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_spinlock.h#L66>`_ can move to the +critical section, but the memory operations in the critical section cannot move +above the lock. In this case, the full memory barrier in the CAS operation can +be replaced to ACQUIRE. On the other hand, the memory operations after the +`unlock <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_spinlock.h#L88>`_ can move to the critical section, but the memory operations in the +critical section cannot move below the unlock. So the full barrier in the STORE +operation can be replaced with RELEASE. + Coding Considerations ---------------------
Signature · 1 line
2.7.4
PATCH 2/10 devtools: prevent use of rte atomic APIs in future patches
In reply to Phil YangIn order to deprecate the rte_atomic APIs, prevent the patches from using rte_atomic APIs.
--- devtools/checkpatches.sh | 9 +++++++++ 1 file changed, 9 insertions(+)
devtools/checkpatches.sh +9 -0
@@ -61,6 +61,15 @@ check_forbidden_additions() { # <patch> -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ "$1" || res=1 + # refrain from new additions of 16/32/64 bits rte_atomic_xxx() + # multiple folders and expressions are separated by spaces + awk -v FOLDERS="lib drivers app examples" \ + -v EXPRESSIONS="rte_atomic[0-9][0-9]_.*\\\(" \ + -v RET_ON_FAIL=1 \ + -v MESSAGE='Using c11 atomic built-ins instead of rte_atomic' \ + -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ + "$1" || res=1 + # svg figures must be included with wildcard extension # because of png conversion for pdf docs awk -v FOLDERS='doc' \
Signature · 1 line
2.7.4
PATCH 3/10 vhost: optimize broadcast rarp sync with c11 atomic
In reply to Phil YangThe rarp packet broadcast flag is synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.
--- lib/librte_vhost/vhost.h | 2 +- lib/librte_vhost/vhost_user.c | 7 +++---- lib/librte_vhost/virtio_net.c | 16 +++++++++------- 3 files changed, 13 insertions(+), 12 deletions(-)
lib/librte_vhost/vhost.h +1 -1
@@ -350,7 +350,7 @@ struct virtio_net {uint32_t flags;uint16_t vhost_hlen;/* to tell if we need broadcast rarp packet */ -rte_atomic16_t broadcast_rarp; +int16_t broadcast_rarp;uint32_t nr_vring;int dequeue_zero_copy;int extbuf;
lib/librte_vhost/vhost_user.c +3 -4
@@ -2145,11 +2145,10 @@ vhost_user_send_rarp(struct virtio_net **pdev, struct VhostUserMsg *msg,* Set the flag to inject a RARP broadcast packet at * rte_vhost_dequeue_burst(). * -* rte_smp_wmb() is for making sure the mac is copied -* before the flag is set. +* __ATOMIC_RELEASE ordering is for making sure the mac is +* copied before the flag is set. */ -rte_smp_wmb (); -rte_atomic16_set (&dev->broadcast_rarp,1 ); +__atomic_store_n (&dev->broadcast_rarp,1 , __ATOMIC_RELEASE); did = dev->vdpa_dev_id; vdpa_dev =rte_vdpa_get_device (did);if (vdpa_dev && vdpa_dev->ops->migration_done)
lib/librte_vhost/virtio_net.c +9 -7
@@ -2203,6 +2203,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,struct virtio_net *dev;struct rte_mbuf *rarp_mbuf =NULL ;struct vhost_virtqueue *vq; +int success =1 ; dev =get_device (vid);if (!dev)
@@ -2249,16 +2250,17 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,* * broadcast_rarp shares a cacheline in the virtio_net structure * with some fields that are accessed during enqueue and -* rte_atomic16_cmpset() causes a write if using cmpxchg. This could -* result in false sharing between enqueue and dequeue. +* __atomic_compare_exchange_n causes a write if performed compare +* and exchange. This could result in false sharing between enqueue +* and dequeue. * * Prevent unnecessary false sharing by reading broadcast_rarp first -* and only performing cmpset if the read indicates it is likely to -* be set. +* and only performing compare and exchange if the read indicates it +* is likely to be set. */ -if (unlikely (rte_atomic16_read (&dev->broadcast_rarp) && -rte_atomic16_cmpset ((volatile uint16_t *) - &dev->broadcast_rarp.cnt,1 ,0 ))) { +if (unlikely (__atomic_load_n (&dev->broadcast_rarp, __ATOMIC_ACQUIRE) && +__atomic_compare_exchange_n (&dev->broadcast_rarp, + &success,0 ,0 , __ATOMIC_RELEASE, __ATOMIC_RELAXED))) { rarp_mbuf =rte_net_make_rarp_packet (mbuf_pool, &dev->mac);if (rarp_mbuf ==NULL ) {
Signature · 1 line
2.7.4
PATCH 4/10 ipsec: optimize with c11 atomic for sa outbound sqn update
In reply to Phil YangFor SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.
--- lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/sa.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
lib/librte_ipsec/ipsec_sqn.h +2 -1
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa->sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED);else { sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h +1 -1
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;struct {
Signature · 1 line
2.7.4
PATCH 5/10 service: remove rte prefix from static functions
In reply to Phil Yang--- lib/librte_eal/common/rte_service.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +9 -9
@@ -333,7 +333,7 @@ rte_service_runstate_get(uint32_t id) }static inline void -rte_service_runner_do_callback (struct rte_service_spec_impl *s, +service_runner_do_callback (struct rte_service_spec_impl *s,struct core_state *cs,uint32_t service_idx) {void *userdata = s->spec.callback_userdata;
@@ -376,10 +376,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ; -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);rte_atomic32_clear (&s->execute_lock); }else -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);return 0 ; }
@@ -433,7 +433,7 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) }static int32_t -rte_service_runner_func (void *arg) +service_runner_func (void *arg) {RTE_SET_USED (arg);uint32_t i;
@@ -703,7 +703,7 @@ rte_service_lcore_start(uint32_t lcore)*/ lcore_states[lcore].runstate =RUNSTATE_RUNNING ; -int ret =rte_eal_remote_launch (rte_service_runner_func,0 , lcore); +int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */ return ret; }
@@ -782,7 +782,7 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id, }static void -rte_service_dump_one (FILE *f,struct rte_service_spec_impl *s, +service_dump_one (FILE *f,struct rte_service_spec_impl *s,uint64_t all_cycles,uint32_t reset) {/* avoid divide by zero */
@@ -815,7 +815,7 @@ rte_service_attr_reset_all(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );int reset =1 ; -rte_service_dump_one (NULL , s,0 , reset); +service_dump_one (NULL , s,0 , reset);return 0 ; }
@@ -873,7 +873,7 @@ rte_service_dump(FILE *f, uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );fprintf (f,"Service %s Summary\n" , s->spec.name);uint32_t reset =0 ; -rte_service_dump_one (f, s, total_cycles, reset); +service_dump_one (f, s, total_cycles, reset);return 0 ; }
@@ -883,7 +883,7 @@ rte_service_dump(FILE *f, uint32_t id)if (!service_valid (i))continue ;uint32_t reset =0 ; -rte_service_dump_one (f, &rte_services[i], total_cycles, reset); +service_dump_one (f, &rte_services[i], total_cycles, reset); }fprintf (f,"Service Cores Summary\n" );
Signature · 1 line
2.7.4
PATCH 6/10 service: remove redundant code
In reply to Phil YangThe service id validation is verified in the calling function, remove the redundant code inside the service_update function.
--- lib/librte_eal/common/rte_service.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-)
lib/librte_eal/common/rte_service.c +12 -19
@@ -549,21 +549,10 @@ rte_service_start_with_defaults(void) }static int32_t -service_update (struct rte_service_spec *service,uint32_t lcore, +service_update (uint32_t sid,uint32_t lcore,uint32_t *set,uint32_t *enabled) { -uint32_t i; -int32_t sid =-1 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if ((struct rte_service_spec *)&rte_services[i] == service && -service_valid (i)) { - sid = i; -break ; - } - } - -if (sid ==-1 || lcore >=RTE_MAX_LCORE ) +if (lcore >=RTE_MAX_LCORE )return -EINVAL ;if (!lcore_states[lcore].is_service_core)
@@ -595,19 +584,23 @@ service_update(struct rte_service_spec *service, uint32_t lcore,int32_t rte_service_map_lcore_set (uint32_t id,uint32_t lcore,uint32_t enabled) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* validate ID, or return error value */ +if (id >=RTE_SERVICE_NUM_MAX || !service_valid (id)) +return -EINVAL ; +uint32_t on = enabled >0 ; -return service_update (&s->spec, lcore, &on,0 ); +return service_update (id, lcore, &on,0 ); }int32_t rte_service_map_lcore_get (uint32_t id,uint32_t lcore) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* validate ID, or return error value */ +if (id >=RTE_SERVICE_NUM_MAX || !service_valid (id)) +return -EINVAL ; +uint32_t enabled; -int ret =service_update (&s->spec, lcore,0 , &enabled); +int ret =service_update (id, lcore,0 , &enabled);if (ret ==0 )return enabled;return ret;
Signature · 1 line
2.7.4
PATCH 7/10 service: avoid race condition for MT unsafe service
In reply to Phil YangFrom: Honnappa Nagarahalli <[email protected]>
There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.
--- lib/librte_eal/common/rte_service.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {uint8_t internal_flags;/* per service statistics */ +/* Indicates how many cores the service is mapped to run on. +* It does not indicate the number of cores the service is running +* on currently. +*/ rte_atomic32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent;
@@ -367,12 +371,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -/* check do we need cmpset, if MT safe or <= 1 core -* mapped, atomic ops are not required. -*/ -const int use_atomics = (service_mt_safe (s) ==0 ) && - (rte_atomic32_read (&s->num_mapped_cores) >1 ); -if (use_atomics) { +if (service_mt_safe (s) ==0 ) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
Signature · 1 line
2.7.4
PATCH 8/10 service: identify service running on another core correctly
In reply to Phil YangFrom: Honnappa Nagarahalli <[email protected]>
The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes more costlier than using atomic operations. Assume that the application passes the right parameters and reduce the number of instructions for all cases.
--- lib/librte_eal/common/rte_service.c | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
@@ -357,7 +357,7 @@ service_runner_do_callback(struct rte_service_spec_impl *s,/* Expects the service 's' is valid. */ static int32_t service_run (uint32_t i,struct core_state *cs,uint64_t service_mask, -struct rte_service_spec_impl *s) +struct rte_service_spec_impl *s,uint32_t serialize_mt_unsafe) {if (!s)return -EINVAL ;
@@ -371,7 +371,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -if (service_mt_safe (s) ==0 ) { +if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
@@ -409,24 +409,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); -/* Atomically add this core to the mapped cores first, then examine if -* we can run the service. This avoids a race condition between -* checking the value, and atomically adding to the mapped count. +/* Increment num_mapped_cores to indicate that the service is +* is running on a core. */ -if (serialize_mt_unsafe) -rte_atomic32_inc (&s->num_mapped_cores); +rte_atomic32_inc (&s->num_mapped_cores); -if (service_mt_safe (s) ==0 && -rte_atomic32_read (&s->num_mapped_cores) >1 ) { -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); -return -EBUSY ; - } - -int ret =service_run (id, cs,UINT64_MAX , s); +int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); +rte_atomic32_dec (&s->num_mapped_cores);return ret; }
@@ -446,7 +436,7 @@ service_runner_func(void *arg)if (!service_valid (i))continue ;/* return value ignored as no change to code flow */ -service_run (i, cs, service_mask,service_get (i)); +service_run (i, cs, service_mask,service_get (i),1 ); } cs->loops++;
Signature · 1 line
2.7.4
PATCH 9/10 service: optimize with c11 one-way barrier
In reply to Phil YangThe num_mapped_cores and execute_lock are synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.
--- lib/librte_eal/common/rte_service.c | 50 ++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 15 deletions(-)
lib/librte_eal/common/rte_service.c +35 -15
@@ -42,7 +42,7 @@ struct rte_service_spec_impl {* running this service callback. When not set, a core may take the * lock and then run the service callback. */ -rte_atomic32_t execute_lock; +uint32_t execute_lock;/* API set/get-able variables */ int8_t app_runstate;
@@ -54,7 +54,7 @@ struct rte_service_spec_impl {* It does not indicate the number of cores the service is running * on currently. */ -rte_atomic32_t num_mapped_cores; +int32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent; } __rte_cache_aligned;
@@ -329,7 +329,8 @@ rte_service_runstate_get(uint32_t id)rte_smp_rmb ();int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); -int lcore_mapped = (rte_atomic32_read (&s->num_mapped_cores) >0 ); +int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, + __ATOMIC_RELAXED) >0 );return (s->app_runstate ==RUNSTATE_RUNNING ) && (s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -372,11 +373,20 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ;if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) { -if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 )) +uint32_t expected =0 ; +/* ACQUIRE ordering here is to prevent the callback +* function from hoisting up before the execute_lock +* setting. +*/ +if (!__atomic_compare_exchange_n (&s->execute_lock, &expected,1 , +0 , __ATOMIC_ACQUIRE, __ATOMIC_RELAXED))return -EBUSY ;service_runner_do_callback (s, cs, i); -rte_atomic32_clear (&s->execute_lock); +/* RELEASE ordering here is used to pair with ACQUIRE +* above to achieve lock semantic. +*/ +__atomic_store_n (&s->execute_lock,0 , __ATOMIC_RELEASE); }else service_runner_do_callback (s, cs, i);
@@ -412,11 +422,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)/* Increment num_mapped_cores to indicate that the service is * is running on a core. */ -rte_atomic32_inc (&s->num_mapped_cores); +__atomic_add_fetch (&s->num_mapped_cores,1 , __ATOMIC_ACQUIRE);int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -rte_atomic32_dec (&s->num_mapped_cores); +__atomic_sub_fetch (&s->num_mapped_cores,1 , __ATOMIC_RELEASE);return ret; }
@@ -549,24 +559,32 @@ service_update(uint32_t sid, uint32_t lcore,uint64_t sid_mask =UINT64_C (1 ) << sid;if (set) { -uint64_t lcore_mapped = lcore_states[lcore].service_mask & - sid_mask; +/* When multiple threads try to update the same lcore +* service concurrently, e.g. set lcore map followed +* by clear lcore map, the unsynchronized service_mask +* values have issues on the num_mapped_cores value +* consistency. So we use ACQUIRE ordering to pair with +* the RELEASE ordering to synchronize the service_mask. +*/ +uint64_t lcore_mapped =__atomic_load_n ( + &lcore_states[lcore].service_mask, + __ATOMIC_ACQUIRE) & sid_mask;if (*set && !lcore_mapped) { lcore_states[lcore].service_mask |= sid_mask; -rte_atomic32_inc (&rte_services[sid].num_mapped_cores); +__atomic_add_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELEASE); }if (!*set && lcore_mapped) { lcore_states[lcore].service_mask &= ~(sid_mask); -rte_atomic32_dec (&rte_services[sid].num_mapped_cores); +__atomic_sub_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELEASE); } }if (enabled) *enabled = !!(lcore_states[lcore].service_mask & (sid_mask)); -rte_smp_wmb (); -return 0 ; }
@@ -622,7 +640,8 @@ rte_service_lcore_reset_all(void) } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) -rte_atomic32_set (&rte_services[i].num_mapped_cores,0 ); +__atomic_store_n (&rte_services[i].num_mapped_cores,0 , + __ATOMIC_RELAXED);rte_smp_wmb ();
@@ -705,7 +724,8 @@ rte_service_lcore_stop(uint32_t lcore)int32_t enabled = service_mask & (UINT64_C (1 ) << i);int32_t service_running =rte_service_runstate_get (i);int32_t only_core = (1 == -rte_atomic32_read (&rte_services[i].num_mapped_cores)); +__atomic_load_n (&rte_services[i].num_mapped_cores, + __ATOMIC_RELAXED));/* if the core is mapped, and the service is running, and this * is the only core that is mapped, the service would cease to
Signature · 1 line
2.7.4
PATCH 10/10 service: relax barriers with C11 atomic operations
In reply to Phil YangTo guarantee the inter-threads visibility of the shareable domain, it uses a lot of rte_smp_r/wmb in the service library. This patch relaxed these barriers for service by using c11 atomic one-way barrier operations.
--- lib/librte_eal/common/rte_service.c | 45 ++++++++++++++++++++----------------- 1 file changed, 25 insertions(+), 20 deletions(-)
lib/librte_eal/common/rte_service.c +25 -20
@@ -176,9 +176,11 @@ rte_service_set_stats_enable(uint32_t id, int32_t enabled)SERVICE_VALID_GET_OR_ERR_RET (id, s,0 );if (enabled) - s->internal_flags |=SERVICE_F_STATS_ENABLED ; +__atomic_or_fetch (&s->internal_flags,SERVICE_F_STATS_ENABLED , + __ATOMIC_RELEASE);else - s->internal_flags &= ~(SERVICE_F_STATS_ENABLED ); +__atomic_and_fetch (&s->internal_flags, + ~(SERVICE_F_STATS_ENABLED ), __ATOMIC_RELEASE);return 0 ; }
@@ -190,9 +192,11 @@ rte_service_set_runstate_mapped_check(uint32_t id, int32_t enabled)SERVICE_VALID_GET_OR_ERR_RET (id, s,0 );if (enabled) - s->internal_flags |=SERVICE_F_START_CHECK ; +__atomic_or_fetch (&s->internal_flags,SERVICE_F_START_CHECK , + __ATOMIC_RELEASE);else - s->internal_flags &= ~(SERVICE_F_START_CHECK ); +__atomic_and_fetch (&s->internal_flags, ~(SERVICE_F_START_CHECK ), + __ATOMIC_RELEASE);return 0 ; }
@@ -261,8 +265,8 @@ rte_service_component_register(const struct rte_service_spec *spec, s->spec = *spec; s->internal_flags |=SERVICE_F_REGISTERED |SERVICE_F_START_CHECK ; -rte_smp_wmb (); - rte_service_count++; +/* make sure the counter update after the state change. */ +__atomic_add_fetch (&rte_service_count,1 , __ATOMIC_RELEASE);if (id_ptr) *id_ptr = free_slot;
@@ -278,9 +282,10 @@ rte_service_component_unregister(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); rte_service_count--; -rte_smp_wmb (); - s->internal_flags &= ~(SERVICE_F_REGISTERED ); +/* make sure the counter update before the state change. */ +__atomic_and_fetch (&s->internal_flags, ~(SERVICE_F_REGISTERED ), + __ATOMIC_RELEASE);/* clear the run-bit in all cores */ for (i =0 ; i <RTE_MAX_LCORE ; i++)
@@ -298,11 +303,12 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );if (runstate) - s->comp_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->comp_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -313,11 +319,12 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );if (runstate) - s->app_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->app_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->app_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->app_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -439,7 +446,8 @@ service_runner_func(void *arg)const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (lcore_states[lcore].runstate ==RUNSTATE_RUNNING ) { +while (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) {const uint64_t service_mask = cs->service_mask;for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) {
@@ -450,8 +458,6 @@ service_runner_func(void *arg) } cs->loops++; - -rte_smp_rmb (); } lcore_config[lcore].state =WAIT ;
@@ -660,9 +666,8 @@ rte_service_lcore_add(uint32_t lcore)/* ensure that after adding a core the mask and state are defaults */ lcore_states[lcore].service_mask =0 ; - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; - -rte_smp_wmb (); +__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);return rte_eal_wait_lcore (lcore); }
Signature · 1 line
2.7.4
v2 cover generic rte atomic APIs deprecate proposal
In reply to Phil YangDPDK provides generic rte_atomic APIs to do several atomic operations. These APIs are using the deprecated __sync built-ins and enforce full memory barriers on aarch64. However, full barriers are not necessary in many use cases. In order to address such use cases, C language offers C11 atomic APIs. The C11 atomic APIs provide finer memory barrier control by making use of the memory ordering parameter provided by the user. Various patches submitted in the past [1] and the patches in this series indicate significant performance gains on multiple aarch64 CPUs and no performance loss on x86.
But the existing rte_atomic API implementations cannot be changed as the APIs do not take the memory ordering parameter. The only choice available is replacing the usage of the rte_atomic APIs with C11 atomic APIs. In order to make this change, the following steps are proposed:
[1] deprecate rte_atomic APIs so that future patches do not use rte_atomic APIs (a script is added to flag the usages).
[2] refactor the code that uses rte_atomic APIs to use c11 atomic APIs.
This patchset contains:
1) the checkpatch script changes to flag rte_atomic API usage in patches.
2) changes to programmer guide describing writing efficient code for aarch64.
3) changes to various libraries to make use of c11 atomic APIs.
We are planning to replicate this idea across all the other libraries, drivers, examples, test applications. In the next phase, we will add changes to the mbuf, the EAL interrupts and the event timer adapter libraries.
v2:
1. fix Clang '-Wincompatible-pointer-types' WARNING.
2. fix typos.
Honnappa Nagarahalli (2):
service: avoid race condition for MT unsafe service
service: identify service running on another core correctly
Phil Yang (8):
doc: add generic atomic deprecation section
devtools: prevent use of rte atomic APIs in future patches
vhost: optimize broadcast rarp sync with c11 atomic
ipsec: optimize with c11 atomic for sa outbound sqn update
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 one-way barrier
service: relax barriers with C11 atomic operations
devtools/checkpatches.sh | 9 ++ doc/guides/prog_guide/writing_efficient_code.rst | 60 +++++++- lib/librte_eal/common/rte_service.c | 175 ++++++++++++----------- lib/librte_ipsec/ipsec_sqn.h | 3 +- lib/librte_ipsec/sa.h | 2 +- lib/librte_vhost/vhost.h | 2 +- lib/librte_vhost/vhost_user.c | 7 +- lib/librte_vhost/virtio_net.c | 16 ++- 8 files changed, 175 insertions(+), 99 deletions(-)
Signature · 1 line
2.7.4
PATCH v2 1/10 doc: add generic atomic deprecation section
Add deprecating the generic rte_atomic_xx APIs to c11 atomic built-ins guide and examples.
--- doc/guides/prog_guide/writing_efficient_code.rst | 60 +++++++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-)
doc/guides/prog_guide/writing_efficient_code.rst +59 -1
@@ -167,7 +167,13 @@ but with the added cost of lower throughput. Locks and Atomic Operations --------------------------- -Atomic operations imply a lock prefix before the instruction, +This section describes some key considerations when using locks and atomic +operations in the DPDK environment. + +Locks +~~~~~ + +On x86, atomic operations imply a lock prefix before the instruction, causing the processor's LOCK# signal to be asserted during execution of the following instruction. This has a big impact on performance in a multicore environment.
@@ -176,6 +182,58 @@ It can often be replaced by other solutions like per-lcore variables. Also, some locking techniques are more efficient than others. For instance, the Read-Copy-Update (RCU) algorithm can frequently replace simple rwlocks. +Atomic Operations: Use C11 Atomic Built-ins +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +DPDK `generic rte_atomic <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_atomic.h>`_ operations are +implemented by `__sync built-ins <https://gcc.gnu.org/onlinedocs/gcc/_005f_005fsync-Builtins.html>`_. +These __sync built-ins result in full barriers on aarch64, which are unnecessary +in many use cases. They can be replaced by `__atomic built-ins <https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html>`_ that +conform to the C11 memory model and provide finer memory order control. + +So replacing the rte_atomic operations with __atomic built-ins might improve +performance for aarch64 machines. `More details <https://www.dpdk.org/wp-content/uploads/sites/35/2019/10/StateofC11Code.pdf>`_. + +Some typical optimization cases are listed below: + +Atomicity +^^^^^^^^^ + +Some use cases require atomicity alone, the ordering of the memory operations +does not matter. For example the packets statistics in the `vhost <https://github.com/DPDK/dpdk/blob/v20.02/examples/vhost/main.c#L796>`_ example application. + +It just updates the number of transmitted packets, no subsequent logic depends +on these counters. So the RELAXED memory ordering is sufficient: + +.. code-block:: c + + static __rte_always_inline void + virtio_xmit(struct vhost_dev *dst_vdev, struct vhost_dev *src_vdev, + struct rte_mbuf *m) + { + ... + ... + if (enable_stats) { + __atomic_add_fetch(&dst_vdev->stats.rx_total_atomic, 1, __ATOMIC_RELAXED); + __atomic_add_fetch(&dst_vdev->stats.rx_atomic, ret, __ATOMIC_RELAXED); + ... + } + } + +One-way Barrier +^^^^^^^^^^^^^^^ + +Some use cases allow for memory reordering in one way while requiring memory +ordering in the other direction. + +For example, the memory operations before the `lock <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_spinlock.h#L66>`_ can move to the +critical section, but the memory operations in the critical section cannot move +above the lock. In this case, the full memory barrier in the CAS operation can +be replaced to ACQUIRE. On the other hand, the memory operations after the +`unlock <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_spinlock.h#L88>`_ can move to the critical section, but the memory operations in the +critical section cannot move below the unlock. So the full barrier in the STORE +operation can be replaced with RELEASE. + Coding Considerations ---------------------
Signature · 1 line
2.7.4
PATCH v2 2/10 devtools: prevent use of rte atomic APIs in future patches
In reply to Phil YangIn order to deprecate the rte_atomic APIs, prevent the patches from using rte_atomic APIs.
--- devtools/checkpatches.sh | 9 +++++++++ 1 file changed, 9 insertions(+)
devtools/checkpatches.sh +9 -0
@@ -61,6 +61,15 @@ check_forbidden_additions() { # <patch> -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ "$1" || res=1 + # refrain from new additions of 16/32/64 bits rte_atomic_xxx() + # multiple folders and expressions are separated by spaces + awk -v FOLDERS="lib drivers app examples" \ + -v EXPRESSIONS="rte_atomic[0-9][0-9]_.*\\\(" \ + -v RET_ON_FAIL=1 \ + -v MESSAGE='Using c11 atomic built-ins instead of rte_atomic' \ + -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ + "$1" || res=1 + # svg figures must be included with wildcard extension # because of png conversion for pdf docs awk -v FOLDERS='doc' \
Signature · 1 line
2.7.4
PATCH v2 3/10 vhost: optimize broadcast rarp sync with c11 atomic
In reply to Phil YangThe rarp packet broadcast flag is synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.
--- lib/librte_vhost/vhost.h | 2 +- lib/librte_vhost/vhost_user.c | 7 +++---- lib/librte_vhost/virtio_net.c | 16 +++++++++------- 3 files changed, 13 insertions(+), 12 deletions(-)
lib/librte_vhost/vhost.h +1 -1
@@ -350,7 +350,7 @@ struct virtio_net {uint32_t flags;uint16_t vhost_hlen;/* to tell if we need broadcast rarp packet */ -rte_atomic16_t broadcast_rarp; +int16_t broadcast_rarp;uint32_t nr_vring;int dequeue_zero_copy;int extbuf;
lib/librte_vhost/vhost_user.c +3 -4
@@ -2145,11 +2145,10 @@ vhost_user_send_rarp(struct virtio_net **pdev, struct VhostUserMsg *msg,* Set the flag to inject a RARP broadcast packet at * rte_vhost_dequeue_burst(). * -* rte_smp_wmb() is for making sure the mac is copied -* before the flag is set. +* __ATOMIC_RELEASE ordering is for making sure the mac is +* copied before the flag is set. */ -rte_smp_wmb (); -rte_atomic16_set (&dev->broadcast_rarp,1 ); +__atomic_store_n (&dev->broadcast_rarp,1 , __ATOMIC_RELEASE); did = dev->vdpa_dev_id; vdpa_dev =rte_vdpa_get_device (did);if (vdpa_dev && vdpa_dev->ops->migration_done)
lib/librte_vhost/virtio_net.c +9 -7
@@ -2203,6 +2203,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,struct virtio_net *dev;struct rte_mbuf *rarp_mbuf =NULL ;struct vhost_virtqueue *vq; +int16_t success =1 ; dev =get_device (vid);if (!dev)
@@ -2249,16 +2250,17 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,* * broadcast_rarp shares a cacheline in the virtio_net structure * with some fields that are accessed during enqueue and -* rte_atomic16_cmpset() causes a write if using cmpxchg. This could -* result in false sharing between enqueue and dequeue. +* __atomic_compare_exchange_n causes a write if performed compare +* and exchange. This could result in false sharing between enqueue +* and dequeue. * * Prevent unnecessary false sharing by reading broadcast_rarp first -* and only performing cmpset if the read indicates it is likely to -* be set. +* and only performing compare and exchange if the read indicates it +* is likely to be set. */ -if (unlikely (rte_atomic16_read (&dev->broadcast_rarp) && -rte_atomic16_cmpset ((volatile uint16_t *) - &dev->broadcast_rarp.cnt,1 ,0 ))) { +if (unlikely (__atomic_load_n (&dev->broadcast_rarp, __ATOMIC_ACQUIRE) && +__atomic_compare_exchange_n (&dev->broadcast_rarp, + &success,0 ,0 , __ATOMIC_RELEASE, __ATOMIC_RELAXED))) { rarp_mbuf =rte_net_make_rarp_packet (mbuf_pool, &dev->mac);if (rarp_mbuf ==NULL ) {
Signature · 1 line
2.7.4
PATCH v2 4/10 ipsec: optimize with c11 atomic for sa outbound sqn update
In reply to Phil YangFor SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.
--- lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/sa.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
lib/librte_ipsec/ipsec_sqn.h +2 -1
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa->sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED);else { sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h +1 -1
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;struct {
Signature · 1 line
2.7.4
PATCH v2 5/10 service: remove rte prefix from static functions
In reply to Phil Yang--- lib/librte_eal/common/rte_service.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +9 -9
@@ -333,7 +333,7 @@ rte_service_runstate_get(uint32_t id) }static inline void -rte_service_runner_do_callback (struct rte_service_spec_impl *s, +service_runner_do_callback (struct rte_service_spec_impl *s,struct core_state *cs,uint32_t service_idx) {void *userdata = s->spec.callback_userdata;
@@ -376,10 +376,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ; -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);rte_atomic32_clear (&s->execute_lock); }else -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);return 0 ; }
@@ -433,7 +433,7 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) }static int32_t -rte_service_runner_func (void *arg) +service_runner_func (void *arg) {RTE_SET_USED (arg);uint32_t i;
@@ -703,7 +703,7 @@ rte_service_lcore_start(uint32_t lcore)*/ lcore_states[lcore].runstate =RUNSTATE_RUNNING ; -int ret =rte_eal_remote_launch (rte_service_runner_func,0 , lcore); +int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */ return ret; }
@@ -782,7 +782,7 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id, }static void -rte_service_dump_one (FILE *f,struct rte_service_spec_impl *s, +service_dump_one (FILE *f,struct rte_service_spec_impl *s,uint64_t all_cycles,uint32_t reset) {/* avoid divide by zero */
@@ -815,7 +815,7 @@ rte_service_attr_reset_all(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );int reset =1 ; -rte_service_dump_one (NULL , s,0 , reset); +service_dump_one (NULL , s,0 , reset);return 0 ; }
@@ -873,7 +873,7 @@ rte_service_dump(FILE *f, uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );fprintf (f,"Service %s Summary\n" , s->spec.name);uint32_t reset =0 ; -rte_service_dump_one (f, s, total_cycles, reset); +service_dump_one (f, s, total_cycles, reset);return 0 ; }
@@ -883,7 +883,7 @@ rte_service_dump(FILE *f, uint32_t id)if (!service_valid (i))continue ;uint32_t reset =0 ; -rte_service_dump_one (f, &rte_services[i], total_cycles, reset); +service_dump_one (f, &rte_services[i], total_cycles, reset); }fprintf (f,"Service Cores Summary\n" );
Signature · 1 line
2.7.4
PATCH v2 6/10 service: remove redundant code
In reply to Phil YangThe service id validation is verified in the calling function, remove the redundant code inside the service_update function.
--- lib/librte_eal/common/rte_service.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-)
lib/librte_eal/common/rte_service.c +12 -19
@@ -549,21 +549,10 @@ rte_service_start_with_defaults(void) }static int32_t -service_update (struct rte_service_spec *service,uint32_t lcore, +service_update (uint32_t sid,uint32_t lcore,uint32_t *set,uint32_t *enabled) { -uint32_t i; -int32_t sid =-1 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if ((struct rte_service_spec *)&rte_services[i] == service && -service_valid (i)) { - sid = i; -break ; - } - } - -if (sid ==-1 || lcore >=RTE_MAX_LCORE ) +if (lcore >=RTE_MAX_LCORE )return -EINVAL ;if (!lcore_states[lcore].is_service_core)
@@ -595,19 +584,23 @@ service_update(struct rte_service_spec *service, uint32_t lcore,int32_t rte_service_map_lcore_set (uint32_t id,uint32_t lcore,uint32_t enabled) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* validate ID, or return error value */ +if (id >=RTE_SERVICE_NUM_MAX || !service_valid (id)) +return -EINVAL ; +uint32_t on = enabled >0 ; -return service_update (&s->spec, lcore, &on,0 ); +return service_update (id, lcore, &on,0 ); }int32_t rte_service_map_lcore_get (uint32_t id,uint32_t lcore) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* validate ID, or return error value */ +if (id >=RTE_SERVICE_NUM_MAX || !service_valid (id)) +return -EINVAL ; +uint32_t enabled; -int ret =service_update (&s->spec, lcore,0 , &enabled); +int ret =service_update (id, lcore,0 , &enabled);if (ret ==0 )return enabled;return ret;
Signature · 1 line
2.7.4
PATCH v2 7/10 service: avoid race condition for MT unsafe service
In reply to Phil YangFrom: Honnappa Nagarahalli <[email protected]>
There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.
--- lib/librte_eal/common/rte_service.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {uint8_t internal_flags;/* per service statistics */ +/* Indicates how many cores the service is mapped to run on. +* It does not indicate the number of cores the service is running +* on currently. +*/ rte_atomic32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent;
@@ -367,12 +371,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -/* check do we need cmpset, if MT safe or <= 1 core -* mapped, atomic ops are not required. -*/ -const int use_atomics = (service_mt_safe (s) ==0 ) && - (rte_atomic32_read (&s->num_mapped_cores) >1 ); -if (use_atomics) { +if (service_mt_safe (s) ==0 ) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
Signature · 1 line
2.7.4
PATCH v2 8/10 service: identify service running on another core correctly
In reply to Phil YangFrom: Honnappa Nagarahalli <[email protected]>
The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases.
--- lib/librte_eal/common/rte_service.c | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
@@ -357,7 +357,7 @@ service_runner_do_callback(struct rte_service_spec_impl *s,/* Expects the service 's' is valid. */ static int32_t service_run (uint32_t i,struct core_state *cs,uint64_t service_mask, -struct rte_service_spec_impl *s) +struct rte_service_spec_impl *s,uint32_t serialize_mt_unsafe) {if (!s)return -EINVAL ;
@@ -371,7 +371,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -if (service_mt_safe (s) ==0 ) { +if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
@@ -409,24 +409,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); -/* Atomically add this core to the mapped cores first, then examine if -* we can run the service. This avoids a race condition between -* checking the value, and atomically adding to the mapped count. +/* Increment num_mapped_cores to indicate that the service +* is running on a core. */ -if (serialize_mt_unsafe) -rte_atomic32_inc (&s->num_mapped_cores); +rte_atomic32_inc (&s->num_mapped_cores); -if (service_mt_safe (s) ==0 && -rte_atomic32_read (&s->num_mapped_cores) >1 ) { -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); -return -EBUSY ; - } - -int ret =service_run (id, cs,UINT64_MAX , s); +int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); +rte_atomic32_dec (&s->num_mapped_cores);return ret; }
@@ -446,7 +436,7 @@ service_runner_func(void *arg)if (!service_valid (i))continue ;/* return value ignored as no change to code flow */ -service_run (i, cs, service_mask,service_get (i)); +service_run (i, cs, service_mask,service_get (i),1 ); } cs->loops++;
Signature · 1 line
2.7.4
PATCH v2 9/10 service: optimize with c11 one-way barrier
In reply to Phil YangThe num_mapped_cores and execute_lock are synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.
--- lib/librte_eal/common/rte_service.c | 50 ++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 15 deletions(-)
lib/librte_eal/common/rte_service.c +35 -15
@@ -42,7 +42,7 @@ struct rte_service_spec_impl {* running this service callback. When not set, a core may take the * lock and then run the service callback. */ -rte_atomic32_t execute_lock; +uint32_t execute_lock;/* API set/get-able variables */ int8_t app_runstate;
@@ -54,7 +54,7 @@ struct rte_service_spec_impl {* It does not indicate the number of cores the service is running * on currently. */ -rte_atomic32_t num_mapped_cores; +int32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent; } __rte_cache_aligned;
@@ -329,7 +329,8 @@ rte_service_runstate_get(uint32_t id)rte_smp_rmb ();int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); -int lcore_mapped = (rte_atomic32_read (&s->num_mapped_cores) >0 ); +int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, + __ATOMIC_RELAXED) >0 );return (s->app_runstate ==RUNSTATE_RUNNING ) && (s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -372,11 +373,20 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ;if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) { -if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 )) +uint32_t expected =0 ; +/* ACQUIRE ordering here is to prevent the callback +* function from hoisting up before the execute_lock +* setting. +*/ +if (!__atomic_compare_exchange_n (&s->execute_lock, &expected,1 , +0 , __ATOMIC_ACQUIRE, __ATOMIC_RELAXED))return -EBUSY ;service_runner_do_callback (s, cs, i); -rte_atomic32_clear (&s->execute_lock); +/* RELEASE ordering here is used to pair with ACQUIRE +* above to achieve lock semantic. +*/ +__atomic_store_n (&s->execute_lock,0 , __ATOMIC_RELEASE); }else service_runner_do_callback (s, cs, i);
@@ -412,11 +422,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)/* Increment num_mapped_cores to indicate that the service * is running on a core. */ -rte_atomic32_inc (&s->num_mapped_cores); +__atomic_add_fetch (&s->num_mapped_cores,1 , __ATOMIC_ACQUIRE);int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -rte_atomic32_dec (&s->num_mapped_cores); +__atomic_sub_fetch (&s->num_mapped_cores,1 , __ATOMIC_RELEASE);return ret; }
@@ -549,24 +559,32 @@ service_update(uint32_t sid, uint32_t lcore,uint64_t sid_mask =UINT64_C (1 ) << sid;if (set) { -uint64_t lcore_mapped = lcore_states[lcore].service_mask & - sid_mask; +/* When multiple threads try to update the same lcore +* service concurrently, e.g. set lcore map followed +* by clear lcore map, the unsynchronized service_mask +* values have issues on the num_mapped_cores value +* consistency. So we use ACQUIRE ordering to pair with +* the RELEASE ordering to synchronize the service_mask. +*/ +uint64_t lcore_mapped =__atomic_load_n ( + &lcore_states[lcore].service_mask, + __ATOMIC_ACQUIRE) & sid_mask;if (*set && !lcore_mapped) { lcore_states[lcore].service_mask |= sid_mask; -rte_atomic32_inc (&rte_services[sid].num_mapped_cores); +__atomic_add_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELEASE); }if (!*set && lcore_mapped) { lcore_states[lcore].service_mask &= ~(sid_mask); -rte_atomic32_dec (&rte_services[sid].num_mapped_cores); +__atomic_sub_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELEASE); } }if (enabled) *enabled = !!(lcore_states[lcore].service_mask & (sid_mask)); -rte_smp_wmb (); -return 0 ; }
@@ -622,7 +640,8 @@ rte_service_lcore_reset_all(void) } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) -rte_atomic32_set (&rte_services[i].num_mapped_cores,0 ); +__atomic_store_n (&rte_services[i].num_mapped_cores,0 , + __ATOMIC_RELAXED);rte_smp_wmb ();
@@ -705,7 +724,8 @@ rte_service_lcore_stop(uint32_t lcore)int32_t enabled = service_mask & (UINT64_C (1 ) << i);int32_t service_running =rte_service_runstate_get (i);int32_t only_core = (1 == -rte_atomic32_read (&rte_services[i].num_mapped_cores)); +__atomic_load_n (&rte_services[i].num_mapped_cores, + __ATOMIC_RELAXED));/* if the core is mapped, and the service is running, and this * is the only core that is mapped, the service would cease to
Signature · 1 line
2.7.4
PATCH v2 10/10 service: relax barriers with C11 atomic operations
In reply to Phil YangTo guarantee the inter-threads visibility of the shareable domain, it uses a lot of rte_smp_r/wmb in the service library. This patch relaxed these barriers for service by using c11 atomic one-way barrier operations.
--- lib/librte_eal/common/rte_service.c | 45 ++++++++++++++++++++----------------- 1 file changed, 25 insertions(+), 20 deletions(-)
lib/librte_eal/common/rte_service.c +25 -20
@@ -176,9 +176,11 @@ rte_service_set_stats_enable(uint32_t id, int32_t enabled)SERVICE_VALID_GET_OR_ERR_RET (id, s,0 );if (enabled) - s->internal_flags |=SERVICE_F_STATS_ENABLED ; +__atomic_or_fetch (&s->internal_flags,SERVICE_F_STATS_ENABLED , + __ATOMIC_RELEASE);else - s->internal_flags &= ~(SERVICE_F_STATS_ENABLED ); +__atomic_and_fetch (&s->internal_flags, + ~(SERVICE_F_STATS_ENABLED ), __ATOMIC_RELEASE);return 0 ; }
@@ -190,9 +192,11 @@ rte_service_set_runstate_mapped_check(uint32_t id, int32_t enabled)SERVICE_VALID_GET_OR_ERR_RET (id, s,0 );if (enabled) - s->internal_flags |=SERVICE_F_START_CHECK ; +__atomic_or_fetch (&s->internal_flags,SERVICE_F_START_CHECK , + __ATOMIC_RELEASE);else - s->internal_flags &= ~(SERVICE_F_START_CHECK ); +__atomic_and_fetch (&s->internal_flags, ~(SERVICE_F_START_CHECK ), + __ATOMIC_RELEASE);return 0 ; }
@@ -261,8 +265,8 @@ rte_service_component_register(const struct rte_service_spec *spec, s->spec = *spec; s->internal_flags |=SERVICE_F_REGISTERED |SERVICE_F_START_CHECK ; -rte_smp_wmb (); - rte_service_count++; +/* make sure the counter update after the state change. */ +__atomic_add_fetch (&rte_service_count,1 , __ATOMIC_RELEASE);if (id_ptr) *id_ptr = free_slot;
@@ -278,9 +282,10 @@ rte_service_component_unregister(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); rte_service_count--; -rte_smp_wmb (); - s->internal_flags &= ~(SERVICE_F_REGISTERED ); +/* make sure the counter update before the state change. */ +__atomic_and_fetch (&s->internal_flags, ~(SERVICE_F_REGISTERED ), + __ATOMIC_RELEASE);/* clear the run-bit in all cores */ for (i =0 ; i <RTE_MAX_LCORE ; i++)
@@ -298,11 +303,12 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );if (runstate) - s->comp_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->comp_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -313,11 +319,12 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );if (runstate) - s->app_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->app_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->app_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->app_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -439,7 +446,8 @@ service_runner_func(void *arg)const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (lcore_states[lcore].runstate ==RUNSTATE_RUNNING ) { +while (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) {const uint64_t service_mask = cs->service_mask;for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) {
@@ -450,8 +458,6 @@ service_runner_func(void *arg) } cs->loops++; - -rte_smp_rmb (); } lcore_config[lcore].state =WAIT ;
@@ -660,9 +666,8 @@ rte_service_lcore_add(uint32_t lcore)/* ensure that after adding a core the mask and state are defaults */ lcore_states[lcore].service_mask =0 ; - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; - -rte_smp_wmb (); +__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);return rte_eal_wait_lcore (lcore); }
Signature · 1 line
2.7.4
v3 cover generic rte atomic APIs deprecate proposal
In reply to Phil YangDPDK provides generic rte_atomic APIs to do several atomic operations. These APIs are using the deprecated __sync built-ins and enforce full memory barriers on aarch64. However, full barriers are not necessary in many use cases. In order to address such use cases, C language offers C11 atomic APIs. The C11 atomic APIs provide finer memory barrier control by making use of the memory ordering parameter provided by the user. Various patches submitted in the past [2] and the patches in this series indicate significant performance gains on multiple aarch64 CPUs and no performance loss on x86.
But the existing rte_atomic API implementations cannot be changed as the APIs do not take the memory ordering parameter. The only choice available is replacing the usage of the rte_atomic APIs with C11 atomic APIs. In order to make this change, the following steps are proposed:
[1] deprecate rte_atomic APIs so that future patches do not use rte_atomic APIs (a script is added to flag the usages).
[2] refactor the code that uses rte_atomic APIs to use c11 atomic APIs.
This patchset contains:
1) the checkpatch script changes to flag rte_atomic API usage in patches.
2) changes to programmer guide describing writing efficient code for aarch64.
3) changes to various libraries to make use of c11 atomic APIs.
We are planning to replicate this idea across all the other libraries, drivers, examples, test applications. In the next phase, we will add changes to the mbuf, the EAL interrupts and the event timer adapter libraries.
v3:
add libatomic dependency for 32-bit clang
v2:
1. fix Clang '-Wincompatible-pointer-types' WARNING.
2. fix typos.
Honnappa Nagarahalli (2):
service: avoid race condition for MT unsafe service
service: identify service running on another core correctly
Phil Yang (10):
doc: add generic atomic deprecation section
devtools: prevent use of rte atomic APIs in future patches eal/build: add libatomic dependency for 32-bit clang
build: remove redundant code
vhost: optimize broadcast rarp sync with c11 atomic
ipsec: optimize with c11 atomic for sa outbound sqn update
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 one-way barrier
service: relax barriers with C11 atomic operations
devtools/checkpatches.sh | 9 ++ doc/guides/prog_guide/writing_efficient_code.rst | 60 +++++++- drivers/event/octeontx/meson.build | 5 - drivers/event/octeontx2/meson.build | 5 - drivers/event/opdl/meson.build | 5 - lib/librte_eal/common/rte_service.c | 175 ++++++++++++----------- lib/librte_eal/meson.build | 6 + lib/librte_ipsec/ipsec_sqn.h | 3 +- lib/librte_ipsec/sa.h | 2 +- lib/librte_rcu/meson.build | 5 - lib/librte_vhost/vhost.h | 2 +- lib/librte_vhost/vhost_user.c | 7 +- lib/librte_vhost/virtio_net.c | 16 ++- 13 files changed, 181 insertions(+), 119 deletions(-)
Signature · 1 line
2.7.4
PATCH v3 1/12 doc: add generic atomic deprecation section
Add deprecating the generic rte_atomic_xx APIs to c11 atomic built-ins guide and examples.
--- doc/guides/prog_guide/writing_efficient_code.rst | 60 +++++++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-)
doc/guides/prog_guide/writing_efficient_code.rst +59 -1
@@ -167,7 +167,13 @@ but with the added cost of lower throughput. Locks and Atomic Operations --------------------------- -Atomic operations imply a lock prefix before the instruction, +This section describes some key considerations when using locks and atomic +operations in the DPDK environment. + +Locks +~~~~~ + +On x86, atomic operations imply a lock prefix before the instruction, causing the processor's LOCK# signal to be asserted during execution of the following instruction. This has a big impact on performance in a multicore environment.
@@ -176,6 +182,58 @@ It can often be replaced by other solutions like per-lcore variables. Also, some locking techniques are more efficient than others. For instance, the Read-Copy-Update (RCU) algorithm can frequently replace simple rwlocks. +Atomic Operations: Use C11 Atomic Built-ins +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +DPDK `generic rte_atomic <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_atomic.h>`_ operations are +implemented by `__sync built-ins <https://gcc.gnu.org/onlinedocs/gcc/_005f_005fsync-Builtins.html>`_. +These __sync built-ins result in full barriers on aarch64, which are unnecessary +in many use cases. They can be replaced by `__atomic built-ins <https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html>`_ that +conform to the C11 memory model and provide finer memory order control. + +So replacing the rte_atomic operations with __atomic built-ins might improve +performance for aarch64 machines. `More details <https://www.dpdk.org/wp-content/uploads/sites/35/2019/10/StateofC11Code.pdf>`_. + +Some typical optimization cases are listed below: + +Atomicity +^^^^^^^^^ + +Some use cases require atomicity alone, the ordering of the memory operations +does not matter. For example the packets statistics in the `vhost <https://github.com/DPDK/dpdk/blob/v20.02/examples/vhost/main.c#L796>`_ example application. + +It just updates the number of transmitted packets, no subsequent logic depends +on these counters. So the RELAXED memory ordering is sufficient: + +.. code-block:: c + + static __rte_always_inline void + virtio_xmit(struct vhost_dev *dst_vdev, struct vhost_dev *src_vdev, + struct rte_mbuf *m) + { + ... + ... + if (enable_stats) { + __atomic_add_fetch(&dst_vdev->stats.rx_total_atomic, 1, __ATOMIC_RELAXED); + __atomic_add_fetch(&dst_vdev->stats.rx_atomic, ret, __ATOMIC_RELAXED); + ... + } + } + +One-way Barrier +^^^^^^^^^^^^^^^ + +Some use cases allow for memory reordering in one way while requiring memory +ordering in the other direction. + +For example, the memory operations before the `lock <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_spinlock.h#L66>`_ can move to the +critical section, but the memory operations in the critical section cannot move +above the lock. In this case, the full memory barrier in the CAS operation can +be replaced to ACQUIRE. On the other hand, the memory operations after the +`unlock <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_spinlock.h#L88>`_ can move to the critical section, but the memory operations in the +critical section cannot move below the unlock. So the full barrier in the STORE +operation can be replaced with RELEASE. + Coding Considerations ---------------------
Signature · 1 line
2.7.4
PATCH v3 2/12 devtools: prevent use of rte atomic APIs in future patches
In reply to Phil YangIn order to deprecate the rte_atomic APIs, prevent the patches from using rte_atomic APIs.
--- devtools/checkpatches.sh | 9 +++++++++ 1 file changed, 9 insertions(+)
devtools/checkpatches.sh +9 -0
@@ -61,6 +61,15 @@ check_forbidden_additions() { # <patch> -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ "$1" || res=1 + # refrain from new additions of 16/32/64 bits rte_atomic_xxx() + # multiple folders and expressions are separated by spaces + awk -v FOLDERS="lib drivers app examples" \ + -v EXPRESSIONS="rte_atomic[0-9][0-9]_.*\\\(" \ + -v RET_ON_FAIL=1 \ + -v MESSAGE='Using c11 atomic built-ins instead of rte_atomic' \ + -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ + "$1" || res=1 + # svg figures must be included with wildcard extension # because of png conversion for pdf docs awk -v FOLDERS='doc' \
Signature · 1 line
2.7.4
PATCH v3 3/12 eal/build: add libatomic dependency for 32-bit clang
In reply to Phil YangWhen compiling with clang on 32-bit platforms, we are missing copies of 64-bit atomic functions. We can solve this by linking against libatomic for the drivers and libs which need those atomic ops.
--- lib/librte_eal/meson.build | 6 ++++++ 1 file changed, 6 insertions(+)
lib/librte_eal/meson.build +6 -0
@@ -20,6 +20,12 @@ endif if cc.has_function('getentropy', prefix : '#include <unistd.h>') cflags += '-DRTE_LIBEAL_USE_GETENTROPY' endif + +# for clang 32-bit compiles we need libatomic for 64-bit atomic ops +if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false + ext_deps += cc.find_library('atomic') +endif + sources = common_sources + env_sources objs = common_objs + env_objs headers = common_headers + env_headers
Signature · 1 line
2.7.4
Re: [dpdk-dev] [PATCH v3 03/12] eal/build: add libatomic dependency for 32-bit clang
Quoted 30 lines
-----Original Message-----
From: dev <[email protected]> On Behalf Of Phil Yang
Sent: Tuesday, March 17, 2020 9:18 AM
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>
Subject: [dpdk-dev] [PATCH v3 03/12] eal/build: add libatomic dependency for 32-bit clangWhen compiling with clang on 32-bit platforms, we are missing copies of 64-bit atomic functions. We can solve this by linking against libatomic for the drivers and libs which need those atomic ops.
Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>--- lib/librte_eal/meson.build | 6 ++++++ 1 file changed, 6 insertions(+)lib/librte_eal/meson.build +0
-0
lib/librte_eal/meson.build endif · line 28
Quoted 9 lines
lib/librte_eal/meson.build +6
-0@@ -20,6 +20,12 @@ endif if cc.has_function('getentropy', prefix : '#include <unistd.h>') cflags += '-DRTE_LIBEAL_USE_GETENTROPY' endif + +# for clang 32-bit compiles we need libatomic for 64-bit atomic ops +if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false + ext_deps += cc.find_library('atomic') +endif +
This should be unneeded since:
https://git.dpdk.org/dpdk/commit/?id=da4eae278b56e698c64d0c39939a7a55c5b6abdd
Thanks,
Phil Yang
Quoted 5 lines
lib/librte_eal/meson.build +0
-1sources = common_sources + env_sources objs = common_objs + env_objs headers = common_headers + env_headers --2.7.4
PATCH v3 4/12 build: remove redundant code
In reply to Phil YangAll these libs and drivers are built upon the eal lib. So when compiling with clang on 32-bit platforms linking against libatomic for the eal lib is sufficient. Remove the redundant code.
--- drivers/event/octeontx/meson.build | 5 ----- drivers/event/octeontx2/meson.build | 5 ----- drivers/event/opdl/meson.build | 5 ----- lib/librte_rcu/meson.build | 5 ----- 4 files changed, 20 deletions(-)
drivers/event/octeontx/meson.build +0 -5
@@ -11,8 +11,3 @@ sources = files('ssovf_worker.c', ) deps += ['common_octeontx', 'mempool_octeontx', 'bus_vdev', 'pmd_octeontx'] - -# for clang 32-bit compiles we need libatomic for 64-bit atomic ops -if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false - ext_deps += cc.find_library('atomic') -endif
drivers/event/octeontx2/meson.build +0 -5
@@ -20,11 +20,6 @@ if not dpdk_conf.get('RTE_ARCH_64') extra_flags += ['-Wno-int-to-pointer-cast', '-Wno-pointer-to-int-cast'] endif -# for clang 32-bit compiles we need libatomic for 64-bit atomic ops -if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false - ext_deps += cc.find_library('atomic') -endif - foreach flag: extra_flags if cc.has_argument(flag) cflags += flag
drivers/event/opdl/meson.build +0 -5
@@ -10,8 +10,3 @@ sources = files( 'opdl_test.c', ) deps += ['bus_vdev'] - -# for clang 32-bit compiles we need libatomic for 64-bit atomic ops -if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false - ext_deps += cc.find_library('atomic') -endif
lib/librte_rcu/meson.build +0 -5
@@ -5,8 +5,3 @@ allow_experimental_apis = true sources = files('rte_rcu_qsbr.c') headers = files('rte_rcu_qsbr.h') - -# for clang 32-bit compiles we need libatomic for 64-bit atomic ops -if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false - ext_deps += cc.find_library('atomic') -endif
Signature · 1 line
2.7.4
Re: [dpdk-dev] [PATCH v3 04/12] build: remove redundant code
Quoted 20 lines
-----Original Message-----
From: dev <[email protected]> On Behalf Of Phil Yang
Sent: Tuesday, March 17, 2020 9:18 AM
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>
Subject: [dpdk-dev] [PATCH v3 04/12] build: remove redundant codeAll these libs and drivers are built upon the eal lib. So when compiling with clang on 32-bit platforms linking against libatomic for the eal lib is sufficient. Remove the redundant code.
Suggested-by Ruifeng Wang <[email protected]>Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>
Quoted 6 lines
--- drivers/event/octeontx/meson.build | 5 ----- drivers/event/octeontx2/meson.build | 5 ----- drivers/event/opdl/meson.build | 5 ----- lib/librte_rcu/meson.build | 5 ----- 4 files changed, 20 deletions(-)
This should be unneeded since:
https://git.dpdk.org/dpdk/commit/?id=da4eae278b56e698c64d0c39939a7a55c5b6abdd
Abandon this patch.
Thanks,
Phil Yang
<snip>
PATCH v3 5/12 vhost: optimize broadcast rarp sync with c11 atomic
In reply to Phil YangThe rarp packet broadcast flag is synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.
--- lib/librte_vhost/vhost.h | 2 +- lib/librte_vhost/vhost_user.c | 7 +++---- lib/librte_vhost/virtio_net.c | 16 +++++++++------- 3 files changed, 13 insertions(+), 12 deletions(-)
lib/librte_vhost/vhost.h +1 -1
@@ -350,7 +350,7 @@ struct virtio_net {uint32_t flags;uint16_t vhost_hlen;/* to tell if we need broadcast rarp packet */ -rte_atomic16_t broadcast_rarp; +int16_t broadcast_rarp;uint32_t nr_vring;int dequeue_zero_copy;int extbuf;
lib/librte_vhost/vhost_user.c +3 -4
@@ -2145,11 +2145,10 @@ vhost_user_send_rarp(struct virtio_net **pdev, struct VhostUserMsg *msg,* Set the flag to inject a RARP broadcast packet at * rte_vhost_dequeue_burst(). * -* rte_smp_wmb() is for making sure the mac is copied -* before the flag is set. +* __ATOMIC_RELEASE ordering is for making sure the mac is +* copied before the flag is set. */ -rte_smp_wmb (); -rte_atomic16_set (&dev->broadcast_rarp,1 ); +__atomic_store_n (&dev->broadcast_rarp,1 , __ATOMIC_RELEASE); did = dev->vdpa_dev_id; vdpa_dev =rte_vdpa_get_device (did);if (vdpa_dev && vdpa_dev->ops->migration_done)
lib/librte_vhost/virtio_net.c +9 -7
@@ -2203,6 +2203,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,struct virtio_net *dev;struct rte_mbuf *rarp_mbuf =NULL ;struct vhost_virtqueue *vq; +int16_t success =1 ; dev =get_device (vid);if (!dev)
@@ -2249,16 +2250,17 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,* * broadcast_rarp shares a cacheline in the virtio_net structure * with some fields that are accessed during enqueue and -* rte_atomic16_cmpset() causes a write if using cmpxchg. This could -* result in false sharing between enqueue and dequeue. +* __atomic_compare_exchange_n causes a write if performed compare +* and exchange. This could result in false sharing between enqueue +* and dequeue. * * Prevent unnecessary false sharing by reading broadcast_rarp first -* and only performing cmpset if the read indicates it is likely to -* be set. +* and only performing compare and exchange if the read indicates it +* is likely to be set. */ -if (unlikely (rte_atomic16_read (&dev->broadcast_rarp) && -rte_atomic16_cmpset ((volatile uint16_t *) - &dev->broadcast_rarp.cnt,1 ,0 ))) { +if (unlikely (__atomic_load_n (&dev->broadcast_rarp, __ATOMIC_ACQUIRE) && +__atomic_compare_exchange_n (&dev->broadcast_rarp, + &success,0 ,0 , __ATOMIC_RELEASE, __ATOMIC_RELAXED))) { rarp_mbuf =rte_net_make_rarp_packet (mbuf_pool, &dev->mac);if (rarp_mbuf ==NULL ) {
Signature · 1 line
2.7.4
PATCH v2 1/1 vhost: optimize broadcast rarp sync with c11 atomic
The rarp packet broadcast flag is synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.
---
v2:
split from the 'generic rte atomic APIs deprecate proposal' patchset.
lib/librte_vhost/vhost.h | 2 +- lib/librte_vhost/vhost_user.c | 7 +++---- lib/librte_vhost/virtio_net.c | 16 +++++++++------- 3 files changed, 13 insertions(+), 12 deletions(-)
lib/librte_vhost/vhost.h +1 -1
@@ -350,7 +350,7 @@ struct virtio_net {uint32_t flags;uint16_t vhost_hlen;/* to tell if we need broadcast rarp packet */ -rte_atomic16_t broadcast_rarp; +int16_t broadcast_rarp;uint32_t nr_vring;int dequeue_zero_copy;int extbuf;
lib/librte_vhost/vhost_user.c +3 -4
@@ -2145,11 +2145,10 @@ vhost_user_send_rarp(struct virtio_net **pdev, struct VhostUserMsg *msg,* Set the flag to inject a RARP broadcast packet at * rte_vhost_dequeue_burst(). * -* rte_smp_wmb() is for making sure the mac is copied -* before the flag is set. +* __ATOMIC_RELEASE ordering is for making sure the mac is +* copied before the flag is set. */ -rte_smp_wmb (); -rte_atomic16_set (&dev->broadcast_rarp,1 ); +__atomic_store_n (&dev->broadcast_rarp,1 , __ATOMIC_RELEASE); did = dev->vdpa_dev_id; vdpa_dev =rte_vdpa_get_device (did);if (vdpa_dev && vdpa_dev->ops->migration_done)
lib/librte_vhost/virtio_net.c +9 -7
@@ -2203,6 +2203,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,struct virtio_net *dev;struct rte_mbuf *rarp_mbuf =NULL ;struct vhost_virtqueue *vq; +int16_t success =1 ; dev =get_device (vid);if (!dev)
@@ -2249,16 +2250,17 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,* * broadcast_rarp shares a cacheline in the virtio_net structure * with some fields that are accessed during enqueue and -* rte_atomic16_cmpset() causes a write if using cmpxchg. This could -* result in false sharing between enqueue and dequeue. +* __atomic_compare_exchange_n causes a write if performed compare +* and exchange. This could result in false sharing between enqueue +* and dequeue. * * Prevent unnecessary false sharing by reading broadcast_rarp first -* and only performing cmpset if the read indicates it is likely to -* be set. +* and only performing compare and exchange if the read indicates it +* is likely to be set. */ -if (unlikely (rte_atomic16_read (&dev->broadcast_rarp) && -rte_atomic16_cmpset ((volatile uint16_t *) - &dev->broadcast_rarp.cnt,1 ,0 ))) { +if (unlikely (__atomic_load_n (&dev->broadcast_rarp, __ATOMIC_ACQUIRE) && +__atomic_compare_exchange_n (&dev->broadcast_rarp, + &success,0 ,0 , __ATOMIC_RELEASE, __ATOMIC_RELAXED))) { rarp_mbuf =rte_net_make_rarp_packet (mbuf_pool, &dev->mac);if (rarp_mbuf ==NULL ) {
Signature · 1 line
2.7.4
Re: [dpdk-dev] [PATCH v2] vhost: optimize broadcast rarp sync with c11 atomic
Quoted 8 lines · Phil Yang
The rarp packet broadcast flag is synchronized with
rte_atomic_XXAPIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.Signed-off-by Phil Yang <[email protected]>Reviewed-by Gavin Hu <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>Reviewed-by Joyce Kong <[email protected]>
Quoted 8 lines
---
v2:
split from the 'generic rte atomic APIs deprecate proposal' patchset.lib/librte_vhost/vhost.h | 2 +- lib/librte_vhost/vhost_user.c | 7 +++---- lib/librte_vhost/virtio_net.c | 16 +++++++++------- 3 files changed, 13 insertions(+), 12 deletions(-)
Thanks,
Maxime
Maxime Coquelin
AppliedRe: [dpdk-dev] [PATCH v2] vhost: optimize broadcast rarp sync with c11 atomic
In reply to Phil YangQuoted 8 lines · Phil Yang
The rarp packet broadcast flag is synchronized with
rte_atomic_XXAPIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.Signed-off-by Phil Yang <[email protected]>Reviewed-by Gavin Hu <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>Reviewed-by Joyce Kong <[email protected]>
Quoted 8 lines
---
v2:
split from the 'generic rte atomic APIs deprecate proposal' patchset.lib/librte_vhost/vhost.h | 2 +- lib/librte_vhost/vhost_user.c | 7 +++---- lib/librte_vhost/virtio_net.c | 16 +++++++++------- 3 files changed, 13 insertions(+), 12 deletions(-)
Applied to dpdk-next-virtio/master
Thanks,
Maxime
PATCH v3 6/12 ipsec: optimize with c11 atomic for sa outbound sqn update
In reply to Phil YangFor SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.
--- lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/sa.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
lib/librte_ipsec/ipsec_sqn.h +2 -1
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa->sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED);else { sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h +1 -1
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;struct {
Signature · 1 line
2.7.4
Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update
Hi Phil,
Quoted 20 lines
For SA outbound packets,
rte_atomic64_add_returnis used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implementedrte_atomic_XXAPI on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>--- lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/sa.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)lib/librte_ipsec/ipsec_sqn.h +0
-0
lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132
Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2
-1@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa->sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED);
One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit __atomic builtins (gcc seems to always generate cmpxchng8b for such cases). Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.
Quoted 8 lines
lib/librte_ipsec/ipsec_sqn.h +0
-0else { sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;lib/librte_ipsec/sa.h +0
-0
lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125
Quoted 7 lines
lib/librte_ipsec/sa.h +1
-1@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;
If we don't need rte_atomic64 here anymore,
then I think we can collapse the union to just:uint64_t outb;
Quoted 3 lines
lib/librte_ipsec/sa.h +0
-1struct { --2.7.4
Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update
<snip>
Quoted 27 lines
Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update
Hi Phil,
Quoted 22 lines
For SA outbound packets,
rte_atomic64_add_returnis used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implementedrte_atomic_XXAPI on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>--- lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/sa.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/lib/librte_ipsec/ipsec_sqn.hindex 0c2f76a..e884af7 100644lib/librte_ipsec/ipsec_sqn.h +0
-0@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)
Quoted 12 lines
lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132
Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2
-1n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa- sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED);One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit__atomicbuiltins (gcc seems to always generate cmpxchng8b for such cases). Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.
I had looked at this some time back for GCC. The function call is generated only if the underlying platform does not support the atomic instructions for the operand size. Otherwise, gcc generates the instructions directly. I would think the behavior would be the same for clang.
Quoted 24 lines
Quoted 9 lines
lib/librte_ipsec/ipsec_sqn.h +0
-0else {lib/librte_ipsec/ipsec_sqn.h +0
-0sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0
-0d22451b..cab9a2e 100644
lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125
Quoted 7 lines
lib/librte_ipsec/sa.h +1
-1@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;If we don't need
rte_atomic64here anymore, then I think we can collapse the union to just:uint64_toutb;Quoted 3 lines
lib/librte_ipsec/sa.h +0
-1struct { --2.7.4
Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update
Quoted 54 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Monday, March 23, 2020 7:08 PM
To: Ananyev, Konstantin <[email protected]>; Phil Yang <[email protected]>; [email protected]; Van Haaren, Harry <[email protected]>; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update<snip>
Quoted 27 lines
Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update
Hi Phil,
Quoted 22 lines
For SA outbound packets,
rte_atomic64_add_returnis used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implementedrte_atomic_XXAPI on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>--- lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/sa.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/lib/librte_ipsec/ipsec_sqn.hindex 0c2f76a..e884af7 100644lib/librte_ipsec/ipsec_sqn.h +0
-0@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)Quoted 12 lines
lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132
Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2
-1n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa- sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED);One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit__atomicbuiltins (gcc seems to always generate cmpxchng8b for such cases). Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.
Quoted 3 lines
I had looked at this some time back for GCC. The function call is generated only if the underlying platform does not support the atomic instructions for the operand size. Otherwise, gcc generates the instructions directly.
I would think the behavior would be the same for clang.
From what I see not really.
As an example:
$ cat tatm11.c #include <stdint.h>
struct x {
uint64_t v __attribute__((aligned(8)));
};uint64_tffxadd1(struct x *x, uint32_t n, uint32_t m)
{
return __atomic_add_fetch(&x->v, n, __ATOMIC_RELAXED);
}uint64_tffxadd11(uint64_t *v, uint32_t n, uint32_t m)
{
return __atomic_add_fetch(v, n, __ATOMIC_RELAXED);
}gcc for i686 will generate code with cmpxchng8b for both cases.
clang will generate cmpxchng8b for ffxadd1() - when data is explicitly 8B aligned, but will emit a function call for ffxadd11().
Quoted 25 lines
Quoted 25 lines
Quoted 10 lines
lib/librte_ipsec/ipsec_sqn.h +0
-0else {lib/librte_ipsec/ipsec_sqn.h +0
-0sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0
-0d22451b..cab9a2e 100644
lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125
Quoted 7 lines
lib/librte_ipsec/sa.h +1
-1@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;If we don't need
rte_atomic64here anymore, then I think we can collapse the union to just:uint64_toutb;Quoted 3 lines
lib/librte_ipsec/sa.h +0
-1struct { --2.7.4
Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update
<snip>
Quoted 59 lines
Quoted 27 lines
Quoted 27 lines
Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update
Hi Phil,
Quoted 22 lines
For SA outbound packets,
rte_atomic64_add_returnis used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implementedrte_atomic_XXAPI on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>--- lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/sa.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/lib/librte_ipsec/ipsec_sqn.hindex 0c2f76a..e884af7 100644lib/librte_ipsec/ipsec_sqn.h +0
-0@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)Quoted 10 lines
Quoted 10 lines
lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132
Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2
-1n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa- sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED);One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit__atomicbuiltins (gcc seems to always generate cmpxchng8b forsuch cases).
Quoted 4 lines
Quoted 2 lines
Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.I had looked at this some time back for GCC. The function call is generated only if the underlying platform does not support the atomic
instructions for the operand size. Otherwise, gcc generates the instructions directly.
Quoted 1 line
I would think the behavior would be the same for clang.
From what I see not really.
As an example:$ cat tatm11.c #include <stdint.h>struct x { uint64_t v __attribute__((aligned(8))); };
uint64_tffxadd1(struct x *x, uint32_t n, uint32_t m) { return __atomic_add_fetch(&x->v, n, __ATOMIC_RELAXED); }
Quoted 7 lines
uint64_tffxadd11(uint64_t *v, uint32_t n, uint32_t m) { return __atomic_add_fetch(v, n, __ATOMIC_RELAXED); }gcc for i686 will generate code with cmpxchng8b for both cases. clang will generate cmpxchng8b for
ffxadd1()- when data is explicitly 8B aligned, but will emit a function call forffxadd11().
Does it require libatomic to be linked in this case? Clang documentation calls out unaligned case where it would generate the function call [1]. On aarch64, the atomic instructions need the address to be aligned.
[1] https://clang.llvm.org/docs/Toolchain.html#atomics-library
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 11 lines
lib/librte_ipsec/ipsec_sqn.h +0
-0else {lib/librte_ipsec/ipsec_sqn.h +0
-0sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0
-0d22451b..cab9a2e 100644
lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125
Quoted 7 lines
lib/librte_ipsec/sa.h +1
-1@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;If we don't need
rte_atomic64here anymore, then I think we can collapse the union to just:uint64_toutb;Quoted 3 lines
lib/librte_ipsec/sa.h +0
-1struct { --2.7.4
Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update
Quoted 54 lines
Quoted 54 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
For SA outbound packets,
rte_atomic64_add_returnis used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implementedrte_atomic_XXAPI on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>--- lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/sa.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/lib/librte_ipsec/ipsec_sqn.hindex 0c2f76a..e884af7 100644lib/librte_ipsec/ipsec_sqn.h +0
-0@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)Quoted 10 lines
Quoted 10 lines
lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132
Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2
-1n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa- sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED);One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit__atomicbuiltins (gcc seems to always generate cmpxchng8b forsuch cases).
Quoted 4 lines
Quoted 2 lines
Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.I had looked at this some time back for GCC. The function call is generated only if the underlying platform does not support the atomic
instructions for the operand size. Otherwise, gcc generates the instructions directly.
Quoted 1 line
I would think the behavior would be the same for clang.
From what I see not really.
As an example:$ cat tatm11.c #include <stdint.h>struct x { uint64_t v __attribute__((aligned(8))); };
uint64_tffxadd1(struct x *x, uint32_t n, uint32_t m) { return __atomic_add_fetch(&x->v, n, __ATOMIC_RELAXED); }
Quoted 8 lines
Quoted 7 lines
uint64_tffxadd11(uint64_t *v, uint32_t n, uint32_t m) { return __atomic_add_fetch(v, n, __ATOMIC_RELAXED); }gcc for i686 will generate code with cmpxchng8b for both cases. clang will generate cmpxchng8b for
ffxadd1()- when data is explicitly 8B aligned, but will emit a function call forffxadd11().Does it require libatomic to be linked in this case?
Yes, it does.
In fact same story even with current dpdk.org master. To make i686-native-linuxapp-clang successfully, I have to explicitly add EXTRA_LDFLAGS="-latomic".
To be more specific:
$ for i in i686-native-linuxapp-clang/lib/*.a; do x=`nm $i | grep __atomic_`; if [[ -n "${x}" ]]; then echo $i; echo $x; fi; done
i686-native-linuxapp-clang/lib/librte_distributor.a
U __atomic_load_8 U __atomic_store_8
i686-native-linuxapp-clang/lib/librte_pmd_opdl_event.a
U __atomic_load_8 U __atomic_store_8
i686-native-linuxapp-clang/lib/librte_rcu.a
U __atomic_compare_exchange_8 U __atomic_load_8As there were no complains so far, it makes me think that probably no-one using clang for IA-32 builds.
Quoted 2 lines
Clang documentation calls out unaligned case where it would generate the function call [1].
Seems so, and it treats uin64_t as 4B aligned for IA.
Quoted 1 line
On aarch64, the atomic instructions need the address to be aligned.
For that particular case (cmpxchng8b) there is no such restrictions for IA-32. Again, as I said before, gcc manages to emit code without function calls for exactly the same source.
Quoted 29 lines
[1] https://clang.llvm.org/docs/Toolchain.html#atomics-library
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 11 lines
lib/librte_ipsec/ipsec_sqn.h +0
-0else {lib/librte_ipsec/ipsec_sqn.h +0
-0sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0
-0d22451b..cab9a2e 100644
lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125
Quoted 7 lines
lib/librte_ipsec/sa.h +1
-1@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;If we don't need
rte_atomic64here anymore, then I think we can collapse the union to just:uint64_toutb;Quoted 3 lines
lib/librte_ipsec/sa.h +0
-1struct { --2.7.4
Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update
Quoted 88 lines
-----Original Message-----
From: dev <[email protected]> On Behalf Of Ananyev, Konstantin
Sent: Tuesday, March 24, 2020 1:10 PM
To: Honnappa Nagarahalli <[email protected]>; Phil Yang <[email protected]>; [email protected]; Van Haaren, Harry <[email protected]>; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; nd <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn updateQuoted 53 lines
Quoted 53 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
For SA outbound packets,
rte_atomic64_add_returnis used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implementedrte_atomic_XXAPI on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>--- lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/sa.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/lib/librte_ipsec/ipsec_sqn.hindex 0c2f76a..e884af7 100644lib/librte_ipsec/ipsec_sqn.h +0
-0@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)Quoted 10 lines
Quoted 10 lines
lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132
Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2
-1n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa- sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED);One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit__atomicbuiltins (gcc seems to always generate cmpxchng8b forsuch cases).
Quoted 4 lines
Quoted 2 lines
Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.I had looked at this some time back for GCC. The function call is generated only if the underlying platform does not support the atomic
instructions for the operand size. Otherwise, gcc generates the instructions directly.
Quoted 1 line
I would think the behavior would be the same for clang.
From what I see not really.
As an example:$ cat tatm11.c #include <stdint.h>struct x { uint64_t v __attribute__((aligned(8))); };
uint64_tffxadd1(struct x *x, uint32_t n, uint32_t m) { return __atomic_add_fetch(&x->v, n, __ATOMIC_RELAXED); }Quoted 8 lines
Quoted 7 lines
uint64_tffxadd11(uint64_t *v, uint32_t n, uint32_t m) { return __atomic_add_fetch(v, n, __ATOMIC_RELAXED); }gcc for i686 will generate code with cmpxchng8b for both cases. clang will generate cmpxchng8b for
ffxadd1()- when data is explicitly 8B aligned, but will emit a function call forffxadd11().Does it require libatomic to be linked in this case?
Yes, it does.
In fact same story even with current dpdk.org master. To make i686-native-linuxapp-clang successfully, I have to explicitly add EXTRA_LDFLAGS="-latomic".
To be more specific:$ for i in i686-native-linuxapp-clang/lib/*.a; do x=`nm $i | grep __atomic_`; if [[ -n "${x}" ]]; then echo $i; echo $x; fi; done i686-native-linuxapp-clang/lib/librte_distributor.a U __atomic_load_8 U __atomic_store_8 i686-native-linuxapp-clang/lib/librte_pmd_opdl_event.a U __atomic_load_8 U __atomic_store_8 i686-native-linuxapp-clang/lib/librte_rcu.a U __atomic_compare_exchange_8 U __atomic_load_8
Quoted 7 lines
As there were no complains so far, it makes me think that probably no-one using clang for IA-32 builds.
Quoted 2 lines
Clang documentation calls out unaligned case where it would generate the function call [1].
Seems so, and it treats
uin64_tas 4B aligned for IA.
correction: for IA-32
Quoted 36 lines
Quoted 2 lines
On aarch64, the atomic instructions need the address to be aligned.
For that particular case (cmpxchng8b) there is no such restrictions for IA-32. Again, as I said before, gcc manages to emit code without function calls for exactly the same source.
Quoted 29 lines
[1] https://clang.llvm.org/docs/Toolchain.html#atomics-library
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 11 lines
lib/librte_ipsec/ipsec_sqn.h +0
-0else {lib/librte_ipsec/ipsec_sqn.h +0
-0sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0
-0d22451b..cab9a2e 100644
lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125
Quoted 7 lines
lib/librte_ipsec/sa.h +1
-1@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;If we don't need
rte_atomic64here anymore, then I think we can collapse the union to just:uint64_toutb;Quoted 3 lines
lib/librte_ipsec/sa.h +0
-1struct { --2.7.4
Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update
In reply to Ananyev, KonstantinHi Konstantin,
<snip>
Quoted 67 lines
Quoted 2 lines
Quoted 2 lines
Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa
outbound
Quoted 6 lines
Quoted 6 lines
sqn update
Hi Phil,
Quoted 2 lines
For SA outbound packets,
rte_atomic64_add_returnis used togenerate
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
SQN atomically. This introduced an unnecessary full barrier by calling the '
__sync' builtin implementedrte_atomic_XXAPI on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>--- lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/sa.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/lib/librte_ipsec/ipsec_sqn.hindex 0c2f76a..e884af7 100644lib/librte_ipsec/ipsec_sqn.h +0
-0@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)Quoted 10 lines
Quoted 10 lines
lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132
Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2
-1n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa- sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED);One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit__atomicbuiltins (gcc seems to always generate cmpxchng8b for suchcases).
Quoted 3 lines
Quoted 2 lines
Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.I had looked at this some time back for GCC. The function call is generated
only if the underlying platform does not support the atomic
Quoted 1 line
instructions for the operand size. Otherwise, gcc generates the instructions
directly.
Quoted 1 line
I would think the behavior would be the same for clang.
From what I see not really.
As an example:$ cat tatm11.c #include <stdint.h>struct x { uint64_t v __attribute__((aligned(8))); };
uint64_tffxadd1(struct x *x, uint32_t n, uint32_t m){ return __atomic_add_fetch(&x->v, n, __ATOMIC_RELAXED); }
uint64_tffxadd11(uint64_t *v, uint32_t n, uint32_t m)
Quoted 8 lines
{ return __atomic_add_fetch(v, n, __ATOMIC_RELAXED); }gcc for i686 will generate code with cmpxchng8b for both cases. clang will generate cmpxchng8b for
ffxadd1()- when data is explicitly 8B aligned,
but will emit a function call forffxadd11().
I guess your testbed is an i386 platform. However, what I see here is different.
Testbed i686: Ubuntu 18.04.4 LTS/GCC 8.3/ Clang 9.0.0-2 Both Clang and GCC for i686 generate code with xadd for these two cases.
Testbed i386: Ubuntu 16.04 LTS (Installed libatomic)/GCC 5.4.0/ Clang 4.0.0 GCC will generate code with cmpxchng8b for both cases.
Clang generated code emits a function call for both cases.
Thanks,
Phil
Quoted 27 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
lib/librte_ipsec/ipsec_sqn.h +0
-0else {lib/librte_ipsec/ipsec_sqn.h +0
-0sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0
-0d22451b..cab9a2e 100644
lib/librte_ipsec/sa.h +0
-0@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ Quoted 8 lines
Quoted 8 lines
lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125
Quoted 6 lines
lib/librte_ipsec/sa.h +1
-1union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;If we don't need
rte_atomic64here anymore, then I think we cancollapse the
Quoted 6 lines
Quoted 6 lines
union to just:
uint64_toutb;Quoted 3 lines
lib/librte_ipsec/sa.h +0
-1struct { --2.7.4
Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update
Hi Phil,
Quoted 77 lines
<snip>
Quoted 67 lines
Quoted 2 lines
Quoted 2 lines
Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa
outbound
Quoted 6 lines
Quoted 6 lines
sqn update
Hi Phil,
Quoted 2 lines
For SA outbound packets,
rte_atomic64_add_returnis used togenerate
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
SQN atomically. This introduced an unnecessary full barrier by calling the '
__sync' builtin implementedrte_atomic_XXAPI on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>--- lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/sa.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/lib/librte_ipsec/ipsec_sqn.hindex 0c2f76a..e884af7 100644lib/librte_ipsec/ipsec_sqn.h +0
-0@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)Quoted 10 lines
Quoted 10 lines
lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132
Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2
-1n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa- sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED);One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit__atomicbuiltins (gcc seems to always generate cmpxchng8b for suchcases).
Quoted 3 lines
Quoted 2 lines
Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.I had looked at this some time back for GCC. The function call is generated
only if the underlying platform does not support the atomic
Quoted 1 line
instructions for the operand size. Otherwise, gcc generates the instructions
directly.
Quoted 1 line
I would think the behavior would be the same for clang.
From what I see not really.
As an example:$ cat tatm11.c #include <stdint.h>struct x { uint64_t v __attribute__((aligned(8))); };
uint64_tffxadd1(struct x *x, uint32_t n, uint32_t m){ return __atomic_add_fetch(&x->v, n, __ATOMIC_RELAXED); }
uint64_tffxadd11(uint64_t *v, uint32_t n, uint32_t m)Quoted 8 lines
{ return __atomic_add_fetch(v, n, __ATOMIC_RELAXED); }gcc for i686 will generate code with cmpxchng8b for both cases. clang will generate cmpxchng8b for
ffxadd1()- when data is explicitly 8B aligned,
but will emit a function call forffxadd11().
Quoted 4 lines
I guess your testbed is an i386 platform. However, what I see here is different.
Testbed i686: Ubuntu 18.04.4 LTS/GCC 8.3/ Clang 9.0.0-2 Both Clang and GCC for i686 generate code with xadd for these two cases.
I suppose you meant x86_64 here (-m64), right?
Quoted 4 lines
Testbed i386: Ubuntu 16.04 LTS (Installed libatomic)/GCC 5.4.0/ Clang 4.0.0 GCC will generate code with cmpxchng8b for both cases.
Clang generated code emits a function call for both cases.
That's exactly what I am talking about above.X86_64 (64 bit binary) - no function calls for both gcc and clang i686 (32 bit binary) - no function calls with gcc, functions calls with clang when explicit alignment is not specified.
As I said in my initial email, that's probably not a big deal - from what I was told so far we don't officially support clang for IA-32 and I don't know does anyone uses it at all right now. Though if someone thinks it is a potential problem here -
it is better to flag it at early stage.
So once again my questions to the community:
1/ Does anyone builds/uses DPDK with i686-clang?
2/ If there are anyone, can these persons try to evaluate how big perf drop it would cause for them?
3/ Is there an option to switch to i686-gcc (supported one)? Konstantin
Quoted 27 lines
Quoted 27 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
lib/librte_ipsec/ipsec_sqn.h +0
-0else {lib/librte_ipsec/ipsec_sqn.h +0
-0sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0
-0d22451b..cab9a2e 100644
lib/librte_ipsec/sa.h +0
-0@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ Quoted 8 lines
Quoted 8 lines
lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125
Quoted 6 lines
lib/librte_ipsec/sa.h +1
-1union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;If we don't need
rte_atomic64here anymore, then I think we cancollapse the
Quoted 6 lines
Quoted 6 lines
union to just:
uint64_toutb;Quoted 3 lines
lib/librte_ipsec/sa.h +0
-1struct { --2.7.4
Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update
Quoted 100 lines
-----Original Message-----
From: Ananyev, Konstantin <[email protected]>
Sent: Tuesday, March 24, 2020 7:04 PM
To: Phil Yang <[email protected]>; Honnappa Nagarahalli <[email protected]>; [email protected]; Van Haaren, Harry <[email protected]>; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn updateHi Phil,
Quoted 6 lines
<snip>
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa
outbound
Quoted 2 lines
Quoted 2 lines
sqn update
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Hi Phil,
Quoted 2 lines
For SA outbound packets,
rte_atomic64_add_returnis used togenerate
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
SQN atomically. This introduced an unnecessary full barrier by calling the '
__sync' builtin implementedrte_atomic_XXAPI on aarch64.This
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.
Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>--- lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/sa.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa
Quoted 5 lines
(quoted diff) +0
-0
b/lib/librte_ipsec/ipsec_sqn.hindex 0c2f76a..e884af7 100644lib/librte_ipsec/ipsec_sqn.h +0
-0@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa*sa,
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132
Quoted 8 lines
lib/librte_ipsec/ipsec_sqn.h +2
-1uint32_t *num) n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa- sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED);One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
__atomicbuiltins (gcc seems to always generate cmpxchng8b for suchcases).
Quoted 3 lines
Quoted 2 lines
Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.I had looked at this some time back for GCC. The function call is
generated
Quoted 2 lines
Quoted 2 lines
only if the underlying platform does not support the atomic
Quoted 1 line
instructions for the operand size. Otherwise, gcc generates the
instructions
Quoted 30 lines
Quoted 29 lines
directly.
Quoted 1 line
I would think the behavior would be the same for clang.
From what I see not really.
As an example:$ cat tatm11.c #include <stdint.h>struct x { uint64_t v __attribute__((aligned(8))); };
uint64_tffxadd1(struct x *x, uint32_t n, uint32_t m){ return __atomic_add_fetch(&x->v, n, __ATOMIC_RELAXED); }
uint64_tffxadd11(uint64_t *v, uint32_t n, uint32_t m){ return __atomic_add_fetch(v, n, __ATOMIC_RELAXED); }gcc for i686 will generate code with cmpxchng8b for both cases. clang will generate cmpxchng8b for
ffxadd1()- when data is explicitly 8B aligned,
but will emit a function call forffxadd11().
Quoted 7 lines
Quoted 1 line
I guess your testbed is an i386 platform. However, what I see here is
different.
Quoted 3 lines
Testbed i686: Ubuntu 18.04.4 LTS/GCC 8.3/ Clang 9.0.0-2 Both Clang and GCC for i686 generate code with xadd for these two cases.
I suppose you meant
x86_64here (-m64), right?
Yes. It is x86_64 here.
Quoted 51 lines
Quoted 6 lines
Testbed i386: Ubuntu 16.04 LTS (Installed libatomic)/GCC 5.4.0/ Clang 4.0.0 GCC will generate code with cmpxchng8b for both cases.
Clang generated code emits a function call for both cases.That's exactly what I am talking about above.
X86_64(64 bit binary) - no function calls for both gcc and clang i686 (32 bit binary) - no function calls with gcc, functions calls with clang when explicit alignment is not specified.As I said in my initial email, that's probably not a big deal - from what I was told so far we don't officially support clang for IA-32 and I don't know does anyone uses it at all right now. Though if someone thinks it is a potential problem here -
it is better to flag it at early stage.
So once again my questions to the community:
1/ Does anyone builds/uses DPDK with i686-clang?
2/ If there are anyone, can these persons try to evaluate how big perf drop it would cause for them?
3/ Is there an option to switch to i686-gcc (supported one)? KonstantinQuoted 27 lines
Quoted 27 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
lib/librte_ipsec/ipsec_sqn.h +0
-0else {lib/librte_ipsec/ipsec_sqn.h +0
-0sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0
-0d22451b..cab9a2e 100644
lib/librte_ipsec/sa.h +0
-0@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ Quoted 8 lines
Quoted 8 lines
lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125
Quoted 6 lines
lib/librte_ipsec/sa.h +1
-1union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;If we don't need
rte_atomic64here anymore, then I think we cancollapse the
Quoted 6 lines
Quoted 6 lines
union to just:
uint64_toutb;Quoted 3 lines
lib/librte_ipsec/sa.h +0
-1struct { --2.7.4
PATCH v2 1/1 ipsec: optimize with c11 atomic for sa outbound sqn update
In reply to Phil YangFor SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.
---
v2:
split from the "generic rte atomic APIs deprecate proposal" patchset.
lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/meson.build | 5 +++++ lib/librte_ipsec/sa.h | 2 +- 3 files changed, 8 insertions(+), 2 deletions(-)
lib/librte_ipsec/ipsec_sqn.h +2 -1
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa->sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED);else { sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;
lib/librte_ipsec/meson.build +5 -0
@@ -6,3 +6,8 @@ sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c', 'ipsec_sad.c') headers = files('rte_ipsec.h', 'rte_ipsec_group.h', 'rte_ipsec_sa.h', 'rte_ipsec_sad.h') deps += ['mbuf', 'net', 'cryptodev', 'security', 'hash'] + +# for clang 32-bit compiles we need libatomic for 64-bit atomic ops +if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false + ext_deps += cc.find_library('atomic') +endif
lib/librte_ipsec/sa.h +1 -1
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;struct {
Signature · 1 line
2.7.4
Re: [dpdk-dev] [PATCH v2] ipsec: optimize with c11 atomic for sa outbound sqn update
Quoted 16 lines · Phil Yang
For SA outbound packets,
rte_atomic64_add_returnis used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implementedrte_atomic_XXAPI on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>lib/librte_ipsec/meson.build +0
-0
lib/librte_ipsec/meson.build sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c', 'ipsec_sad.c') · line 13
Quoted 8 lines
lib/librte_ipsec/meson.build +5
-0@@ -6,3 +6,8 @@ sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c', 'ipsec_sad.c') headers = files('rte_ipsec.h', 'rte_ipsec_group.h', 'rte_ipsec_sa.h', 'rte_ipsec_sad.h') deps += ['mbuf', 'net', 'cryptodev', 'security', 'hash'] + +# for clang 32-bit compiles we need libatomic for 64-bit atomic ops +if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false + ext_deps += cc.find_library('atomic') +endif
The following patch has been merged in master now. You don't need this anymore.
commit da4eae278b56e698c64d0c39939a7a55c5b6abdd
Author: Pavan Nikhilesh <[email protected]> Date: Sun Apr 19 15:31:01 2020 +0530
build: add global libatomic dependency for 32-bit clang
Add libatomic as a global dependency when compiling for 32-bit using clang. As we need libatomic for 64-bit atomic ops.
Re: [dpdk-dev] [PATCH v2] ipsec: optimize with c11 atomic for sa outbound sqn update
Quoted 49 lines
-----Original Message-----
From: Jerin Jacob <[email protected]>
Sent: Friday, April 24, 2020 1:45 AM
To: Phil Yang <[email protected]>
Cc: Ananyev, Konstantin <[email protected]>; dpdk-dev <[email protected]>; [email protected]; Bernard Iremonger <[email protected]>; Vladimir Medvedkin <[email protected]>; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v2] ipsec: optimize with c11 atomic for sa outbound sqn updatelib/librte_ipsec/meson.build sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c',
Quoted 16 lines · Phil Yang
For SA outbound packets,
rte_atomic64_add_returnis used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implementedrte_atomic_XXAPI on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>lib/librte_ipsec/meson.build +0
-0@@ -6,3 +6,8 @@ sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c','
ipsec_sad.c')lib/librte_ipsec/meson.build sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c', · line 6
Quoted 1 line
lib/librte_ipsec/meson.build +0
-0headers = files('rte_ipsec.h', 'rte_ipsec_group.h', 'rte_ipsec_sa.h','
rte_ipsec_sad.h')lib/librte_ipsec/meson.build sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c', 'ipsec_sad.c') · line 13
Quoted 7 lines
lib/librte_ipsec/meson.build +5
-0deps += ['mbuf', 'net', 'cryptodev', 'security', 'hash'] + +# for clang 32-bit compiles we need libatomic for 64-bit atomic ops +if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false + ext_deps += cc.find_library('atomic') +endifThe following patch has been merged in master now. You don't need this anymore.
commit
da4eae278b56e698c64d0c39939a7a55c5b6abddAuthor: Pavan Nikhilesh <[email protected]> Date: Sun Apr 19 15:31:01 2020 +0530
Quoted 7 lines
build: add global libatomic dependency for 32-bit clang
Add libatomic as a global dependency when compiling for 32-bit using clang. As we need libatomic for 64-bit atomic ops.
Signed-off-by Pavan Nikhilesh <[email protected]>Acked-by Bruce Richardson <[email protected]>
Great, we don't need to add it module by module anymore. Updated in v3. Thank you very much.
Thanks,
Phil
Re: [dpdk-dev] [PATCH v2] ipsec: optimize with c11 atomic for sa outbound sqn update
In reply to Phil YangQuoted 52 lines
For SA outbound packets,
rte_atomic64_add_returnis used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implementedrte_atomic_XXAPI on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>---
v2:
split from the "generic rte atomic APIs deprecate proposal" patchset.lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/meson.build | 5 +++++ lib/librte_ipsec/sa.h | 2 +- 3 files changed, 8 insertions(+), 2 deletions(-)lib/librte_ipsec/ipsec_sqn.h +2
-1@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa->sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED);else { sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;lib/librte_ipsec/meson.build +5
-0@@ -6,3 +6,8 @@ sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c', 'ipsec_sad.c') headers = files('rte_ipsec.h', 'rte_ipsec_group.h', 'rte_ipsec_sa.h', 'rte_ipsec_sad.h') deps += ['mbuf', 'net', 'cryptodev', 'security', 'hash'] + +# for clang 32-bit compiles we need libatomic for 64-bit atomic ops +if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false + ext_deps += cc.find_library('atomic') +endiflib/librte_ipsec/sa.h +0
-0
lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 126
Quoted 8 lines
lib/librte_ipsec/sa.h +1
-1@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;struct {
Seems you missed my comments for previous version, so I put here:
If we don't need rte_atomic64 here anymore,
then I think we can collapse the union to just:uint64_t outb;
Konstantin
Re: [dpdk-dev] [PATCH v2] ipsec: optimize with c11 atomic for sa outbound sqn update
Quoted 79 lines
-----Original Message-----
From: Ananyev, Konstantin <[email protected]>
Sent: Friday, April 24, 2020 2:11 AM
To: Phil Yang <[email protected]>; [email protected]
Cc: [email protected]; Iremonger, Bernard <[email protected]>; Medvedkin, Vladimir <[email protected]>; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v2] ipsec: optimize with c11 atomic for sa outbound sqn updatelib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa,
Quoted 25 lines
For SA outbound packets,
rte_atomic64_add_returnis used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implementedrte_atomic_XXAPI on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>---
v2:
split from the "generic rte atomic APIs deprecate proposal" patchset.lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/meson.build | 5 +++++ lib/librte_ipsec/sa.h | 2 +- 3 files changed, 8 insertions(+), 2 deletions(-)lib/librte_ipsec/ipsec_sqn.h +0
-0@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa,
uint32_t*num)lib/librte_ipsec/meson.build sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c',
Quoted 15 lines
lib/librte_ipsec/ipsec_sqn.h +2
-1n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa- sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED); else { sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn;lib/librte_ipsec/meson.build +0
-0@@ -6,3 +6,8 @@ sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c','
ipsec_sad.c')lib/librte_ipsec/meson.build sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c', · line 6
Quoted 1 line
lib/librte_ipsec/meson.build +0
-0headers = files('rte_ipsec.h', 'rte_ipsec_group.h', 'rte_ipsec_sa.h','
rte_ipsec_sad.h')Quoted 12 lines
lib/librte_ipsec/meson.build +5
-0deps += ['mbuf', 'net', 'cryptodev', 'security', 'hash'] + +# for clang 32-bit compiles we need libatomic for 64-bit atomic ops +if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false + ext_deps += cc.find_library('atomic') +endiflib/librte_ipsec/sa.h +0
-0lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 126
Quoted 8 lines
lib/librte_ipsec/sa.h +1
-1@@ -120,7 +120,7 @@ struct rte_ipsec_sa {*/ union {union { -rte_atomic64_t atom; +uint64_t atom;uint64_t raw; } outb;struct {Seems you missed my comments for previous version, so I put here:
Quoted 3 lines
If we don't need
rte_atomic64here anymore, then I think we can collapse the union to just:uint64_toutb;
My bad, I missed this comment.
Updated in v3. Please review it.
Thanks,
Phil
Quoted 2 lines
Konstantin
PATCH v3 1/1 ipsec: optimize with c11 atomic for sa outbound sqn update
In reply to Phil YangFor SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. Use c11 atomics with RELAXED ordering for outbound SQN update instead of rte_atomic ops which enforce unnecessary barriers on aarch64.
---
v3:
1. since libatomic dependency for 32-bit clang added globally, so remove the redundant code.
2. collapse union outb to unint64_t outb as the rte_atomic is no needed.
v2:
split from the "generic rte atomic APIs deprecate proposal" patchset.
lib/librte_ipsec/ipsec_sqn.h | 6 +++--- lib/librte_ipsec/sa.c | 2 +- lib/librte_ipsec/sa.h | 5 +---- 3 files changed, 5 insertions(+), 8 deletions(-)
lib/librte_ipsec/ipsec_sqn.h +3 -3
@@ -128,10 +128,10 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) n = *num;if (SQN_ATOMIC (sa)) - sqn = (uint64_t )rte_atomic64_add_return (&sa->sqn.outb.atom, n); + sqn =__atomic_add_fetch (&sa->sqn.outb, n, __ATOMIC_RELAXED);else { - sqn = sa->sqn.outb.raw + n; - sa->sqn.outb.raw = sqn; + sqn = sa->sqn.outb + n; + sa->sqn.outb = sqn; }/* overflow */
lib/librte_ipsec/sa.c +1 -1
@@ -283,7 +283,7 @@ esp_outb_init(struct rte_ipsec_sa *sa, uint32_t hlen) {uint8_t algo_type; - sa->sqn.outb.raw =1 ; + sa->sqn.outb =1 ; algo_type = sa->algo_type;
lib/librte_ipsec/sa.h +1 -4
@@ -119,10 +119,7 @@ struct rte_ipsec_sa {* place from other frequently accesed data. */ union { -union { -rte_atomic64_t atom; -uint64_t raw; - } outb; +uint64_t outb;struct {uint32_t rdidx;/* read index */ uint32_t wridx;/* write index */
Signature · 1 line
2.7.4
Re: [dpdk-dev] [PATCH v3] ipsec: optimize with c11 atomic for sa outbound sqn update
Quoted 10 lines
For SA outbound packets,
rte_atomic64_add_returnis used to generate SQN atomically. Use c11 atomics with RELAXED ordering for outbound SQN update instead ofrte_atomicops which enforce unnecessary barriers on aarch64.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>---
Quoted 1 line
2.7.4
Akhil Goyal
AppliedRe: [dpdk-dev] [PATCH v3] ipsec: optimize with c11 atomic for sa outbound sqn update
Quoted 7 lines
Quoted 7 lines
For SA outbound packets,
rte_atomic64_add_returnis used to generate SQN atomically. Use c11 atomics with RELAXED ordering for outbound SQN update instead ofrte_atomicops which enforce unnecessary barriers on aarch64.
Quoted 7 lines
Quoted 4 lines
Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>---
Applied to dpdk-next-crypto
Thanks.
PATCH v3 7/12 service: remove rte prefix from static functions
In reply to Phil Yang--- lib/librte_eal/common/rte_service.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +9 -9
@@ -336,7 +336,7 @@ rte_service_runstate_get(uint32_t id) }static inline void -rte_service_runner_do_callback (struct rte_service_spec_impl *s, +service_runner_do_callback (struct rte_service_spec_impl *s,struct core_state *cs,uint32_t service_idx) {void *userdata = s->spec.callback_userdata;
@@ -379,10 +379,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ; -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);rte_atomic32_clear (&s->execute_lock); }else -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);return 0 ; }
@@ -436,7 +436,7 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) }static int32_t -rte_service_runner_func (void *arg) +service_runner_func (void *arg) {RTE_SET_USED (arg);uint32_t i;
@@ -706,7 +706,7 @@ rte_service_lcore_start(uint32_t lcore)*/ lcore_states[lcore].runstate =RUNSTATE_RUNNING ; -int ret =rte_eal_remote_launch (rte_service_runner_func,0 , lcore); +int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */ return ret; }
@@ -785,7 +785,7 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id, }static void -rte_service_dump_one (FILE *f,struct rte_service_spec_impl *s, +service_dump_one (FILE *f,struct rte_service_spec_impl *s,uint64_t all_cycles,uint32_t reset) {/* avoid divide by zero */
@@ -818,7 +818,7 @@ rte_service_attr_reset_all(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );int reset =1 ; -rte_service_dump_one (NULL , s,0 , reset); +service_dump_one (NULL , s,0 , reset);return 0 ; }
@@ -876,7 +876,7 @@ rte_service_dump(FILE *f, uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );fprintf (f,"Service %s Summary\n" , s->spec.name);uint32_t reset =0 ; -rte_service_dump_one (f, s, total_cycles, reset); +service_dump_one (f, s, total_cycles, reset);return 0 ; }
@@ -886,7 +886,7 @@ rte_service_dump(FILE *f, uint32_t id)if (!service_valid (i))continue ;uint32_t reset =0 ; -rte_service_dump_one (f, &rte_services[i], total_cycles, reset); +service_dump_one (f, &rte_services[i], total_cycles, reset); }fprintf (f,"Service Cores Summary\n" );
Signature · 1 line
2.7.4
Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions
Quoted 10 lines
From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 07/12] service: remove rte prefix from static functions
Quoted 6 lines
Fixes 3cf5eb1546ed ("service: fix and refactor atomic service accesses")Fixes 21698354c832 ("service: introduce service cores concept")Signed-off-by Phil Yang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>
This patchset needs a rebase since the EAL file movement got merged, however I'll review here so we can include some Acks etc and make progress.
Is this really a "Fix"? The internal function names were not exported in the .map file, so are not part of public ABI. This is an internal naming improvement (thanks for doing cleanup), but I don't think the
Fixes: tags make sense?
Also I'm not sure if we want to port this patch back to stable? Changing (internal) function names seems like unnecessary churn, and hence risk to a stable release, without any benefit?
---
<snip patch diff>
Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions
Quoted 40 lines
-----Original Message-----
From: Van Haaren, Harry <[email protected]>
Sent: Friday, April 3, 2020 7:58 PM
To: Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]
Subject: RE: [PATCH v3 07/12] service: remove rte prefix from static functionsQuoted 3 lines
From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, HarryQuoted 2 lines
Ananyev, Konstantin <[email protected]>; [email protected]; [email protected];
Quoted 1 line
Quoted 1 line
Quoted 9 lines
[email protected]; [email protected]; [email protected]
Subject: [PATCH v3 07/12] service: remove rte prefix from static functionsFixes 3cf5eb1546ed ("service: fix and refactor atomic service accesses")Fixes 21698354c832 ("service: introduce service cores concept")Signed-off-by Phil Yang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>This patchset needs a rebase since the EAL file movement got merged, however I'll review here so we can include some Acks etc and make progress.
Quoted 8 lines
Is this really a "Fix"? The internal function names were not exported in the .map file, so are not part of public ABI. This is an internal naming improvement (thanks for doing cleanup), but I don't think the
Fixes: tags make sense?Also I'm not sure if we want to port this patch back to stable? Changing (internal) function names seems like unnecessary churn, and hence risk to a stable release, without any benefit?
OK.
I will remove these tags in the next version and split the service core patches from the original series into a series by itself.
Thanks,
Phil
Quoted 4 lines
---
<snip patch diff>
Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions
Quoted 11 lines
-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Wednesday, April 8, 2020 11:15 AM
To: Van Haaren, Harry <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v3 07/12] service: remove rte prefix from static functions
<snip>
Quoted 11 lines
Quoted 8 lines
Is this really a "Fix"? The internal function names were not exported in the .map file, so are not part of public ABI. This is an internal naming improvement (thanks for doing cleanup), but I don't think the
Fixes: tags make sense?Also I'm not sure if we want to port this patch back to stable? Changing (internal) function names seems like unnecessary churn, and hence risk to a stable release, without any benefit?
OK.
I will remove these tags in the next version and split the service core patches from the original series into a series by itself.
Cool - good idea to split.
Perhaps we should focus on getting bugfixes in for the existing code, before doing cleanup? It would make backports easier if churn is minimal.
Suggesting patches order (first to last)
1. bugfixes/things to backport
2. cleanups
3. C11 atomic optimizations
Quoted 2 lines
Thanks,
Phil
Thanks, and I'll get to reading/reviewing your and Honnappa's feedback later today.
-H
Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions
Quoted 48 lines
-----Original Message-----
From: Van Haaren, Harry <[email protected]>
Sent: Wednesday, April 8, 2020 6:37 PM
To: Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v3 07/12] service: remove rte prefix from static functionsQuoted 4 lines
-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Wednesday, April 8, 2020 11:15 AM
To: Van Haaren, Harry <[email protected]>;Quoted 2 lines
Ananyev, Konstantin <[email protected]>; [email protected]; [email protected];
Quoted 1 line
Quoted 2 lines
Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce
Kong
Quoted 1 line
<[email protected]>; nd <[email protected]>; [email protected]; nd
Quoted 1 line
Subject: RE: [PATCH v3 07/12] service: remove rte prefix from static
functions
<snip>Quoted 7 lines
Quoted 7 lines
Is this really a "Fix"? The internal function names were not exported in the .map file, so are not part of public ABI. This is an internal naming improvement (thanks for doing cleanup), but I don't think the
Fixes: tags make sense?Also I'm not sure if we want to port this patch back to stable? Changing (internal) function names seems like unnecessary churn, and hence risk
to a
Quoted 4 lines
Quoted 1 line
stable release, without any benefit?
OK.
I will remove these tags in the next version and split the service core patches from the original series into a series by itself.Cool - good idea to split.
Quoted 7 lines
Perhaps we should focus on getting bugfixes in for the existing code, before doing cleanup? It would make backports easier if churn is minimal.
Suggesting patches order (first to last)
1. bugfixes/things to backport
2. cleanups
3. C11 atomic optimizations
That is a good idea. I will follow this order.
Quoted 9 lines
Quoted 4 lines
Thanks,
PhilThanks, and I'll get to reading/reviewing your and Honnappa's feedback later today.
-H
Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions
In reply to Phil Yang<snip>
Quoted 38 lines
Subject: [PATCH v3 07/12] service: remove rte prefix from static functions
Fixes 3cf5eb1546ed ("service: fix and refactor atomic service accesses")Fixes 21698354c832 ("service: introduce service cores concept")Signed-off-by Phil Yang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>--- lib/librte_eal/common/rte_service.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)lib/librte_eal/common/rte_service.c +1
-1@@ -336,7 +336,7 @@ rte_service_runstate_get(uint32_t id) }static inline void -rte_service_runner_do_callback (struct rte_service_spec_impl *s, +service_runner_do_callback (struct rte_service_spec_impl *s,struct core_state *cs,uint32_t service_idx) { void *userdata = s->spec.callback_userdata; @@ -379,10 +379,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))lib/librte_eal/common/rte_service.c +2
-2return -EBUSY ; -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);rte_atomic32_clear (&s->execute_lock); }else -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);return 0 ; }
lib/librte_eal/common/rte_service.c rte_service_run_iter_on_app_lcore(uint32_t id, · line 442
Quoted 8 lines
lib/librte_eal/common/rte_service.c +1
-1@@ -436,7 +436,7 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) }static int32_t -rte_service_runner_func (void *arg) +service_runner_func (void *arg) {RTE_SET_USED (arg);uint32_t i;
This is a minor comment.
Since you are touching 'service_runner_func', please consider doing the below improvement:
struct core_state *cs = &lcore_states[lcore]; while (lcore_states[lcore].runstate == RUNSTATE_RUNNING) {
The while above can be changed as follows to make it more readable
while (cs->runstate == RUNSTATE_RUNNING) {
Quoted 12 lines
lib/librte_eal/common/rte_service.c +1
-1@@ -706,7 +706,7 @@ rte_service_lcore_start(uint32_t lcore)*/ lcore_states[lcore].runstate =RUNSTATE_RUNNING ; -int ret =rte_eal_remote_launch (rte_service_runner_func,0 , lcore); +int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */ return ret; }@@ -785,7 +785,7 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id, }
<snip>
Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions
Quoted 67 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Monday, April 6, 2020 5:35 AM
To: Phil Yang <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 07/12] service: remove rte prefix from static functions<snip>
Quoted 18 lines
Subject: [PATCH v3 07/12] service: remove rte prefix from static functions
Fixes 3cf5eb1546ed ("service: fix and refactor atomic service accesses")Fixes 21698354c832 ("service: introduce service cores concept")Signed-off-by Phil Yang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>--- lib/librte_eal/common/rte_service.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)lib/librte_eal/common/rte_service.c +0
-0lib/librte_eal/common/rte_service.c rte_service_runstate_get(uint32_t id) } · line 342
Quoted 8 lines
lib/librte_eal/common/rte_service.c +1
-1@@ -336,7 +336,7 @@ rte_service_runstate_get(uint32_t id) } static inline void -rte_service_runner_do_callback(struct rte_service_spec_impl *s, +service_runner_do_callback(struct rte_service_spec_impl *s, struct core_state *cs, uint32_t service_idx) { void *userdata = s->spec.callback_userdata; @@ -379,10 +379,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0,1))
Quoted 12 lines
lib/librte_eal/common/rte_service.c +2
-2return -EBUSY ; -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);rte_atomic32_clear (&s->execute_lock); }else -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);return 0 ; }lib/librte_eal/common/rte_service.c rte_service_run_iter_on_app_lcore(uint32_t id, · line 442
Quoted 8 lines
lib/librte_eal/common/rte_service.c +1
-1@@ -436,7 +436,7 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) }static int32_t -rte_service_runner_func (void *arg) +service_runner_func (void *arg) {RTE_SET_USED (arg);uint32_t i;This is a minor comment.
Since you are touching 'service_runner_func', please consider doing the below improvement:
Quoted 6 lines
struct
core_state*cs = &lcore_states[lcore]; while (lcore_states[lcore].runstate ==RUNSTATE_RUNNING) {The while above can be changed as follows to make it more readable
while (
cs->runstate==RUNSTATE_RUNNING) {
OK. I will clean it up in the next version.
Thanks,
Phil
Quoted 14 lines
Quoted 13 lines
lib/librte_eal/common/rte_service.c +1
-1@@ -706,7 +706,7 @@ rte_service_lcore_start(uint32_t lcore)*/ lcore_states[lcore].runstate =RUNSTATE_RUNNING ; -int ret =rte_eal_remote_launch (rte_service_runner_func,0 , lcore); +int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */ return ret; }@@ -785,7 +785,7 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id, }<snip>
v2 cover use c11 atomics for service core lib
In reply to Phil YangThe rte_atomic ops and rte_smp barriers enforce DMB barriers on aarch64. Using c11 atomics with explicit memory ordering instead of the rte_atomic ops and rte_smp barriers for inter-threads synchronization can uplift the performance on aarch64 and no performance loss on x86.
This patchset contains:
1) fix race condition for MT unsafe service.
2) clean up redundant code.
3) use c11 atomics for service core lib to avoid unnecessary barriers.
v2:
Still waiting on Harry for the final solution on the MT unsafe race condition issue. But I have incorporated the comments so far.
1. add 'Fixes' tag for bug-fix patches.
2. remove 'Fixes' tag for code cleanup patches.
3. remove unused parameter for service_dump_one function.
4. replace the execute_lock atomic CAS operation to spinlock_try_lock.
5. use c11 atomics with RELAXED memory ordering for num_mapped_cores.
6. relax barriers for guard variables runstate, comp_runstate and app_runstate with c11 one-way barriers.
Honnappa Nagarahalli (2):
service: fix race condition for MT unsafe service
service: identify service running on another core correctly
Phil Yang (4):
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 atomics
service: relax barriers with C11 atomics
lib/librte_eal/common/rte_service.c | 234 +++++++++++++++++++----------------- lib/librte_eal/meson.build | 4 + 2 files changed, 130 insertions(+), 108 deletions(-)
Signature · 1 line
2.7.4
PATCH v2 1/6 service: fix race condition for MT unsafe service
From: Honnappa Nagarahalli <[email protected]>
The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.
--- lib/librte_eal/common/rte_service.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {uint8_t internal_flags;/* per service statistics */ +/* Indicates how many cores the service is mapped to run on. +* It does not indicate the number of cores the service is running +* on currently. +*/ rte_atomic32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -/* check do we need cmpset, if MT safe or <= 1 core -* mapped, atomic ops are not required. -*/ -const int use_atomics = (service_mt_safe (s) ==0 ) && - (rte_atomic32_read (&s->num_mapped_cores) >1 ); -if (use_atomics) { +if (service_mt_safe (s) ==0 ) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
Signature · 1 line
2.7.4
Re: [dpdk-dev] [PATCH v2 1/6] service: fix race condition for MT unsafe service
Quoted 18 lines
-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Thursday, April 23, 2020 5:31 PM
To: Van Haaren, Harry <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: [PATCH v2 1/6] service: fix race condition for MT unsafe serviceFrom: Honnappa Nagarahalli <[email protected]>
The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use '
execute_lock' always when the service is MT unsafe.
Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")Signed-off-by Honnappa Nagarahalli <[email protected]>Reviewed-by Phil Yang <[email protected]>---
Thanks for spinning a new revision - based on ML discussion previously, it seems like the "use service-run-count" to avoid this race would be a complex solution.
Suggesting the following;
1) Take the approach as per this patch, to always take the atomic, fixing the race condition.
2) Add an API to service-cores, which allows "committing" of mappings. Committing the mapping would imply that the mappings will not be changed in future. With runtime-remapping being removed from the equation, the existing branch-over-atomic optimization is valid again.
So this would offer applications two situations
A) No application change: possible performance regression due to atomic always taken.
B) Call "commit" API, and regain the performance as per previous DPDK versions.
Thoughts/opinions on the above? I've flagged the rest of the patchset for review ASAP. Regards, -Harry
Quoted 36 lines
lib/librte_eal/common/rte_service.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)lib/librte_eal/common/rte_service.c +5
-7@@ -50,6 +50,10 @@ struct rte_service_spec_impl {uint8_t internal_flags;/* per service statistics */ +/* Indicates how many cores the service is mapped to run on. +* It does not indicate the number of cores the service is running +* on currently. +*/ rte_atomic32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent;@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -/* check do we need cmpset, if MT safe or <= 1 core -* mapped, atomic ops are not required. -*/ -const int use_atomics = (service_mt_safe (s) ==0 ) && - (rte_atomic32_read (&s->num_mapped_cores) >1 ); -if (use_atomics) { +if (service_mt_safe (s) ==0 ) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ; --2.7.4
Re: [dpdk-dev] [PATCH v2 1/6] service: fix race condition for MT unsafe service
Hi Harry,
Thanks for getting back on this.
<snip>
Quoted 17 lines
Quoted 10 lines
Subject: [PATCH v2 1/6] service: fix race condition for MT unsafe service
From: Honnappa Nagarahalli <[email protected]>
The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use '
execute_lock' always when the service is MT unsafe.Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")Signed-off-by Honnappa Nagarahalli <[email protected]>Reviewed-by Phil Yang <[email protected]>---
Quoted 7 lines
Thanks for spinning a new revision - based on ML discussion previously, it seems like the "use service-run-count" to avoid this race would be a complex solution.
Suggesting the following;
1) Take the approach as per this patch, to always take the atomic, fixing the race condition.
Ok
Quoted 4 lines
2) Add an API to service-cores, which allows "committing" of mappings. Committing the mapping would imply that the mappings will not be changed in future. With runtime-remapping being removed from the equation, the existing branch-over-atomic optimization is valid again.
Ok. Just to make sure I understand this:
a) on the data plane, if commit API is called (probably a new state variable) and num_mapped_cores is set to 1, there is no need to take the lock.
b) possible implementation of the commit API would check if num_mapped_cores for the service is set to 1 and set a variable to indicate that the lock is not required.
What do you think about asking the application to set the service capability to MT_SAFE if it knows that the service will run on a single core? This would require us to change the documentation and does not require additional code.
Quoted 47 lines
So this would offer applications two situations
A) No application change: possible performance regression due to atomic always taken.
B) Call "commit" API, and regain the performance as per previous DPDK versions.Thoughts/opinions on the above? I've flagged the rest of the patchset for review ASAP. Regards, -Harry
Quoted 37 lines
lib/librte_eal/common/rte_service.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)lib/librte_eal/common/rte_service.c +5
-6@@ -50,6 +50,10 @@ struct rte_service_spec_impl {uint8_t internal_flags;/* per service statistics */ +/* Indicates how many cores the service is mapped to run on. +* It does not indicate the number of cores the service is running +* on currently. +*/ rte_atomic32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent;@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs,uint64_t service_mask, cs->service_active_on_lcore[i] = 1; -/* check do we need cmpset, if MT safe or <= 1 core -* mapped, atomic ops are not required. -*/ -const int use_atomics = (service_mt_safe (s) ==0 ) && - (rte_atomic32_read (&s- num_mapped_cores) > 1); -if (use_atomics) { +if (service_mt_safe (s) ==0 ) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return
-EBUSY;--
2.7.4
Re: [dpdk-dev] [PATCH v2 1/6] service: fix race condition for MT unsafe service
Quoted 35 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Wednesday, April 29, 2020 11:49 PM
To: Van Haaren, Harry <[email protected]>; Phil Yang <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Gavin Hu <[email protected]>; nd <[email protected]>; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v2 1/6] service: fix race condition for MT unsafe serviceHi Harry,
Thanks for getting back on this.<snip>
Quoted 17 lines
Quoted 16 lines
Subject: [PATCH v2 1/6] service: fix race condition for MT unsafe service
From: Honnappa Nagarahalli <[email protected]>
The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use '
execute_lock' always when the service is MT unsafe.Fixes e9139a32f6e8 ("service: add function to run on app lcore")Signed-off-by Honnappa Nagarahalli <[email protected]>Reviewed-by Phil Yang <[email protected]>---
Quoted 8 lines
Quoted 7 lines
Thanks for spinning a new revision - based on ML discussion previously, it seems like the "use service-run-count" to avoid this race would be a complex solution.
Suggesting the following;
1) Take the approach as per this patch, to always take the atomic, fixing the race condition.Ok
I've micro-benchmarked this code change inside the service cores autotest, and it introduces around 35 cycles of overhead per service call. This is not ideal, but given it's a bugfix, and by far the simplest method to fix this race-condition. Having discussed and investigated multiple other solutions, I believe this is the right solution. Thanks Honnappa and Phil for identifying and driving a solution.
I suggest to post the benchmarking unit-test addition patch, and integrate that *before* the series under review here gets merged? This makes benchmarking of the "before bugfix" performance in future easier should it be required.
Quoted 11 lines
Quoted 4 lines
2) Add an API to service-cores, which allows "committing" of mappings. Committing the mapping would imply that the mappings will not be changed in future. With runtime-remapping being removed from the equation, the existing branch-over-atomic optimization is valid again.
Ok. Just to make sure I understand this:
a) on the data plane, if commit API is called (probably a new state variable) andnum_mapped_coresis set to 1, there is no need to take the lock.
b) possible implementation of the commit API would check ifnum_mapped_coresfor the service is set to 1 and set a variable to indicate that the lock is not required.
Quoted 3 lines
What do you think about asking the application to set the service capability to
MT_SAFEif it knows that the service will run on a single core? This would require us to change the documentation and does not require additional code.
That's a nice idea - I like that if applications want to micro-optimize around the atomic, that they have a workaround/solution to do so, particularly that it doesn't require code-changes and backporting.
Will send review and send feedback on the patches themselves. Regards, -Harry
Quoted 11 lines
Quoted 11 lines
So this would offer applications two situations
A) No application change: possible performance regression due to atomic always taken.
B) Call "commit" API, and regain the performance as per previous DPDK versions.Thoughts/opinions on the above? I've flagged the rest of the patchset for review ASAP. Regards, -Harry
Quoted 2 lines
lib/librte_eal/common/rte_service.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
<snip patch changes>
Re: [dpdk-dev] [PATCH v2 1/6] service: fix race condition for MT unsafe service
<snip>
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 11 lines
Subject: [PATCH v2 1/6] service: fix race condition for MT unsafe service
From: Honnappa Nagarahalli <[email protected]>
The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use '
execute_lock' always when the service is MT unsafe.Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")Signed-off-by Honnappa Nagarahalli <[email protected]>Reviewed-by Phil Yang <[email protected]>---
Thanks for spinning a new revision - based on ML discussion previously, it seems like the "use service-run-count" to avoid this race would be a complex solution.
Suggesting the following;
1) Take the approach as per this patch, to always take the atomic, fixing the race condition.
Quoted 8 lines
Quoted 1 line
Ok
I've micro-benchmarked this code change inside the service cores autotest, and it introduces around 35 cycles of overhead per service call. This is not ideal, but given it's a bugfix, and by far the simplest method to fix this race- condition. Having discussed and investigated multiple other solutions, I believe this is the right solution.
Thanks Honnappa and Phil for identifying and driving a solution.
You are welcome. Thank you for your timely responses.
Quoted 6 lines
I suggest to post the benchmarking unit-test addition patch, and integrate that
*before* the series under review here gets merged? This makes benchmarking of the "before bugfix" performance in future easier should it be required.
I do not see any issues, would be happy to review. I think we still have time to catch up with RC2 (May 8th). You had also mentioned about calling out that, the control plane APIs are not MT safe. Should I add that to this patch?
Quoted 38 lines
Quoted 9 lines
Quoted 6 lines
2) Add an API to service-cores, which allows "committing" of mappings. Committing the mapping would imply that the mappings will not be changed in future. With runtime-remapping being removed from the equation, the existing branch-over-atomic optimization is valid again.
Ok. Just to make sure I understand this:
a) on the data plane, if commit API is called (probably a new state variable) andnum_mapped_coresis set to 1, there is no need to take thelock.
Quoted 7 lines
b) possible implementation of the commit API would check if
num_mapped_coresfor the service is set to 1 and set a variable to indicate that the lock is not required.What do you think about asking the application to set the service capability to
MT_SAFEif it knows that the service will run on a single core? This would require us to change the documentation and doesnot require additional code.
That's a nice idea - I like that if applications want to micro-optimize around the atomic, that they have a workaround/solution to do so, particularly that it doesn't require code-changes and backporting.
Will send review and send feedback on the patches themselves. Regards, -Harry
Quoted 11 lines
Quoted 11 lines
So this would offer applications two situations
A) No application change: possible performance regression due to atomic always taken.
B) Call "commit" API, and regain the performance as per previous DPDK versions.Thoughts/opinions on the above? I've flagged the rest of the patchset for review ASAP. Regards, -Harry
Quoted 2 lines
lib/librte_eal/common/rte_service.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)<snip patch changes>
Re: [dpdk-dev] [PATCH v2 1/6] service: fix race condition for MT unsafe service
Quoted 41 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Friday, May 1, 2020 3:56 PM
To: Van Haaren, Harry <[email protected]>; Phil Yang <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Gavin Hu <[email protected]>; nd <[email protected]>; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v2 1/6] service: fix race condition for MT unsafe service<snip>
Quoted 27 lines
Quoted 26 lines
Quoted 18 lines
Quoted 11 lines
Subject: [PATCH v2 1/6] service: fix race condition for MT unsafe service
From: Honnappa Nagarahalli <[email protected]>
The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use '
execute_lock' always when the service is MT unsafe.Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")Signed-off-by Honnappa Nagarahalli <[email protected]>Reviewed-by Phil Yang <[email protected]>---
Quoted 7 lines
Thanks for spinning a new revision - based on ML discussion previously, it seems like the "use service-run-count" to avoid this race would be a complex solution.
Suggesting the following;
1) Take the approach as per this patch, to always take the atomic, fixing the race condition.Ok
Quoted 7 lines
Quoted 6 lines
I've micro-benchmarked this code change inside the service cores autotest, and it introduces around 35 cycles of overhead per service call. This is not ideal, but given it's a bugfix, and by far the simplest method to fix this race- condition. Having discussed and investigated multiple other solutions, I believe this is the right solution.
Thanks Honnappa and Phil for identifying and driving a solution.You are welcome. Thank you for your timely responses.
Perhaps not so timely after all ... I'll review C11 patches Tuesday morning, it's a long weekend in Ireland!
Quoted 6 lines
Quoted 5 lines
I suggest to post the benchmarking unit-test addition patch, and integrate that
*before* the series under review here gets merged? This makes benchmarking of the "before bugfix" performance in future easier should it be required.I do not see any issues, would be happy to review.
Thanks for volunteering, you're on CC when sent, for convenience:
http://patches.dpdk.org/patch/69651/
Quoted 1 line
I think we still have time to catch up with RC2 (May 8th).
Agree, merge into RC2 would be great.
Quoted 2 lines
You had also mentioned about calling out that, the control plane APIs are not MT safe. Should I add that to this patch?
Yes, that'd be great.
<snip discussion details>
Cheers, -Harry
PATCH v2 2/6 service: identify service running on another core correctly
In reply to Phil YangFrom: Honnappa Nagarahalli <[email protected]>
The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases.
--- lib/librte_eal/common/rte_service.c | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
@@ -360,7 +360,7 @@ rte_service_runner_do_callback(struct rte_service_spec_impl *s,/* Expects the service 's' is valid. */ static int32_t service_run (uint32_t i,struct core_state *cs,uint64_t service_mask, -struct rte_service_spec_impl *s) +struct rte_service_spec_impl *s,uint32_t serialize_mt_unsafe) {if (!s)return -EINVAL ;
@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -if (service_mt_safe (s) ==0 ) { +if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); -/* Atomically add this core to the mapped cores first, then examine if -* we can run the service. This avoids a race condition between -* checking the value, and atomically adding to the mapped count. +/* Increment num_mapped_cores to indicate that the service +* is running on a core. */ -if (serialize_mt_unsafe) -rte_atomic32_inc (&s->num_mapped_cores); +rte_atomic32_inc (&s->num_mapped_cores); -if (service_mt_safe (s) ==0 && -rte_atomic32_read (&s->num_mapped_cores) >1 ) { -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); -return -EBUSY ; - } - -int ret =service_run (id, cs,UINT64_MAX , s); +int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); +rte_atomic32_dec (&s->num_mapped_cores);return ret; }
@@ -449,7 +439,7 @@ rte_service_runner_func(void *arg)if (!service_valid (i))continue ;/* return value ignored as no change to code flow */ -service_run (i, cs, service_mask,service_get (i)); +service_run (i, cs, service_mask,service_get (i),1 ); } cs->loops++;
Signature · 1 line
2.7.4
PATCH v2 3/6 service: remove rte prefix from static functions
In reply to Phil Yangclean up rte prefix from static functions. remove unused parameter for service_dump_one function.
--- lib/librte_eal/common/rte_service.c | 34 +++++++++++----------------------- 1 file changed, 11 insertions(+), 23 deletions(-)
lib/librte_eal/common/rte_service.c +11 -23
@@ -340,7 +340,7 @@ rte_service_runstate_get(uint32_t id) }static inline void -rte_service_runner_do_callback (struct rte_service_spec_impl *s, +service_runner_do_callback (struct rte_service_spec_impl *s,struct core_state *cs,uint32_t service_idx) {void *userdata = s->spec.callback_userdata;
@@ -378,10 +378,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ; -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);rte_atomic32_clear (&s->execute_lock); }else -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);return 0 ; }
@@ -425,14 +425,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) }static int32_t -rte_service_runner_func (void *arg) +service_runner_func (void *arg) {RTE_SET_USED (arg);uint32_t i;const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (lcore_states[lcore].runstate ==RUNSTATE_RUNNING ) { +while (cs->runstate ==RUNSTATE_RUNNING ) {const uint64_t service_mask = cs->service_mask;for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) {
@@ -693,9 +693,9 @@ rte_service_lcore_start(uint32_t lcore)/* set core to run state first, and then launch otherwise it will * return immediately as runstate keeps it in the service poll loop */ - lcore_states[lcore].runstate =RUNSTATE_RUNNING ; + cs->runstate =RUNSTATE_RUNNING ; -int ret =rte_eal_remote_launch (rte_service_runner_func,0 , lcore); +int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */ return ret; }
@@ -774,13 +774,9 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id, }static void -rte_service_dump_one (FILE *f,struct rte_service_spec_impl *s, -uint64_t all_cycles,uint32_t reset) +service_dump_one (FILE *f,struct rte_service_spec_impl *s,uint32_t reset) {/* avoid divide by zero */ -if (all_cycles ==0 ) - all_cycles =1 ; -int calls =1 ;if (s->calls !=0 ) calls = s->calls;
@@ -807,7 +803,7 @@ rte_service_attr_reset_all(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );int reset =1 ; -rte_service_dump_one (NULL , s,0 , reset); +service_dump_one (NULL , s, reset);return 0 ; }
@@ -851,21 +847,13 @@ rte_service_dump(FILE *f, uint32_t id)uint32_t i;int print_one = (id !=UINT32_MAX ); -uint64_t total_cycles =0 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if (!service_valid (i)) -continue ; - total_cycles += rte_services[i].cycles_spent; - } -/* print only the specified service */ if (print_one) {struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );fprintf (f,"Service %s Summary\n" , s->spec.name);uint32_t reset =0 ; -rte_service_dump_one (f, s, total_cycles, reset); +service_dump_one (f, s, reset);return 0 ; }
@@ -875,7 +863,7 @@ rte_service_dump(FILE *f, uint32_t id)if (!service_valid (i))continue ;uint32_t reset =0 ; -rte_service_dump_one (f, &rte_services[i], total_cycles, reset); +service_dump_one (f, &rte_services[i], reset); }fprintf (f,"Service Cores Summary\n" );
Signature · 1 line
2.7.4
PATCH v2 4/6 service: remove redundant code
In reply to Phil YangThe service id validation is duplicated, remove the redundant code in the calling functions.
--- lib/librte_eal/common/rte_service.c | 28 ++++++---------------------- 1 file changed, 6 insertions(+), 22 deletions(-)
lib/librte_eal/common/rte_service.c +6 -22
@@ -541,24 +541,12 @@ rte_service_start_with_defaults(void) }static int32_t -service_update (struct rte_service_spec *service,uint32_t lcore, +service_update (uint32_t sid,uint32_t lcore,uint32_t *set,uint32_t *enabled) { -uint32_t i; -int32_t sid =-1 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if ((struct rte_service_spec *)&rte_services[i] == service && -service_valid (i)) { - sid = i; -break ; - } - } - -if (sid ==-1 || lcore >=RTE_MAX_LCORE ) -return -EINVAL ; - -if (!lcore_states[lcore].is_service_core) +/* validate ID, or return error value */ +if (sid >=RTE_SERVICE_NUM_MAX || !service_valid (sid) || + lcore >=RTE_MAX_LCORE || !lcore_states[lcore].is_service_core)return -EINVAL ;uint64_t sid_mask =UINT64_C (1 ) << sid;
@@ -587,19 +575,15 @@ service_update(struct rte_service_spec *service, uint32_t lcore,int32_t rte_service_map_lcore_set (uint32_t id,uint32_t lcore,uint32_t enabled) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );uint32_t on = enabled >0 ; -return service_update (&s->spec, lcore, &on,0 ); +return service_update (id, lcore, &on,0 ); }int32_t rte_service_map_lcore_get (uint32_t id,uint32_t lcore) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );uint32_t enabled; -int ret =service_update (&s->spec, lcore,0 , &enabled); +int ret =service_update (id, lcore,0 , &enabled);if (ret ==0 )return enabled;return ret;
Signature · 1 line
2.7.4
PATCH v2 5/6 service: optimize with c11 atomics
In reply to Phil YangThe num_mapped_cores is used as a statistics. Use c11 atomics with RELAXED ordering for num_mapped_cores instead of rte_atomic ops which enforce unnessary barriers on aarch64.
Replace execute_lock operations to spinlock_try_lock to avoid duplicate code.
--- lib/librte_eal/common/rte_service.c | 32 ++++++++++++++++++-------------- lib/librte_eal/meson.build | 4 ++++ 2 files changed, 22 insertions(+), 14 deletions(-)
lib/librte_eal/common/rte_service.c +18 -14
@@ -20,6 +20,7 @@#include <rte_atomic.h> #include <rte_memory.h> #include <rte_malloc.h> +#include <rte_spinlock.h> #include "eal_private.h"
@@ -38,11 +39,11 @@ struct rte_service_spec_impl {/* public part of the struct */ struct rte_service_spec spec; -/* atomic lock that when set indicates a service core is currently +/* spin lock that when set indicates a service core is currently * running this service callback. When not set, a core may take the * lock and then run the service callback. */ -rte_atomic32_t execute_lock; +rte_spinlock_t execute_lock;/* API set/get-able variables */ int8_t app_runstate;
@@ -54,7 +55,7 @@ struct rte_service_spec_impl {* It does not indicate the number of cores the service is running * on currently. */ -rte_atomic32_t num_mapped_cores; +uint32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent; } __rte_cache_aligned;
@@ -332,7 +333,8 @@ rte_service_runstate_get(uint32_t id)rte_smp_rmb ();int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); -int lcore_mapped = (rte_atomic32_read (&s->num_mapped_cores) >0 ); +int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, + __ATOMIC_RELAXED) >0 );return (s->app_runstate ==RUNSTATE_RUNNING ) && (s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -375,11 +377,11 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ;if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) { -if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 )) +if (!rte_spinlock_trylock (&s->execute_lock))return -EBUSY ;service_runner_do_callback (s, cs, i); -rte_atomic32_clear (&s->execute_lock); +rte_spinlock_unlock (&s->execute_lock); }else service_runner_do_callback (s, cs, i);
@@ -415,11 +417,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)/* Increment num_mapped_cores to indicate that the service * is running on a core. */ -rte_atomic32_inc (&s->num_mapped_cores); +__atomic_add_fetch (&s->num_mapped_cores,1 , __ATOMIC_RELAXED);int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -rte_atomic32_dec (&s->num_mapped_cores); +__atomic_sub_fetch (&s->num_mapped_cores,1 , __ATOMIC_RELAXED);return ret; }
@@ -556,19 +558,19 @@ service_update(uint32_t sid, uint32_t lcore,if (*set && !lcore_mapped) { lcore_states[lcore].service_mask |= sid_mask; -rte_atomic32_inc (&rte_services[sid].num_mapped_cores); +__atomic_add_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELAXED); }if (!*set && lcore_mapped) { lcore_states[lcore].service_mask &= ~(sid_mask); -rte_atomic32_dec (&rte_services[sid].num_mapped_cores); +__atomic_sub_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELAXED); } }if (enabled) *enabled = !!(lcore_states[lcore].service_mask & (sid_mask)); -rte_smp_wmb (); -return 0 ; }
@@ -616,7 +618,8 @@ rte_service_lcore_reset_all(void) } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) -rte_atomic32_set (&rte_services[i].num_mapped_cores,0 ); +__atomic_store_n (&rte_services[i].num_mapped_cores,0 , + __ATOMIC_RELAXED);rte_smp_wmb ();
@@ -699,7 +702,8 @@ rte_service_lcore_stop(uint32_t lcore)int32_t enabled = service_mask & (UINT64_C (1 ) << i);int32_t service_running =rte_service_runstate_get (i);int32_t only_core = (1 == -rte_atomic32_read (&rte_services[i].num_mapped_cores)); +__atomic_load_n (&rte_services[i].num_mapped_cores, + __ATOMIC_RELAXED));/* if the core is mapped, and the service is running, and this * is the only core that is mapped, the service would cease to
lib/librte_eal/meson.build +4 -0
@@ -21,3 +21,7 @@ endif if cc.has_header('getopt.h') cflags += ['-DHAVE_GETOPT_H', '-DHAVE_GETOPT', '-DHAVE_GETOPT_LONG'] endif +# for clang 32-bit compiles we need libatomic for 64-bit atomic ops +if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false + ext_deps += cc.find_library('atomic') +endif
Signature · 1 line
2.7.4
PATCH v2 6/6 service: relax barriers with C11 atomics
In reply to Phil YangThe runstate, comp_runstate and app_runstate are used as guard variables in the service core lib. To guarantee the inter-threads visibility of these guard variables, it uses rte_smp_r/wmb. This patch use c11 atomic built-ins to relax these barriers.
--- lib/librte_eal/common/rte_service.c | 115 ++++++++++++++++++++++++++---------- 1 file changed, 84 insertions(+), 31 deletions(-)
lib/librte_eal/common/rte_service.c +84 -31
@@ -265,7 +265,6 @@ rte_service_component_register(const struct rte_service_spec *spec, s->spec = *spec; s->internal_flags |=SERVICE_F_REGISTERED |SERVICE_F_START_CHECK ; -rte_smp_wmb (); rte_service_count++;if (id_ptr)
@@ -282,7 +281,6 @@ rte_service_component_unregister(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); rte_service_count--; -rte_smp_wmb (); s->internal_flags &= ~(SERVICE_F_REGISTERED );
@@ -301,12 +299,17 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* comp_runstate act as the guard variable. Use store-release +* memory order. This synchronizes with load-acquire in +* service_run and service_runstate_get function. +*/ if (runstate) - s->comp_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->comp_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -316,12 +319,17 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* app_runstate act as the guard variable. Use store-release +* memory order. This synchronizes with load-acquire in +* service_run runstate_get function. +*/ if (runstate) - s->app_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->app_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->app_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->app_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -330,15 +338,24 @@ rte_service_runstate_get(uint32_t id) {struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); -rte_smp_rmb (); -int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); -int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, +/* comp_runstate and app_runstate act as the guard variables. +* Use load-acquire memory order. This synchronizes with +* store-release in service state set functions. +*/ +if (__atomic_load_n (&s->comp_runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING && +__atomic_load_n (&s->app_runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) { +int check_disabled = !(s->internal_flags & +SERVICE_F_START_CHECK ); +int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, __ATOMIC_RELAXED) >0 ); -return (s->app_runstate ==RUNSTATE_RUNNING ) && - (s->comp_runstate ==RUNSTATE_RUNNING ) && - (check_disabled | lcore_mapped); +return (check_disabled | lcore_mapped); + }else +return 0 ; + } static inline void
@@ -367,9 +384,15 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,if (!s)return -EINVAL ; -if (s->comp_runstate !=RUNSTATE_RUNNING || - s->app_runstate !=RUNSTATE_RUNNING || - !(service_mask & (UINT64_C (1 ) << i))) { +/* comp_runstate and app_runstate act as the guard variables. +* Use load-acquire memory order. This synchronizes with +* store-release in service state set functions. +*/ +if (__atomic_load_n (&s->comp_runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_RUNNING || +__atomic_load_n (&s->app_runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_RUNNING || + !(service_mask & (UINT64_C (1 ) << i))) { cs->service_active_on_lcore[i] =0 ;return -ENOEXEC ; }
@@ -434,7 +457,12 @@ service_runner_func(void *arg)const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (cs->runstate ==RUNSTATE_RUNNING ) { +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +while (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) {const uint64_t service_mask = cs->service_mask;for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) {
@@ -445,8 +473,6 @@ service_runner_func(void *arg) } cs->loops++; - -rte_smp_rmb (); } lcore_config[lcore].state =WAIT ;
@@ -614,15 +640,18 @@ rte_service_lcore_reset_all(void)if (lcore_states[i].is_service_core) { lcore_states[i].service_mask =0 ;set_lcore_state (i,ROLE_RTE ); - lcore_states[i].runstate =RUNSTATE_STOPPED ; +/* runstate act as guard variable Use +* store-release memory order here to synchronize +* with load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[i].runstate, +RUNSTATE_STOPPED , __ATOMIC_RELEASE); } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++)__atomic_store_n (&rte_services[i].num_mapped_cores,0 , __ATOMIC_RELAXED); -rte_smp_wmb (); -return 0 ; }
@@ -638,9 +667,11 @@ rte_service_lcore_add(uint32_t lcore)/* ensure that after adding a core the mask and state are defaults */ lcore_states[lcore].service_mask =0 ; - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; - -rte_smp_wmb (); +/* Use store-release memory order here to synchronize with +* load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);return rte_eal_wait_lcore (lcore); }
@@ -655,7 +686,12 @@ rte_service_lcore_del(uint32_t lcore)if (!cs->is_service_core)return -EINVAL ; -if (cs->runstate !=RUNSTATE_STOPPED ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_STOPPED )return -EBUSY ;set_lcore_state (lcore,ROLE_RTE );
@@ -674,13 +710,21 @@ rte_service_lcore_start(uint32_t lcore)if (!cs->is_service_core)return -EINVAL ; -if (cs->runstate ==RUNSTATE_RUNNING ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING )return -EALREADY ;/* set core to run state first, and then launch otherwise it will * return immediately as runstate keeps it in the service poll loop */ - cs->runstate =RUNSTATE_RUNNING ; +/* Use load-acquire memory order here to synchronize with +* store-release in runstate update functions. +*/ +__atomic_store_n (&cs->runstate,RUNSTATE_RUNNING , __ATOMIC_RELEASE);int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */
@@ -693,7 +737,12 @@ rte_service_lcore_stop(uint32_t lcore)if (lcore >=RTE_MAX_LCORE )return -EINVAL ; -if (lcore_states[lcore].runstate ==RUNSTATE_STOPPED ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&lcore_states[lcore].runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_STOPPED )return -EALREADY ;uint32_t i;
@@ -713,7 +762,11 @@ rte_service_lcore_stop(uint32_t lcore)return -EBUSY ; } - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; +/* Use store-release memory order here to synchronize with +* load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);return 0 ; }
Signature · 1 line
2.7.4
v3 cover use c11 atomics for service core lib
In reply to Phil YangThe rte_atomic ops and rte_smp barriers enforce DMB barriers on aarch64. Using c11 atomics with explicit memory ordering instead of the rte_atomic ops and rte_smp barriers for inter-threads synchronization can uplift the performance on aarch64 and no performance loss on x86.
This patchset contains:
1) fix race condition for MT unsafe service.
2) clean up redundant code.
3) use c11 atomics for service core lib to avoid unnecessary barriers.
v2:
Still waiting on Harry for the final solution on the MT unsafe race condition issue. But I have incorporated the comments so far.
1. add 'Fixes' tag for bug-fix patches.
2. remove 'Fixes' tag for code cleanup patches.
3. remove unused parameter for service_dump_one function.
4. replace the execute_lock atomic CAS operation to spinlock_try_lock.
5. use c11 atomics with RELAXED memory ordering for num_mapped_cores.
6. relax barriers for guard variables runstate, comp_runstate and app_runstate with c11 one-way barriers.
v3:
Sending this version since Phil is on holiday.
1. Updated the API documentation to indicate how the locking can be avoided.
Honnappa Nagarahalli (2):
service: fix race condition for MT unsafe service
service: identify service running on another core correctly
Phil Yang (4):
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 atomics
service: relax barriers with C11 atomics
lib/librte_eal/common/rte_service.c | 234 ++++++++++-------- lib/librte_eal/include/rte_service.h | 8 +- .../include/rte_service_component.h | 6 +- lib/librte_eal/meson.build | 4 + 4 files changed, 141 insertions(+), 111 deletions(-)
Signature · 1 line
2.17.1
PATCH v3 1/6 service: fix race condition for MT unsafe service
The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.
If the service is known to be mmapped on a single lcore, setting the service capability to MT safe will avoid taking the lock and improve the performance.
--- lib/librte_eal/common/rte_service.c | 11 +++++------ lib/librte_eal/include/rte_service.h | 8 ++++++-- lib/librte_eal/include/rte_service_component.h | 6 +++++- 3 files changed, 16 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {uint8_t internal_flags;/* per service statistics */ +/* Indicates how many cores the service is mapped to run on. +* It does not indicate the number of cores the service is running +* on currently. +*/ rte_atomic32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -/* check do we need cmpset, if MT safe or <= 1 core -* mapped, atomic ops are not required. -*/ -const int use_atomics = (service_mt_safe (s) ==0 ) && - (rte_atomic32_read (&s->num_mapped_cores) >1 ); -if (use_atomics) { +if (service_mt_safe (s) ==0 ) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
lib/librte_eal/include/rte_service.h +6 -2
@@ -104,12 +104,16 @@ int32_t rte_service_probe_capability(uint32_t id, uint32_t capability);* Each core can be added or removed from running a specific service. This * function enables or disables *lcore* to run *service_id*. * -* If multiple cores are enabled on a service, an atomic is used to ensure that -* only one cores runs the service at a time. The exception to this is when +* If multiple cores are enabled on a service, a lock is used to ensure that +* only one core runs the service at a time. The exception to this is when * a service indicates that it is multi-thread safe by setting the capability * called RTE_SERVICE_CAP_MT_SAFE. With the multi-thread safe capability set, * the service function can be run on multiple threads at the same time. * +* If the service is known to be mapped to a single lcore, setting the +* capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve +* better performance by avoiding the use of lock. +* * @param service_id the service to apply the lcore to * @param lcore The lcore that will be mapped to service * @param enable Zero to unmap or disable the core, non-zero to enable
lib/librte_eal/include/rte_service_component.h +5 -1
@@ -43,7 +43,7 @@ struct rte_service_spec {/** * Register a new service. * -* A service represents a component that the requires CPU time periodically to +* A service represents a component that requires CPU time periodically to * achieve its purpose. * * For example the eventdev SW PMD requires CPU cycles to perform its
@@ -56,6 +56,10 @@ struct rte_service_spec {* *rte_service_component_runstate_set*, which indicates that the service * component is ready to be executed. * +* If the service is known to be mapped to a single lcore, setting the +* capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve +* better performance. +* * @param spec The specification of the service to register * @param[out] service_id A pointer to a uint32_t, which will be filled in * during registration of the service. It is set to the integers
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v3 1/6] service: fix race condition for MT unsafe service
Quoted 12 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, May 2, 2020 1:03 AM
To: [email protected]; [email protected]; Van Haaren, Harry <[email protected]>
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 1/6] service: fix race condition for MT unsafe service
Quoted 6 lines
The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use '
execute_lock' always when the service is MT unsafe.If the service is known to be mmapped on a single lcore,
mmapped is a typo? Just mapped.
Quoted 8 lines
setting the service capability to MT safe will avoid taking the lock and improve the performance.
Fixes e9139a32f6e8 ("service: add function to run on app lcore")Signed-off-by Honnappa Nagarahalli <[email protected]>Reviewed-by Phil Yang <[email protected]>
<snip diff>
PATCH v3 2/6 service: identify service running on another core correctly
In reply to Honnappa NagarahalliThe logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases.
--- lib/librte_eal/common/rte_service.c | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
@@ -360,7 +360,7 @@ rte_service_runner_do_callback(struct rte_service_spec_impl *s,/* Expects the service 's' is valid. */ static int32_t service_run (uint32_t i,struct core_state *cs,uint64_t service_mask, -struct rte_service_spec_impl *s) +struct rte_service_spec_impl *s,uint32_t serialize_mt_unsafe) {if (!s)return -EINVAL ;
@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -if (service_mt_safe (s) ==0 ) { +if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); -/* Atomically add this core to the mapped cores first, then examine if -* we can run the service. This avoids a race condition between -* checking the value, and atomically adding to the mapped count. +/* Increment num_mapped_cores to indicate that the service +* is running on a core. */ -if (serialize_mt_unsafe) -rte_atomic32_inc (&s->num_mapped_cores); +rte_atomic32_inc (&s->num_mapped_cores); -if (service_mt_safe (s) ==0 && -rte_atomic32_read (&s->num_mapped_cores) >1 ) { -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); -return -EBUSY ; - } - -int ret =service_run (id, cs,UINT64_MAX , s); +int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); +rte_atomic32_dec (&s->num_mapped_cores);return ret; }
@@ -449,7 +439,7 @@ rte_service_runner_func(void *arg)if (!service_valid (i))continue ;/* return value ignored as no change to code flow */ -service_run (i, cs, service_mask,service_get (i)); +service_run (i, cs, service_mask,service_get (i),1 ); } cs->loops++;
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v3 2/6] service: identify service running on another core correctly
Quoted 13 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, May 2, 2020 1:03 AM
To: [email protected]; [email protected]; Van Haaren, Harry <[email protected]>
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 2/6] service: identify service running on another core correctly
Quoted 8 lines
The logic to identify if the MT unsafe service is running on another core can return
-EBUSYspuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases.Fixes 8d39d3e237c2 ("service: fix race in service on app lcore function")
Add "fix" to the title, suggestion:
service: fix identification of service running on other lcore
Quoted 2 lines
Signed-off-by Honnappa Nagarahalli <[email protected]>Reviewed-by Phil Yang <[email protected]>
I believe there may be some optimizations we can apply after this patchset as the "num_mapped_cores" variable is no longer used in a significant way for the atomic selection, however lets leave that optimization outside of 20.05 scope.
With title (see above) & comment (see below) updated.
Quoted 1 line
---
<snip some diff>
Quoted 4 lines
lib/librte_eal/common/rte_service.c +0
-0@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
lib/librte_eal/common/rte_service.c rte_service_run_iter_on_app_lcore(uint32_t id, · line 419
Quoted 10 lines
lib/librte_eal/common/rte_service.c +3
-5-/* Atomically add this core to the mapped cores first, then examine if -* we can run the service. This avoids a race condition between -* checking the value, and atomically adding to the mapped count. +/* Increment num_mapped_cores to indicate that the service +* is running on a core. */ -if (serialize_mt_unsafe) -rte_atomic32_inc (&s->num_mapped_cores); +rte_atomic32_inc (&s->num_mapped_cores);
The comment for the added lines here are a little confusing to me, the "num_mapped_cores" does not indicate that the service "is running on a core", it indicates the number of mapped lcores to that service. Suggestion below?
/* Increment num_mapped_cores to reflect that this core is
* now mapped capable of running the service.
*/
lib/librte_eal/common/rte_service.c rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) · line 420
Quoted 9 lines
lib/librte_eal/common/rte_service.c +1
-8-if (service_mt_safe (s) ==0 && -rte_atomic32_read (&s->num_mapped_cores) >1 ) { -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); -return -EBUSY ; - } - -int ret =service_run (id, cs,UINT64_MAX , s); +int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe);
<snip rest of diff>
PATCH v3 3/6 service: remove rte prefix from static functions
In reply to Honnappa NagarahalliFrom: Phil Yang <[email protected]>
clean up rte prefix from static functions. remove unused parameter for service_dump_one function.
--- lib/librte_eal/common/rte_service.c | 34 ++++++++++------------------- 1 file changed, 11 insertions(+), 23 deletions(-)
lib/librte_eal/common/rte_service.c +11 -23
@@ -340,7 +340,7 @@ rte_service_runstate_get(uint32_t id) }static inline void -rte_service_runner_do_callback (struct rte_service_spec_impl *s, +service_runner_do_callback (struct rte_service_spec_impl *s,struct core_state *cs,uint32_t service_idx) {void *userdata = s->spec.callback_userdata;
@@ -378,10 +378,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ; -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);rte_atomic32_clear (&s->execute_lock); }else -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);return 0 ; }
@@ -425,14 +425,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) }static int32_t -rte_service_runner_func (void *arg) +service_runner_func (void *arg) {RTE_SET_USED (arg);uint32_t i;const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (lcore_states[lcore].runstate ==RUNSTATE_RUNNING ) { +while (cs->runstate ==RUNSTATE_RUNNING ) {const uint64_t service_mask = cs->service_mask;for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) {
@@ -693,9 +693,9 @@ rte_service_lcore_start(uint32_t lcore)/* set core to run state first, and then launch otherwise it will * return immediately as runstate keeps it in the service poll loop */ - lcore_states[lcore].runstate =RUNSTATE_RUNNING ; + cs->runstate =RUNSTATE_RUNNING ; -int ret =rte_eal_remote_launch (rte_service_runner_func,0 , lcore); +int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */ return ret; }
@@ -774,13 +774,9 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id, }static void -rte_service_dump_one (FILE *f,struct rte_service_spec_impl *s, -uint64_t all_cycles,uint32_t reset) +service_dump_one (FILE *f,struct rte_service_spec_impl *s,uint32_t reset) {/* avoid divide by zero */ -if (all_cycles ==0 ) - all_cycles =1 ; -int calls =1 ;if (s->calls !=0 ) calls = s->calls;
@@ -807,7 +803,7 @@ rte_service_attr_reset_all(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );int reset =1 ; -rte_service_dump_one (NULL , s,0 , reset); +service_dump_one (NULL , s, reset);return 0 ; }
@@ -851,21 +847,13 @@ rte_service_dump(FILE *f, uint32_t id)uint32_t i;int print_one = (id !=UINT32_MAX ); -uint64_t total_cycles =0 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if (!service_valid (i)) -continue ; - total_cycles += rte_services[i].cycles_spent; - } -/* print only the specified service */ if (print_one) {struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );fprintf (f,"Service %s Summary\n" , s->spec.name);uint32_t reset =0 ; -rte_service_dump_one (f, s, total_cycles, reset); +service_dump_one (f, s, reset);return 0 ; }
@@ -875,7 +863,7 @@ rte_service_dump(FILE *f, uint32_t id)if (!service_valid (i))continue ;uint32_t reset =0 ; -rte_service_dump_one (f, &rte_services[i], total_cycles, reset); +service_dump_one (f, &rte_services[i], reset); }fprintf (f,"Service Cores Summary\n" );
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v3 3/6] service: remove rte prefix from static functions
Quoted 12 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, May 2, 2020 1:03 AM
To: [email protected]; [email protected]; Van Haaren, Harry <[email protected]>
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v3 3/6] service: remove rte prefix from static functions
Quoted 7 lines
From: Phil Yang <[email protected]>
clean up rte prefix from static functions. remove unused parameter for
service_dump_onefunction.Signed-off-by Phil Yang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>
PATCH v3 4/6 service: remove redundant code
In reply to Honnappa NagarahalliFrom: Phil Yang <[email protected]>
The service id validation is duplicated, remove the redundant code in the calling functions.
--- lib/librte_eal/common/rte_service.c | 28 ++++++---------------------- 1 file changed, 6 insertions(+), 22 deletions(-)
lib/librte_eal/common/rte_service.c +6 -22
@@ -541,24 +541,12 @@ rte_service_start_with_defaults(void) }static int32_t -service_update (struct rte_service_spec *service,uint32_t lcore, +service_update (uint32_t sid,uint32_t lcore,uint32_t *set,uint32_t *enabled) { -uint32_t i; -int32_t sid =-1 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if ((struct rte_service_spec *)&rte_services[i] == service && -service_valid (i)) { - sid = i; -break ; - } - } - -if (sid ==-1 || lcore >=RTE_MAX_LCORE ) -return -EINVAL ; - -if (!lcore_states[lcore].is_service_core) +/* validate ID, or return error value */ +if (sid >=RTE_SERVICE_NUM_MAX || !service_valid (sid) || + lcore >=RTE_MAX_LCORE || !lcore_states[lcore].is_service_core)return -EINVAL ;uint64_t sid_mask =UINT64_C (1 ) << sid;
@@ -587,19 +575,15 @@ service_update(struct rte_service_spec *service, uint32_t lcore,int32_t rte_service_map_lcore_set (uint32_t id,uint32_t lcore,uint32_t enabled) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );uint32_t on = enabled >0 ; -return service_update (&s->spec, lcore, &on,0 ); +return service_update (id, lcore, &on,0 ); }int32_t rte_service_map_lcore_get (uint32_t id,uint32_t lcore) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );uint32_t enabled; -int ret =service_update (&s->spec, lcore,0 , &enabled); +int ret =service_update (id, lcore,0 , &enabled);if (ret ==0 )return enabled;return ret;
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v3 4/6] service: remove redundant code
Quoted 12 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, May 2, 2020 1:03 AM
To: [email protected]; [email protected]; Van Haaren, Harry <[email protected]>
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v3 4/6] service: remove redundant code
Quoted 7 lines
From: Phil Yang <[email protected]>
The service id validation is duplicated, remove the redundant code in the calling functions.
Signed-off-by Phil Yang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>
<snip diff>
PATCH v3 5/6 service: optimize with c11 atomics
In reply to Honnappa NagarahalliFrom: Phil Yang <[email protected]>
The num_mapped_cores is used as a statistics. Use c11 atomics with RELAXED ordering for num_mapped_cores instead of rte_atomic ops which enforce unnessary barriers on aarch64.
Replace execute_lock operations to spinlock_try_lock to avoid duplicate code.
--- lib/librte_eal/common/rte_service.c | 32 ++++++++++++++++------------- lib/librte_eal/meson.build | 4 ++++ 2 files changed, 22 insertions(+), 14 deletions(-)
lib/librte_eal/common/rte_service.c +18 -14
@@ -20,6 +20,7 @@#include <rte_atomic.h> #include <rte_memory.h> #include <rte_malloc.h> +#include <rte_spinlock.h> #include "eal_private.h"
@@ -38,11 +39,11 @@ struct rte_service_spec_impl {/* public part of the struct */ struct rte_service_spec spec; -/* atomic lock that when set indicates a service core is currently +/* spin lock that when set indicates a service core is currently * running this service callback. When not set, a core may take the * lock and then run the service callback. */ -rte_atomic32_t execute_lock; +rte_spinlock_t execute_lock;/* API set/get-able variables */ int8_t app_runstate;
@@ -54,7 +55,7 @@ struct rte_service_spec_impl {* It does not indicate the number of cores the service is running * on currently. */ -rte_atomic32_t num_mapped_cores; +uint32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent; } __rte_cache_aligned;
@@ -332,7 +333,8 @@ rte_service_runstate_get(uint32_t id)rte_smp_rmb ();int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); -int lcore_mapped = (rte_atomic32_read (&s->num_mapped_cores) >0 ); +int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, + __ATOMIC_RELAXED) >0 );return (s->app_runstate ==RUNSTATE_RUNNING ) && (s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -375,11 +377,11 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ;if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) { -if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 )) +if (!rte_spinlock_trylock (&s->execute_lock))return -EBUSY ;service_runner_do_callback (s, cs, i); -rte_atomic32_clear (&s->execute_lock); +rte_spinlock_unlock (&s->execute_lock); }else service_runner_do_callback (s, cs, i);
@@ -415,11 +417,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)/* Increment num_mapped_cores to indicate that the service * is running on a core. */ -rte_atomic32_inc (&s->num_mapped_cores); +__atomic_add_fetch (&s->num_mapped_cores,1 , __ATOMIC_RELAXED);int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -rte_atomic32_dec (&s->num_mapped_cores); +__atomic_sub_fetch (&s->num_mapped_cores,1 , __ATOMIC_RELAXED);return ret; }
@@ -556,19 +558,19 @@ service_update(uint32_t sid, uint32_t lcore,if (*set && !lcore_mapped) { lcore_states[lcore].service_mask |= sid_mask; -rte_atomic32_inc (&rte_services[sid].num_mapped_cores); +__atomic_add_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELAXED); }if (!*set && lcore_mapped) { lcore_states[lcore].service_mask &= ~(sid_mask); -rte_atomic32_dec (&rte_services[sid].num_mapped_cores); +__atomic_sub_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELAXED); } }if (enabled) *enabled = !!(lcore_states[lcore].service_mask & (sid_mask)); -rte_smp_wmb (); -return 0 ; }
@@ -616,7 +618,8 @@ rte_service_lcore_reset_all(void) } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) -rte_atomic32_set (&rte_services[i].num_mapped_cores,0 ); +__atomic_store_n (&rte_services[i].num_mapped_cores,0 , + __ATOMIC_RELAXED);rte_smp_wmb ();
@@ -699,7 +702,8 @@ rte_service_lcore_stop(uint32_t lcore)int32_t enabled = service_mask & (UINT64_C (1 ) << i);int32_t service_running =rte_service_runstate_get (i);int32_t only_core = (1 == -rte_atomic32_read (&rte_services[i].num_mapped_cores)); +__atomic_load_n (&rte_services[i].num_mapped_cores, + __ATOMIC_RELAXED));/* if the core is mapped, and the service is running, and this * is the only core that is mapped, the service would cease to
lib/librte_eal/meson.build +4 -0
@@ -21,3 +21,7 @@ endif if cc.has_header('getopt.h') cflags += ['-DHAVE_GETOPT_H', '-DHAVE_GETOPT', '-DHAVE_GETOPT_LONG'] endif +# for clang 32-bit compiles we need libatomic for 64-bit atomic ops +if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false + ext_deps += cc.find_library('atomic') +endif
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v3 5/6] service: optimize with c11 atomics
Quoted 18 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, May 2, 2020 1:03 AM
To: [email protected]; [email protected]; Van Haaren, Harry <[email protected]>
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v3 5/6] service: optimize with c11 atomicsFrom: Phil Yang <[email protected]>
The
num_mapped_coresis used as a statistics. Use c11 atomics with RELAXED ordering fornum_mapped_coresinstead ofrte_atomicops which enforce unnessary barriers on aarch64.
Quoted 5 lines
Replace
execute_lockoperations tospinlock_try_lockto avoid duplicate code.Signed-off-by Phil Yang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>
<snip diff>
PATCH v3 6/6 service: relax barriers with C11 atomics
In reply to Honnappa NagarahalliFrom: Phil Yang <[email protected]>
The runstate, comp_runstate and app_runstate are used as guard variables in the service core lib. To guarantee the inter-threads visibility of these guard variables, it uses rte_smp_r/wmb. This patch use c11 atomic built-ins to relax these barriers.
--- lib/librte_eal/common/rte_service.c | 115 ++++++++++++++++++++-------- 1 file changed, 84 insertions(+), 31 deletions(-)
lib/librte_eal/common/rte_service.c +84 -31
@@ -265,7 +265,6 @@ rte_service_component_register(const struct rte_service_spec *spec, s->spec = *spec; s->internal_flags |=SERVICE_F_REGISTERED |SERVICE_F_START_CHECK ; -rte_smp_wmb (); rte_service_count++;if (id_ptr)
@@ -282,7 +281,6 @@ rte_service_component_unregister(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); rte_service_count--; -rte_smp_wmb (); s->internal_flags &= ~(SERVICE_F_REGISTERED );
@@ -301,12 +299,17 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* comp_runstate act as the guard variable. Use store-release +* memory order. This synchronizes with load-acquire in +* service_run and service_runstate_get function. +*/ if (runstate) - s->comp_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->comp_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -316,12 +319,17 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* app_runstate act as the guard variable. Use store-release +* memory order. This synchronizes with load-acquire in +* service_run runstate_get function. +*/ if (runstate) - s->app_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->app_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->app_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->app_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -330,15 +338,24 @@ rte_service_runstate_get(uint32_t id) {struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); -rte_smp_rmb (); -int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); -int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, +/* comp_runstate and app_runstate act as the guard variables. +* Use load-acquire memory order. This synchronizes with +* store-release in service state set functions. +*/ +if (__atomic_load_n (&s->comp_runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING && +__atomic_load_n (&s->app_runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) { +int check_disabled = !(s->internal_flags & +SERVICE_F_START_CHECK ); +int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, __ATOMIC_RELAXED) >0 ); -return (s->app_runstate ==RUNSTATE_RUNNING ) && - (s->comp_runstate ==RUNSTATE_RUNNING ) && - (check_disabled | lcore_mapped); +return (check_disabled | lcore_mapped); + }else +return 0 ; + } static inline void
@@ -367,9 +384,15 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,if (!s)return -EINVAL ; -if (s->comp_runstate !=RUNSTATE_RUNNING || - s->app_runstate !=RUNSTATE_RUNNING || - !(service_mask & (UINT64_C (1 ) << i))) { +/* comp_runstate and app_runstate act as the guard variables. +* Use load-acquire memory order. This synchronizes with +* store-release in service state set functions. +*/ +if (__atomic_load_n (&s->comp_runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_RUNNING || +__atomic_load_n (&s->app_runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_RUNNING || + !(service_mask & (UINT64_C (1 ) << i))) { cs->service_active_on_lcore[i] =0 ;return -ENOEXEC ; }
@@ -434,7 +457,12 @@ service_runner_func(void *arg)const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (cs->runstate ==RUNSTATE_RUNNING ) { +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +while (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) {const uint64_t service_mask = cs->service_mask;for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) {
@@ -445,8 +473,6 @@ service_runner_func(void *arg) } cs->loops++; - -rte_smp_rmb (); } lcore_config[lcore].state =WAIT ;
@@ -614,15 +640,18 @@ rte_service_lcore_reset_all(void)if (lcore_states[i].is_service_core) { lcore_states[i].service_mask =0 ;set_lcore_state (i,ROLE_RTE ); - lcore_states[i].runstate =RUNSTATE_STOPPED ; +/* runstate act as guard variable Use +* store-release memory order here to synchronize +* with load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[i].runstate, +RUNSTATE_STOPPED , __ATOMIC_RELEASE); } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++)__atomic_store_n (&rte_services[i].num_mapped_cores,0 , __ATOMIC_RELAXED); -rte_smp_wmb (); -return 0 ; }
@@ -638,9 +667,11 @@ rte_service_lcore_add(uint32_t lcore)/* ensure that after adding a core the mask and state are defaults */ lcore_states[lcore].service_mask =0 ; - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; - -rte_smp_wmb (); +/* Use store-release memory order here to synchronize with +* load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);return rte_eal_wait_lcore (lcore); }
@@ -655,7 +686,12 @@ rte_service_lcore_del(uint32_t lcore)if (!cs->is_service_core)return -EINVAL ; -if (cs->runstate !=RUNSTATE_STOPPED ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_STOPPED )return -EBUSY ;set_lcore_state (lcore,ROLE_RTE );
@@ -674,13 +710,21 @@ rte_service_lcore_start(uint32_t lcore)if (!cs->is_service_core)return -EINVAL ; -if (cs->runstate ==RUNSTATE_RUNNING ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING )return -EALREADY ;/* set core to run state first, and then launch otherwise it will * return immediately as runstate keeps it in the service poll loop */ - cs->runstate =RUNSTATE_RUNNING ; +/* Use load-acquire memory order here to synchronize with +* store-release in runstate update functions. +*/ +__atomic_store_n (&cs->runstate,RUNSTATE_RUNNING , __ATOMIC_RELEASE);int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */
@@ -693,7 +737,12 @@ rte_service_lcore_stop(uint32_t lcore)if (lcore >=RTE_MAX_LCORE )return -EINVAL ; -if (lcore_states[lcore].runstate ==RUNSTATE_STOPPED ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&lcore_states[lcore].runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_STOPPED )return -EALREADY ;uint32_t i;
@@ -713,7 +762,11 @@ rte_service_lcore_stop(uint32_t lcore)return -EBUSY ; } - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; +/* Use store-release memory order here to synchronize with +* load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);return 0 ; }
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v3 6/6] service: relax barriers with C11 atomics
Quoted 14 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, May 2, 2020 1:03 AM
To: [email protected]; [email protected]; Van Haaren, Harry <[email protected]>
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v3 6/6] service: relax barriers with C11 atomicsFrom: Phil Yang <[email protected]>
Quoted 7 lines
The runstate,
comp_runstateandapp_runstateare used as guard variables in the service core lib. To guarantee the inter-threads visibility of these guard variables, it uses rte_smp_r/wmb. This patch use c11 atomic built-ins to relax these barriers.Signed-off-by Phil Yang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>
<snip diff>
v4 cover use c11 atomics for service core lib
In reply to Phil YangThe rte_atomic ops and rte_smp barriers enforce DMB barriers on aarch64. Using c11 atomics with explicit memory ordering instead of the rte_atomic ops and rte_smp barriers for inter-threads synchronization can uplift the performance on aarch64 and no performance loss on x86.
This patchset contains:
1) fix race condition for MT unsafe service.
2) clean up redundant code.
3) use c11 atomics for service core lib to avoid unnecessary barriers.
v2:
Still waiting on Harry for the final solution on the MT unsafe race condition issue. But I have incorporated the comments so far.
1. add 'Fixes' tag for bug-fix patches.
2. remove 'Fixes' tag for code cleanup patches.
3. remove unused parameter for service_dump_one function.
4. replace the execute_lock atomic CAS operation to spinlock_try_lock.
5. use c11 atomics with RELAXED memory ordering for num_mapped_cores.
6. relax barriers for guard variables runstate, comp_runstate and app_runstate with c11 one-way barriers.
v3:
Sending this version since Phil is on holiday.
1. Updated the API documentation to indicate how the locking can be avoided.
v4:
1. Fix the nits in 2/6 commit message and comments in code.
Honnappa Nagarahalli (2):
service: fix race condition for MT unsafe service
service: fix identification of service running on other lcore
Phil Yang (4):
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 atomics
service: relax barriers with C11 atomics
lib/librte_eal/common/rte_service.c | 234 ++++++++++-------- lib/librte_eal/include/rte_service.h | 8 +- .../include/rte_service_component.h | 6 +- lib/librte_eal/meson.build | 4 + 4 files changed, 141 insertions(+), 111 deletions(-)
Signature · 1 line
2.17.1
PATCH v4 1/6 service: fix race condition for MT unsafe service
The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.
If the service is known to be mmapped on a single lcore, setting the service capability to MT safe will avoid taking the lock and improve the performance.
--- lib/librte_eal/common/rte_service.c | 11 +++++------ lib/librte_eal/include/rte_service.h | 8 ++++++-- lib/librte_eal/include/rte_service_component.h | 6 +++++- 3 files changed, 16 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {uint8_t internal_flags;/* per service statistics */ +/* Indicates how many cores the service is mapped to run on. +* It does not indicate the number of cores the service is running +* on currently. +*/ rte_atomic32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -/* check do we need cmpset, if MT safe or <= 1 core -* mapped, atomic ops are not required. -*/ -const int use_atomics = (service_mt_safe (s) ==0 ) && - (rte_atomic32_read (&s->num_mapped_cores) >1 ); -if (use_atomics) { +if (service_mt_safe (s) ==0 ) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
lib/librte_eal/include/rte_service.h +6 -2
@@ -104,12 +104,16 @@ int32_t rte_service_probe_capability(uint32_t id, uint32_t capability);* Each core can be added or removed from running a specific service. This * function enables or disables *lcore* to run *service_id*. * -* If multiple cores are enabled on a service, an atomic is used to ensure that -* only one cores runs the service at a time. The exception to this is when +* If multiple cores are enabled on a service, a lock is used to ensure that +* only one core runs the service at a time. The exception to this is when * a service indicates that it is multi-thread safe by setting the capability * called RTE_SERVICE_CAP_MT_SAFE. With the multi-thread safe capability set, * the service function can be run on multiple threads at the same time. * +* If the service is known to be mapped to a single lcore, setting the +* capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve +* better performance by avoiding the use of lock. +* * @param service_id the service to apply the lcore to * @param lcore The lcore that will be mapped to service * @param enable Zero to unmap or disable the core, non-zero to enable
lib/librte_eal/include/rte_service_component.h +5 -1
@@ -43,7 +43,7 @@ struct rte_service_spec {/** * Register a new service. * -* A service represents a component that the requires CPU time periodically to +* A service represents a component that requires CPU time periodically to * achieve its purpose. * * For example the eventdev SW PMD requires CPU cycles to perform its
@@ -56,6 +56,10 @@ struct rte_service_spec {* *rte_service_component_runstate_set*, which indicates that the service * component is ready to be executed. * +* If the service is known to be mapped to a single lcore, setting the +* capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve +* better performance. +* * @param spec The specification of the service to register * @param[out] service_id A pointer to a uint32_t, which will be filled in * during registration of the service. It is set to the integers
Signature · 1 line
2.17.1
PATCH v4 2/6 service: fix identification of service running on other lcore
In reply to Honnappa NagarahalliThe logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduce the number of instructions for all cases.
--- lib/librte_eal/common/rte_service.c | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
@@ -360,7 +360,7 @@ rte_service_runner_do_callback(struct rte_service_spec_impl *s,/* Expects the service 's' is valid. */ static int32_t service_run (uint32_t i,struct core_state *cs,uint64_t service_mask, -struct rte_service_spec_impl *s) +struct rte_service_spec_impl *s,uint32_t serialize_mt_unsafe) {if (!s)return -EINVAL ;
@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -if (service_mt_safe (s) ==0 ) { +if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); -/* Atomically add this core to the mapped cores first, then examine if -* we can run the service. This avoids a race condition between -* checking the value, and atomically adding to the mapped count. +/* Increment num_mapped_cores to reflect that this core is +* now mapped capable of running the service. */ -if (serialize_mt_unsafe) -rte_atomic32_inc (&s->num_mapped_cores); +rte_atomic32_inc (&s->num_mapped_cores); -if (service_mt_safe (s) ==0 && -rte_atomic32_read (&s->num_mapped_cores) >1 ) { -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); -return -EBUSY ; - } - -int ret =service_run (id, cs,UINT64_MAX , s); +int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); +rte_atomic32_dec (&s->num_mapped_cores);return ret; }
@@ -449,7 +439,7 @@ rte_service_runner_func(void *arg)if (!service_valid (i))continue ;/* return value ignored as no change to code flow */ -service_run (i, cs, service_mask,service_get (i)); +service_run (i, cs, service_mask,service_get (i),1 ); } cs->loops++;
Signature · 1 line
2.17.1
PATCH v4 3/6 service: remove rte prefix from static functions
In reply to Honnappa NagarahalliFrom: Phil Yang <[email protected]>
clean up rte prefix from static functions. remove unused parameter for service_dump_one function.
--- lib/librte_eal/common/rte_service.c | 34 ++++++++++------------------- 1 file changed, 11 insertions(+), 23 deletions(-)
lib/librte_eal/common/rte_service.c +11 -23
@@ -340,7 +340,7 @@ rte_service_runstate_get(uint32_t id) }static inline void -rte_service_runner_do_callback (struct rte_service_spec_impl *s, +service_runner_do_callback (struct rte_service_spec_impl *s,struct core_state *cs,uint32_t service_idx) {void *userdata = s->spec.callback_userdata;
@@ -378,10 +378,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ; -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);rte_atomic32_clear (&s->execute_lock); }else -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);return 0 ; }
@@ -425,14 +425,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) }static int32_t -rte_service_runner_func (void *arg) +service_runner_func (void *arg) {RTE_SET_USED (arg);uint32_t i;const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (lcore_states[lcore].runstate ==RUNSTATE_RUNNING ) { +while (cs->runstate ==RUNSTATE_RUNNING ) {const uint64_t service_mask = cs->service_mask;for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) {
@@ -693,9 +693,9 @@ rte_service_lcore_start(uint32_t lcore)/* set core to run state first, and then launch otherwise it will * return immediately as runstate keeps it in the service poll loop */ - lcore_states[lcore].runstate =RUNSTATE_RUNNING ; + cs->runstate =RUNSTATE_RUNNING ; -int ret =rte_eal_remote_launch (rte_service_runner_func,0 , lcore); +int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */ return ret; }
@@ -774,13 +774,9 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id, }static void -rte_service_dump_one (FILE *f,struct rte_service_spec_impl *s, -uint64_t all_cycles,uint32_t reset) +service_dump_one (FILE *f,struct rte_service_spec_impl *s,uint32_t reset) {/* avoid divide by zero */ -if (all_cycles ==0 ) - all_cycles =1 ; -int calls =1 ;if (s->calls !=0 ) calls = s->calls;
@@ -807,7 +803,7 @@ rte_service_attr_reset_all(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );int reset =1 ; -rte_service_dump_one (NULL , s,0 , reset); +service_dump_one (NULL , s, reset);return 0 ; }
@@ -851,21 +847,13 @@ rte_service_dump(FILE *f, uint32_t id)uint32_t i;int print_one = (id !=UINT32_MAX ); -uint64_t total_cycles =0 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if (!service_valid (i)) -continue ; - total_cycles += rte_services[i].cycles_spent; - } -/* print only the specified service */ if (print_one) {struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );fprintf (f,"Service %s Summary\n" , s->spec.name);uint32_t reset =0 ; -rte_service_dump_one (f, s, total_cycles, reset); +service_dump_one (f, s, reset);return 0 ; }
@@ -875,7 +863,7 @@ rte_service_dump(FILE *f, uint32_t id)if (!service_valid (i))continue ;uint32_t reset =0 ; -rte_service_dump_one (f, &rte_services[i], total_cycles, reset); +service_dump_one (f, &rte_services[i], reset); }fprintf (f,"Service Cores Summary\n" );
Signature · 1 line
2.17.1
PATCH v4 4/6 service: remove redundant code
In reply to Honnappa NagarahalliFrom: Phil Yang <[email protected]>
The service id validation is duplicated, remove the redundant code in the calling functions.
--- lib/librte_eal/common/rte_service.c | 28 ++++++---------------------- 1 file changed, 6 insertions(+), 22 deletions(-)
lib/librte_eal/common/rte_service.c +6 -22
@@ -541,24 +541,12 @@ rte_service_start_with_defaults(void) }static int32_t -service_update (struct rte_service_spec *service,uint32_t lcore, +service_update (uint32_t sid,uint32_t lcore,uint32_t *set,uint32_t *enabled) { -uint32_t i; -int32_t sid =-1 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if ((struct rte_service_spec *)&rte_services[i] == service && -service_valid (i)) { - sid = i; -break ; - } - } - -if (sid ==-1 || lcore >=RTE_MAX_LCORE ) -return -EINVAL ; - -if (!lcore_states[lcore].is_service_core) +/* validate ID, or return error value */ +if (sid >=RTE_SERVICE_NUM_MAX || !service_valid (sid) || + lcore >=RTE_MAX_LCORE || !lcore_states[lcore].is_service_core)return -EINVAL ;uint64_t sid_mask =UINT64_C (1 ) << sid;
@@ -587,19 +575,15 @@ service_update(struct rte_service_spec *service, uint32_t lcore,int32_t rte_service_map_lcore_set (uint32_t id,uint32_t lcore,uint32_t enabled) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );uint32_t on = enabled >0 ; -return service_update (&s->spec, lcore, &on,0 ); +return service_update (id, lcore, &on,0 ); }int32_t rte_service_map_lcore_get (uint32_t id,uint32_t lcore) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );uint32_t enabled; -int ret =service_update (&s->spec, lcore,0 , &enabled); +int ret =service_update (id, lcore,0 , &enabled);if (ret ==0 )return enabled;return ret;
Signature · 1 line
2.17.1
PATCH v4 5/6 service: optimize with c11 atomics
In reply to Honnappa NagarahalliFrom: Phil Yang <[email protected]>
The num_mapped_cores is used as a statistics. Use c11 atomics with RELAXED ordering for num_mapped_cores instead of rte_atomic ops which enforce unnessary barriers on aarch64.
Replace execute_lock operations to spinlock_try_lock to avoid duplicate code.
--- lib/librte_eal/common/rte_service.c | 32 ++++++++++++++++------------- lib/librte_eal/meson.build | 4 ++++ 2 files changed, 22 insertions(+), 14 deletions(-)
lib/librte_eal/common/rte_service.c +18 -14
@@ -20,6 +20,7 @@#include <rte_atomic.h> #include <rte_memory.h> #include <rte_malloc.h> +#include <rte_spinlock.h> #include "eal_private.h"
@@ -38,11 +39,11 @@ struct rte_service_spec_impl {/* public part of the struct */ struct rte_service_spec spec; -/* atomic lock that when set indicates a service core is currently +/* spin lock that when set indicates a service core is currently * running this service callback. When not set, a core may take the * lock and then run the service callback. */ -rte_atomic32_t execute_lock; +rte_spinlock_t execute_lock;/* API set/get-able variables */ int8_t app_runstate;
@@ -54,7 +55,7 @@ struct rte_service_spec_impl {* It does not indicate the number of cores the service is running * on currently. */ -rte_atomic32_t num_mapped_cores; +uint32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent; } __rte_cache_aligned;
@@ -332,7 +333,8 @@ rte_service_runstate_get(uint32_t id)rte_smp_rmb ();int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); -int lcore_mapped = (rte_atomic32_read (&s->num_mapped_cores) >0 ); +int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, + __ATOMIC_RELAXED) >0 );return (s->app_runstate ==RUNSTATE_RUNNING ) && (s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -375,11 +377,11 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ;if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) { -if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 )) +if (!rte_spinlock_trylock (&s->execute_lock))return -EBUSY ;service_runner_do_callback (s, cs, i); -rte_atomic32_clear (&s->execute_lock); +rte_spinlock_unlock (&s->execute_lock); }else service_runner_do_callback (s, cs, i);
@@ -415,11 +417,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)/* Increment num_mapped_cores to reflect that this core is * now mapped capable of running the service. */ -rte_atomic32_inc (&s->num_mapped_cores); +__atomic_add_fetch (&s->num_mapped_cores,1 , __ATOMIC_RELAXED);int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -rte_atomic32_dec (&s->num_mapped_cores); +__atomic_sub_fetch (&s->num_mapped_cores,1 , __ATOMIC_RELAXED);return ret; }
@@ -556,19 +558,19 @@ service_update(uint32_t sid, uint32_t lcore,if (*set && !lcore_mapped) { lcore_states[lcore].service_mask |= sid_mask; -rte_atomic32_inc (&rte_services[sid].num_mapped_cores); +__atomic_add_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELAXED); }if (!*set && lcore_mapped) { lcore_states[lcore].service_mask &= ~(sid_mask); -rte_atomic32_dec (&rte_services[sid].num_mapped_cores); +__atomic_sub_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELAXED); } }if (enabled) *enabled = !!(lcore_states[lcore].service_mask & (sid_mask)); -rte_smp_wmb (); -return 0 ; }
@@ -616,7 +618,8 @@ rte_service_lcore_reset_all(void) } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) -rte_atomic32_set (&rte_services[i].num_mapped_cores,0 ); +__atomic_store_n (&rte_services[i].num_mapped_cores,0 , + __ATOMIC_RELAXED);rte_smp_wmb ();
@@ -699,7 +702,8 @@ rte_service_lcore_stop(uint32_t lcore)int32_t enabled = service_mask & (UINT64_C (1 ) << i);int32_t service_running =rte_service_runstate_get (i);int32_t only_core = (1 == -rte_atomic32_read (&rte_services[i].num_mapped_cores)); +__atomic_load_n (&rte_services[i].num_mapped_cores, + __ATOMIC_RELAXED));/* if the core is mapped, and the service is running, and this * is the only core that is mapped, the service would cease to
lib/librte_eal/meson.build +4 -0
@@ -21,3 +21,7 @@ endif if cc.has_header('getopt.h') cflags += ['-DHAVE_GETOPT_H', '-DHAVE_GETOPT', '-DHAVE_GETOPT_LONG'] endif +# for clang 32-bit compiles we need libatomic for 64-bit atomic ops +if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false + ext_deps += cc.find_library('atomic') +endif
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v4 5/6] service: optimize with c11 atomics
Quoted 29 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Wednesday, May 6, 2020 5:18 AM
To: [email protected]; Phil Yang <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: [PATCH v4 5/6] service: optimize with c11 atomicsFrom: Phil Yang <[email protected]>
The
num_mapped_coresis used as a statistics. Use c11 atomics with RELAXED ordering fornum_mapped_coresinstead ofrte_atomicops which enforce unnessary barriers on aarch64.Replace
execute_lockoperations tospinlock_try_lockto avoid duplicate code.Signed-off-by Phil Yang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>Acked-by Harry van Haaren <[email protected]>--- lib/librte_eal/common/rte_service.c | 32 ++++++++++++++++------------- lib/librte_eal/meson.build | 4 ++++ 2 files changed, 22 insertions(+), 14 deletions(-)
<snip>
lib/librte_eal/meson.build endif · line 27
Quoted 13 lines
lib/librte_eal/meson.build +0
-0@@ -21,3 +21,7 @@ endif if cc.has_header('getopt.h') cflags += ['-DHAVE_GETOPT_H', '-DHAVE_GETOPT', '-
DHAVE_GETOPT_LONG']
endiflib/librte_eal/meson.build +4
-0+# for clang 32-bit compiles we need libatomic for 64-bit atomic ops +if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false + ext_deps += cc.find_library('atomic') +endif
We can remove this as it has been added in global. "da4eae278b56 - build: add global libatomic dependency for 32-bit clang"
I've updated it in v5.
Thanks,
Phil
Quoted 2 lines
--
2.17.1
PATCH v4 6/6 service: relax barriers with C11 atomics
In reply to Honnappa NagarahalliFrom: Phil Yang <[email protected]>
The runstate, comp_runstate and app_runstate are used as guard variables in the service core lib. To guarantee the inter-threads visibility of these guard variables, it uses rte_smp_r/wmb. This patch use c11 atomic built-ins to relax these barriers.
--- lib/librte_eal/common/rte_service.c | 115 ++++++++++++++++++++-------- 1 file changed, 84 insertions(+), 31 deletions(-)
lib/librte_eal/common/rte_service.c +84 -31
@@ -265,7 +265,6 @@ rte_service_component_register(const struct rte_service_spec *spec, s->spec = *spec; s->internal_flags |=SERVICE_F_REGISTERED |SERVICE_F_START_CHECK ; -rte_smp_wmb (); rte_service_count++;if (id_ptr)
@@ -282,7 +281,6 @@ rte_service_component_unregister(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); rte_service_count--; -rte_smp_wmb (); s->internal_flags &= ~(SERVICE_F_REGISTERED );
@@ -301,12 +299,17 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* comp_runstate act as the guard variable. Use store-release +* memory order. This synchronizes with load-acquire in +* service_run and service_runstate_get function. +*/ if (runstate) - s->comp_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->comp_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -316,12 +319,17 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* app_runstate act as the guard variable. Use store-release +* memory order. This synchronizes with load-acquire in +* service_run runstate_get function. +*/ if (runstate) - s->app_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->app_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->app_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->app_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -330,15 +338,24 @@ rte_service_runstate_get(uint32_t id) {struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); -rte_smp_rmb (); -int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); -int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, +/* comp_runstate and app_runstate act as the guard variables. +* Use load-acquire memory order. This synchronizes with +* store-release in service state set functions. +*/ +if (__atomic_load_n (&s->comp_runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING && +__atomic_load_n (&s->app_runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) { +int check_disabled = !(s->internal_flags & +SERVICE_F_START_CHECK ); +int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, __ATOMIC_RELAXED) >0 ); -return (s->app_runstate ==RUNSTATE_RUNNING ) && - (s->comp_runstate ==RUNSTATE_RUNNING ) && - (check_disabled | lcore_mapped); +return (check_disabled | lcore_mapped); + }else +return 0 ; + } static inline void
@@ -367,9 +384,15 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,if (!s)return -EINVAL ; -if (s->comp_runstate !=RUNSTATE_RUNNING || - s->app_runstate !=RUNSTATE_RUNNING || - !(service_mask & (UINT64_C (1 ) << i))) { +/* comp_runstate and app_runstate act as the guard variables. +* Use load-acquire memory order. This synchronizes with +* store-release in service state set functions. +*/ +if (__atomic_load_n (&s->comp_runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_RUNNING || +__atomic_load_n (&s->app_runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_RUNNING || + !(service_mask & (UINT64_C (1 ) << i))) { cs->service_active_on_lcore[i] =0 ;return -ENOEXEC ; }
@@ -434,7 +457,12 @@ service_runner_func(void *arg)const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (cs->runstate ==RUNSTATE_RUNNING ) { +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +while (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) {const uint64_t service_mask = cs->service_mask;for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) {
@@ -445,8 +473,6 @@ service_runner_func(void *arg) } cs->loops++; - -rte_smp_rmb (); } lcore_config[lcore].state =WAIT ;
@@ -614,15 +640,18 @@ rte_service_lcore_reset_all(void)if (lcore_states[i].is_service_core) { lcore_states[i].service_mask =0 ;set_lcore_state (i,ROLE_RTE ); - lcore_states[i].runstate =RUNSTATE_STOPPED ; +/* runstate act as guard variable Use +* store-release memory order here to synchronize +* with load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[i].runstate, +RUNSTATE_STOPPED , __ATOMIC_RELEASE); } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++)__atomic_store_n (&rte_services[i].num_mapped_cores,0 , __ATOMIC_RELAXED); -rte_smp_wmb (); -return 0 ; }
@@ -638,9 +667,11 @@ rte_service_lcore_add(uint32_t lcore)/* ensure that after adding a core the mask and state are defaults */ lcore_states[lcore].service_mask =0 ; - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; - -rte_smp_wmb (); +/* Use store-release memory order here to synchronize with +* load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);return rte_eal_wait_lcore (lcore); }
@@ -655,7 +686,12 @@ rte_service_lcore_del(uint32_t lcore)if (!cs->is_service_core)return -EINVAL ; -if (cs->runstate !=RUNSTATE_STOPPED ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_STOPPED )return -EBUSY ;set_lcore_state (lcore,ROLE_RTE );
@@ -674,13 +710,21 @@ rte_service_lcore_start(uint32_t lcore)if (!cs->is_service_core)return -EINVAL ; -if (cs->runstate ==RUNSTATE_RUNNING ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING )return -EALREADY ;/* set core to run state first, and then launch otherwise it will * return immediately as runstate keeps it in the service poll loop */ - cs->runstate =RUNSTATE_RUNNING ; +/* Use load-acquire memory order here to synchronize with +* store-release in runstate update functions. +*/ +__atomic_store_n (&cs->runstate,RUNSTATE_RUNNING , __ATOMIC_RELEASE);int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */
@@ -693,7 +737,12 @@ rte_service_lcore_stop(uint32_t lcore)if (lcore >=RTE_MAX_LCORE )return -EINVAL ; -if (lcore_states[lcore].runstate ==RUNSTATE_STOPPED ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&lcore_states[lcore].runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_STOPPED )return -EALREADY ;uint32_t i;
@@ -713,7 +762,11 @@ rte_service_lcore_stop(uint32_t lcore)return -EBUSY ; } - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; +/* Use store-release memory order here to synchronize with +* load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);return 0 ; }
Signature · 1 line
2.17.1
v5 cover use c11 atomics for service core lib
In reply to Honnappa NagarahalliThe rte_atomic ops and rte_smp barriers enforce DMB barriers on aarch64. Using c11 atomics with explicit memory ordering instead of the rte_atomic ops and rte_smp barriers for inter-threads synchronization can uplift the performance on aarch64 and no performance loss on x86.
This patchset contains:
1) fix race condition for MT unsafe service.
2) clean up redundant code.
3) use c11 atomics for service core lib to avoid unnecessary barriers.
v2:
Still waiting on Harry for the final solution on the MT unsafe race condition issue. But I have incorporated the comments so far.
1. add 'Fixes' tag for bug-fix patches.
2. remove 'Fixes' tag for code cleanup patches.
3. remove unused parameter for service_dump_one function.
4. replace the execute_lock atomic CAS operation to spinlock_try_lock.
5. use c11 atomics with RELAXED memory ordering for num_mapped_cores.
6. relax barriers for guard variables runstate, comp_runstate and app_runstate with c11 one-way barriers.
v3:
Sending this version since Phil is on holiday.
1. Updated the API documentation to indicate how the locking can be avoided.
v4:
1. Fix the nits in 2/6 commit message and comments in code.
v5:
1. Remove redundant libatomic clang dependency claim code as it has been added in global. (The commit da4eae278b56)
Honnappa Nagarahalli (2):
service: fix race condition for MT unsafe service
service: fix identification of service running on other lcore
Phil Yang (4):
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 atomics
service: relax barriers with C11 atomics
lib/librte_eal/common/rte_service.c | 234 +++++++++++++------------ lib/librte_eal/include/rte_service.h | 8 +- lib/librte_eal/include/rte_service_component.h | 6 +- 3 files changed, 137 insertions(+), 111 deletions(-)
Signature · 1 line
2.7.4
PATCH v5 1/6 service: fix race condition for MT unsafe service
From: Honnappa Nagarahalli <[email protected]>
The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.
If the service is known to be mmapped on a single lcore, setting the service capability to MT safe will avoid taking the lock and improve the performance.
--- lib/librte_eal/common/rte_service.c | 11 +++++------ lib/librte_eal/include/rte_service.h | 8 ++++++-- lib/librte_eal/include/rte_service_component.h | 6 +++++- 3 files changed, 16 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {uint8_t internal_flags;/* per service statistics */ +/* Indicates how many cores the service is mapped to run on. +* It does not indicate the number of cores the service is running +* on currently. +*/ rte_atomic32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -/* check do we need cmpset, if MT safe or <= 1 core -* mapped, atomic ops are not required. -*/ -const int use_atomics = (service_mt_safe (s) ==0 ) && - (rte_atomic32_read (&s->num_mapped_cores) >1 ); -if (use_atomics) { +if (service_mt_safe (s) ==0 ) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
lib/librte_eal/include/rte_service.h +6 -2
@@ -104,12 +104,16 @@ int32_t rte_service_probe_capability(uint32_t id, uint32_t capability);* Each core can be added or removed from running a specific service. This * function enables or disables *lcore* to run *service_id*. * -* If multiple cores are enabled on a service, an atomic is used to ensure that -* only one cores runs the service at a time. The exception to this is when +* If multiple cores are enabled on a service, a lock is used to ensure that +* only one core runs the service at a time. The exception to this is when * a service indicates that it is multi-thread safe by setting the capability * called RTE_SERVICE_CAP_MT_SAFE. With the multi-thread safe capability set, * the service function can be run on multiple threads at the same time. * +* If the service is known to be mapped to a single lcore, setting the +* capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve +* better performance by avoiding the use of lock. +* * @param service_id the service to apply the lcore to * @param lcore The lcore that will be mapped to service * @param enable Zero to unmap or disable the core, non-zero to enable
lib/librte_eal/include/rte_service_component.h +5 -1
@@ -43,7 +43,7 @@ struct rte_service_spec {/** * Register a new service. * -* A service represents a component that the requires CPU time periodically to +* A service represents a component that requires CPU time periodically to * achieve its purpose. * * For example the eventdev SW PMD requires CPU cycles to perform its
@@ -56,6 +56,10 @@ struct rte_service_spec {* *rte_service_component_runstate_set*, which indicates that the service * component is ready to be executed. * +* If the service is known to be mapped to a single lcore, setting the +* capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve +* better performance. +* * @param spec The specification of the service to register * @param[out] service_id A pointer to a uint32_t, which will be filled in * during registration of the service. It is set to the integers
Signature · 1 line
2.7.4
PATCH v5 2/6 service: fix identification of service running on other lcore
In reply to Phil YangFrom: Honnappa Nagarahalli <[email protected]>
The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduce the number of instructions for all cases.
--- lib/librte_eal/common/rte_service.c | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
@@ -360,7 +360,7 @@ rte_service_runner_do_callback(struct rte_service_spec_impl *s,/* Expects the service 's' is valid. */ static int32_t service_run (uint32_t i,struct core_state *cs,uint64_t service_mask, -struct rte_service_spec_impl *s) +struct rte_service_spec_impl *s,uint32_t serialize_mt_unsafe) {if (!s)return -EINVAL ;
@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -if (service_mt_safe (s) ==0 ) { +if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); -/* Atomically add this core to the mapped cores first, then examine if -* we can run the service. This avoids a race condition between -* checking the value, and atomically adding to the mapped count. +/* Increment num_mapped_cores to reflect that this core is +* now mapped capable of running the service. */ -if (serialize_mt_unsafe) -rte_atomic32_inc (&s->num_mapped_cores); +rte_atomic32_inc (&s->num_mapped_cores); -if (service_mt_safe (s) ==0 && -rte_atomic32_read (&s->num_mapped_cores) >1 ) { -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); -return -EBUSY ; - } - -int ret =service_run (id, cs,UINT64_MAX , s); +int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); +rte_atomic32_dec (&s->num_mapped_cores);return ret; }
@@ -449,7 +439,7 @@ rte_service_runner_func(void *arg)if (!service_valid (i))continue ;/* return value ignored as no change to code flow */ -service_run (i, cs, service_mask,service_get (i)); +service_run (i, cs, service_mask,service_get (i),1 ); } cs->loops++;
Signature · 1 line
2.7.4
PATCH v5 3/6 service: remove rte prefix from static functions
In reply to Phil Yangclean up rte prefix from static functions. remove unused parameter for service_dump_one function.
--- lib/librte_eal/common/rte_service.c | 34 +++++++++++----------------------- 1 file changed, 11 insertions(+), 23 deletions(-)
lib/librte_eal/common/rte_service.c +11 -23
@@ -340,7 +340,7 @@ rte_service_runstate_get(uint32_t id) }static inline void -rte_service_runner_do_callback (struct rte_service_spec_impl *s, +service_runner_do_callback (struct rte_service_spec_impl *s,struct core_state *cs,uint32_t service_idx) {void *userdata = s->spec.callback_userdata;
@@ -378,10 +378,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ; -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);rte_atomic32_clear (&s->execute_lock); }else -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);return 0 ; }
@@ -425,14 +425,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) }static int32_t -rte_service_runner_func (void *arg) +service_runner_func (void *arg) {RTE_SET_USED (arg);uint32_t i;const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (lcore_states[lcore].runstate ==RUNSTATE_RUNNING ) { +while (cs->runstate ==RUNSTATE_RUNNING ) {const uint64_t service_mask = cs->service_mask;for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) {
@@ -693,9 +693,9 @@ rte_service_lcore_start(uint32_t lcore)/* set core to run state first, and then launch otherwise it will * return immediately as runstate keeps it in the service poll loop */ - lcore_states[lcore].runstate =RUNSTATE_RUNNING ; + cs->runstate =RUNSTATE_RUNNING ; -int ret =rte_eal_remote_launch (rte_service_runner_func,0 , lcore); +int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */ return ret; }
@@ -774,13 +774,9 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id, }static void -rte_service_dump_one (FILE *f,struct rte_service_spec_impl *s, -uint64_t all_cycles,uint32_t reset) +service_dump_one (FILE *f,struct rte_service_spec_impl *s,uint32_t reset) {/* avoid divide by zero */ -if (all_cycles ==0 ) - all_cycles =1 ; -int calls =1 ;if (s->calls !=0 ) calls = s->calls;
@@ -807,7 +803,7 @@ rte_service_attr_reset_all(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );int reset =1 ; -rte_service_dump_one (NULL , s,0 , reset); +service_dump_one (NULL , s, reset);return 0 ; }
@@ -851,21 +847,13 @@ rte_service_dump(FILE *f, uint32_t id)uint32_t i;int print_one = (id !=UINT32_MAX ); -uint64_t total_cycles =0 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if (!service_valid (i)) -continue ; - total_cycles += rte_services[i].cycles_spent; - } -/* print only the specified service */ if (print_one) {struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );fprintf (f,"Service %s Summary\n" , s->spec.name);uint32_t reset =0 ; -rte_service_dump_one (f, s, total_cycles, reset); +service_dump_one (f, s, reset);return 0 ; }
@@ -875,7 +863,7 @@ rte_service_dump(FILE *f, uint32_t id)if (!service_valid (i))continue ;uint32_t reset =0 ; -rte_service_dump_one (f, &rte_services[i], total_cycles, reset); +service_dump_one (f, &rte_services[i], reset); }fprintf (f,"Service Cores Summary\n" );
Signature · 1 line
2.7.4
PATCH v5 4/6 service: remove redundant code
In reply to Phil YangThe service id validation is duplicated, remove the redundant code in the calling functions.
--- lib/librte_eal/common/rte_service.c | 28 ++++++---------------------- 1 file changed, 6 insertions(+), 22 deletions(-)
lib/librte_eal/common/rte_service.c +6 -22
@@ -541,24 +541,12 @@ rte_service_start_with_defaults(void) }static int32_t -service_update (struct rte_service_spec *service,uint32_t lcore, +service_update (uint32_t sid,uint32_t lcore,uint32_t *set,uint32_t *enabled) { -uint32_t i; -int32_t sid =-1 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if ((struct rte_service_spec *)&rte_services[i] == service && -service_valid (i)) { - sid = i; -break ; - } - } - -if (sid ==-1 || lcore >=RTE_MAX_LCORE ) -return -EINVAL ; - -if (!lcore_states[lcore].is_service_core) +/* validate ID, or return error value */ +if (sid >=RTE_SERVICE_NUM_MAX || !service_valid (sid) || + lcore >=RTE_MAX_LCORE || !lcore_states[lcore].is_service_core)return -EINVAL ;uint64_t sid_mask =UINT64_C (1 ) << sid;
@@ -587,19 +575,15 @@ service_update(struct rte_service_spec *service, uint32_t lcore,int32_t rte_service_map_lcore_set (uint32_t id,uint32_t lcore,uint32_t enabled) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );uint32_t on = enabled >0 ; -return service_update (&s->spec, lcore, &on,0 ); +return service_update (id, lcore, &on,0 ); }int32_t rte_service_map_lcore_get (uint32_t id,uint32_t lcore) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );uint32_t enabled; -int ret =service_update (&s->spec, lcore,0 , &enabled); +int ret =service_update (id, lcore,0 , &enabled);if (ret ==0 )return enabled;return ret;
Signature · 1 line
2.7.4
PATCH v5 5/6 service: optimize with c11 atomics
In reply to Phil YangThe num_mapped_cores is used as a statistics. Use c11 atomics with RELAXED ordering for num_mapped_cores instead of rte_atomic ops which enforce unnessary barriers on aarch64.
Replace execute_lock operations to spinlock_try_lock to avoid duplicate code.
--- lib/librte_eal/common/rte_service.c | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-)
lib/librte_eal/common/rte_service.c +18 -14
@@ -20,6 +20,7 @@#include <rte_atomic.h> #include <rte_memory.h> #include <rte_malloc.h> +#include <rte_spinlock.h> #include "eal_private.h"
@@ -38,11 +39,11 @@ struct rte_service_spec_impl {/* public part of the struct */ struct rte_service_spec spec; -/* atomic lock that when set indicates a service core is currently +/* spin lock that when set indicates a service core is currently * running this service callback. When not set, a core may take the * lock and then run the service callback. */ -rte_atomic32_t execute_lock; +rte_spinlock_t execute_lock;/* API set/get-able variables */ int8_t app_runstate;
@@ -54,7 +55,7 @@ struct rte_service_spec_impl {* It does not indicate the number of cores the service is running * on currently. */ -rte_atomic32_t num_mapped_cores; +uint32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent; } __rte_cache_aligned;
@@ -332,7 +333,8 @@ rte_service_runstate_get(uint32_t id)rte_smp_rmb ();int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); -int lcore_mapped = (rte_atomic32_read (&s->num_mapped_cores) >0 ); +int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, + __ATOMIC_RELAXED) >0 );return (s->app_runstate ==RUNSTATE_RUNNING ) && (s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -375,11 +377,11 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ;if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) { -if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 )) +if (!rte_spinlock_trylock (&s->execute_lock))return -EBUSY ;service_runner_do_callback (s, cs, i); -rte_atomic32_clear (&s->execute_lock); +rte_spinlock_unlock (&s->execute_lock); }else service_runner_do_callback (s, cs, i);
@@ -415,11 +417,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)/* Increment num_mapped_cores to reflect that this core is * now mapped capable of running the service. */ -rte_atomic32_inc (&s->num_mapped_cores); +__atomic_add_fetch (&s->num_mapped_cores,1 , __ATOMIC_RELAXED);int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -rte_atomic32_dec (&s->num_mapped_cores); +__atomic_sub_fetch (&s->num_mapped_cores,1 , __ATOMIC_RELAXED);return ret; }
@@ -556,19 +558,19 @@ service_update(uint32_t sid, uint32_t lcore,if (*set && !lcore_mapped) { lcore_states[lcore].service_mask |= sid_mask; -rte_atomic32_inc (&rte_services[sid].num_mapped_cores); +__atomic_add_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELAXED); }if (!*set && lcore_mapped) { lcore_states[lcore].service_mask &= ~(sid_mask); -rte_atomic32_dec (&rte_services[sid].num_mapped_cores); +__atomic_sub_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELAXED); } }if (enabled) *enabled = !!(lcore_states[lcore].service_mask & (sid_mask)); -rte_smp_wmb (); -return 0 ; }
@@ -616,7 +618,8 @@ rte_service_lcore_reset_all(void) } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) -rte_atomic32_set (&rte_services[i].num_mapped_cores,0 ); +__atomic_store_n (&rte_services[i].num_mapped_cores,0 , + __ATOMIC_RELAXED);rte_smp_wmb ();
@@ -699,7 +702,8 @@ rte_service_lcore_stop(uint32_t lcore)int32_t enabled = service_mask & (UINT64_C (1 ) << i);int32_t service_running =rte_service_runstate_get (i);int32_t only_core = (1 == -rte_atomic32_read (&rte_services[i].num_mapped_cores)); +__atomic_load_n (&rte_services[i].num_mapped_cores, + __ATOMIC_RELAXED));/* if the core is mapped, and the service is running, and this * is the only core that is mapped, the service would cease to
Signature · 1 line
2.7.4
PATCH v5 6/6 service: relax barriers with C11 atomics
In reply to Phil YangThe runstate, comp_runstate and app_runstate are used as guard variables in the service core lib. To guarantee the inter-threads visibility of these guard variables, it uses rte_smp_r/wmb. This patch use c11 atomic built-ins to relax these barriers.
--- lib/librte_eal/common/rte_service.c | 115 ++++++++++++++++++++++++++---------- 1 file changed, 84 insertions(+), 31 deletions(-)
lib/librte_eal/common/rte_service.c +84 -31
@@ -265,7 +265,6 @@ rte_service_component_register(const struct rte_service_spec *spec, s->spec = *spec; s->internal_flags |=SERVICE_F_REGISTERED |SERVICE_F_START_CHECK ; -rte_smp_wmb (); rte_service_count++;if (id_ptr)
@@ -282,7 +281,6 @@ rte_service_component_unregister(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); rte_service_count--; -rte_smp_wmb (); s->internal_flags &= ~(SERVICE_F_REGISTERED );
@@ -301,12 +299,17 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* comp_runstate act as the guard variable. Use store-release +* memory order. This synchronizes with load-acquire in +* service_run and service_runstate_get function. +*/ if (runstate) - s->comp_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->comp_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -316,12 +319,17 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* app_runstate act as the guard variable. Use store-release +* memory order. This synchronizes with load-acquire in +* service_run runstate_get function. +*/ if (runstate) - s->app_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->app_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->app_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->app_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -330,15 +338,24 @@ rte_service_runstate_get(uint32_t id) {struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); -rte_smp_rmb (); -int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); -int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, +/* comp_runstate and app_runstate act as the guard variables. +* Use load-acquire memory order. This synchronizes with +* store-release in service state set functions. +*/ +if (__atomic_load_n (&s->comp_runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING && +__atomic_load_n (&s->app_runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) { +int check_disabled = !(s->internal_flags & +SERVICE_F_START_CHECK ); +int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, __ATOMIC_RELAXED) >0 ); -return (s->app_runstate ==RUNSTATE_RUNNING ) && - (s->comp_runstate ==RUNSTATE_RUNNING ) && - (check_disabled | lcore_mapped); +return (check_disabled | lcore_mapped); + }else +return 0 ; + } static inline void
@@ -367,9 +384,15 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,if (!s)return -EINVAL ; -if (s->comp_runstate !=RUNSTATE_RUNNING || - s->app_runstate !=RUNSTATE_RUNNING || - !(service_mask & (UINT64_C (1 ) << i))) { +/* comp_runstate and app_runstate act as the guard variables. +* Use load-acquire memory order. This synchronizes with +* store-release in service state set functions. +*/ +if (__atomic_load_n (&s->comp_runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_RUNNING || +__atomic_load_n (&s->app_runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_RUNNING || + !(service_mask & (UINT64_C (1 ) << i))) { cs->service_active_on_lcore[i] =0 ;return -ENOEXEC ; }
@@ -434,7 +457,12 @@ service_runner_func(void *arg)const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (cs->runstate ==RUNSTATE_RUNNING ) { +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +while (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) {const uint64_t service_mask = cs->service_mask;for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) {
@@ -445,8 +473,6 @@ service_runner_func(void *arg) } cs->loops++; - -rte_smp_rmb (); } lcore_config[lcore].state =WAIT ;
@@ -614,15 +640,18 @@ rte_service_lcore_reset_all(void)if (lcore_states[i].is_service_core) { lcore_states[i].service_mask =0 ;set_lcore_state (i,ROLE_RTE ); - lcore_states[i].runstate =RUNSTATE_STOPPED ; +/* runstate act as guard variable Use +* store-release memory order here to synchronize +* with load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[i].runstate, +RUNSTATE_STOPPED , __ATOMIC_RELEASE); } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++)__atomic_store_n (&rte_services[i].num_mapped_cores,0 , __ATOMIC_RELAXED); -rte_smp_wmb (); -return 0 ; }
@@ -638,9 +667,11 @@ rte_service_lcore_add(uint32_t lcore)/* ensure that after adding a core the mask and state are defaults */ lcore_states[lcore].service_mask =0 ; - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; - -rte_smp_wmb (); +/* Use store-release memory order here to synchronize with +* load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);return rte_eal_wait_lcore (lcore); }
@@ -655,7 +686,12 @@ rte_service_lcore_del(uint32_t lcore)if (!cs->is_service_core)return -EINVAL ; -if (cs->runstate !=RUNSTATE_STOPPED ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_STOPPED )return -EBUSY ;set_lcore_state (lcore,ROLE_RTE );
@@ -674,13 +710,21 @@ rte_service_lcore_start(uint32_t lcore)if (!cs->is_service_core)return -EINVAL ; -if (cs->runstate ==RUNSTATE_RUNNING ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING )return -EALREADY ;/* set core to run state first, and then launch otherwise it will * return immediately as runstate keeps it in the service poll loop */ - cs->runstate =RUNSTATE_RUNNING ; +/* Use load-acquire memory order here to synchronize with +* store-release in runstate update functions. +*/ +__atomic_store_n (&cs->runstate,RUNSTATE_RUNNING , __ATOMIC_RELEASE);int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */
@@ -693,7 +737,12 @@ rte_service_lcore_stop(uint32_t lcore)if (lcore >=RTE_MAX_LCORE )return -EINVAL ; -if (lcore_states[lcore].runstate ==RUNSTATE_STOPPED ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&lcore_states[lcore].runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_STOPPED )return -EALREADY ;uint32_t i;
@@ -713,7 +762,11 @@ rte_service_lcore_stop(uint32_t lcore)return -EBUSY ; } - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; +/* Use store-release memory order here to synchronize with +* load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);return 0 ; }
Signature · 1 line
2.7.4
v6 cover use c11 atomics for service core lib
In reply to Phil YangThe rte_atomic ops and rte_smp barriers enforce DMB barriers on aarch64. Using c11 atomics with explicit memory ordering instead of the rte_atomic ops and rte_smp barriers for inter-threads synchronization can uplift the performance on aarch64 and no performance loss on x86.
This patchset contains:
1) fix race condition for MT unsafe service.
2) clean up redundant code.
3) use c11 atomics for service core lib to avoid unnecessary barriers.
v2:
Still waiting on Harry for the final solution on the MT unsafe race condition issue. But I have incorporated the comments so far.
1. add 'Fixes' tag for bug-fix patches.
2. remove 'Fixes' tag for code cleanup patches.
3. remove unused parameter for service_dump_one function.
4. replace the execute_lock atomic CAS operation to spinlock_try_lock.
5. use c11 atomics with RELAXED memory ordering for num_mapped_cores.
6. relax barriers for guard variables runstate, comp_runstate and app_runstate with c11 one-way barriers.
v3:
Sending this version since Phil is on holiday.
1. Updated the API documentation to indicate how the locking can be avoided.
v4:
1. Fix the nits in 2/6 commit message and comments in code.
v5:
1. Remove redundant libatomic clang dependency claim code as it has been added in global. (The commit da4eae278b56)
v6:
1. Fix coding style issue. Remove illegal Change-ID tag in patch 5/6.
Honnappa Nagarahalli (2):
service: fix race condition for MT unsafe service
service: fix identification of service running on other lcore
Phil Yang (4):
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 atomics
service: relax barriers with C11 atomics
lib/librte_eal/common/rte_service.c | 234 +++++++++++++------------ lib/librte_eal/include/rte_service.h | 8 +- lib/librte_eal/include/rte_service_component.h | 6 +- 3 files changed, 137 insertions(+), 111 deletions(-)
Signature · 1 line
2.7.4
PATCH v6 1/6 service: fix race condition for MT unsafe service
From: Honnappa Nagarahalli <[email protected]>
The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.
If the service is known to be mmapped on a single lcore, setting the service capability to MT safe will avoid taking the lock and improve the performance.
--- lib/librte_eal/common/rte_service.c | 11 +++++------ lib/librte_eal/include/rte_service.h | 8 ++++++-- lib/librte_eal/include/rte_service_component.h | 6 +++++- 3 files changed, 16 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {uint8_t internal_flags;/* per service statistics */ +/* Indicates how many cores the service is mapped to run on. +* It does not indicate the number of cores the service is running +* on currently. +*/ rte_atomic32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -/* check do we need cmpset, if MT safe or <= 1 core -* mapped, atomic ops are not required. -*/ -const int use_atomics = (service_mt_safe (s) ==0 ) && - (rte_atomic32_read (&s->num_mapped_cores) >1 ); -if (use_atomics) { +if (service_mt_safe (s) ==0 ) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
lib/librte_eal/include/rte_service.h +6 -2
@@ -104,12 +104,16 @@ int32_t rte_service_probe_capability(uint32_t id, uint32_t capability);* Each core can be added or removed from running a specific service. This * function enables or disables *lcore* to run *service_id*. * -* If multiple cores are enabled on a service, an atomic is used to ensure that -* only one cores runs the service at a time. The exception to this is when +* If multiple cores are enabled on a service, a lock is used to ensure that +* only one core runs the service at a time. The exception to this is when * a service indicates that it is multi-thread safe by setting the capability * called RTE_SERVICE_CAP_MT_SAFE. With the multi-thread safe capability set, * the service function can be run on multiple threads at the same time. * +* If the service is known to be mapped to a single lcore, setting the +* capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve +* better performance by avoiding the use of lock. +* * @param service_id the service to apply the lcore to * @param lcore The lcore that will be mapped to service * @param enable Zero to unmap or disable the core, non-zero to enable
lib/librte_eal/include/rte_service_component.h +5 -1
@@ -43,7 +43,7 @@ struct rte_service_spec {/** * Register a new service. * -* A service represents a component that the requires CPU time periodically to +* A service represents a component that requires CPU time periodically to * achieve its purpose. * * For example the eventdev SW PMD requires CPU cycles to perform its
@@ -56,6 +56,10 @@ struct rte_service_spec {* *rte_service_component_runstate_set*, which indicates that the service * component is ready to be executed. * +* If the service is known to be mapped to a single lcore, setting the +* capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve +* better performance. +* * @param spec The specification of the service to register * @param[out] service_id A pointer to a uint32_t, which will be filled in * during registration of the service. It is set to the integers
Signature · 1 line
2.7.4
PATCH v6 2/6 service: fix identification of service running on other lcore
In reply to Phil YangFrom: Honnappa Nagarahalli <[email protected]>
The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduce the number of instructions for all cases.
--- lib/librte_eal/common/rte_service.c | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
@@ -360,7 +360,7 @@ rte_service_runner_do_callback(struct rte_service_spec_impl *s,/* Expects the service 's' is valid. */ static int32_t service_run (uint32_t i,struct core_state *cs,uint64_t service_mask, -struct rte_service_spec_impl *s) +struct rte_service_spec_impl *s,uint32_t serialize_mt_unsafe) {if (!s)return -EINVAL ;
@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -if (service_mt_safe (s) ==0 ) { +if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); -/* Atomically add this core to the mapped cores first, then examine if -* we can run the service. This avoids a race condition between -* checking the value, and atomically adding to the mapped count. +/* Increment num_mapped_cores to reflect that this core is +* now mapped capable of running the service. */ -if (serialize_mt_unsafe) -rte_atomic32_inc (&s->num_mapped_cores); +rte_atomic32_inc (&s->num_mapped_cores); -if (service_mt_safe (s) ==0 && -rte_atomic32_read (&s->num_mapped_cores) >1 ) { -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); -return -EBUSY ; - } - -int ret =service_run (id, cs,UINT64_MAX , s); +int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); +rte_atomic32_dec (&s->num_mapped_cores);return ret; }
@@ -449,7 +439,7 @@ rte_service_runner_func(void *arg)if (!service_valid (i))continue ;/* return value ignored as no change to code flow */ -service_run (i, cs, service_mask,service_get (i)); +service_run (i, cs, service_mask,service_get (i),1 ); } cs->loops++;
Signature · 1 line
2.7.4
PATCH v6 3/6 service: remove rte prefix from static functions
In reply to Phil Yangclean up rte prefix from static functions. remove unused parameter for service_dump_one function.
--- lib/librte_eal/common/rte_service.c | 34 +++++++++++----------------------- 1 file changed, 11 insertions(+), 23 deletions(-)
lib/librte_eal/common/rte_service.c +11 -23
@@ -340,7 +340,7 @@ rte_service_runstate_get(uint32_t id) }static inline void -rte_service_runner_do_callback (struct rte_service_spec_impl *s, +service_runner_do_callback (struct rte_service_spec_impl *s,struct core_state *cs,uint32_t service_idx) {void *userdata = s->spec.callback_userdata;
@@ -378,10 +378,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ; -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);rte_atomic32_clear (&s->execute_lock); }else -rte_service_runner_do_callback (s, cs, i); +service_runner_do_callback (s, cs, i);return 0 ; }
@@ -425,14 +425,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) }static int32_t -rte_service_runner_func (void *arg) +service_runner_func (void *arg) {RTE_SET_USED (arg);uint32_t i;const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (lcore_states[lcore].runstate ==RUNSTATE_RUNNING ) { +while (cs->runstate ==RUNSTATE_RUNNING ) {const uint64_t service_mask = cs->service_mask;for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) {
@@ -693,9 +693,9 @@ rte_service_lcore_start(uint32_t lcore)/* set core to run state first, and then launch otherwise it will * return immediately as runstate keeps it in the service poll loop */ - lcore_states[lcore].runstate =RUNSTATE_RUNNING ; + cs->runstate =RUNSTATE_RUNNING ; -int ret =rte_eal_remote_launch (rte_service_runner_func,0 , lcore); +int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */ return ret; }
@@ -774,13 +774,9 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id, }static void -rte_service_dump_one (FILE *f,struct rte_service_spec_impl *s, -uint64_t all_cycles,uint32_t reset) +service_dump_one (FILE *f,struct rte_service_spec_impl *s,uint32_t reset) {/* avoid divide by zero */ -if (all_cycles ==0 ) - all_cycles =1 ; -int calls =1 ;if (s->calls !=0 ) calls = s->calls;
@@ -807,7 +803,7 @@ rte_service_attr_reset_all(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );int reset =1 ; -rte_service_dump_one (NULL , s,0 , reset); +service_dump_one (NULL , s, reset);return 0 ; }
@@ -851,21 +847,13 @@ rte_service_dump(FILE *f, uint32_t id)uint32_t i;int print_one = (id !=UINT32_MAX ); -uint64_t total_cycles =0 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if (!service_valid (i)) -continue ; - total_cycles += rte_services[i].cycles_spent; - } -/* print only the specified service */ if (print_one) {struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );fprintf (f,"Service %s Summary\n" , s->spec.name);uint32_t reset =0 ; -rte_service_dump_one (f, s, total_cycles, reset); +service_dump_one (f, s, reset);return 0 ; }
@@ -875,7 +863,7 @@ rte_service_dump(FILE *f, uint32_t id)if (!service_valid (i))continue ;uint32_t reset =0 ; -rte_service_dump_one (f, &rte_services[i], total_cycles, reset); +service_dump_one (f, &rte_services[i], reset); }fprintf (f,"Service Cores Summary\n" );
Signature · 1 line
2.7.4
PATCH v6 4/6 service: remove redundant code
In reply to Phil YangThe service id validation is duplicated, remove the redundant code in the calling functions.
--- lib/librte_eal/common/rte_service.c | 28 ++++++---------------------- 1 file changed, 6 insertions(+), 22 deletions(-)
lib/librte_eal/common/rte_service.c +6 -22
@@ -541,24 +541,12 @@ rte_service_start_with_defaults(void) }static int32_t -service_update (struct rte_service_spec *service,uint32_t lcore, +service_update (uint32_t sid,uint32_t lcore,uint32_t *set,uint32_t *enabled) { -uint32_t i; -int32_t sid =-1 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if ((struct rte_service_spec *)&rte_services[i] == service && -service_valid (i)) { - sid = i; -break ; - } - } - -if (sid ==-1 || lcore >=RTE_MAX_LCORE ) -return -EINVAL ; - -if (!lcore_states[lcore].is_service_core) +/* validate ID, or return error value */ +if (sid >=RTE_SERVICE_NUM_MAX || !service_valid (sid) || + lcore >=RTE_MAX_LCORE || !lcore_states[lcore].is_service_core)return -EINVAL ;uint64_t sid_mask =UINT64_C (1 ) << sid;
@@ -587,19 +575,15 @@ service_update(struct rte_service_spec *service, uint32_t lcore,int32_t rte_service_map_lcore_set (uint32_t id,uint32_t lcore,uint32_t enabled) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );uint32_t on = enabled >0 ; -return service_update (&s->spec, lcore, &on,0 ); +return service_update (id, lcore, &on,0 ); }int32_t rte_service_map_lcore_get (uint32_t id,uint32_t lcore) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );uint32_t enabled; -int ret =service_update (&s->spec, lcore,0 , &enabled); +int ret =service_update (id, lcore,0 , &enabled);if (ret ==0 )return enabled;return ret;
Signature · 1 line
2.7.4
PATCH v6 5/6 service: optimize with c11 atomics
In reply to Phil YangThe num_mapped_cores is used as a statistics. Use c11 atomics with RELAXED ordering for num_mapped_cores instead of rte_atomic ops which enforce unnessary barriers on aarch64.
Replace execute_lock operations to spinlock_try_lock to avoid duplicate code.
--- lib/librte_eal/common/rte_service.c | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-)
lib/librte_eal/common/rte_service.c +18 -14
@@ -20,6 +20,7 @@#include <rte_atomic.h> #include <rte_memory.h> #include <rte_malloc.h> +#include <rte_spinlock.h> #include "eal_private.h"
@@ -38,11 +39,11 @@ struct rte_service_spec_impl {/* public part of the struct */ struct rte_service_spec spec; -/* atomic lock that when set indicates a service core is currently +/* spin lock that when set indicates a service core is currently * running this service callback. When not set, a core may take the * lock and then run the service callback. */ -rte_atomic32_t execute_lock; +rte_spinlock_t execute_lock;/* API set/get-able variables */ int8_t app_runstate;
@@ -54,7 +55,7 @@ struct rte_service_spec_impl {* It does not indicate the number of cores the service is running * on currently. */ -rte_atomic32_t num_mapped_cores; +uint32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent; } __rte_cache_aligned;
@@ -332,7 +333,8 @@ rte_service_runstate_get(uint32_t id)rte_smp_rmb ();int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); -int lcore_mapped = (rte_atomic32_read (&s->num_mapped_cores) >0 ); +int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, + __ATOMIC_RELAXED) >0 );return (s->app_runstate ==RUNSTATE_RUNNING ) && (s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -375,11 +377,11 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ;if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) { -if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 )) +if (!rte_spinlock_trylock (&s->execute_lock))return -EBUSY ;service_runner_do_callback (s, cs, i); -rte_atomic32_clear (&s->execute_lock); +rte_spinlock_unlock (&s->execute_lock); }else service_runner_do_callback (s, cs, i);
@@ -415,11 +417,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)/* Increment num_mapped_cores to reflect that this core is * now mapped capable of running the service. */ -rte_atomic32_inc (&s->num_mapped_cores); +__atomic_add_fetch (&s->num_mapped_cores,1 , __ATOMIC_RELAXED);int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -rte_atomic32_dec (&s->num_mapped_cores); +__atomic_sub_fetch (&s->num_mapped_cores,1 , __ATOMIC_RELAXED);return ret; }
@@ -556,19 +558,19 @@ service_update(uint32_t sid, uint32_t lcore,if (*set && !lcore_mapped) { lcore_states[lcore].service_mask |= sid_mask; -rte_atomic32_inc (&rte_services[sid].num_mapped_cores); +__atomic_add_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELAXED); }if (!*set && lcore_mapped) { lcore_states[lcore].service_mask &= ~(sid_mask); -rte_atomic32_dec (&rte_services[sid].num_mapped_cores); +__atomic_sub_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELAXED); } }if (enabled) *enabled = !!(lcore_states[lcore].service_mask & (sid_mask)); -rte_smp_wmb (); -return 0 ; }
@@ -616,7 +618,8 @@ rte_service_lcore_reset_all(void) } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) -rte_atomic32_set (&rte_services[i].num_mapped_cores,0 ); +__atomic_store_n (&rte_services[i].num_mapped_cores,0 , + __ATOMIC_RELAXED);rte_smp_wmb ();
@@ -699,7 +702,8 @@ rte_service_lcore_stop(uint32_t lcore)int32_t enabled = service_mask & (UINT64_C (1 ) << i);int32_t service_running =rte_service_runstate_get (i);int32_t only_core = (1 == -rte_atomic32_read (&rte_services[i].num_mapped_cores)); +__atomic_load_n (&rte_services[i].num_mapped_cores, + __ATOMIC_RELAXED));/* if the core is mapped, and the service is running, and this * is the only core that is mapped, the service would cease to
Signature · 1 line
2.7.4
PATCH v6 6/6 service: relax barriers with C11 atomics
In reply to Phil YangThe runstate, comp_runstate and app_runstate are used as guard variables in the service core lib. To guarantee the inter-threads visibility of these guard variables, it uses rte_smp_r/wmb. This patch use c11 atomic built-ins to relax these barriers.
--- lib/librte_eal/common/rte_service.c | 115 ++++++++++++++++++++++++++---------- 1 file changed, 84 insertions(+), 31 deletions(-)
lib/librte_eal/common/rte_service.c +84 -31
@@ -265,7 +265,6 @@ rte_service_component_register(const struct rte_service_spec *spec, s->spec = *spec; s->internal_flags |=SERVICE_F_REGISTERED |SERVICE_F_START_CHECK ; -rte_smp_wmb (); rte_service_count++;if (id_ptr)
@@ -282,7 +281,6 @@ rte_service_component_unregister(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); rte_service_count--; -rte_smp_wmb (); s->internal_flags &= ~(SERVICE_F_REGISTERED );
@@ -301,12 +299,17 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* comp_runstate act as the guard variable. Use store-release +* memory order. This synchronizes with load-acquire in +* service_run and service_runstate_get function. +*/ if (runstate) - s->comp_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->comp_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -316,12 +319,17 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* app_runstate act as the guard variable. Use store-release +* memory order. This synchronizes with load-acquire in +* service_run runstate_get function. +*/ if (runstate) - s->app_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->app_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->app_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->app_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -330,15 +338,24 @@ rte_service_runstate_get(uint32_t id) {struct rte_service_spec_impl *s;SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); -rte_smp_rmb (); -int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); -int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, +/* comp_runstate and app_runstate act as the guard variables. +* Use load-acquire memory order. This synchronizes with +* store-release in service state set functions. +*/ +if (__atomic_load_n (&s->comp_runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING && +__atomic_load_n (&s->app_runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) { +int check_disabled = !(s->internal_flags & +SERVICE_F_START_CHECK ); +int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, __ATOMIC_RELAXED) >0 ); -return (s->app_runstate ==RUNSTATE_RUNNING ) && - (s->comp_runstate ==RUNSTATE_RUNNING ) && - (check_disabled | lcore_mapped); +return (check_disabled | lcore_mapped); + }else +return 0 ; + } static inline void
@@ -367,9 +384,15 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,if (!s)return -EINVAL ; -if (s->comp_runstate !=RUNSTATE_RUNNING || - s->app_runstate !=RUNSTATE_RUNNING || - !(service_mask & (UINT64_C (1 ) << i))) { +/* comp_runstate and app_runstate act as the guard variables. +* Use load-acquire memory order. This synchronizes with +* store-release in service state set functions. +*/ +if (__atomic_load_n (&s->comp_runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_RUNNING || +__atomic_load_n (&s->app_runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_RUNNING || + !(service_mask & (UINT64_C (1 ) << i))) { cs->service_active_on_lcore[i] =0 ;return -ENOEXEC ; }
@@ -434,7 +457,12 @@ service_runner_func(void *arg)const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (cs->runstate ==RUNSTATE_RUNNING ) { +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +while (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) {const uint64_t service_mask = cs->service_mask;for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) {
@@ -445,8 +473,6 @@ service_runner_func(void *arg) } cs->loops++; - -rte_smp_rmb (); } lcore_config[lcore].state =WAIT ;
@@ -614,15 +640,18 @@ rte_service_lcore_reset_all(void)if (lcore_states[i].is_service_core) { lcore_states[i].service_mask =0 ;set_lcore_state (i,ROLE_RTE ); - lcore_states[i].runstate =RUNSTATE_STOPPED ; +/* runstate act as guard variable Use +* store-release memory order here to synchronize +* with load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[i].runstate, +RUNSTATE_STOPPED , __ATOMIC_RELEASE); } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++)__atomic_store_n (&rte_services[i].num_mapped_cores,0 , __ATOMIC_RELAXED); -rte_smp_wmb (); -return 0 ; }
@@ -638,9 +667,11 @@ rte_service_lcore_add(uint32_t lcore)/* ensure that after adding a core the mask and state are defaults */ lcore_states[lcore].service_mask =0 ; - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; - -rte_smp_wmb (); +/* Use store-release memory order here to synchronize with +* load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);return rte_eal_wait_lcore (lcore); }
@@ -655,7 +686,12 @@ rte_service_lcore_del(uint32_t lcore)if (!cs->is_service_core)return -EINVAL ; -if (cs->runstate !=RUNSTATE_STOPPED ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) !=RUNSTATE_STOPPED )return -EBUSY ;set_lcore_state (lcore,ROLE_RTE );
@@ -674,13 +710,21 @@ rte_service_lcore_start(uint32_t lcore)if (!cs->is_service_core)return -EINVAL ; -if (cs->runstate ==RUNSTATE_RUNNING ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING )return -EALREADY ;/* set core to run state first, and then launch otherwise it will * return immediately as runstate keeps it in the service poll loop */ - cs->runstate =RUNSTATE_RUNNING ; +/* Use load-acquire memory order here to synchronize with +* store-release in runstate update functions. +*/ +__atomic_store_n (&cs->runstate,RUNSTATE_RUNNING , __ATOMIC_RELEASE);int ret =rte_eal_remote_launch (service_runner_func,0 , lcore);/* returns -EBUSY if the core is already launched, 0 on success */
@@ -693,7 +737,12 @@ rte_service_lcore_stop(uint32_t lcore)if (lcore >=RTE_MAX_LCORE )return -EINVAL ; -if (lcore_states[lcore].runstate ==RUNSTATE_STOPPED ) +/* runstate act as the guard variable. Use load-acquire +* memory order here to synchronize with store-release +* in runstate update functions. +*/ +if (__atomic_load_n (&lcore_states[lcore].runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_STOPPED )return -EALREADY ;uint32_t i;
@@ -713,7 +762,11 @@ rte_service_lcore_stop(uint32_t lcore)return -EBUSY ; } - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; +/* Use store-release memory order here to synchronize with +* load-acquire in runstate read functions. +*/ +__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);return 0 ; }
Signature · 1 line
2.7.4
Re: [dpdk-dev] [PATCH v6 0/6] use c11 atomics for service core lib
In reply to Phil YangQuoted 10 lines · Phil Yang
The
rte_atomicops andrte_smpbarriers enforce DMB barriers on aarch64. Using c11 atomics with explicit memory ordering instead of therte_atomicops andrte_smpbarriers for inter-threads synchronization can uplift the performance on aarch64 and no performance loss on x86.This patchset contains:
1) fix race condition for MT unsafe service.
2) clean up redundant code.
3) use c11 atomics for service core lib to avoid unnecessary barriers.
Series applied, thanks for the cleanup and fixes.
Signature · 1 line
David Marchand
PATCH v3 8/12 service: remove redundant code
In reply to Phil YangThe service id validation is verified in the calling function, remove the redundant code inside the service_update function.
--- lib/librte_eal/common/rte_service.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-)
lib/librte_eal/common/rte_service.c +12 -19
@@ -552,21 +552,10 @@ rte_service_start_with_defaults(void) }static int32_t -service_update (struct rte_service_spec *service,uint32_t lcore, +service_update (uint32_t sid,uint32_t lcore,uint32_t *set,uint32_t *enabled) { -uint32_t i; -int32_t sid =-1 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if ((struct rte_service_spec *)&rte_services[i] == service && -service_valid (i)) { - sid = i; -break ; - } - } - -if (sid ==-1 || lcore >=RTE_MAX_LCORE ) +if (lcore >=RTE_MAX_LCORE )return -EINVAL ;if (!lcore_states[lcore].is_service_core)
@@ -598,19 +587,23 @@ service_update(struct rte_service_spec *service, uint32_t lcore,int32_t rte_service_map_lcore_set (uint32_t id,uint32_t lcore,uint32_t enabled) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* validate ID, or return error value */ +if (id >=RTE_SERVICE_NUM_MAX || !service_valid (id)) +return -EINVAL ; +uint32_t on = enabled >0 ; -return service_update (&s->spec, lcore, &on,0 ); +return service_update (id, lcore, &on,0 ); }int32_t rte_service_map_lcore_get (uint32_t id,uint32_t lcore) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* validate ID, or return error value */ +if (id >=RTE_SERVICE_NUM_MAX || !service_valid (id)) +return -EINVAL ; +uint32_t enabled; -int ret =service_update (&s->spec, lcore,0 , &enabled); +int ret =service_update (id, lcore,0 , &enabled);if (ret ==0 )return enabled;return ret;
Signature · 1 line
2.7.4
Re: [dpdk-dev] [PATCH v3 08/12] service: remove redundant code
Quoted 10 lines
From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 08/12] service: remove redundant code
Quoted 8 lines
The service id validation is verified in the calling function, remove the redundant code inside the
service_updatefunction.Fixes 21698354c832 ("service: introduce service cores concept")Signed-off-by Phil Yang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>
Same comment as patch 7/12, is this really a "Fix"? This functionality is not "broken" in the current code? And is there value in porting to stable? I'd see this as unnecessary churn.
As before, it is a valid cleanup (thanks), and I'd like to take it for new DPDK releases.
Happy to Ack without Fixes or Cc Stable, if that's acceptable to you?
Quoted 66 lines
--- lib/librte_eal/common/rte_service.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-)lib/librte_eal/common/rte_service.c +12
-19@@ -552,21 +552,10 @@ rte_service_start_with_defaults(void) }static int32_t -service_update (struct rte_service_spec *service,uint32_t lcore, +service_update (uint32_t sid,uint32_t lcore,uint32_t *set,uint32_t *enabled) { -uint32_t i; -int32_t sid =-1 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if ((struct rte_service_spec *)&rte_services[i] == service && -service_valid (i)) { - sid = i; -break ; - } - } - -if (sid ==-1 || lcore >=RTE_MAX_LCORE ) +if (lcore >=RTE_MAX_LCORE )return -EINVAL ;if (!lcore_states[lcore].is_service_core)@@ -598,19 +587,23 @@ service_update(struct rte_service_spec *service, uint32_t lcore, int32_t rte_service_map_lcore_set(uint32_t id, uint32_t lcore, uint32_t enabled) { - struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* validate ID, or return error value */ +if (id >=RTE_SERVICE_NUM_MAX || !service_valid (id)) +return -EINVAL ; +uint32_t on = enabled >0 ; -return service_update (&s->spec, lcore, &on,0 ); +return service_update (id, lcore, &on,0 ); }int32_t rte_service_map_lcore_get (uint32_t id,uint32_t lcore) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* validate ID, or return error value */ +if (id >=RTE_SERVICE_NUM_MAX || !service_valid (id)) +return -EINVAL ; +uint32_t enabled; -int ret =service_update (&s->spec, lcore,0 , &enabled); +int ret =service_update (id, lcore,0 , &enabled);if (ret ==0 )return enabled;return ret;
--
2.7.4
Re: [dpdk-dev] [PATCH v3 08/12] service: remove redundant code
<snip>
Quoted 11 lines
Quoted 9 lines
The service id validation is verified in the calling function, remove the redundant code inside the
service_updatefunction.Fixes 21698354c832 ("service: introduce service cores concept")Signed-off-by Phil Yang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>
Quoted 8 lines
Same comment as patch 7/12, is this really a "Fix"? This functionality is not "broken" in the current code? And is there value in porting to stable? I'd see this as unnecessary churn.
As before, it is a valid cleanup (thanks), and I'd like to take it for new DPDK releases.
Happy to Ack without Fixes or Cc Stable, if that's acceptable to you?
Agreed.
Quoted 16 lines
Quoted 16 lines
--- lib/librte_eal/common/rte_service.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-)lib/librte_eal/common/rte_service.c +0
-0@@ -552,21 +552,10 @@ rte_service_start_with_defaults(void) }
Quoted 4 lines
lib/librte_eal/common/rte_service.c rte_service_start_with_defaults(void) · line 556
Quoted 4 lines
lib/librte_eal/common/rte_service.c +1
-1static int32_t -service_update (struct rte_service_spec *service,uint32_t lcore, +service_update (uint32_t sid,uint32_t lcore,uint32_t *set,uint32_t *enabled)
'set' parameter does not need be passed by reference, pass by value is enough.
Quoted 7 lines
Quoted 7 lines
lib/librte_eal/common/rte_service.c +0
-6{ -uint32_t i; -int32_t sid =-1 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if ((struct rte_service_spec *)&rte_services[i] == service && -service_valid (i)) {
Quoted 8 lines
lib/librte_eal/common/rte_service.c rte_service_start_with_defaults(void) · line 559
Quoted 8 lines
lib/librte_eal/common/rte_service.c +1
-6- sid = i; -break ; - } - } - -if (sid ==-1 || lcore >=RTE_MAX_LCORE ) +if (lcore >=RTE_MAX_LCORE )return -EINVAL ;
The validations look somewhat inconsistent in service_update function, we are validating some parameters and not some. Suggest bringing the validation of the service id also into this function and remove it from the calling functions.
Quoted 12 lines
lib/librte_eal/common/rte_service.c service_update(struct rte_service_spec *service, · line 593
Quoted 12 lines
lib/librte_eal/common/rte_service.c +4
-2if (!lcore_states[lcore].is_service_core)@@ -598,19 +587,23 @@ service_update(struct rte_service_spec *service, uint32_t lcore, int32_t rte_service_map_lcore_set(uint32_t id, uint32_t lcore, uint32_t enabled) { - struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* validate ID, or return error value */ +if (id >=RTE_SERVICE_NUM_MAX || !service_valid (id)) +return -EINVAL ; +uint32_t on = enabled >0 ;
We do not need the above line. 'enabled' can be passed directly to 'service_update'.
Quoted 20 lines
Quoted 20 lines
lib/librte_eal/common/rte_service.c +6
-5-return service_update (&s->spec, lcore, &on,0 ); +return service_update (id, lcore, &on,0 ); }int32_t rte_service_map_lcore_get (uint32_t id,uint32_t lcore) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* validate ID, or return error value */ +if (id >=RTE_SERVICE_NUM_MAX || !service_valid (id)) +return -EINVAL ; +uint32_t enabled; -int ret =service_update (&s->spec, lcore,0 , &enabled); +int ret =service_update (id, lcore,0 , &enabled);if (ret ==0 )return enabled;return ret; --2.7.4
Re: [dpdk-dev] [PATCH v3 08/12] service: remove redundant code
Quoted 29 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Monday, April 6, 2020 2:35 AM
To: Van Haaren, Harry <[email protected]>; Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 08/12] service: remove redundant code<snip>
Quoted 7 lines
Quoted 7 lines
The service id validation is verified in the calling function, remove the redundant code inside the
service_updatefunction.Fixes 21698354c832 ("service: introduce service cores concept")Quoted 6 lines
Quoted 2 lines
Signed-off-by Phil Yang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>Same comment as patch 7/12, is this really a "Fix"? This functionality is not "broken" in the current code? And is there value in porting to stable? I'd
Quoted 8 lines
see
Quoted 6 lines
this as unnecessary churn.
As before, it is a valid cleanup (thanks), and I'd like to take it for new DPDK releases.
Happy to Ack without Fixes or Cc Stable, if that's acceptable to you?
Agreed.
Agreed.
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
--- lib/librte_eal/common/rte_service.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-)lib/librte_eal/common/rte_service.c +0
-0@@ -552,21 +552,10 @@ rte_service_start_with_defaults(void) }
Quoted 6 lines
Quoted 4 lines
lib/librte_eal/common/rte_service.c rte_service_start_with_defaults(void) · line 556
Quoted 4 lines
lib/librte_eal/common/rte_service.c +1
-1static int32_t -service_update (struct rte_service_spec *service,uint32_t lcore, +service_update (uint32_t sid,uint32_t lcore,uint32_t *set,uint32_t *enabled)'set' parameter does not need be passed by reference, pass by value is enough.
Agreed.
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
lib/librte_eal/common/rte_service.c +0
-10{ -uint32_t i; -int32_t sid =-1 ; - -for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) { -if ((struct rte_service_spec *)&rte_services[i] == service && -service_valid (i)) { -sid = i; -break ; -} -}
Quoted 8 lines
Quoted 4 lines
lib/librte_eal/common/rte_service.c rte_service_start_with_defaults(void) · line 559
Quoted 4 lines
lib/librte_eal/common/rte_service.c +1
-2- -if (sid ==-1 || lcore >=RTE_MAX_LCORE ) +if (lcore >=RTE_MAX_LCORE )return -EINVAL ;The validations look somewhat inconsistent in
service_updatefunction, we are validating some parameters and not some.
Suggest bringing the validation of the service id also into this function and remove it from the calling functions.
Agreed. I will update it in the next version.
Quoted 5 lines
Quoted 4 lines
lib/librte_eal/common/rte_service.c service_update(struct rte_service_spec
Quoted 4 lines
lib/librte_eal/common/rte_service.c +0
-0if (!lcore_states[lcore].is_service_core)@@ -598,19 +587,23 @@ service_update(struct rte_service_spec*service,
Quoted 11 lines
Quoted 9 lines
lib/librte_eal/common/rte_service.c service_update(struct rte_service_spec *service, uint32_t lcore, · line 594
Quoted 9 lines
lib/librte_eal/common/rte_service.c +4
-2uint32_t lcore, int32_t rte_service_map_lcore_set(uint32_t id, uint32_t lcore, uint32_t enabled) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* validate ID, or return error value */ +if (id >=RTE_SERVICE_NUM_MAX || !service_valid (id)) +return -EINVAL ; +uint32_t on = enabled >0 ;We do not need the above line. 'enabled' can be passed directly to '
service_update'.
Agreed.
Quoted 22 lines
Quoted 21 lines
Quoted 21 lines
lib/librte_eal/common/rte_service.c +6
-5-return service_update (&s->spec, lcore, &on,0 ); +return service_update (id, lcore, &on,0 ); }int32_t rte_service_map_lcore_get (uint32_t id,uint32_t lcore) { -struct rte_service_spec_impl *s; -SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); +/* validate ID, or return error value */ +if (id >=RTE_SERVICE_NUM_MAX || !service_valid (id)) +return -EINVAL ; +uint32_t enabled; -int ret =service_update (&s->spec, lcore,0 , &enabled); +int ret =service_update (id, lcore,0 , &enabled);if (ret ==0 )return enabled;return ret; --2.7.4
PATCH v3 9/12 service: avoid race condition for MT unsafe service
In reply to Phil YangFrom: Honnappa Nagarahalli <[email protected]>
There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.
--- lib/librte_eal/common/rte_service.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {uint8_t internal_flags;/* per service statistics */ +/* Indicates how many cores the service is mapped to run on. +* It does not indicate the number of cores the service is running +* on currently. +*/ rte_atomic32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -/* check do we need cmpset, if MT safe or <= 1 core -* mapped, atomic ops are not required. -*/ -const int use_atomics = (service_mt_safe (s) ==0 ) && - (rte_atomic32_read (&s->num_mapped_cores) >1 ); -if (use_atomics) { +if (service_mt_safe (s) ==0 ) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
Signature · 1 line
2.7.4
Re: [dpdk-dev] [PATCH v3 09/12] service: avoid race condition for MT unsafe service
Quoted 19 lines
From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: [PATCH v3 09/12] service: avoid race condition for MT unsafe serviceFrom: Honnappa Nagarahalli <[email protected]>
There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use '
execute_lock' always when the service is MT unsafe.
Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")Signed-off-by Honnappa Nagarahalli <[email protected]>Reviewed-by Phil Yang <[email protected]>Reviewed-by Gavin Hu <[email protected]>
We should put "fix" in the title, once converged on an implementation.
Regarding Fixes and stable backport, we should consider if fixing this in stable with a performance degradation, fixing with more complex solution, or documenting a known issue a better solution.
This fix (always taking the atomic lock) will have a negative performance impact on existing code using services. We should investigate a way to fix it without causing datapath performance degradation.
I think there is a way to achieve this by moving more checks/time to the control path (lcore updating the map), and not forcing the datapath lcore to always take an atomic.
In this particular case, we have a counter for number of iterations that a service has done. If this increments we know that the lcore running the service has re-entered the critical section, so would see an updated "needs atomic" flag.
This approach may introduce a predictable branch on the datapath, however the cost of a predictable branch vs always taking an atomic is order(s?) of magnitude, so a branch is much preferred.
It must be possible to avoid the datapath overhead using a scheme like this. It will likely be more complex than your proposed change below, however if it avoids datapath performance drops I feel that a more complex solution is worth investigating at least.
A unit test is required to validate a fix like this - although perhaps found by inspection/review, a real-world test to validate would give confidence.
Thoughts on such an approach?
Quoted 37 lines
--- lib/librte_eal/common/rte_service.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)lib/librte_eal/common/rte_service.c +5
-7@@ -50,6 +50,10 @@ struct rte_service_spec_impl {uint8_t internal_flags;/* per service statistics */ +/* Indicates how many cores the service is mapped to run on. +* It does not indicate the number of cores the service is running +* on currently. +*/ rte_atomic32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent;@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -/* check do we need cmpset, if MT safe or <= 1 core -* mapped, atomic ops are not required. -*/ -const int use_atomics = (service_mt_safe (s) ==0 ) && - (rte_atomic32_read (&s->num_mapped_cores) >1 ); -if (use_atomics) { +if (service_mt_safe (s) ==0 ) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ; --2.7.4
Re: [dpdk-dev] [PATCH v3 09/12] service: avoid race condition for MT unsafe service
<snip>
Quoted 11 lines
Quoted 11 lines
Subject: [PATCH v3 09/12] service: avoid race condition for MT unsafe service
From: Honnappa Nagarahalli <[email protected]>
There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use '
execute_lock' always when the service is MT unsafe.
Quoted 8 lines
Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")Signed-off-by Honnappa Nagarahalli <[email protected]>Reviewed-by Phil Yang <[email protected]>Reviewed-by Gavin Hu <[email protected]>We should put "fix" in the title, once converged on an implementation.
Ok, will replace 'avoid' with 'fix' (once we agree on the solution)
Quoted 9 lines
Regarding Fixes and stable backport, we should consider if fixing this in stable with a performance degradation, fixing with more complex solution, or documenting a known issue a better solution.
This fix (always taking the atomic lock) will have a negative performance impact on existing code using services. We should investigate a way to fix it without causing datapath performance degradation.
Trying to gauge the impact on the existing applications...
The documentation does not explicitly disallow run time mapping of cores to service.
1) If the applications are mapping the cores to services at run time, they are running with a bug. IMO, bug fix resulting in a performance drop should be acceptable.
2) If the service is configured to run on single core (num_mapped_cores == 1), but service is set to MT unsafe - this will have a (possible) performance impact.
a) This can be solved by setting the service to MT safe and can be documented. This might be a reasonable solution for applications which are compiling with
future DPDK releases.
b) We can also solve this using symbol versioning - the old version of this function will use the old code, the new version of this function will use the code in
this patch. So, if the application is run with future DPDK releases without recompiling, it will continue to use the old version. If the application is compiled
with future releases, they can use solution in 2a. We also should think if this is an appropriate solution as this would force 1) to recompile to get the fix.
3) If the service is configured to run on multiple cores (num_mapped_cores > 1), then for those applications, the lock is being taken already. These applications might see some improvements as this patch removes few instructions.
Quoted 4 lines
I think there is a way to achieve this by moving more checks/time to the control path (lcore updating the map), and not forcing the datapath lcore to always take an atomic.
I think 2a above is the solution.
Quoted 2 lines
In this particular case, we have a counter for number of iterations that a
Which counter are you thinking about?
All the counters I checked are not atomic operations currently. If we are going to use counters they have to be atomic, which means additional cycles in the data path.
Quoted 4 lines
service has done. If this increments we know that the lcore running the service has re-entered the critical section, so would see an updated "needs atomic" flag.
Quoted 8 lines
This approach may introduce a predictable branch on the datapath, however the cost of a predictable branch vs always taking an atomic is order(s?) of magnitude, so a branch is much preferred.
It must be possible to avoid the datapath overhead using a scheme like this. It will likely be more complex than your proposed change below, however if it avoids datapath performance drops I feel that a more complex solution is worth investigating at least.
I do not completely understand the approach you are proposing, may be you can elaborate more. But, it seems to be based on a counter approach. Following is my assessment on what happens if we use a counter. Let us say we kept track of how many cores are running the service currently. We need an atomic counter other than 'num_mapped_cores'. Let us call that counter 'num_current_cores'. The code to call the service would look like below.
1) rte_atomic32_inc(&num_current_cores); /* this results in a full memory barrier */
2) if (__atomic_load_n(&num_current_cores, __ATOMIC_ACQUIRE) == 1) { /* rte_atomic_read is not enough here as it does not provide the required memory barrier for any architecture */
3) run_service(); /* Call the service */
4) }
5) rte_atomic32_sub(&num_current_cores); /* Calling rte_atomic32_clear is not enough as it is not an atomic operation and does not provide the required memory barrier */
But the above code has race conditions in lines 1 and 2. It is possible that none of the cores will ever get to run the service as they all could simultaneously increment the counter. Hence lines 1 and 2 together need to be atomic, which is nothing but 'compare-exchange' operation.
BTW, the current code has a bug where it calls 'rte_atomic_clear(&s->execute_lock)', it is missing memory barriers which results in clearing the execute_lock before the service has completed running. I suggest changing the 'execute_lock' to rte_spinlock_t and using rte_spinlock_try_lock and rte_spinlock_unlock APIs.
Quoted 3 lines
A unit test is required to validate a fix like this - although perhaps found by inspection/review, a real-world test to validate would give confidence.
Agree, need to have a test case.
Quoted 44 lines
Thoughts on such an approach?
Quoted 38 lines
--- lib/librte_eal/common/rte_service.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)lib/librte_eal/common/rte_service.c +5
-6@@ -50,6 +50,10 @@ struct rte_service_spec_impl {uint8_t internal_flags;/* per service statistics */ +/* Indicates how many cores the service is mapped to run on. +* It does not indicate the number of cores the service is running +* on currently. +*/ rte_atomic32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent;@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs,uint64_t service_mask, cs->service_active_on_lcore[i] = 1; -/* check do we need cmpset, if MT safe or <= 1 core -* mapped, atomic ops are not required. -*/ -const int use_atomics = (service_mt_safe (s) ==0 ) && - (rte_atomic32_read (&s- num_mapped_cores) > 1); -if (use_atomics) { +if (service_mt_safe (s) ==0 ) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return
-EBUSY;--
2.7.4
Re: [dpdk-dev] [PATCH v3 09/12] service: avoid race condition for MT unsafe service
Quoted 76 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, April 4, 2020 7:03 PM
To: Van Haaren, Harry <[email protected]>; Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 09/12] service: avoid race condition for MT unsafe service<snip>
Quoted 11 lines
Quoted 11 lines
Subject: [PATCH v3 09/12] service: avoid race condition for MT unsafe service
From: Honnappa Nagarahalli <[email protected]>
There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use '
execute_lock' always when the service is MT unsafe.Quoted 8 lines
Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")Signed-off-by Honnappa Nagarahalli <[email protected]>Reviewed-by Phil Yang <[email protected]>Reviewed-by Gavin Hu <[email protected]>We should put "fix" in the title, once converged on an implementation.
Ok, will replace 'avoid' with 'fix' (once we agree on the solution)
Quoted 2 lines
Regarding Fixes and stable backport, we should consider if fixing this in
stable
Quoted 6 lines
with a performance degradation, fixing with more complex solution, or documenting a known issue a better solution.
This fix (always taking the atomic lock) will have a negative performance impact on existing code using services. We should investigate a way to fix
it
Quoted 1 line
without causing datapath performance degradation.
Trying to gauge the impact on the existing applications...
The documentation does not explicitly disallow run time mapping of cores to service.
1) If the applications are mapping the cores to services at run time, they are running with a bug. IMO, bug fix resulting in a performance drop should be acceptable.
2) If the service is configured to run on single core (num_mapped_cores== 1), but service is set to MT unsafe - this will have a (possible) performance impact.
a) This can be solved by setting the service to MT safe and can be
documented. This might be a reasonable solution for applications which are compiling with
future DPDK releases.
b) We can also solve this using symbol versioning - the old version of
this function will use the old code, the new version of this function will use the code in
this patch. So, if the application is run with future DPDK
releases without recompiling, it will continue to use the old version. If the application is compiled
with future releases, they can use solution in 2a. We also
should think if this is an appropriate solution as this would force 1) to recompile to get the fix.
3) If the service is configured to run on multiple cores (num_mapped_cores> 1), then for those applications, the lock is being taken already. These applications might see some improvements as this patch removes few instructions.Quoted 1 line
Quoted 4 lines
Quoted 3 lines
I think there is a way to achieve this by moving more checks/time to the control path (lcore updating the map), and not forcing the datapath lcore to always take an atomic.
I think 2a above is the solution.
2a above is e.g. the Eventdev SW routines like Rx/Tx scheduler services. We should strive to not reduce datapath performance at all here.
Quoted 5 lines
Quoted 1 line
In this particular case, we have a counter for number of iterations that a
Which counter are you thinking about?
All the counters I checked are not atomic operations currently. If we are going to use counters they have to be atomic, which means additional cycles in the data path.
I'll try to explain the concept better, take this example:
- One service core is mapped to a MT_UNSAFE service, like event/sw pmd
- Application wants to map a 2nd lcore to the same service
- You point out that today there is a race over the lock
-- the lock is not taken if (num_mapped_lcores == 1)
-- this avoids an atomic lock/unlock on the datapath
To achieve our desired goal;
- control thread doing mapping performs the following operations
-- write service->num_mapped_lcores++ (atomic not required, only single-writer allowed by APIs) -- MFENCE (full st-ld barrier) to flush write, and force later loads to issue after -- read the "calls_per_service" counter for each lcores, add them up.
---- Wait :)
-- re-read the "calls_per_service", and ensure the count has changed.
---- The fact that the calls_per_service has changed ensures the service-lcore
has seen the new "num_mapped_cores" value, and has now taken the lock!
-- *now* it is safe to map the 2nd lcore to the service
There is a caveat here that the increments to the "calls_per_service" variable must become globally-observable. To force this immediately would require a write memory barrier, which could impact datapath performance. Given the service is now taking a lock, the unlock() thereof would ensure the "calls_per_service" is flushed to memory.
Note: we could use calls_per_service, or add a new variable to the service struct. Writes to this do not need to be atomic, as it is either mapped to a single core, or else there's a lock around it.
Quoted 8 lines
Quoted 8 lines
service has done. If this increments we know that the lcore running the service has re-entered the critical section, so would see an updated "needs atomic" flag.
This approach may introduce a predictable branch on the datapath, however the cost of a predictable branch vs always taking an atomic is order(s?) of magnitude, so a branch is much preferred.
Quoted 8 lines
Quoted 1 line
It must be possible to avoid the datapath overhead using a scheme like this.
It
Quoted 3 lines
will likely be more complex than your proposed change below, however if it avoids datapath performance drops I feel that a more complex solution is worth investigating at least.
I do not completely understand the approach you are proposing, may be you can elaborate more.
Expanded above, showing a possible solution that does not require additional atomics on the datapath.
Quoted 22 lines
But, it seems to be based on a counter approach. Following is
my assessment on what happens if we use a counter. Let us say we kept track of how many cores are running the service currently. We need an atomic counter other than 'num_mapped_cores'. Let us call that counter 'num_current_cores'. The code to call the service would look like below.1)
rte_atomic32_inc(&num_current_cores); /* this results in a full memory barrier */
2) if (__atomic_load_n(&num_current_cores, __ATOMIC_ACQUIRE)== 1) { /*rte_atomic_readis not enough here as it does not provide the required memory barrier for any architecture */
3)run_service(); /* Call the service */
4) }
5)rte_atomic32_sub(&num_current_cores); /* Callingrte_atomic32_clearis not enough as it is not an atomic operation and does not provide the required memory barrier */But the above code has race conditions in lines 1 and 2. It is possible that none of the cores will ever get to run the service as they all could simultaneously increment the counter. Hence lines 1 and 2 together need to be atomic, which is nothing but 'compare-exchange' operation.
Quoted 5 lines
BTW, the current code has a bug where it calls 'rte_atomic_clear(&s-
Quoted 1 line
execute_lock)', it is missing memory barriers which results in clearing the
execute_lockbefore the service has completed running. I suggest changing the 'execute_lock' torte_spinlock_tand usingrte_spinlock_try_lockandrte_spinlock_unlockAPIs.
I don't think a spinlock is what we want here:
The idea is that a service-lcore can be mapped to multiple services. If one service is already being run (by another thread), we do not want to spin here waiting for it to become "free" to run by this thread, it should continue to the next service that it is mapped to.
Quoted 10 lines
Quoted 2 lines
A unit test is required to validate a fix like this - although perhaps found
by
Quoted 1 line
inspection/review, a real-world test to validate would give confidence.
Agree, need to have a test case.
Quoted 4 lines
Thoughts on such an approach?
<snip patch contents>
Re: [dpdk-dev] [PATCH v3 09/12] service: avoid race condition for MT unsafe service
<snip>
Quoted 63 lines
Quoted 63 lines
Quoted 12 lines
Quoted 12 lines
Subject: [PATCH v3 09/12] service: avoid race condition for MT unsafe service
From: Honnappa Nagarahalli <[email protected]>
There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use '
execute_lock' always when the service is MT unsafe.Quoted 8 lines
Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")Signed-off-by Honnappa Nagarahalli <[email protected]>Reviewed-by Phil Yang <[email protected]>Reviewed-by Gavin Hu <[email protected]>We should put "fix" in the title, once converged on an implementation.
Ok, will replace 'avoid' with 'fix' (once we agree on the solution)
Quoted 3 lines
Regarding Fixes and stable backport, we should consider if fixing this in
stable
Quoted 7 lines
with a performance degradation, fixing with more complex solution, or documenting a known issue a better solution.
This fix (always taking the atomic lock) will have a negative performance impact on existing code using services. We should investigate a way to fix
it
Quoted 1 line
without causing datapath performance degradation.
Trying to gauge the impact on the existing applications...
The documentation does not explicitly disallow run time mapping of cores to service.
1) If the applications are mapping the cores to services at run time, they are running with a bug. IMO, bug fix resulting in a performance drop should be acceptable.
2) If the service is configured to run on single core (num_mapped_cores== 1), but service is set to MT unsafe - this will have a (possible) performance impact.
a) This can be solved by setting the service to MT safe and can be
documented. This might be a reasonable solution for applications which are compiling with
future DPDK releases.
b) We can also solve this using symbol versioning - the old version
of this function will use the old code, the new version of this function will use the code in
this patch. So, if the application is run with
future DPDK releases without recompiling, it will continue to use the old version. If the application is compiled
with future releases, they can use solution in 2a.
We also should think if this is an appropriate solution as this would force 1) to recompile to get the fix.
3) If the service is configured to run on multiple cores (num_mapped_cores> 1), then for those applications, the lock is being taken already. These applications might see some improvements as this patch removes few instructions.Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
I think there is a way to achieve this by moving more checks/time to the control path (lcore updating the map), and not forcing the datapath lcore to always take an atomic.
I think 2a above is the solution.
2a above is e.g. the Eventdev SW routines like Rx/Tx scheduler services.
I scanned through the code briefly
I see that Eth RX/TX, Crypto adapters are setting the MT_SAFE capabilities, can be ignored.
Timer adaptor and some others do not set MT_SAFE. Seems like the cores to run on are mapped during run time. But it is not clear to me if it can get mapped to run on multiple cores. If they are, they are running with the bug. But, these are all internal to DPDK and can be fixed.
Are there no performance tests in these components that we can run?
Quoted 17 lines
We should strive to not reduce datapath performance at all here.
Quoted 6 lines
Quoted 2 lines
In this particular case, we have a counter for number of iterations that a
Which counter are you thinking about?
All the counters I checked are not atomic operations currently. If we are going to use counters they have to be atomic, which means additional cycles in the data path.I'll try to explain the concept better, take this example:
- One service core is mapped to aMT_UNSAFEservice, like event/sw pmd
- Application wants to map a 2nd lcore to the same service
- You point out that today there is a race over the lock
-- the lock is not taken if (num_mapped_lcores== 1)
-- this avoids an atomic lock/unlock on the datapath
Quoted 4 lines
To achieve our desired goal;
- control thread doing mapping performs the following operations-- write service->num_mapped_lcores++ (atomic not required, only single-writer allowed by APIs)
This has to be atomic because of rte_service_run_iter_on_app_lcore API. Performance should be fine as this API is not called frequently. But need to consider the implications of more than one thread updating num_mapped_cores.
Quoted 2 lines
-- MFENCE (full st-ld barrier) to flush write, and force later loads to issue
after
I am not exactly sure what MFENCE on x86 does. On Arm platforms, the full barrier (DMB ISH) just makes sure that memory operations are not re-ordered around it. It does not say anything about when that store is visible to other cores. It will be visible at some point in time to cores. But, I do not think we need to be worried about flushing to memory.
Quoted 1 line
-- read the "
calls_per_service" counter for each lcores, add them up.
This can be trimmed down to the single core the service is mapped to currently, no need to add all the counters.
Quoted 2 lines
---- Wait :)
-- re-read the "calls_per_service", and ensure the count has changed.
Basically, polling. This causes more traffic on the interconnect between the cores. But might be ok since this API might not be called frequently.
Quoted 6 lines
---- The fact that the
calls_per_servicehas changed ensures the service-
lcore
has seen the new "num_mapped_cores" value, and has now taken the
lock!
-- *now* it is safe to map the 2nd lcore to the service
Quoted 6 lines
There is a caveat here that the increments to the "
calls_per_service" variable must become globally-observable. To force this immediately would require a write memory barrier, which could impact datapath performance. Given the service is now taking a lock, theunlock()thereof would ensure the "calls_per_service"
is flushed to memory.
If we increment this variable only when the lock is held, we should be fine. We could have a separate variable.
Quoted 5 lines
Note: we could use
calls_per_service, or add a new variable to the service struct.
Writes to this do not need to be atomic, as it is either mapped to a single core, or else there's a lock around it.
I think it is better to have a separate variable that is updated only when the lock is held.
I do not see any change in API sequence. We do this hand-shake only if the service is running (which is all controlled in the writer thread), correct?
This does not solve the problem with rte_service_run_iter_on_app_lcore getting called on multiple cores concurrently for the same service.
Quoted 55 lines
Quoted 11 lines
Quoted 11 lines
service has done. If this increments we know that the lcore running the service has re-entered the critical section, so would see an updated "needs atomic" flag.
This approach may introduce a predictable branch on the datapath, however the cost of a predictable branch vs always taking an atomic is order(s?) of magnitude, so a branch is much preferred.
It must be possible to avoid the datapath overhead using a scheme like
this.
Quoted 7 lines
It
Quoted 3 lines
will likely be more complex than your proposed change below, however if it avoids datapath performance drops I feel that a more complex solution is worth investigating at least.
I do not completely understand the approach you are proposing, may be you can elaborate more.
Expanded above, showing a possible solution that does not require additional atomics on the datapath.
Quoted 4 lines
But, it seems to be based on a counter approach. Following is my assessment on what happens if we use a counter. Let us say we kept track of how many cores are running the service currently. We need an atomic counter other than '
num_mapped_cores'. Let us call that counter'
num_current_cores'.Quoted 13 lines
The code to call the service would look like below.
1)
rte_atomic32_inc(&num_current_cores); /* this results in a full memory barrier */
2) if (__atomic_load_n(&num_current_cores, __ATOMIC_ACQUIRE)== 1) { /*rte_atomic_readis not enough here as it does not provide the required memory barrier for any architecture */
3)run_service(); /* Call the service */
4) }
5)rte_atomic32_sub(&num_current_cores); /* Callingrte_atomic32_clearis not enough as it is not an atomic operation and does not provide the required memory barrier */Quoted 4 lines
But the above code has race conditions in lines 1 and 2. It is possible that none of the cores will ever get to run the service as they all could simultaneously increment the counter. Hence lines 1 and 2 together need to be atomic, which is nothing but 'compare-exchange'
operation.
Quoted 7 lines
BTW, the current code has a bug where it calls 'rte_atomic_clear(&s-
Quoted 2 lines
execute_lock)', it is missing memory barriers which results in clearing the
execute_lockbefore the service has completed running. I suggest changing the 'execute_lock' torte_spinlock_tand usingrte_spinlock_try_lockandrte_spinlock_unlockAPIs.
Quoted 6 lines
I don't think a spinlock is what we want here:
The idea is that a service-lcore can be mapped to multiple services. If one service is already being run (by another thread), we do not want to spin here waiting for it to become "free" to run by this thread, it should continue to the next service that it is mapped to.
Agree. I am suggesting to use 'rte_spinlock_try_lock' (does not spin) which is nothing but 'compare-exchange'. Since the API is available, we should make use of it instead of repeating the code.
Quoted 14 lines
Quoted 13 lines
Quoted 5 lines
A unit test is required to validate a fix like this - although perhaps found
by
Quoted 1 line
inspection/review, a real-world test to validate would give confidence.
Agree, need to have a test case.
Quoted 4 lines
Thoughts on such an approach?
<snip patch contents>
Re: [dpdk-dev] [PATCH v3 09/12] service: avoid race condition for MT unsafe service
Quoted 84 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Thursday, April 9, 2020 2:32 AM
To: Van Haaren, Harry <[email protected]>; Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 09/12] service: avoid race condition for MT unsafe service<snip>
Quoted 68 lines
Quoted 67 lines
Quoted 12 lines
Quoted 12 lines
Subject: [PATCH v3 09/12] service: avoid race condition for MT unsafe service
From: Honnappa Nagarahalli <[email protected]>
There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use '
execute_lock' always when the service is MT unsafe.Quoted 8 lines
Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")Signed-off-by Honnappa Nagarahalli <[email protected]>Reviewed-by Phil Yang <[email protected]>Reviewed-by Gavin Hu <[email protected]>We should put "fix" in the title, once converged on an implementation.
Ok, will replace 'avoid' with 'fix' (once we agree on the solution)
Quoted 3 lines
Regarding Fixes and stable backport, we should consider if fixing this in
stable
Quoted 7 lines
with a performance degradation, fixing with more complex solution, or documenting a known issue a better solution.
This fix (always taking the atomic lock) will have a negative performance impact on existing code using services. We should investigate a way to fix
it
Quoted 1 line
without causing datapath performance degradation.
Trying to gauge the impact on the existing applications...
The documentation does not explicitly disallow run time mapping of cores to service.
1) If the applications are mapping the cores to services at run time, they are running with a bug. IMO, bug fix resulting in a performance drop should be acceptable.
2) If the service is configured to run on single core (num_mapped_cores== 1), but service is set to MT unsafe - this will have a (possible) performance impact.
a) This can be solved by setting the service to MT safe and can be
documented. This might be a reasonable solution for applications which are compiling with
future DPDK releases.
b) We can also solve this using symbol versioning - the old version
of this function will use the old code, the new version of this function will use the code in
this patch. So, if the application is run with
future DPDK releases without recompiling, it will continue to use the old version. If the application is compiled
with future releases, they can use solution in 2a.
We also should think if this is an appropriate solution as this would force 1) to recompile to get the fix.
3) If the service is configured to run on multiple cores (num_mapped_cores> 1), then for those applications, the lock is being taken already. These applications might see some improvements as this patch removes few instructions.Quoted 4 lines
I think there is a way to achieve this by moving more checks/time to the control path (lcore updating the map), and not forcing the datapath lcore to always take an atomic.
I think 2a above is the solution.
Quoted 7 lines
Quoted 1 line
2a above is e.g. the Eventdev SW routines like Rx/Tx scheduler services.
I scanned through the code briefly
I see that Eth RX/TX, Crypto adapters are setting theMT_SAFEcapabilities, can be ignored.
Timer adaptor and some others do not setMT_SAFE. Seems like the cores to run on are mapped during run time. But it is not clear to me if it can get mapped to run on multiple cores. If they are, they are running with the bug.
EAL will map each service to a single lcore. It will "round-robin" if there are more services than service-lcores to run them on. So agree that DPDK's default mappings will not suffer this issue.
Quoted 37 lines
But, these are all internal to DPDK and can be fixed. Are there no performance tests in these components that we can run?
Quoted 17 lines
We should strive to not reduce datapath performance at all here.
Quoted 6 lines
Quoted 2 lines
In this particular case, we have a counter for number of iterations that a
Which counter are you thinking about?
All the counters I checked are not atomic operations currently. If we are going to use counters they have to be atomic, which means additional cycles in the data path.I'll try to explain the concept better, take this example:
- One service core is mapped to aMT_UNSAFEservice, like event/sw pmd
- Application wants to map a 2nd lcore to the same service
- You point out that today there is a race over the lock
-- the lock is not taken if (num_mapped_lcores== 1)
-- this avoids an atomic lock/unlock on the datapathQuoted 4 lines
To achieve our desired goal;
- control thread doing mapping performs the following operations-- write service->num_mapped_lcores++ (atomic not required, only single-writer allowed by APIs)
This has to be atomic because of
rte_service_run_iter_on_app_lcoreAPI. Performance should be fine as this API is not called frequently. But need to consider the implications of more than one thread updatingnum_mapped_cores.Quoted 1 line
-- MFENCE (full st-ld barrier) to flush write, and force later loads to
issue
Quoted 1 line
after
I am not exactly sure what MFENCE on x86 does. On Arm platforms, the full barrier (DMB ISH) just makes sure that memory operations are not re-ordered around it. It does not say anything about when that store is visible to other cores. It will be visible at some point in time to cores.
But, I do not think we need to be worried about flushing to memory.
Quoted 3 lines
Quoted 1 line
-- read the "
calls_per_service" counter for each lcores, add them up.This can be trimmed down to the single core the service is mapped to currently, no need to add all the counters.
Correct - however that requires figuring out which lcore is running the service. Anyway, agree - it's an implementation detail as to exactly how we detect it.
Quoted 5 lines
Quoted 3 lines
---- Wait :)
-- re-read the "calls_per_service", and ensure the count has changed.Basically, polling. This causes more traffic on the interconnect between the cores. But might be ok since this API might not be called frequently.
Agree this will not be called frequently, and that some polling here will not be a problem.
Quoted 8 lines
Quoted 1 line
---- The fact that the
calls_per_servicehas changed ensures theservice-
Quoted 6 lines
lcore
has seen the new "num_mapped_cores" value, and has now taken the
lock!
-- *now* it is safe to map the 2nd lcore to the serviceThere is a caveat here that the increments to the "
calls_per_service"
Quoted 8 lines
variable
Quoted 5 lines
must become globally-observable. To force this immediately would require a write memory barrier, which could impact datapath performance. Given the service is now taking a lock, the
unlock()thereof would ensure the "calls_per_service"
is flushed to memory.If we increment this variable only when the lock is held, we should be fine. We could have a separate variable.
Sure, if a separate variable is preferred that's fine with me.
Quoted 9 lines
Quoted 3 lines
Note: we could use
calls_per_service, or add a new variable to the service struct.
Writes to this do not need to be atomic, as it is either mapped to a singlecore,
Quoted 1 line
or else there's a lock around it.
I think it is better to have a separate variable that is updated only when the lock is held.
I do not see any change in API sequence. We do this hand-shake only if the service is running (which is all controlled in the writer thread), correct?
Yes this increment can be localized to just the branch when the unlock() occurs, as that is the only time it could make a difference.
Quoted 2 lines
This does not solve the problem with
rte_service_run_iter_on_app_lcoregetting called on multiple cores concurrently for the same service.
Agreed. This "on_app_lcore" API was an addition required to enable unit-testing in a sane way, to run iterations of eg Eventdev PMD.
I am in favor of documenting that the application is responsible to ensure the service being run on a specific application lcore is not concurrently running on another application lcore.
Quoted 57 lines
Quoted 49 lines
Quoted 9 lines
Quoted 9 lines
service has done. If this increments we know that the lcore running the service has re-entered the critical section, so would see an updated "needs atomic" flag.
This approach may introduce a predictable branch on the datapath, however the cost of a predictable branch vs always taking an atomic is order(s?) of magnitude, so a branch is much preferred.
It must be possible to avoid the datapath overhead using a scheme like
this.
Quoted 7 lines
It
Quoted 3 lines
will likely be more complex than your proposed change below, however if it avoids datapath performance drops I feel that a more complex solution is worth investigating at least.
I do not completely understand the approach you are proposing, may be you can elaborate more.
Expanded above, showing a possible solution that does not require additional atomics on the datapath.
Quoted 4 lines
But, it seems to be based on a counter approach. Following is my assessment on what happens if we use a counter. Let us say we kept track of how many cores are running the service currently. We need an atomic counter other than '
num_mapped_cores'. Let us call that counter'
num_current_cores'.Quoted 13 lines
The code to call the service would look like below.
1)
rte_atomic32_inc(&num_current_cores); /* this results in a full memory barrier */
2) if (__atomic_load_n(&num_current_cores, __ATOMIC_ACQUIRE)== 1) { /*rte_atomic_readis not enough here as it does not provide the required memory barrier for any architecture */
3)run_service(); /* Call the service */
4) }
5)rte_atomic32_sub(&num_current_cores); /* Callingrte_atomic32_clearis not enough as it is not an atomic operation and does not provide the required memory barrier */Quoted 4 lines
But the above code has race conditions in lines 1 and 2. It is possible that none of the cores will ever get to run the service as they all could simultaneously increment the counter. Hence lines 1 and 2 together need to be atomic, which is nothing but 'compare-exchange'
operation.
Quoted 4 lines
BTW, the current code has a bug where it calls 'rte_atomic_clear(&s-
Quoted 2 lines
execute_lock)', it is missing memory barriers which results in clearing theQuoted 8 lines
Quoted 3 lines
execute_lockbefore the service has completed running. I suggest changing the 'execute_lock' torte_spinlock_tand usingrte_spinlock_try_lockandrte_spinlock_unlockAPIs.I don't think a spinlock is what we want here:
The idea is that a service-lcore can be mapped to multiple services. If one service is already being run (by another thread), we do not want to
Quoted 7 lines
spin
Quoted 1 line
here waiting for it to become "free" to run by this thread, it should
continue
Quoted 1 line
to the next service that it is mapped to.
Agree. I am suggesting to use '
rte_spinlock_try_lock' (does not spin) which is nothing but 'compare-exchange'. Since the API is available, we should make use of it instead of repeating the code.
Ah apologies, I misread the spinlock usage. Sure if the spinlock_t code is preferred I'm ok with a change. It would be clean to have a separate patch in the patchset to make this change, and have it later in the set than the changes for backporting to ease integration with stable branch.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 2 lines
A unit test is required to validate a fix like this - although perhaps found
by
Quoted 1 line
inspection/review, a real-world test to validate would give confidence.
Agree, need to have a test case.
Quoted 4 lines
Thoughts on such an approach?
<snip patch contents>
Re: [dpdk-dev] [PATCH v3 09/12] service: avoid race condition for MT unsafe service
<snip>
Quoted 79 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Subject: [PATCH v3 09/12] service: avoid race condition for MT unsafe service
From: Honnappa Nagarahalli <[email protected]>
There has possible that a MT unsafe service might get configured to run on another core while the service is running
currently.
Quoted 63 lines
Quoted 63 lines
Quoted 62 lines
Quoted 8 lines
Quoted 8 lines
This might result in the MT unsafe service running on multiple cores simultaneously. Use '
execute_lock' always when the service is MT unsafe.Fixes e9139a32f6e8 ("service: add function to run on applcore")
Cc: [email protected]Quoted 6 lines
Quoted 4 lines
Signed-off-by: Honnappa Nagarahalli
<[email protected]>Reviewed-by Phil Yang <[email protected]>Reviewed-by Gavin Hu <[email protected]>We should put "fix" in the title, once converged on an implementation.
Ok, will replace 'avoid' with 'fix' (once we agree on the solution)
Quoted 3 lines
Regarding Fixes and stable backport, we should consider if fixing this in
stable
Quoted 7 lines
with a performance degradation, fixing with more complex solution, or documenting a known issue a better solution.
This fix (always taking the atomic lock) will have a negative performance impact on existing code using services. We should investigate a way to fix
it
Quoted 1 line
without causing datapath performance degradation.
Trying to gauge the impact on the existing applications... The documentation does not explicitly disallow run time mapping of cores to service.
1) If the applications are mapping the cores to services at run time, they are running with a bug. IMO, bug fix resulting in a performance drop should be acceptable.
2) If the service is configured to run on single core (num_mapped_cores== 1), but service is set to MT unsafe - this will have a (possible) performance impact.
a) This can be solved by setting the service to MT safe and can
be documented. This might be a reasonable solution for applications which are compiling with
future DPDK releases.
b) We can also solve this using symbol versioning - the old
version of this function will use the old code, the new version of this function will use the code in
this patch. So, if the application is run with
future DPDK releases without recompiling, it will continue to use the old version. If the application is compiled
with future releases, they can use solution in 2a.
We also should think if this is an appropriate solution as this would force 1) to recompile to get the fix.
3) If the service is configured to run on multiple cores (num_mapped_cores> 1), then for those applications, the lock is being taken already. These applications might see some improvements as this patch removes few instructions.Quoted 4 lines
I think there is a way to achieve this by moving more checks/time to the control path (lcore updating the map), and not forcing the datapath lcore to always take an atomic.
I think 2a above is the solution.
Quoted 7 lines
Quoted 1 line
2a above is e.g. the Eventdev SW routines like Rx/Tx scheduler services.
I scanned through the code briefly
I see that Eth RX/TX, Crypto adapters are setting theMT_SAFEcapabilities, can be ignored.
Timer adaptor and some others do not setMT_SAFE. Seems like the cores to run on are mapped during run time. But it is not clear to me if it can get mapped to run on multiple cores. If they are, they are running with
Quoted 21 lines
the bug.
EAL will map each service to a single lcore. It will "round-robin" if there are more services than service-lcores to run them on. So agree that DPDK's default mappings will not suffer this issue.
Quoted 14 lines
But, these are all internal to DPDK and can be fixed. Are there no performance tests in these components that we can run?
Quoted 11 lines
We should strive to not reduce datapath performance at all here.
Quoted 6 lines
Quoted 2 lines
In this particular case, we have a counter for number of iterations that a
Which counter are you thinking about?
All the counters I checked are not atomic operations currently. If we are going to use counters they have to be atomic, which means additional cycles in the data path.I'll try to explain the concept better, take this example:
I tried to implement this algorithm, but there are few issues, please see below.
Quoted 26 lines
Quoted 7 lines
Quoted 7 lines
- One service core is mapped to a
MT_UNSAFEservice, like event/sw
pmd
- Application wants to map a 2nd lcore to the same service
- You point out that today there is a race over the lock -- the lock is not taken if (num_mapped_lcores== 1) -- this avoids an atomic lock/unlock on the datapathQuoted 7 lines
Quoted 4 lines
To achieve our desired goal;
- control thread doing mapping performs the following operations-- write service->num_mapped_lcores++ (atomic not required, onlysingle- writer allowed by APIs)
This has to be atomic because of
rte_service_run_iter_on_app_lcoreAPI. Performance should be fine as this API is not called frequently. But need to consider the implications of more than one thread updating
num_mapped_cores.Quoted 11 lines
Quoted 3 lines
-- MFENCE (full st-ld barrier) to flush write, and force later
loads toissue
Quoted 1 line
after
I am not exactly sure what MFENCE on x86 does. On Arm platforms, the full barrier (DMB ISH) just makes sure that memory operations are not re-ordered around it. It does not say anything about when that store is visible to other cores. It will be visible at some point in time to cores. But, I do not think we need to be worried about flushing to memory.
Quoted 10 lines
Quoted 3 lines
Quoted 1 line
-- read the "
calls_per_service" counter for each lcores, add them up.This can be trimmed down to the single core the service is mapped to currently, no need to add all the counters.
Correct - however that requires figuring out which lcore is running the service. Anyway, agree - it's an implementation detail as to exactly how we detect it.
Quoted 3 lines
Quoted 3 lines
---- Wait :)
-- re-read the "calls_per_service", and ensure the count has changed.
Here, there is an assumption that the service core function is running on the service core. If the service core is not running, the code will be stuck in this polling loop.
I could not come up with a good way to check if the service core is running. Checking the app_runstate and comp_runstate is not enough as they just indicate that the service is ready to run. Using the counter 'calls_per_service' introduces race conditions.
Only way I can think of is asking the user to follow a specific sequence of APIs to ensure the service core is running before calling rte_service_map_lcore_set.
Quoted 143 lines
Quoted 2 lines
Basically, polling. This causes more traffic on the interconnect between the cores. But might be ok since this API might not be called
frequently.
Agree this will not be called frequently, and that some polling here will not be a problem.
Quoted 8 lines
Quoted 1 line
---- The fact that the
calls_per_servicehas changed ensures theservice-
Quoted 6 lines
lcore
has seen the new "num_mapped_cores" value, and has now
taken the lock!
-- *now* it is safe to map the 2nd lcore to the serviceThere is a caveat here that the increments to the "
calls_per_service"Quoted 8 lines
variable
Quoted 5 lines
must become globally-observable. To force this immediately would require a write memory barrier, which could impact datapath performance. Given the service is now taking a lock, the
unlock()thereof would ensure the "calls_per_service"
is flushed to memory.If we increment this variable only when the lock is held, we should be fine. We could have a separate variable.
Sure, if a separate variable is preferred that's fine with me.
Quoted 10 lines
Quoted 4 lines
Note: we could use
calls_per_service, or add a new variable to the service struct.
Writes to this do not need to be atomic, as it is either mapped to a singlecore,
Quoted 1 line
or else there's a lock around it.
I think it is better to have a separate variable that is updated only when the lock is held.
I do not see any change in API sequence. We do this hand-shake only if the service is running (which is all controlled in the writer thread), correct?Yes this increment can be localized to just the branch when the
unlock()occurs, as that is the only time it could make a difference.Quoted 2 lines
This does not solve the problem with
rte_service_run_iter_on_app_lcoregetting called on multiple cores concurrently for the same service.Agreed. This "
on_app_lcore" API was an addition required to enable unit- testing in a sane way, to run iterations of eg Eventdev PMD.I am in favor of documenting that the application is responsible to ensure the service being run on a specific application lcore is not concurrently running on another application lcore.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
service has done. If this increments we know that the lcore running the service has re-entered the critical section, so would see an updated "needs atomic" flag.
This approach may introduce a predictable branch on the datapath, however the cost of a predictable branch vs always taking an atomic is order(s?) of magnitude, so a branch is much
preferred.
Quoted 56 lines
Quoted 51 lines
Quoted 3 lines
Quoted 3 lines
It must be possible to avoid the datapath overhead using a scheme like
this.
Quoted 7 lines
It
Quoted 3 lines
will likely be more complex than your proposed change below, however if it avoids datapath performance drops I feel that a more complex solution is worth investigating at least.
I do not completely understand the approach you are proposing, may be you can elaborate more.
Expanded above, showing a possible solution that does not require additional atomics on the datapath.
Quoted 5 lines
But, it seems to be based on a counter approach. Following is my assessment on what happens if we use a counter. Let us say we kept track of how many cores are running the service currently. We need an atomic counter other than '
num_mapped_cores'. Let us call that counter'
num_current_cores'.Quoted 14 lines
The code to call the service would look like below.
1)
rte_atomic32_inc(&num_current_cores); /* this results in a full memory barrier */
2) if (__atomic_load_n(&num_current_cores, __ATOMIC_ACQUIRE)== 1) {
/*rte_atomic_readis not enough here as it does not provide the required memory barrier for any architecture */
3)run_service(); /* Call the service */
4) }
5)rte_atomic32_sub(&num_current_cores); /* Callingrte_atomic32_clearis not enough as it is not an atomic operation and does not provide the required memory barrier */Quoted 5 lines
But the above code has race conditions in lines 1 and 2. It is possible that none of the cores will ever get to run the service as they all could simultaneously increment the counter. Hence lines 1 and
2 together need to be atomic, which is nothing but 'compare-exchange'operation.
Quoted 8 lines
BTW, the current code has a bug where it calls
'rte_atomic_clear(&s-Quoted 2 lines
execute_lock)', it is missing memory barriers which results in clearing the
execute_lockbefore the service has completed running. I suggest changing the 'execute_lock' torte_spinlock_tand usingrte_spinlock_try_lockandrte_spinlock_unlockAPIs.Quoted 5 lines
I don't think a spinlock is what we want here:
The idea is that a service-lcore can be mapped to multiple services. If one service is already being run (by another thread), we do not want to
Quoted 8 lines
spin
Quoted 2 lines
here waiting for it to become "free" to run by this thread, it should
continue
Quoted 1 line
to the next service that it is mapped to.
Agree. I am suggesting to use '
rte_spinlock_try_lock' (does not spin) which is nothing but 'compare-exchange'. Since the API is available, we should make use of it instead of repeating the code.Ah apologies, I misread the spinlock usage. Sure if the
spinlock_tcode is preferred I'm ok with a change. It would be clean to have a separate patch in the patchset to make this change, and have it later in the set than the changes for backporting to ease integration with stable branch.Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 2 lines
A unit test is required to validate a fix like this - although perhaps found
by
Quoted 1 line
inspection/review, a real-world test to validate would give confidence.
Agree, need to have a test case.
Quoted 4 lines
Thoughts on such an approach?
<snip patch contents>
Re: [dpdk-dev] [PATCH v3 09/12] service: avoid race condition for MT unsafe service
Quoted 14 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, April 18, 2020 7:22 AM
To: Van Haaren, Harry <[email protected]>; Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 09/12] service: avoid race condition for MT unsafe service
Quoted 1 line
<snip>
<snip snip>
Quoted 27 lines
Quoted 27 lines
Quoted 7 lines
Quoted 4 lines
To achieve our desired goal;
- control thread doing mapping performs the following operations-- write service->num_mapped_lcores++ (atomic not required, onlysingle- writer allowed by APIs)
This has to be atomic because of
rte_service_run_iter_on_app_lcoreAPI. Performance should be fine as this API is not called frequently. But need to consider the implications of more than one thread updating
num_mapped_cores.Quoted 11 lines
Quoted 3 lines
-- MFENCE (full st-ld barrier) to flush write, and force later
loads toissue
Quoted 1 line
after
I am not exactly sure what MFENCE on x86 does. On Arm platforms, the full barrier (DMB ISH) just makes sure that memory operations are not re-ordered around it. It does not say anything about when that store is visible to other cores. It will be visible at some point in time to cores. But, I do not think we need to be worried about flushing to memory.
Quoted 3 lines
Quoted 1 line
-- read the "
calls_per_service" counter for each lcores, add them up.This can be trimmed down to the single core the service is mapped to currently, no need to add all the counters.
Correct - however that requires figuring out which lcore is running the service. Anyway, agree - it's an implementation detail as to exactly how we detect it.
Quoted 1 line
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
---- Wait :)
-- re-read the "calls_per_service", and ensure the count has changed.Here, there is an assumption that the service core function is running on the service core. If the service core is not running, the code will be stuck in this polling loop.
Right - we could add a timeout ehre, but that just moves the problem somewhere else (the application) which now needs to handle error rets, and possibly retries. It could be a possible solution.. I'm not in favour of it at the moment, but it needs some more time to think.
Quoted 7 lines
I could not come up with a good way to check if the service core is running. Checking the
app_runstateandcomp_runstateis not enough as they just indicate that the service is ready to run. Using the counter 'calls_per_service' introduces race conditions.Only way I can think of is asking the user to follow a specific sequence of APIs to ensure the service core is running before calling
rte_service_map_lcore_set.
Good point - I'm thinking about this - but haven't come to an obvious conclusion yet. I'm considering other ways to detect the core is/isn't running, and also considering just "high-jacking" the service function pointer temporarily with a CAS, which gives some new options on avoiding threads entering the critical section.
As above, I don't have a good solution yet.
<snip irrelevant to above discussion stuff>
PATCH v3 10/12 service: identify service running on another core correctly
In reply to Phil YangFrom: Honnappa Nagarahalli <[email protected]>
The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases.
--- lib/librte_eal/common/rte_service.c | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
@@ -360,7 +360,7 @@ service_runner_do_callback(struct rte_service_spec_impl *s,/* Expects the service 's' is valid. */ static int32_t service_run (uint32_t i,struct core_state *cs,uint64_t service_mask, -struct rte_service_spec_impl *s) +struct rte_service_spec_impl *s,uint32_t serialize_mt_unsafe) {if (!s)return -EINVAL ;
@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -if (service_mt_safe (s) ==0 ) { +if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); -/* Atomically add this core to the mapped cores first, then examine if -* we can run the service. This avoids a race condition between -* checking the value, and atomically adding to the mapped count. +/* Increment num_mapped_cores to indicate that the service +* is running on a core. */ -if (serialize_mt_unsafe) -rte_atomic32_inc (&s->num_mapped_cores); +rte_atomic32_inc (&s->num_mapped_cores); -if (service_mt_safe (s) ==0 && -rte_atomic32_read (&s->num_mapped_cores) >1 ) { -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); -return -EBUSY ; - } - -int ret =service_run (id, cs,UINT64_MAX , s); +int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); +rte_atomic32_dec (&s->num_mapped_cores);return ret; }
@@ -449,7 +439,7 @@ service_runner_func(void *arg)if (!service_valid (i))continue ;/* return value ignored as no change to code flow */ -service_run (i, cs, service_mask,service_get (i)); +service_run (i, cs, service_mask,service_get (i),1 ); } cs->loops++;
Signature · 1 line
2.7.4
Re: [dpdk-dev] [PATCH v3 10/12] service: identify service running on another core correctly
Quoted 20 lines
From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: [PATCH v3 10/12] service: identify service running on another core correctlyFrom: Honnappa Nagarahalli <[email protected]>
The logic to identify if the MT unsafe service is running on another core can return
-EBUSYspuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases.
Quoted 5 lines
Signed-off-by Honnappa Nagarahalli <[email protected]>Reviewed-by Phil Yang <[email protected]>Reviewed-by Gavin Hu <[email protected]>
Is this fixing broken functionality, or does it aim to only "optimize"? Lack of "fixes" tag suggests optimization.
I'm cautious about the commit phrase "Assume that the application ...", if the code was previously checking things, we must not stop checking them now, this may introduce race-conditions in existing applications?
It seems like the "serialize_mt_unsafe" branch is being pushed further down the callgraph, and instead of branching over atomics this patch forces always executing 2 atomics?
This feels like too specific an optimization/tradeoff, without data to backup that there are no regressions on any DPDK supported platforms.
DPDK today doesn't have a micro-benchmark to gather such perf data, but I would welcome one and we can have a data-driven decision.
Hope this point-of-view makes sense, -Harry
Quoted 71 lines
--- lib/librte_eal/common/rte_service.c | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-)lib/librte_eal/common/rte_service.c +1
-1@@ -360,7 +360,7 @@ service_runner_do_callback(struct rte_service_spec_impl *s,/* Expects the service 's' is valid. */ static int32_tservice_run (uint32_t i,struct core_state *cs,uint64_t service_mask, -struct rte_service_spec_impl *s) +struct rte_service_spec_impl *s,uint32_t serialize_mt_unsafe) {if (!s)lib/librte_eal/common/rte_service.c +6
-16return -EINVAL ;@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ; -if (service_mt_safe (s) ==0 ) { +if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL); -/* Atomically add this core to the mapped cores first, then examine if -* we can run the service. This avoids a race condition between -* checking the value, and atomically adding to the mapped count. +/* Increment num_mapped_cores to indicate that the service +* is running on a core. */ -if (serialize_mt_unsafe) -rte_atomic32_inc (&s->num_mapped_cores); +rte_atomic32_inc (&s->num_mapped_cores); -if (service_mt_safe (s) ==0 && -rte_atomic32_read (&s->num_mapped_cores) >1 ) { -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); -return -EBUSY ; - } - -int ret =service_run (id, cs,UINT64_MAX , s); +int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); +rte_atomic32_dec (&s->num_mapped_cores);return ret;}
lib/librte_eal/common/rte_service.c +1
-2@@ -449,7 +439,7 @@ service_runner_func(void *arg)if (!service_valid (i))continue ;/* return value ignored as no change to code flow */ -service_run (i, cs, service_mask,service_get (i)); +service_run (i, cs, service_mask,service_get (i),1 ); } cs->loops++; --2.7.4
Re: [dpdk-dev] [PATCH v3 10/12] service: identify service running on another core correctly
<snip>
Quoted 11 lines
Quoted 11 lines
Subject: [PATCH v3 10/12] service: identify service running on another core correctly
From: Honnappa Nagarahalli <[email protected]>
The logic to identify if the MT unsafe service is running on another core can return
-EBUSYspuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases.
Quoted 8 lines
Quoted 5 lines
Signed-off-by Honnappa Nagarahalli <[email protected]>Reviewed-by Phil Yang <[email protected]>Reviewed-by Gavin Hu <[email protected]>Is this fixing broken functionality, or does it aim to only "optimize"? Lack of "fixes" tag suggests optimization.
Good point, it is missing the 'Fixes' tag. Relooking at the code, following are few problems that I observe, please correct me if I am wrong:
1) Looking at the API rte_service_map_lcore_set, 'num_mapped_cores' keeps track of the number of cores the service is mapped to. However, in the function 'rte_service_run_iter_on_app_lcore', the 'num_mapped_cores' is incremented only if 'serialize_mt_unsafe' is set. This will return incorrect result in 'rte_service_runstate_get' API when 'serialize_mt_unsafe' is not set.
2) Since incrementing 'num_mapped_cores' and checking its value is not an atomic operation, it introduces race conditions. Consider the current code snippet from the function 'rte_service_run_iter_on_app_lcore'.
1) if (serialize_mt_unsafe)
2) rte_atomic32_inc(&s->num_mapped_cores);
3) if (service_mt_safe(s) == 0 && rte_atomic32_read(&s->num_mapped_cores) > 1) {
4) if (serialize_mt_unsafe)
5) rte_atomic32_dec(&s->num_mapped_cores);
6) return -EBUSY;
7) }
It is possible that more than 1 thread is executing line 3) concurrently, in which case all of them will hit line 6). Due to this, it is possible that the service might never run or it might take multiple attempts to run the service wasting cycles.
If you agree these are bugs, I can add 'fixes' tag.
Quoted 4 lines
I'm cautious about the commit phrase "Assume that the application ...", if the code was previously checking things, we must not stop checking them now, this may introduce race-conditions in existing applications?
What I meant here is, let us believe in what the application says. i.e. if the applications sets 'serialize_mt_unsafe' to 1, let us assume that the service is infact 'unsafe'.
Quoted 4 lines
It seems like the "
serialize_mt_unsafe" branch is being pushed further down the callgraph, and instead of branching over atomics this patch forces always executing 2 atomics?
Yes, that is correct. Please see explanation in 1) above.
Quoted 3 lines
This feels like too specific an optimization/tradeoff, without data to backup that there are no regressions on any DPDK supported platforms.
Apologies for missing the 'Fixes' tag, this patch is a bug fix.
Quoted 3 lines
DPDK today doesn't have a micro-benchmark to gather such perf data, but I would welcome one and we can have a data-driven decision.
When this was developed, was the logic to avoid taking the lock measured to provide any improvements?
Quoted 74 lines
Hope this point-of-view makes sense, -Harry
Quoted 71 lines
--- lib/librte_eal/common/rte_service.c | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-)lib/librte_eal/common/rte_service.c +2
-2@@ -360,7 +360,7 @@ service_runner_do_callback(struct rte_service_spec_impl *s, /* Expects the service 's' is valid. */ static int32_t service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, - struct rte_service_spec_impl *s) + struct rte_service_spec_impl *s, uint32_t serialize_mt_unsafe) { if (!s) return -EINVAL;@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs,uint64_t service_mask, cs->service_active_on_lcore[i] = 1; -if (service_mt_safe (s) ==0 ) { +if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) {if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 ))return -EBUSY ;@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id,
uint32_tserialize_mt_unsafe)lib/librte_eal/common/rte_service.c +6
-17SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); -/* Atomically add this core to the mapped cores first, then examine if -* we can run the service. This avoids a race condition between -* checking the value, and atomically adding to the mapped count. +/* Increment num_mapped_cores to indicate that the service +* is running on a core. */ -if (serialize_mt_unsafe) -rte_atomic32_inc (&s->num_mapped_cores); +rte_atomic32_inc (&s->num_mapped_cores); -if (service_mt_safe (s) ==0 && -rte_atomic32_read (&s->num_mapped_cores) >1 ) { -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); -return -EBUSY ; - } - -int ret =service_run (id, cs,UINT64_MAX , s); +int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -if (serialize_mt_unsafe) -rte_atomic32_dec (&s->num_mapped_cores); +rte_atomic32_dec (&s->num_mapped_cores);return ret; }@@ -449,7 +439,7 @@ service_runner_func(void *arg)if (!service_valid (i))continue ;/* return value ignored as no change to code flow */ -service_run (i, cs, service_mask,service_get (i)); +service_run (i, cs, service_mask,service_get (i),1 ); } cs->loops++; --2.7.4
PATCH v3 11/12 service: optimize with c11 one-way barrier
In reply to Phil YangThe num_mapped_cores and execute_lock are synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.
--- lib/librte_eal/common/rte_service.c | 50 ++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 15 deletions(-)
lib/librte_eal/common/rte_service.c +35 -15
@@ -42,7 +42,7 @@ struct rte_service_spec_impl {* running this service callback. When not set, a core may take the * lock and then run the service callback. */ -rte_atomic32_t execute_lock; +uint32_t execute_lock;/* API set/get-able variables */ int8_t app_runstate;
@@ -54,7 +54,7 @@ struct rte_service_spec_impl {* It does not indicate the number of cores the service is running * on currently. */ -rte_atomic32_t num_mapped_cores; +int32_t num_mapped_cores;uint64_t calls;uint64_t cycles_spent; } __rte_cache_aligned;
@@ -332,7 +332,8 @@ rte_service_runstate_get(uint32_t id)rte_smp_rmb ();int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); -int lcore_mapped = (rte_atomic32_read (&s->num_mapped_cores) >0 ); +int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, + __ATOMIC_RELAXED) >0 );return (s->app_runstate ==RUNSTATE_RUNNING ) && (s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -375,11 +376,20 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ;if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) { -if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 )) +uint32_t expected =0 ; +/* ACQUIRE ordering here is to prevent the callback +* function from hoisting up before the execute_lock +* setting. +*/ +if (!__atomic_compare_exchange_n (&s->execute_lock, &expected,1 , +0 , __ATOMIC_ACQUIRE, __ATOMIC_RELAXED))return -EBUSY ;service_runner_do_callback (s, cs, i); -rte_atomic32_clear (&s->execute_lock); +/* RELEASE ordering here is used to pair with ACQUIRE +* above to achieve lock semantic. +*/ +__atomic_store_n (&s->execute_lock,0 , __ATOMIC_RELEASE); }else service_runner_do_callback (s, cs, i);
@@ -415,11 +425,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)/* Increment num_mapped_cores to indicate that the service * is running on a core. */ -rte_atomic32_inc (&s->num_mapped_cores); +__atomic_add_fetch (&s->num_mapped_cores,1 , __ATOMIC_ACQUIRE);int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -rte_atomic32_dec (&s->num_mapped_cores); +__atomic_sub_fetch (&s->num_mapped_cores,1 , __ATOMIC_RELEASE);return ret; }
@@ -552,24 +562,32 @@ service_update(uint32_t sid, uint32_t lcore,uint64_t sid_mask =UINT64_C (1 ) << sid;if (set) { -uint64_t lcore_mapped = lcore_states[lcore].service_mask & - sid_mask; +/* When multiple threads try to update the same lcore +* service concurrently, e.g. set lcore map followed +* by clear lcore map, the unsynchronized service_mask +* values have issues on the num_mapped_cores value +* consistency. So we use ACQUIRE ordering to pair with +* the RELEASE ordering to synchronize the service_mask. +*/ +uint64_t lcore_mapped =__atomic_load_n ( + &lcore_states[lcore].service_mask, + __ATOMIC_ACQUIRE) & sid_mask;if (*set && !lcore_mapped) { lcore_states[lcore].service_mask |= sid_mask; -rte_atomic32_inc (&rte_services[sid].num_mapped_cores); +__atomic_add_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELEASE); }if (!*set && lcore_mapped) { lcore_states[lcore].service_mask &= ~(sid_mask); -rte_atomic32_dec (&rte_services[sid].num_mapped_cores); +__atomic_sub_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELEASE); } }if (enabled) *enabled = !!(lcore_states[lcore].service_mask & (sid_mask)); -rte_smp_wmb (); -return 0 ; }
@@ -625,7 +643,8 @@ rte_service_lcore_reset_all(void) } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) -rte_atomic32_set (&rte_services[i].num_mapped_cores,0 ); +__atomic_store_n (&rte_services[i].num_mapped_cores,0 , + __ATOMIC_RELAXED);rte_smp_wmb ();
@@ -708,7 +727,8 @@ rte_service_lcore_stop(uint32_t lcore)int32_t enabled = service_mask & (UINT64_C (1 ) << i);int32_t service_running =rte_service_runstate_get (i);int32_t only_core = (1 == -rte_atomic32_read (&rte_services[i].num_mapped_cores)); +__atomic_load_n (&rte_services[i].num_mapped_cores, + __ATOMIC_RELAXED));/* if the core is mapped, and the service is running, and this * is the only core that is mapped, the service would cease to
Signature · 1 line
2.7.4
Re: [dpdk-dev] [PATCH v3 11/12] service: optimize with c11 one-way barrier
Quoted 11 lines
-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 11/12] service: optimize with c11 one-way barrier
Quoted 8 lines
The
num_mapped_coresandexecute_lockare synchronized withrte_atomic_XXAPIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>
Based on discussion on-list, it seems the consensus is to not use GCC builtins, but instead use C11 APIs "proper"? If my conclusion is correct, the v+1 of this patchset would require updates to that style API.
Inline comments for context below, -Harry
Quoted 20 lines
--- lib/librte_eal/common/rte_service.c | 50 ++++++++++++++++++++++++++----------(quoted diff) +0
-1- 1 file changed, 35 insertions(+), 15 deletions(-)lib/librte_eal/common/rte_service.c +1
-1@@ -42,7 +42,7 @@ struct rte_service_spec_impl {* running this service callback. When not set, a core may take the * lock and then run the service callback. */ -rte_atomic32_t execute_lock; +uint32_t execute_lock;/* API set/get-able variables */ int8_t app_runstate;
lib/librte_eal/common/rte_service.c struct rte_service_spec_impl { · line 57
Quoted 5 lines
lib/librte_eal/common/rte_service.c +1
-1@@ -54,7 +54,7 @@ struct rte_service_spec_impl {* It does not indicate the number of cores the service is running * on currently. */ -rte_atomic32_t num_mapped_cores; +int32_t num_mapped_cores;
Any reason why "int32_t" or "uint32_t" is used over another? execute_lock is a uint32_t above, num_mapped_cores is an int32_t?
Quoted 17 lines
lib/librte_eal/common/rte_service.c +2
-1uint64_t calls;uint64_t cycles_spent; } __rte_cache_aligned;@@ -332,7 +332,8 @@ rte_service_runstate_get(uint32_t id)rte_smp_rmb ();int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); -int lcore_mapped = (rte_atomic32_read (&s->num_mapped_cores) >0 ); +int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, + __ATOMIC_RELAXED) >0 );return (s->app_runstate ==RUNSTATE_RUNNING ) && (s->comp_runstate == RUNSTATE_RUNNING) &&@@ -375,11 +376,20 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] =1 ;
lib/librte_eal/common/rte_service.c service_run(uint32_t i, struct core_state *cs, uint64_t · line 387
Quoted 10 lines
lib/librte_eal/common/rte_service.c +7
-1if ((service_mt_safe (s) ==0 ) && (serialize_mt_unsafe ==1 )) { -if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 )) +uint32_t expected =0 ; +/* ACQUIRE ordering here is to prevent the callback +* function from hoisting up before the execute_lock +* setting. +*/ +if (!__atomic_compare_exchange_n (&s->execute_lock, &expected,1 , +0 , __ATOMIC_ACQUIRE, __ATOMIC_RELAXED))return -EBUSY ;
Let's try improve the magic "1" and "0" constants, I believe the "1" here
is the desired "new value on success", and the 0 is "bool weak", where our 0/false constant implies a strongly ordered compare exchange?
"Weak is true for weak compare_exchange, which may fail spuriously, and false for the strong variation, which never fails spuriously.", from https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html
const uint32_t on_success_value = 1;
const bool weak = 0;__atomic_compare_exchange_n(&s->execute_lock, &expected, on_success_value, weak, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED);
Although a bit more verbose, I feel this documents usage a lot better, particularly for those who aren't as familiar with the C11 function arguments order.
Admittedly with the API change to not use __builtins, perhaps this comment is moot.
Quoted 31 lines
lib/librte_eal/common/rte_service.c +6
-3service_runner_do_callback (s, cs, i); -rte_atomic32_clear (&s->execute_lock); +/* RELEASE ordering here is used to pair with ACQUIRE +* above to achieve lock semantic. +*/ +__atomic_store_n (&s->execute_lock,0 , __ATOMIC_RELEASE); }else service_runner_do_callback (s, cs, i);@@ -415,11 +425,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)/* Increment num_mapped_cores to indicate that the service * is running on a core. */ - rte_atomic32_inc(&s->num_mapped_cores); + __atomic_add_fetch(&s->num_mapped_cores, 1, __ATOMIC_ACQUIRE);int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -rte_atomic32_dec (&s->num_mapped_cores); +__atomic_sub_fetch (&s->num_mapped_cores,1 , __ATOMIC_RELEASE);return ret;}
lib/librte_eal/common/rte_service.c +0
-2@@ -552,24 +562,32 @@ service_update(uint32_t sid, uint32_t lcore,uint64_t sid_mask =UINT64_C (1 ) << sid;if (set) { -uint64_t lcore_mapped = lcore_states[lcore].service_mask & - sid_mask;
lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 574
Quoted 10 lines
lib/librte_eal/common/rte_service.c +10
-0+/* When multiple threads try to update the same lcore +* service concurrently, e.g. set lcore map followed +* by clear lcore map, the unsynchronized service_mask +* values have issues on the num_mapped_cores value +* consistency. So we use ACQUIRE ordering to pair with +* the RELEASE ordering to synchronize the service_mask. +*/ +uint64_t lcore_mapped =__atomic_load_n ( + &lcore_states[lcore].service_mask, + __ATOMIC_ACQUIRE) & sid_mask;
Thanks for the comment - it helps me understand things a bit better. Some questions/theories to validate;
1) The service_mask ACQUIRE avoids other loads being hoisted above it, correct?
2) There are non-atomic stores to service_mask. Is it correct that the stores themselves aren't the issue, but relative visibility of service_mask stores vs num_mapped_cores? (Detail in (3) below)
lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 585
Quoted 12 lines
lib/librte_eal/common/rte_service.c +4
-2if (*set && !lcore_mapped) { lcore_states[lcore].service_mask |= sid_mask; -rte_atomic32_inc (&rte_services[sid].num_mapped_cores); +__atomic_add_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELEASE); }if (!*set && lcore_mapped) { lcore_states[lcore].service_mask &= ~(sid_mask); -rte_atomic32_dec (&rte_services[sid].num_mapped_cores); +__atomic_sub_fetch (&rte_services[sid].num_mapped_cores, +1 , __ATOMIC_RELEASE); }
3) Here we update the core-local service_mask, and then update the num_mapped_cores with an ATOMIC_RELEASE. The RELEASE here ensures that the previous store to service_mask is guaranteed to be visible on all cores if this store is visible. Why do we care about this property? The service_mask is core local anway.
4) Even with the load ACQ service_mask, and REL num_mapped_cores store, is there not still a race-condition possible where 2 lcores simultaneously load-ACQ the service_mask, and then both do atomic add/sub_fetch with REL?
5) Assuming 4 above race is true, it raises the real question - the service-cores control APIs are not designed to be multi-thread-safe. Orchestration of service/lcore mappings is not meant to be done by multiple threads at the same time. Documenting this loudly may help, I'm happy to send a patch to do so if we're agreed on the above?
Quoted 32 lines
lib/librte_eal/common/rte_service.c +4
-5}if (enabled) *enabled = !!(lcore_states[lcore].service_mask & (sid_mask)); -rte_smp_wmb (); -return 0 ; }@@ -625,7 +643,8 @@ rte_service_lcore_reset_all(void) } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) -rte_atomic32_set (&rte_services[i].num_mapped_cores,0 ); +__atomic_store_n (&rte_services[i].num_mapped_cores,0 , + __ATOMIC_RELAXED);rte_smp_wmb ();@@ -708,7 +727,8 @@ rte_service_lcore_stop(uint32_t lcore)int32_t enabled = service_mask & (UINT64_C (1 ) << i);int32_t service_running =rte_service_runstate_get (i);int32_t only_core = (1 == -rte_atomic32_read (&rte_services[i].num_mapped_cores)); +__atomic_load_n (&rte_services[i].num_mapped_cores, + __ATOMIC_RELAXED));/* if the core is mapped, and the service is running, and this * is the only core that is mapped, the service would cease to -- 2.7.4
Re: [dpdk-dev] [PATCH v3 11/12] service: optimize with c11 one-way barrier
<snip>
Quoted 94 lines
Quoted 11 lines
Subject: [PATCH v3 11/12] service: optimize with c11 one-way barrier
The
num_mapped_coresandexecute_lockare synchronized withrte_atomic_XXAPIs which is a full barrier, DMB, on aarch64. This patch optimized it with
c11 atomic one-way barrier.Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>Based on discussion on-list, it seems the consensus is to not use GCC builtins, but instead use C11 APIs "proper"? If my conclusion is correct, the v+1 of this patchset would require updates to that style API.
Inline comments for context below, -Harry
Quoted 21 lines
--- lib/librte_eal/common/rte_service.c | 50++++++++++++++++++++++++++----------
(quoted diff) +0
-1- 1 file changed, 35 insertions(+), 15 deletions(-)lib/librte_eal/common/rte_service.c +1
-1@@ -42,7 +42,7 @@ struct rte_service_spec_impl {* running this service callback. When not set, a core may take the * lock and then run the service callback. */ -rte_atomic32_t execute_lock; +uint32_t execute_lock;/* API set/get-able variables */ int8_t app_runstate;lib/librte_eal/common/rte_service.c struct rte_service_spec_impl { · line 57
Quoted 5 lines
lib/librte_eal/common/rte_service.c +1
-1@@ -54,7 +54,7 @@ struct rte_service_spec_impl {* It does not indicate the number of cores the service is running * on currently. */ -rte_atomic32_t num_mapped_cores; +int32_t num_mapped_cores;Any reason why "
int32_t" or "uint32_t" is used over another?execute_lockis auint32_tabove,num_mapped_coresis anint32_t?lib/librte_eal/common/rte_service.c rte_service_runstate_get(uint32_t id) · line 335
Quoted 8 lines
lib/librte_eal/common/rte_service.c +0
-1uint64_t calls;uint64_t cycles_spent; } __rte_cache_aligned;@@ -332,7 +332,8 @@ rte_service_runstate_get(uint32_t id)rte_smp_rmb ();int check_disabled = !(s->internal_flags &SERVICE_F_START_CHECK ); - int lcore_mapped = (rte_atomic32_read(&s->num_mapped_cores) >0);
lib/librte_eal/common/rte_service.c rte_service_runstate_get(uint32_t id)
Quoted 5 lines
lib/librte_eal/common/rte_service.c +2
-0+int lcore_mapped = (__atomic_load_n (&s->num_mapped_cores, + __ATOMIC_RELAXED) >0 );return (s->app_runstate ==RUNSTATE_RUNNING ) && (s->comp_runstate == RUNSTATE_RUNNING) && @@ -375,11+376,20 @@
lib/librte_eal/common/rte_service.c service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
Quoted 11 lines
service_run(uint32_t i, struct
core_state*cs,uint64_tservice_mask,lib/librte_eal/common/rte_service.c +6
-1cs->service_active_on_lcore[i] =1 ; if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) { -if (!rte_atomic32_cmpset ((uint32_t *)&s->execute_lock,0 ,1 )) + uint32_t expected = 0; + /* ACQUIRE ordering here is to prevent the callback + * function from hoisting up before the execute_lock + * setting. + */ + if (!__atomic_compare_exchange_n(&s->execute_lock,&expected, 1,
lib/librte_eal/common/rte_service.c service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, · line 386
Quoted 2 lines
lib/librte_eal/common/rte_service.c +1
-0+ 0, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED))return -EBUSY ;Let's try improve the magic "1" and "0" constants, I believe the "1" here is the desired "new value on success", and the 0 is "bool weak", where our 0/false constant implies a strongly ordered compare exchange?
"Weak is true for weak
compare_exchange, which may fail spuriously, and false for the strong variation, which never fails spuriously.", from
https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.htmlconst
uint32_ton_success_value= 1;
const bool weak = 0;__atomic_compare_exchange_n(&s->execute_lock, &expected, on_success_value, weak, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED);
Quoted 6 lines
Although a bit more verbose, I feel this documents usage a lot better, particularly for those who aren't as familiar with the C11 function arguments order.
Admittedly with the API change to not use
__builtins, perhaps this comment is moot.
Suggest changing the execute_lock to rte_spinlock_t and use rte_spinlock_trylock API.
Quoted 71 lines
Quoted 15 lines
lib/librte_eal/common/rte_service.c +4
-1service_runner_do_callback (s, cs, i); -rte_atomic32_clear (&s->execute_lock); +/* RELEASE ordering here is used to pair with ACQUIRE +* above to achieve lock semantic. +*/ +__atomic_store_n (&s->execute_lock,0 , __ATOMIC_RELEASE); }else service_runner_do_callback (s, cs, i);@@ -415,11 +425,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id,
uint32_tserialize_mt_unsafe)lib/librte_eal/common/rte_service.c rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
Quoted 5 lines
lib/librte_eal/common/rte_service.c +1
-1/* Increment num_mapped_cores to indicate that the service * is running on a core. */ -rte_atomic32_inc (&s->num_mapped_cores); + __atomic_add_fetch(&s->num_mapped_cores, 1,
__ATOMIC_ACQUIRE);lib/librte_eal/common/rte_service.c rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
Quoted 5 lines
lib/librte_eal/common/rte_service.c +1
-1int ret =service_run (id, cs,UINT64_MAX , s, serialize_mt_unsafe); -rte_atomic32_dec (&s->num_mapped_cores); + __atomic_sub_fetch(&s->num_mapped_cores, 1,
__ATOMIC_RELEASE);Quoted 9 lines
lib/librte_eal/common/rte_service.c +0
-2return ret; }@@ -552,24 +562,32 @@ service_update(uint32_t sid, uint32_t lcore,uint64_t sid_mask =UINT64_C (1 ) << sid;if (set) { -uint64_t lcore_mapped = lcore_states[lcore].service_mask & - sid_mask;lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 574
Quoted 10 lines
lib/librte_eal/common/rte_service.c +10
-0+/* When multiple threads try to update the same lcore +* service concurrently, e.g. set lcore map followed +* by clear lcore map, the unsynchronized service_mask +* values have issues on the num_mapped_cores value +* consistency. So we use ACQUIRE ordering to pair with +* the RELEASE ordering to synchronize the service_mask. +*/ +uint64_t lcore_mapped =__atomic_load_n ( + &lcore_states[lcore].service_mask, + __ATOMIC_ACQUIRE) & sid_mask;Thanks for the comment - it helps me understand things a bit better. Some questions/theories to validate;
1) Theservice_maskACQUIRE avoids other loads being hoisted above it, correct?2) There are non-atomic stores to
service_mask. Is it correct that the stores themselves aren't the issue, but relative visibility ofservice_maskstores vsnum_mapped_cores? (Detail in (3) below)lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 572
Quoted 3 lines
lib/librte_eal/common/rte_service.c +0
-1if (*set && !lcore_mapped) { lcore_states[lcore].service_mask |= sid_mask; -rte_atomic32_inc(&rte_services[sid].num_mapped_cores);lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 577
Quoted 1 line
lib/librte_eal/common/rte_service.c +1
-0+__atomic_add_fetch(&rte_services[sid].num_mapped_cores,
lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 572
Quoted 5 lines
lib/librte_eal/common/rte_service.c +1
-1+ 1, __ATOMIC_RELEASE); }if (!*set && lcore_mapped) { lcore_states[lcore].service_mask &= ~(sid_mask); -rte_atomic32_dec(&rte_services[sid].num_mapped_cores);lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 582
Quoted 1 line
lib/librte_eal/common/rte_service.c +1
-0+__atomic_sub_fetch(&rte_services[sid].num_mapped_cores,
Quoted 8 lines
lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 584
Quoted 2 lines
lib/librte_eal/common/rte_service.c +1
-0+ 1, __ATOMIC_RELEASE); }3) Here we update the core-local
service_mask, and then update thenum_mapped_coreswith anATOMIC_RELEASE. The RELEASE here ensures that the previous store toservice_maskis guaranteed to be visible on all cores if this store is visible. Why do we care about this property?
Theservice_maskis core local anway.
We are working on concurrency between the reader and writer. The service_mask is local to the core, but it is accessed by a reader and writer.
I think we should wait to conclude on the meaning of 'num_mapped_cores', that will dictate what the order should be. For ex: if it is just for statistics purpose, then we could use just RELAXED memory order and then the order for service_mask will also change.
Quoted 10 lines
4) Even with the load ACQ
service_mask, and RELnum_mapped_coresstore, is there not still a race-condition possible where 2 lcores simultaneously load- ACQ theservice_mask, and then both do atomic add/sub_fetch with REL?5) Assuming 4 above race is true, it raises the real question - the service-cores control APIs are not designed to be multi-thread-safe. Orchestration of service/lcore mappings is not meant to be done by multiple threads at the same time. Documenting this loudly may help, I'm happy to send a patch to do so if we're agreed on the above?
I completely agree here. writer-writer concurrency is another topic and we should (for now at least) say that the control plane APIs are not thread safe.
Quoted 38 lines
Quoted 25 lines
lib/librte_eal/common/rte_service.c +2
-3}if (enabled) *enabled = !!(lcore_states[lcore].service_mask & (sid_mask)); -rte_smp_wmb (); -return 0 ; }@@ -625,7 +643,8 @@ rte_service_lcore_reset_all(void) } }for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) -rte_atomic32_set (&rte_services[i].num_mapped_cores,0 ); +__atomic_store_n (&rte_services[i].num_mapped_cores,0 , + __ATOMIC_RELAXED);rte_smp_wmb ();lib/librte_eal/common/rte_service.c rte_service_lcore_stop(uint32_t lcore) · line 711
Quoted 4 lines
lib/librte_eal/common/rte_service.c +0
-1@@ -708,7 +727,8 @@ rte_service_lcore_stop(uint32_t lcore)int32_t enabled = service_mask & (UINT64_C (1 ) << i);int32_t service_running =rte_service_runstate_get (i); int32_t only_core = (1 == -rte_atomic32_read(&rte_services[i].num_mapped_cores));lib/librte_eal/common/rte_service.c rte_service_lcore_stop(uint32_t lcore) · line 730
Quoted 1 line
lib/librte_eal/common/rte_service.c +1
-0+__atomic_load_n(&rte_services[i].num_mapped_cores,Quoted 6 lines
lib/librte_eal/common/rte_service.c +1
-1+ __ATOMIC_RELAXED));/* if the core is mapped, and the service is running, and this * is the only core that is mapped, the service would cease to -- 2.7.4
Re: [dpdk-dev] [PATCH v3 11/12] service: optimize with c11 one-way barrier
In reply to Van Haaren, HarryQuoted 64 lines
-----Original Message-----
From: Van Haaren, Harry <[email protected]>
Sent: Friday, April 3, 2020 7:58 PM
To: Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 11/12] service: optimize with c11 one-way barrierQuoted 4 lines
-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, HarryQuoted 2 lines
Ananyev, Konstantin <[email protected]>; [email protected]; [email protected];
Quoted 1 line
Quoted 1 line
Quoted 4 lines
[email protected]; [email protected]
Subject: [PATCH v3 11/12] service: optimize with c11 one-way barrierThe
num_mapped_coresandexecute_lockare synchronized with
rte_atomic_XXQuoted 7 lines
APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.
Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>Based on discussion on-list, it seems the consensus is to not use GCC builtins, but instead use C11 APIs "proper"? If my conclusion is correct, the v+1 of this patchset would require updates to that style API.
Inline comments for context below, -Harry
Quoted 2 lines
--- lib/librte_eal/common/rte_service.c | 50++++++++++++++++++++++++++----------
Quoted 15 lines
(quoted diff) +0
-1- 1 file changed, 35 insertions(+), 15 deletions(-)lib/librte_eal/common/rte_service.c +1
-1@@ -42,7 +42,7 @@ struct rte_service_spec_impl {* running this service callback. When not set, a core may take the * lock and then run the service callback. */ -rte_atomic32_t execute_lock; +uint32_t execute_lock;
Quoted 11 lines
lib/librte_eal/common/rte_service.c struct rte_service_spec_impl { · line 57
Quoted 8 lines
lib/librte_eal/common/rte_service.c +1
-1/* API set/get-able variables */ int8_t app_runstate;@@ -54,7 +54,7 @@ struct rte_service_spec_impl {* It does not indicate the number of cores the service is running * on currently. */ -rte_atomic32_t num_mapped_cores; +int32_t num_mapped_cores;Any reason why "
int32_t" or "uint32_t" is used over another?execute_lockis auint32_tabove,num_mapped_coresis anint32_t?
It should be uint32_t for num_mapped_cores.
This value will not be negative after __atomic_sub_fetch operation, because of the sequence of writer and reader accesses are guaranteed by the memory ordering. I will update it in the next version.
Thanks,
Phil
<snip>
PATCH v3 12/12 service: relax barriers with C11 atomic operations
In reply to Phil YangTo guarantee the inter-threads visibility of the shareable domain, it uses a lot of rte_smp_r/wmb in the service library. This patch relaxed these barriers for service by using c11 atomic one-way barrier operations.
--- lib/librte_eal/common/rte_service.c | 45 ++++++++++++++++++++----------------- 1 file changed, 25 insertions(+), 20 deletions(-)
lib/librte_eal/common/rte_service.c +25 -20
@@ -179,9 +179,11 @@ rte_service_set_stats_enable(uint32_t id, int32_t enabled)SERVICE_VALID_GET_OR_ERR_RET (id, s,0 );if (enabled) - s->internal_flags |=SERVICE_F_STATS_ENABLED ; +__atomic_or_fetch (&s->internal_flags,SERVICE_F_STATS_ENABLED , + __ATOMIC_RELEASE);else - s->internal_flags &= ~(SERVICE_F_STATS_ENABLED ); +__atomic_and_fetch (&s->internal_flags, + ~(SERVICE_F_STATS_ENABLED ), __ATOMIC_RELEASE);return 0 ; }
@@ -193,9 +195,11 @@ rte_service_set_runstate_mapped_check(uint32_t id, int32_t enabled)SERVICE_VALID_GET_OR_ERR_RET (id, s,0 );if (enabled) - s->internal_flags |=SERVICE_F_START_CHECK ; +__atomic_or_fetch (&s->internal_flags,SERVICE_F_START_CHECK , + __ATOMIC_RELEASE);else - s->internal_flags &= ~(SERVICE_F_START_CHECK ); +__atomic_and_fetch (&s->internal_flags, ~(SERVICE_F_START_CHECK ), + __ATOMIC_RELEASE);return 0 ; }
@@ -264,8 +268,8 @@ rte_service_component_register(const struct rte_service_spec *spec, s->spec = *spec; s->internal_flags |=SERVICE_F_REGISTERED |SERVICE_F_START_CHECK ; -rte_smp_wmb (); - rte_service_count++; +/* make sure the counter update after the state change. */ +__atomic_add_fetch (&rte_service_count,1 , __ATOMIC_RELEASE);if (id_ptr) *id_ptr = free_slot;
@@ -281,9 +285,10 @@ rte_service_component_unregister(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); rte_service_count--; -rte_smp_wmb (); - s->internal_flags &= ~(SERVICE_F_REGISTERED ); +/* make sure the counter update before the state change. */ +__atomic_and_fetch (&s->internal_flags, ~(SERVICE_F_REGISTERED ), + __ATOMIC_RELEASE);/* clear the run-bit in all cores */ for (i =0 ; i <RTE_MAX_LCORE ; i++)
@@ -301,11 +306,12 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );if (runstate) - s->comp_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->comp_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -316,11 +322,12 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );if (runstate) - s->app_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->app_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->app_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->app_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
@@ -442,7 +449,8 @@ service_runner_func(void *arg)const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (lcore_states[lcore].runstate ==RUNSTATE_RUNNING ) { +while (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) {const uint64_t service_mask = cs->service_mask;for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) {
@@ -453,8 +461,6 @@ service_runner_func(void *arg) } cs->loops++; - -rte_smp_rmb (); } lcore_config[lcore].state =WAIT ;
@@ -663,9 +669,8 @@ rte_service_lcore_add(uint32_t lcore)/* ensure that after adding a core the mask and state are defaults */ lcore_states[lcore].service_mask =0 ; - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; - -rte_smp_wmb (); +__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);return rte_eal_wait_lcore (lcore); }
Signature · 1 line
2.7.4
Re: [dpdk-dev] [PATCH v3 12/12] service: relax barriers with C11 atomic operations
Quoted 29 lines
From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 12/12] service: relax barriers with C11 atomic operationsTo guarantee the inter-threads visibility of the shareable domain, it uses a lot of rte_smp_r/wmb in the service library. This patch relaxed these barriers for service by using c11 atomic one-way barrier operations.
Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>--- lib/librte_eal/common/rte_service.c | 45 ++++++++++++++++++++----------------(quoted diff) +0
-1- 1 file changed, 25 insertions(+), 20 deletions(-)lib/librte_eal/common/rte_service.c +0
-0@@ -179,9 +179,11 @@ rte_service_set_stats_enable(uint32_t id, int32_t enabled)
lib/librte_eal/common/rte_service.c rte_service_set_stats_enable(uint32_t id, int32_t · line 187
Quoted 10 lines
lib/librte_eal/common/rte_service.c +4
-2SERVICE_VALID_GET_OR_ERR_RET (id, s,0 );if (enabled) - s->internal_flags |=SERVICE_F_STATS_ENABLED ; +__atomic_or_fetch (&s->internal_flags,SERVICE_F_STATS_ENABLED , + __ATOMIC_RELEASE);else - s->internal_flags &= ~(SERVICE_F_STATS_ENABLED ); +__atomic_and_fetch (&s->internal_flags, + ~(SERVICE_F_STATS_ENABLED ), __ATOMIC_RELEASE);
Not sure why these have to become stores with RELEASE memory ordering? (More occurances of same Q below, just answer here?)
Quoted 4 lines
lib/librte_eal/common/rte_service.c +0
-0return 0 ; }@@ -193,9 +195,11 @@ rte_service_set_runstate_mapped_check(uint32_t id, int32_t enabled)
lib/librte_eal/common/rte_service.c rte_service_set_runstate_mapped_check(uint32_t id, · line 203
Quoted 10 lines
lib/librte_eal/common/rte_service.c +4
-2SERVICE_VALID_GET_OR_ERR_RET (id, s,0 );if (enabled) - s->internal_flags |=SERVICE_F_START_CHECK ; +__atomic_or_fetch (&s->internal_flags,SERVICE_F_START_CHECK , + __ATOMIC_RELEASE);else - s->internal_flags &= ~(SERVICE_F_START_CHECK ); +__atomic_and_fetch (&s->internal_flags, ~(SERVICE_F_START_CHECK ), + __ATOMIC_RELEASE);
Same as above, why do these require RELEASE?
Remainder of patch below seems to make sense - there's a wmb() involved hence RELEASE m/o.
lib/librte_eal/common/rte_service.c rte_service_component_register(const struct · line 273
Quoted 11 lines
lib/librte_eal/common/rte_service.c +2
-2return 0 ; }@@ -264,8 +268,8 @@ rte_service_component_register(const struct rte_service_spec *spec, s->spec = *spec; s->internal_flags |=SERVICE_F_REGISTERED |SERVICE_F_START_CHECK ; -rte_smp_wmb (); - rte_service_count++; +/* make sure the counter update after the state change. */ +__atomic_add_fetch (&rte_service_count,1 , __ATOMIC_RELEASE);
This makes sense to me - the RELEASE ensures that previous stores to the s->internal_flags are visible to other cores before rte_service_count increments atomically.
Quoted 82 lines
lib/librte_eal/common/rte_service.c +15
-15if (id_ptr) *id_ptr = free_slot;@@ -281,9 +285,10 @@ rte_service_component_unregister(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL ); rte_service_count--; -rte_smp_wmb (); - s->internal_flags &= ~(SERVICE_F_REGISTERED ); +/* make sure the counter update before the state change. */ +__atomic_and_fetch (&s->internal_flags, ~(SERVICE_F_REGISTERED ), + __ATOMIC_RELEASE);/* clear the run-bit in all cores */ for (i =0 ; i <RTE_MAX_LCORE ; i++)@@ -301,11 +306,12 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate) SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);if (runstate) - s->comp_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->comp_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }@@ -316,11 +322,12 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );if (runstate) - s->app_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->app_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->app_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->app_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }@@ -442,7 +449,8 @@ service_runner_func(void *arg)const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (lcore_states[lcore].runstate ==RUNSTATE_RUNNING ) { +while (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) {const uint64_t service_mask = cs->service_mask;for (i =0 ; i <RTE_SERVICE_NUM_MAX ; i++) {@@ -453,8 +461,6 @@ service_runner_func(void *arg) } cs->loops++; - -rte_smp_rmb (); } lcore_config[lcore].state =WAIT ;@@ -663,9 +669,8 @@ rte_service_lcore_add(uint32_t lcore)/* ensure that after adding a core the mask and state are defaults */ lcore_states[lcore].service_mask =0 ; - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; - -rte_smp_wmb (); +__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);return rte_eal_wait_lcore (lcore); } --2.7.4
Re: [dpdk-dev] [PATCH v3 12/12] service: relax barriers with C11 atomic operations
<snip>
Just to get us on the same page on 'techniques to communicate data from writer to reader' (apologies if it is too trivial)
Let us say that the writer has 512B (key point is - cannot be written atomically) that needs to be communicated to the reader.
Since the data cannot be written atomically, we need a guard variable (which can be written atomically, can be a flag or pointer to data). So, the writer will store 512B in non-atomic way and write to guard variable with release memory order. So, if the guard variable is valid (set in the case of flag or not null in the case of pointer), it guarantees that 512B is written.
The reader will read the guard variable with acquire memory order and read the 512B data only if the guard variable is valid. So, the acquire memory order on the guard variable guarantees that the load of 512B does not happen before the guard variable is read. The validity check on the guard variable guarantees that 512B was written before it was read.
The store(guard_variable, RELEASE) on the writer and the load(guard_variable, ACQUIRE) can be said as synchronizing with each other.
(the guard variable technique applies even if we are not using C11 atomics)
Let us say that the writer has 4B (key point is - can be written atomically) that needs to be communicated to the reader. The writer is free to write this atomically with no constraints on memory ordering as long as this data is not acting as a guard variable for any other data.
In my understanding, the sequence of APIs to call to start a service (writer) are as follows:
1) rte_service_init
2) rte_service_component_register
3) <possible configuration of the service>
4) rte_service_component_runstate_set (the reader is allowed at this point to read the information about the service - written by rte_service_component_register API. This API should not be called before rte_service_component_register)
5) <possible configuration of the service>
6) rte_service_runstate_set (the reader is allowed at this point to read the information about the service - written by rte_service_component_register API and run the service. This API can be called anytime. But, the reader should not attempt to run the service before this API is called)
7) rte_lcore_service_add (multiple of these probably, can be called before this, can't be called later)
8) rte_service_map_lcore_set (this can be called anytime. Can be called even if the service is not registered)
9) rte_service_lcore_start (again, this can be called anytime, even before the service is registered)
So, there are 2 guard variables - 'comp_runstate' and 'app_runstate'. Only these 2 need to have RELEASE ordering in writer and ACQUIRE ordering in reader.
We can write test cases with different orders of these API calls to prove that the memory orders we use are sufficient.
Few comments are inline based on this assessment.
Quoted 42 lines
Subject: RE: [PATCH v3 12/12] service: relax barriers with C11 atomic operations
Quoted 9 lines
From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];Quoted 22 lines
Subject: [PATCH v3 12/12] service: relax barriers with C11 atomic operations
To guarantee the inter-threads visibility of the shareable domain, it uses a lot of rte_smp_r/wmb in the service library. This patch relaxed these barriers for service by using c11 atomic one-way barrier operations.
Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>--- lib/librte_eal/common/rte_service.c | 45++++++++++++++++++++----------------
(quoted diff) +0
-1- 1 file changed, 25 insertions(+), 20 deletions(-)lib/librte_eal/common/rte_service.c +0
-0lib/librte_eal/common/rte_service.c rte_service_set_stats_enable(uint32_t id, int32_t · line 183
Quoted 6 lines
lib/librte_eal/common/rte_service.c +1
-1@@ -179,9 +179,11 @@ rte_service_set_stats_enable(uint32_t id, int32_t enabled) SERVICE_VALID_GET_OR_ERR_RET(id, s, 0); if (enabled) - s->internal_flags |= SERVICE_F_STATS_ENABLED; + __atomic_or_fetch(&s->internal_flags,
SERVICE_F_STATS_ENABLED,
Quoted 8 lines
lib/librte_eal/common/rte_service.c rte_service_set_stats_enable(uint32_t id, int32_t enabled) · line 186
Quoted 5 lines
lib/librte_eal/common/rte_service.c +3
-1+ __ATOMIC_RELEASE); else - s->internal_flags &= ~(SERVICE_F_STATS_ENABLED ); + __atomic_and_fetch(&s->internal_flags, + ~(SERVICE_F_STATS_ENABLED), __ATOMIC_RELEASE);Not sure why these have to become stores with RELEASE memory ordering? (More occurances of same Q below, just answer here?)
Agree, 'internal_flags' is not acting as a guard variable, this should be RELAXED (similarly for the others below). Though I suggest keeping it atomic.
Quoted 11 lines
lib/librte_eal/common/rte_service.c rte_service_set_runstate_mapped_check(uint32_t · line 199
Quoted 10 lines
lib/librte_eal/common/rte_service.c +1
-1return 0 ; }@@ -193,9 +195,11 @@ rte_service_set_runstate_mapped_check(uint32_t id, int32_t enabled)SERVICE_VALID_GET_OR_ERR_RET (id, s,0 );if (enabled) - s->internal_flags |=SERVICE_F_START_CHECK ; + __atomic_or_fetch(&s->internal_flags,
SERVICE_F_START_CHECK,
Quoted 8 lines
lib/librte_eal/common/rte_service.c rte_service_set_runstate_mapped_check(uint32_t id, int32_t enabled)
Quoted 4 lines
lib/librte_eal/common/rte_service.c +2
-1+ __ATOMIC_RELEASE); else - s->internal_flags &= ~(SERVICE_F_START_CHECK ); + __atomic_and_fetch(&s->internal_flags,~(SERVICE_F_START_CHECK),
lib/librte_eal/common/rte_service.c rte_service_set_runstate_mapped_check(uint32_t · line 203
Quoted 1 line
lib/librte_eal/common/rte_service.c +1
-0+ __ATOMIC_RELEASE);Same as above, why do these require RELEASE?
Agree
Quoted 13 lines
Remainder of patch below seems to make sense - there's a
wmb()involved hence RELEASE m/o.lib/librte_eal/common/rte_service.c rte_service_component_register(const struct · line 270
Quoted 6 lines
lib/librte_eal/common/rte_service.c +0
-0return 0 ; }@@ -264,8 +268,8 @@ rte_service_component_register(const struct rte_service_spec *spec, s->spec = *spec; s->internal_flags |= SERVICE_F_REGISTERED |
SERVICE_F_START_CHECK;Quoted 1 line
Quoted 8 lines
lib/librte_eal/common/rte_service.c rte_service_component_register(const struct rte_service_spec *spec, · line 272
Quoted 4 lines
lib/librte_eal/common/rte_service.c +2
-2-rte_smp_wmb (); - rte_service_count++; +/* make sure the counter update after the state change. */ +__atomic_add_fetch (&rte_service_count,1 , __ATOMIC_RELEASE);This makes sense to me - the RELEASE ensures that previous stores to the
s->internal_flagsare visible to other cores beforerte_service_countincrements atomically.
rte_service_count is not a guard variable, does not need RELEASE order. It is also not used by the reader. It looks like it is just a statistic being maintained.
Quoted 7 lines
Quoted 7 lines
lib/librte_eal/common/rte_service.c +0
-0if (id_ptr) *id_ptr = free_slot;@@ -281,9 +285,10 @@ rte_service_component_unregister(uint32_t id)SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );
Quoted 7 lines
lib/librte_eal/common/rte_service.c rte_service_component_unregister(uint32_t id) · line 291
Quoted 7 lines
lib/librte_eal/common/rte_service.c +3
-2rte_service_count--; -rte_smp_wmb (); - s->internal_flags &= ~(SERVICE_F_REGISTERED ); +/* make sure the counter update before the state change. */ +__atomic_and_fetch (&s->internal_flags, ~(SERVICE_F_REGISTERED ), + __ATOMIC_RELEASE);
RELAXED is enough.
Quoted 8 lines
Quoted 8 lines
lib/librte_eal/common/rte_service.c +0
-0/* clear the run-bit in all cores */ for (i =0 ; i <RTE_MAX_LCORE ; i++)@@ -301,11 +306,12 @@ rte_service_component_runstate_set(uint32_t id,
uint32_t
runstate)
SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
Quoted 8 lines
lib/librte_eal/common/rte_service.c rte_service_component_runstate_set(uint32_t id, uint32_t runstate) · line 313
Quoted 8 lines
lib/librte_eal/common/rte_service.c +4
-2if (runstate) - s->comp_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->comp_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->comp_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);
Here we need a thread_fence to prevent the memory operations from a subsequent call to 'rte_service_component_unregister' from getting hoisted above this. The user should be forced to call rte_service_component_unregister before calling rte_service_component_runstate_set. I suggest adding a check in rte_service_component_unregister to ensure that the state is set to RUNSTATE_STOPPED. In fact, the user needs to make sure that the service is stopped for sure before calling rte_service_component_unregister.
Quoted 23 lines
lib/librte_eal/common/rte_service.c rte_service_runstate_set(uint32_t id, uint32_t
Quoted 7 lines
lib/librte_eal/common/rte_service.c +0
-1-rte_smp_wmb ();return 0 ; }@@ -316,11 +322,12 @@ rte_service_runstate_set(uint32_t id, uint32_trunstate)
Quoted 15 lines
lib/librte_eal/common/rte_service.c +4
-3SERVICE_VALID_GET_OR_ERR_RET (id, s, -EINVAL );if (runstate) - s->app_runstate =RUNSTATE_RUNNING ; +__atomic_store_n (&s->app_runstate,RUNSTATE_RUNNING , + __ATOMIC_RELEASE);else - s->app_runstate =RUNSTATE_STOPPED ; +__atomic_store_n (&s->app_runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE); -rte_smp_wmb ();return 0 ; }
Quoted 7 lines
lib/librte_eal/common/rte_service.c service_runner_func(void *arg) · line 453
Quoted 7 lines
lib/librte_eal/common/rte_service.c +2
-1@@ -442,7 +449,8 @@ service_runner_func(void *arg)const int lcore =rte_lcore_id ();struct core_state *cs = &lcore_states[lcore]; -while (lcore_states[lcore].runstate ==RUNSTATE_RUNNING ) { +while (__atomic_load_n (&cs->runstate, + __ATOMIC_ACQUIRE) ==RUNSTATE_RUNNING ) {
This can be RELAXED, lcore's runstate is not acting as a guard variable.
However, note that the writer thread wants to communicate the 'runstate' (4B) to the reader thread. This ordering needs to be handled in 'rte_eal_remote_launch' and 'eal_thread_loop' functions. We have to note that in some other use case, the writer wants to communicate more than 4B to reader. Currently, the 'write' and 'read' system calls may have enough barriers to make things work fine. But, I suggest using the ' lcore_config[lcore].f' as the guard variable to make it explicit and not depend on 'write' and 'read'. We can take up the EAL things in a later patch as it does not cause any issues right now.
Quoted 12 lines
lib/librte_eal/common/rte_service.c service_runner_func(void *arg)
Quoted 3 lines
lib/librte_eal/common/rte_service.c +0
-0const uint64_t service_mask = cs->service_mask; for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) { @@ -453,8+461,6 @@
Quoted 8 lines
service_runner_func(void *arg)lib/librte_eal/common/rte_service.c +0
-2} cs->loops++; - -rte_smp_rmb (); }
Quoted 8 lines
lib/librte_eal/common/rte_service.c rte_service_lcore_add(uint32_t lcore) · line 668
Quoted 8 lines
lib/librte_eal/common/rte_service.c +0
-3lcore_config[lcore].state =WAIT ;@@ -663,9 +669,8 @@ rte_service_lcore_add(uint32_t lcore)/* ensure that after adding a core the mask and state are defaults */ lcore_states[lcore].service_mask =0 ; - lcore_states[lcore].runstate =RUNSTATE_STOPPED ; - -rte_smp_wmb ();
Agree.
Quoted 2 lines
lib/librte_eal/common/rte_service.c rte_service_lcore_add(uint32_t lcore) · line 673
Quoted 2 lines
lib/librte_eal/common/rte_service.c +2
-0+__atomic_store_n (&lcore_states[lcore].runstate,RUNSTATE_STOPPED , + __ATOMIC_RELEASE);
This can be relaxed.
Quoted 5 lines
Quoted 5 lines
lib/librte_eal/common/rte_service.c +0
-1return rte_eal_wait_lcore (lcore); } --2.7.4
Re: [dpdk-dev] [PATCH v3 12/12] service: relax barriers with C11 atomic operations
Quoted 33 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Monday, April 6, 2020 6:06 PM
To: Van Haaren, Harry <[email protected]>; Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 12/12] service: relax barriers with C11 atomic operations<snip>
Just to get us on the same page on 'techniques to communicate data from writer to reader' (apologies if it is too trivial)Let us say that the writer has 512B (key point is - cannot be written atomically) that needs to be communicated to the reader.
Since the data cannot be written atomically, we need a guard variable (which can be written atomically, can be a flag or pointer to data). So, the writer will store 512B in non-atomic way and write to guard variable with release memory order. So, if the guard variable is valid (set in the case of flag or not null in the case of pointer), it guarantees that 512B is written.
The reader will read the guard variable with acquire memory order and read the 512B data only if the guard variable is valid. So, the acquire memory order on the guard variable guarantees that the load of 512B does not happen before the guard variable is read. The validity check on the guard variable guarantees that 512B was written before it was read.
Quoted 4 lines
The
store(guard_variable, RELEASE)on the writer and theload(guard_variable, ACQUIRE)can be said as synchronizing with each other.(the guard variable technique applies even if we are not using C11 atomics)
Yep agreed on the above.
Quoted 17 lines
Let us say that the writer has 4B (key point is - can be written atomically) that needs to be communicated to the reader. The writer is free to write this atomically with no constraints on memory ordering as long as this data is not acting as a guard variable for any other data.
In my understanding, the sequence of APIs to call to start a service (writer) are as follows:
1)rte_service_init
2)rte_service_component_register
3) <possible configuration of the service>
4)rte_service_component_runstate_set(the reader is allowed at this point to read the information about the service - written byrte_service_component_registerAPI. This API should not be called beforerte_service_component_register)
5) <possible configuration of the service>
6)rte_service_runstate_set(the reader is allowed at this point to read the information about the service - written byrte_service_component_registerAPI
Quoted 8 lines
and run the service. This API can be called anytime. But, the reader should not attempt to run the service before this API is called)
7)rte_lcore_service_add(multiple of these probably, can be called before this, can't be called later)
8)rte_service_map_lcore_set(this can be called anytime. Can be called even if the service is not registered)
9)rte_service_lcore_start(again, this can be called anytime, even before the service is registered)
I think this can be simplified, if we look at calling threads:
- one thread is the writer/config thread, and is allowed to call anything --- Any updates/changes must be atomically correct in how the other threads can read state.
- service lcores, which fundamentally spin in run(), and call services mapped to it --- here we need to ensure any service mapped to it is atomic, and the service is valid to run.
- other application threads using "run on app lcore" function
--- similar to service lcore, check for service in valid state, and allow to run.
Services are not allowed to be unregistered e.g. while running. I'd like to avoid the "explosion of combinations" by enforcing simple limitations (and documenting them more clearly if/where required).
Quoted 8 lines
So, there are 2 guard variables - '
comp_runstate' and 'app_runstate'. Only these 2 need to have RELEASE ordering in writer and ACQUIRE ordering in reader.We can write test cases with different orders of these API calls to prove that the memory orders we use are sufficient.
Few comments are inline based on this assessment.
Sure. As per other email thread, splitting changes into bugfix/cleanup/C11 would likely help to try keep track of changes etc required per patch, its getting hard to follow the various topics being discussed in parallel.
Quoted 20 lines
Quoted 20 lines
Subject: RE: [PATCH v3 12/12] service: relax barriers with C11 atomic operations
Quoted 9 lines
From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];Quoted 7 lines
Subject: [PATCH v3 12/12] service: relax barriers with C11 atomic operations
To guarantee the inter-threads visibility of the shareable domain, it uses a lot of rte_smp_r/wmb in the service library. This patch relaxed these barriers for service by using c11 atomic one-way barrier operations.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Signed-off-by Phil Yang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Reviewed-by Gavin Hu <[email protected]>--- lib/librte_eal/common/rte_service.c | 45
<snip patch review comments>
Hi Phil & Honnappa,
Quoted 21 lines
-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 00/12] generic rte atomic APIs deprecate proposalDPDK provides generic
rte_atomicAPIs to do several atomic operations. These APIs are using the deprecated__syncbuilt-ins and enforce full memory barriers on aarch64. However, full barriers are not necessary in many use cases. In order to address such use cases, C language offers C11 atomic APIs. The C11 atomic APIs provide finer memory barrier control by making use of the memory ordering parameter provided by the user. Various patches submitted in the past [2] and the patches in this series indicate significant performance gains on multiple aarch64 CPUs and no performance loss on x86.
Quoted 8 lines
But the existing
rte_atomicAPI implementations cannot be changed as the APIs do not take the memory ordering parameter. The only choice available is replacing the usage of therte_atomicAPIs with C11 atomic APIs. In order to make this change, the following steps are proposed:[1] deprecate
rte_atomicAPIs so that future patches do not userte_atomicAPIs (a script is added to flag the usages).
[2] refactor the code that usesrte_atomicAPIs to use c11 atomic APIs.
On [1] above, I feel deprecating DPDKs atomic functions and failing checkpatch is a bit sudden. Perhaps noting that in a future release (20.11?) DPDK will move to a C11 based atomics model is a more gradual step to achieving the goal, and at that point add a checkpatch warning for additions of rte_atomic*?
More on [2] in context below.
The above is my point-of-view, of course I'd like more people from the DPDK community to provide their input too.
Quoted 8 lines
This patchset contains:
1) the checkpatch script changes to flagrte_atomicAPI usage in patches.
2) changes to programmer guide describing writing efficient code for aarch64.
3) changes to various libraries to make use of c11 atomic APIs.We are planning to replicate this idea across all the other libraries, drivers, examples, test applications. In the next phase, we will add changes to the mbuf, the EAL interrupts and the event timer adapter libraries.
About ~6/12 patches of this C11 set are targeting the Service Cores area of DPDK. I have some concerns over increased complexity of C11 implementation vs the (already complex) rte_atomic implementation today. I see other patchsets enabling C11 across other DPDK components, so maybe we should also discuss C11 enabling in a wider context that just service cores?
I don't think it fair to expect all developers to be well versed in C11 atomic semantics like understanding the complex interactions between the various C11 RELEASE, AQUIRE barriers requires.
As maintainer of Service Cores I'm hesitant to accept the large-scale refactor of atomic-implementation, as it could lead to racey bugs that are likely extremely difficult to track down. (The recent race-on-exit has proven the difficulty in reproducing, and that's with an atomics model I'm quite familiar with).
Let me be very clear: I don't wish to block a C11 atomic implementation, but I'd like to discuss how we (DPDK community) can best port-to and maintain a complex multi-threaded service library with best-in-class performance for the workload.
To put some discussions/solutions on the table:
- Shared Maintainership of a component?
Split in functionality and C11 atomics implementation
Obviously there would be collaboration required in such a case.
- Maybe shared maintainership is too much?
A gentlemans/womans agreement of "helping out" with C11 atomics debug is enough?
Hope my concerns are understandable, and of course input/feedback welcomed! -Harry
PS: Apologies for the delay in reply - was OOO on Irish national holiday.
Quoted 41 lines
v3:
add libatomic dependency for 32-bit clangv2:
1. fix Clang '-Wincompatible-pointer-types' WARNING.
2. fix typos.Honnappa Nagarahalli (2):
service: avoid race condition for MT unsafe service
service: identify service running on another core correctlyPhil Yang (10):
doc: add generic atomic deprecation section
devtools: prevent use of rte atomic APIs in future patches eal/build: add libatomic dependency for 32-bit clang
build: remove redundant code
vhost: optimize broadcast rarp sync with c11 atomic
ipsec: optimize with c11 atomic for sa outbound sqn update
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 one-way barrier
service: relax barriers with C11 atomic operationsdevtools/checkpatches.sh | 9 ++ doc/guides/prog_guide/writing_efficient_code.rst | 60 +++++++- drivers/event/octeontx/meson.build | 5 - drivers/event/octeontx2/meson.build | 5 - drivers/event/opdl/meson.build | 5 - lib/librte_eal/common/rte_service.c | 175 ++++++++++++---------- - lib/librte_eal/meson.build | 6 + lib/librte_ipsec/ipsec_sqn.h | 3 +- lib/librte_ipsec/sa.h | 2 +- lib/librte_rcu/meson.build | 5 - lib/librte_vhost/vhost.h | 2 +- lib/librte_vhost/vhost_user.c | 7 +- lib/librte_vhost/virtio_net.c | 16 ++- 13 files changed, 181 insertions(+), 119 deletions(-)--
2.7.4
18/03/2020 15:01, Van Haaren, Harry:
Quoted 67 lines
Hi Phil & Honnappa,
From: Phil Yang <[email protected]>
Quoted 11 lines
DPDK provides generic
rte_atomicAPIs to do several atomic operations. These APIs are using the deprecated__syncbuilt-ins and enforce full memory barriers on aarch64. However, full barriers are not necessary in many use cases. In order to address such use cases, C language offers C11 atomic APIs. The C11 atomic APIs provide finer memory barrier control by making use of the memory ordering parameter provided by the user. Various patches submitted in the past [2] and the patches in this series indicate significant performance gains on multiple aarch64 CPUs and no performance loss on x86.Quoted 8 lines
But the existing
rte_atomicAPI implementations cannot be changed as the APIs do not take the memory ordering parameter. The only choice available is replacing the usage of therte_atomicAPIs with C11 atomic APIs. In order to make this change, the following steps are proposed:[1] deprecate
rte_atomicAPIs so that future patches do not userte_atomicAPIs (a script is added to flag the usages).
[2] refactor the code that usesrte_atomicAPIs to use c11 atomic APIs.On [1] above, I feel deprecating DPDKs atomic functions and failing checkpatch is a bit sudden. Perhaps noting that in a future release (20.11?) DPDK will move to a C11 based atomics model is a more gradual step to achieving the goal, and at that point add a checkpatch warning for additions of rte_atomic*?
More on [2] in context below.
The above is my point-of-view, of course I'd like more people from the DPDK community to provide their input too.
Quoted 8 lines
This patchset contains:
1) the checkpatch script changes to flagrte_atomicAPI usage in patches.
2) changes to programmer guide describing writing efficient code for aarch64.
3) changes to various libraries to make use of c11 atomic APIs.We are planning to replicate this idea across all the other libraries, drivers, examples, test applications. In the next phase, we will add changes to the mbuf, the EAL interrupts and the event timer adapter libraries.
About ~6/12 patches of this C11 set are targeting the Service Cores area of DPDK. I have some concerns over increased complexity of C11 implementation vs the (already complex)
rte_atomicimplementation today. I see other patchsets enabling C11 across other DPDK components, so maybe we should also discuss C11 enabling in a wider context that just service cores?I don't think it fair to expect all developers to be well versed in C11 atomic semantics like understanding the complex interactions between the various C11 RELEASE, AQUIRE barriers requires.
As maintainer of Service Cores I'm hesitant to accept the large-scale refactor of atomic-implementation, as it could lead to racey bugs that are likely extremely difficult to track down. (The recent race-on-exit has proven the difficulty in reproducing, and that's with an atomics model I'm quite familiar with).
Let me be very clear: I don't wish to block a C11 atomic implementation, but I'd like to discuss how we (DPDK community) can best port-to and maintain a complex multi-threaded service library with best-in-class performance for the workload.
To put some discussions/solutions on the table:
- Shared Maintainership of a component?
Split in functionality and C11 atomics implementation
Obviously there would be collaboration required in such a case.
- Maybe shared maintainership is too much?
A gentlemans/womans agreement of "helping out" with C11 atomics debug is enough?
Quoted 1 line
Hope my concerns are understandable, and of course input/feedback welcomed! -Harry
Thanks for raising the issue Harry.
I think we should have at least two official maintainers for C11 atomics in general. C11 conversion is a progressive effort being done, and should be merged step by step. If C11 maintainers fail to fix some issues on time, then we can hold the effort. Does it make sense?