Skip to contentSkip to conversation

generic rte atomic APIs deprecate proposal

PATCH Superseded Phil Yang 12 participants 219 messages Mar 10, 2020 – Jul 20, 2020 Arm/Amlogic Meson SoC support Page 1 of 2

lore
Share & tools
lore

Phil Yang

author
#0 lore raw

cover generic rte atomic APIs deprecate proposal

DPDK provides generic rte_atomic APIs to do several atomic operations. These APIs are using the deprecated __sync built-ins and enforce full memory barriers on aarch64. However, full barriers are not necessary in many use cases. In order to address such use cases, C language offers C11 atomic APIs. The C11 atomic APIs provide finer memory barrier control by making use of the memory ordering parameter provided by the user. Various patches submitted in the past [1] and the patches in this series indicate significant performance gains on multiple aarch64 CPUs and no performance loss on x86.

But the existing rte_atomic API implementations cannot be changed as the APIs do not take the memory ordering parameter. The only choice available is replacing the usage of the rte_atomic APIs with C11 atomic APIs. In order to make this change, the following steps are proposed:

[1] deprecate rte_atomic APIs so that future patches do not use rte_atomic APIs (a script is added to flag the usages).
[2] refactor the code that uses rte_atomic APIs to use c11 atomic APIs.

This patchset contains:
1) the checkpatch script changes to flag rte_atomic API usage in patches.
2) changes to programmer guide describing writing efficient code for aarch64.
3) changes to various libraries to make use of c11 atomic APIs.

We are planning to replicate this idea across all the other libraries, drivers, examples, test applications. In the next phase, we will add changes to the mbuf, the EAL interrupts and the event timer adapter libraries.

Honnappa Nagarahalli (2):
service: avoid race condition for MT unsafe service
service: identify service running on another core correctly

Phil Yang (8):
doc: add generic atomic deprecation section
devtools: prevent use of rte atomic APIs in future patches
vhost: optimize broadcast rarp sync with c11 atomic
ipsec: optimize with c11 atomic for sa outbound sqn update
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 one-way barrier
service: relax barriers with C11 atomic operations

 devtools/checkpatches.sh                         |   9 ++
 doc/guides/prog_guide/writing_efficient_code.rst |  60 +++++++-
 lib/librte_eal/common/rte_service.c              | 175 ++++++++++++-----------
 lib/librte_ipsec/ipsec_sqn.h                     |   3 +-
 lib/librte_ipsec/sa.h                            |   2 +-
 lib/librte_vhost/vhost.h                         |   2 +-
 lib/librte_vhost/vhost_user.c                    |   7 +-
 lib/librte_vhost/virtio_net.c                    |  16 ++-
 8 files changed, 175 insertions(+), 99 deletions(-)
Signature · 1 line
2.7.4
author
#1 lore raw

PATCH 1/10 doc: add generic atomic deprecation section

Add deprecating the generic rte_atomic_xx APIs to c11 atomic built-ins guide and examples.

---
 doc/guides/prog_guide/writing_efficient_code.rst | 60 +++++++++++++++++++++++-
 1 file changed, 59 insertions(+), 1 deletion(-)
doc/guides/prog_guide/writing_efficient_code.rst +59 -1
index 849f63e..b278bc6 100644
@@ -167,7 +167,13 @@ but with the added cost of lower throughput.
 Locks and Atomic Operations
 ---------------------------
 
-Atomic operations imply a lock prefix before the instruction,
+This section describes some key considerations when using locks and atomic
+operations in the DPDK environment.
+
+Locks
+~~~~~
+
+On x86, atomic operations imply a lock prefix before the instruction,
 causing the processor's LOCK# signal to be asserted during execution of the following instruction.
 This has a big impact on performance in a multicore environment.
 
@@ -176,6 +182,58 @@ It can often be replaced by other solutions like per-lcore variables.
 Also, some locking techniques are more efficient than others.
 For instance, the Read-Copy-Update (RCU) algorithm can frequently replace simple rwlocks.
 
+Atomic Operations: Use C11 Atomic Built-ins
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DPDK `generic rte_atomic <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_atomic.h>`_ operations are
+implemented by `__sync built-ins <https://gcc.gnu.org/onlinedocs/gcc/_005f_005fsync-Builtins.html>`_.
+These __sync built-ins result in full barriers on aarch64, which are unnecessary
+in many use cases. They can be replaced by `__atomic built-ins <https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html>`_ that
+conform to the C11 memory model and provide finer memory order control.
+
+So replacing the rte_atomic operations with __atomic built-ins might improve
+performance for aarch64 machines. `More details <https://www.dpdk.org/wp-content/uploads/sites/35/2019/10/StateofC11Code.pdf>`_.
+
+Some typical optimization cases are listed below:
+
+Atomicity
+^^^^^^^^^
+
+Some use cases require atomicity alone, the ordering of the memory operations
+does not matter. For example the packets statistics in the `vhost <https://github.com/DPDK/dpdk/blob/v20.02/examples/vhost/main.c#L796>`_ example application.
+
+It just updates the number of transmitted packets, no subsequent logic depends
+on these counters. So the RELAXED memory ordering is sufficient:
+
+.. code-block:: c
+
+    static __rte_always_inline void
+    virtio_xmit(struct vhost_dev *dst_vdev, struct vhost_dev *src_vdev,
+            struct rte_mbuf *m)
+    {
+        ...
+        ...
+        if (enable_stats) {
+            __atomic_add_fetch(&dst_vdev->stats.rx_total_atomic, 1, __ATOMIC_RELAXED);
+            __atomic_add_fetch(&dst_vdev->stats.rx_atomic, ret, __ATOMIC_RELAXED);
+            ...
+        }
+    }
+
+One-way Barrier
+^^^^^^^^^^^^^^^
+
+Some use cases allow for memory reordering in one way while requiring memory
+ordering in the other direction.
+
+For example, the memory operations before the `lock <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_spinlock.h#L66>`_ can move to the
+critical section, but the memory operations in the critical section cannot move
+above the lock. In this case, the full memory barrier in the CAS operation can
+be replaced to ACQUIRE. On the other hand, the memory operations after the
+`unlock <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_spinlock.h#L88>`_ can move to the critical section, but the memory operations in the
+critical section cannot move below the unlock. So the full barrier in the STORE
+operation can be replaced with RELEASE.
+
 Coding Considerations
 ---------------------
 
Signature · 1 line
2.7.4
author
#2 lore raw

PATCH 2/10 devtools: prevent use of rte atomic APIs in future patches

In reply to Phil Yang

In order to deprecate the rte_atomic APIs, prevent the patches from using rte_atomic APIs.

---
 devtools/checkpatches.sh | 9 +++++++++
 1 file changed, 9 insertions(+)
devtools/checkpatches.sh +9 -0
index 1794468..493f48e 100755
@@ -61,6 +61,15 @@ check_forbidden_additions() { # <patch>
 		-f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
 		"$1" || res=1
 
+	# refrain from new additions of 16/32/64 bits rte_atomic_xxx()
+	# multiple folders and expressions are separated by spaces
+	awk -v FOLDERS="lib drivers app examples" \
+		-v EXPRESSIONS="rte_atomic[0-9][0-9]_.*\\\(" \
+		-v RET_ON_FAIL=1 \
+		-v MESSAGE='Using c11 atomic built-ins instead of rte_atomic' \
+		-f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
+		"$1" || res=1
+
 	# svg figures must be included with wildcard extension
 	# because of png conversion for pdf docs
 	awk -v FOLDERS='doc' \
Signature · 1 line
2.7.4
author
#3 lore raw

PATCH 3/10 vhost: optimize broadcast rarp sync with c11 atomic

In reply to Phil Yang

The rarp packet broadcast flag is synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.

---
 lib/librte_vhost/vhost.h      |  2 +-
 lib/librte_vhost/vhost_user.c |  7 +++----
 lib/librte_vhost/virtio_net.c | 16 +++++++++-------
 3 files changed, 13 insertions(+), 12 deletions(-)
lib/librte_vhost/vhost.h +1 -1
index 2087d14..0e22125 100644
@@ -350,7 +350,7 @@ struct virtio_net {
 	uint32_t		flags;
 	uint16_t		vhost_hlen;
 	/* to tell if we need broadcast rarp packet */
-	rte_atomic16_t		broadcast_rarp;
+	int16_t			broadcast_rarp;
 	uint32_t		nr_vring;
 	int			dequeue_zero_copy;
 	int			extbuf;
lib/librte_vhost/vhost_user.c +3 -4
index bd1be01..857187d 100644
@@ -2145,11 +2145,10 @@ vhost_user_send_rarp(struct virtio_net **pdev, struct VhostUserMsg *msg,
 	 * Set the flag to inject a RARP broadcast packet at
 	 * rte_vhost_dequeue_burst().
 	 *
-	 * rte_smp_wmb() is for making sure the mac is copied
-	 * before the flag is set.
+	 * __ATOMIC_RELEASE ordering is for making sure the mac is
+	 * copied before the flag is set.
 	 */
-	rte_smp_wmb();
-	rte_atomic16_set(&dev->broadcast_rarp, 1);
+	__atomic_store_n(&dev->broadcast_rarp, 1, __ATOMIC_RELEASE);
 	did = dev->vdpa_dev_id;
 	vdpa_dev = rte_vdpa_get_device(did);
 	if (vdpa_dev && vdpa_dev->ops->migration_done)
lib/librte_vhost/virtio_net.c +9 -7
index 37c47c7..d20f60c 100644
@@ -2203,6 +2203,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
 	struct virtio_net *dev;
 	struct rte_mbuf *rarp_mbuf = NULL;
 	struct vhost_virtqueue *vq;
+	int success = 1;
 
 	dev = get_device(vid);
 	if (!dev)
@@ -2249,16 +2250,17 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
 	 *
 	 * broadcast_rarp shares a cacheline in the virtio_net structure
 	 * with some fields that are accessed during enqueue and
-	 * rte_atomic16_cmpset() causes a write if using cmpxchg. This could
-	 * result in false sharing between enqueue and dequeue.
+	 * __atomic_compare_exchange_n causes a write if performed compare
+	 * and exchange. This could result in false sharing between enqueue
+	 * and dequeue.
 	 *
 	 * Prevent unnecessary false sharing by reading broadcast_rarp first
-	 * and only performing cmpset if the read indicates it is likely to
-	 * be set.
+	 * and only performing compare and exchange if the read indicates it
+	 * is likely to be set.
 	 */
-	if (unlikely(rte_atomic16_read(&dev->broadcast_rarp) &&
-			rte_atomic16_cmpset((volatile uint16_t *)
-				&dev->broadcast_rarp.cnt, 1, 0))) {
+	if (unlikely(__atomic_load_n(&dev->broadcast_rarp, __ATOMIC_ACQUIRE) &&
+			__atomic_compare_exchange_n(&dev->broadcast_rarp,
+			&success, 0, 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED))) {
 
 		rarp_mbuf = rte_net_make_rarp_packet(mbuf_pool, &dev->mac);
 		if (rarp_mbuf == NULL) {
Signature · 1 line
2.7.4
author
#4 lore raw

PATCH 4/10 ipsec: optimize with c11 atomic for sa outbound sqn update

In reply to Phil Yang

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/sa.h        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
lib/librte_ipsec/ipsec_sqn.h +2 -1
index 0c2f76a..e884af7 100644
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)
 
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa->sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);
 	else {
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h +1 -1
index d22451b..cab9a2e 100644
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;
 		struct {
Signature · 1 line
2.7.4
author
#5 lore raw

PATCH 5/10 service: remove rte prefix from static functions

In reply to Phil Yang
Fixes 3cf5eb1546ed ("service: fix and refactor atomic service accesses")
Fixes 21698354c832 ("service: introduce service cores concept")
---
 lib/librte_eal/common/rte_service.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +9 -9
index 7e537b8..a691f5d 100644
@@ -333,7 +333,7 @@ rte_service_runstate_get(uint32_t id)
 }
 
 static inline void
-rte_service_runner_do_callback(struct rte_service_spec_impl *s,
+service_runner_do_callback(struct rte_service_spec_impl *s,
 			       struct core_state *cs, uint32_t service_idx)
 {
 	void *userdata = s->spec.callback_userdata;
@@ -376,10 +376,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 		rte_atomic32_clear(&s->execute_lock);
 	} else
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 
 	return 0;
 }
@@ -433,7 +433,7 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 }
 
 static int32_t
-rte_service_runner_func(void *arg)
+service_runner_func(void *arg)
 {
 	RTE_SET_USED(arg);
 	uint32_t i;
@@ -703,7 +703,7 @@ rte_service_lcore_start(uint32_t lcore)
 	 */
 	lcore_states[lcore].runstate = RUNSTATE_RUNNING;
 
-	int ret = rte_eal_remote_launch(rte_service_runner_func, 0, lcore);
+	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
 	return ret;
 }
@@ -782,7 +782,7 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id,
 }
 
 static void
-rte_service_dump_one(FILE *f, struct rte_service_spec_impl *s,
+service_dump_one(FILE *f, struct rte_service_spec_impl *s,
 		     uint64_t all_cycles, uint32_t reset)
 {
 	/* avoid divide by zero */
@@ -815,7 +815,7 @@ rte_service_attr_reset_all(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	int reset = 1;
-	rte_service_dump_one(NULL, s, 0, reset);
+	service_dump_one(NULL, s, 0, reset);
 	return 0;
 }
 
@@ -873,7 +873,7 @@ rte_service_dump(FILE *f, uint32_t id)
 		SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 		fprintf(f, "Service %s Summary\n", s->spec.name);
 		uint32_t reset = 0;
-		rte_service_dump_one(f, s, total_cycles, reset);
+		service_dump_one(f, s, total_cycles, reset);
 		return 0;
 	}
 
@@ -883,7 +883,7 @@ rte_service_dump(FILE *f, uint32_t id)
 		if (!service_valid(i))
 			continue;
 		uint32_t reset = 0;
-		rte_service_dump_one(f, &rte_services[i], total_cycles, reset);
+		service_dump_one(f, &rte_services[i], total_cycles, reset);
 	}
 
 	fprintf(f, "Service Cores Summary\n");
Signature · 1 line
2.7.4
author
#6 lore raw

PATCH 6/10 service: remove redundant code

In reply to Phil Yang

The service id validation is verified in the calling function, remove the redundant code inside the service_update function.

Fixes 21698354c832 ("service: introduce service cores concept")
---
 lib/librte_eal/common/rte_service.c | 31 ++++++++++++-------------------
 1 file changed, 12 insertions(+), 19 deletions(-)
lib/librte_eal/common/rte_service.c +12 -19
index a691f5d..6990dc2 100644
@@ -549,21 +549,10 @@ rte_service_start_with_defaults(void)
 }
 
 static int32_t
-service_update(struct rte_service_spec *service, uint32_t lcore,
+service_update(uint32_t sid, uint32_t lcore,
 		uint32_t *set, uint32_t *enabled)
 {
-	uint32_t i;
-	int32_t sid = -1;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if ((struct rte_service_spec *)&rte_services[i] == service &&
-				service_valid(i)) {
-			sid = i;
-			break;
-		}
-	}
-
-	if (sid == -1 || lcore >= RTE_MAX_LCORE)
+	if (lcore >= RTE_MAX_LCORE)
 		return -EINVAL;
 
 	if (!lcore_states[lcore].is_service_core)
@@ -595,19 +584,23 @@ service_update(struct rte_service_spec *service, uint32_t lcore,
 int32_t
 rte_service_map_lcore_set(uint32_t id, uint32_t lcore, uint32_t enabled)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
+	/* validate ID, or return error value */
+	if (id >= RTE_SERVICE_NUM_MAX || !service_valid(id))
+		return -EINVAL;
+
 	uint32_t on = enabled > 0;
-	return service_update(&s->spec, lcore, &on, 0);
+	return service_update(id, lcore, &on, 0);
 }
 
 int32_t
 rte_service_map_lcore_get(uint32_t id, uint32_t lcore)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
+	/* validate ID, or return error value */
+	if (id >= RTE_SERVICE_NUM_MAX || !service_valid(id))
+		return -EINVAL;
+
 	uint32_t enabled;
-	int ret = service_update(&s->spec, lcore, 0, &enabled);
+	int ret = service_update(id, lcore, 0, &enabled);
 	if (ret == 0)
 		return enabled;
 	return ret;
Signature · 1 line
2.7.4
author
#7 lore raw

PATCH 7/10 service: avoid race condition for MT unsafe service

In reply to Phil Yang

From: Honnappa Nagarahalli <[email protected]>

There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Fixes e9139a32f6e8 ("service: add function to run on app lcore")
---
 lib/librte_eal/common/rte_service.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
index 6990dc2..b37fc56 100644
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {
 	uint8_t internal_flags;
 
 	/* per service statistics */
+	/* Indicates how many cores the service is mapped to run on.
+	 * It does not indicate the number of cores the service is running
+	 * on currently.
+	 */
 	rte_atomic32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
@@ -367,12 +371,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	/* check do we need cmpset, if MT safe or <= 1 core
-	 * mapped, atomic ops are not required.
-	 */
-	const int use_atomics = (service_mt_safe(s) == 0) &&
-				(rte_atomic32_read(&s->num_mapped_cores) > 1);
-	if (use_atomics) {
+	if (service_mt_safe(s) == 0) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
Signature · 1 line
2.7.4
author
#8 lore raw

PATCH 8/10 service: identify service running on another core correctly

In reply to Phil Yang

From: Honnappa Nagarahalli <[email protected]>

The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes more costlier than using atomic operations. Assume that the application passes the right parameters and reduce the number of instructions for all cases.

---
 lib/librte_eal/common/rte_service.c | 26 ++++++++------------------
 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
index b37fc56..0186024 100644
@@ -357,7 +357,7 @@ service_runner_do_callback(struct rte_service_spec_impl *s,
 /* Expects the service 's' is valid. */
 static int32_t
 service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
-	    struct rte_service_spec_impl *s)
+	    struct rte_service_spec_impl *s, uint32_t serialize_mt_unsafe)
 {
 	if (!s)
 		return -EINVAL;
@@ -371,7 +371,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	if (service_mt_safe(s) == 0) {
+	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
@@ -409,24 +409,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
-	/* Atomically add this core to the mapped cores first, then examine if
-	 * we can run the service. This avoids a race condition between
-	 * checking the value, and atomically adding to the mapped count.
+	/* Increment num_mapped_cores to indicate that the service is
+	 * is running on a core.
 	 */
-	if (serialize_mt_unsafe)
-		rte_atomic32_inc(&s->num_mapped_cores);
+	rte_atomic32_inc(&s->num_mapped_cores);
 
-	if (service_mt_safe(s) == 0 &&
-			rte_atomic32_read(&s->num_mapped_cores) > 1) {
-		if (serialize_mt_unsafe)
-			rte_atomic32_dec(&s->num_mapped_cores);
-		return -EBUSY;
-	}
-
-	int ret = service_run(id, cs, UINT64_MAX, s);
+	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	if (serialize_mt_unsafe)
-		rte_atomic32_dec(&s->num_mapped_cores);
+	rte_atomic32_dec(&s->num_mapped_cores);
 
 	return ret;
 }
@@ -446,7 +436,7 @@ service_runner_func(void *arg)
 			if (!service_valid(i))
 				continue;
 			/* return value ignored as no change to code flow */
-			service_run(i, cs, service_mask, service_get(i));
+			service_run(i, cs, service_mask, service_get(i), 1);
 		}
 
 		cs->loops++;
Signature · 1 line
2.7.4
author
#9 lore raw

PATCH 9/10 service: optimize with c11 one-way barrier

In reply to Phil Yang

The num_mapped_cores and execute_lock are synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.

---
 lib/librte_eal/common/rte_service.c | 50 ++++++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 15 deletions(-)
lib/librte_eal/common/rte_service.c +35 -15
index 0186024..efb3c9f 100644
@@ -42,7 +42,7 @@ struct rte_service_spec_impl {
 	 * running this service callback. When not set, a core may take the
 	 * lock and then run the service callback.
 	 */
-	rte_atomic32_t execute_lock;
+	uint32_t execute_lock;
 
 	/* API set/get-able variables */
 	int8_t app_runstate;
@@ -54,7 +54,7 @@ struct rte_service_spec_impl {
 	 * It does not indicate the number of cores the service is running
 	 * on currently.
 	 */
-	rte_atomic32_t num_mapped_cores;
+	int32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
 } __rte_cache_aligned;
@@ -329,7 +329,8 @@ rte_service_runstate_get(uint32_t id)
 	rte_smp_rmb();
 
 	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (rte_atomic32_read(&s->num_mapped_cores) > 0);
+	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+					    __ATOMIC_RELAXED) > 0);
 
 	return (s->app_runstate == RUNSTATE_RUNNING) &&
 		(s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -372,11 +373,20 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 	cs->service_active_on_lcore[i] = 1;
 
 	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
-		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
+		uint32_t expected = 0;
+		/* ACQUIRE ordering here is to prevent the callback
+		 * function from hoisting up before the execute_lock
+		 * setting.
+		 */
+		if (!__atomic_compare_exchange_n(&s->execute_lock, &expected, 1,
+			    0, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED))
 			return -EBUSY;
 
 		service_runner_do_callback(s, cs, i);
-		rte_atomic32_clear(&s->execute_lock);
+		/* RELEASE ordering here is used to pair with ACQUIRE
+		 * above to achieve lock semantic.
+		 */
+		__atomic_store_n(&s->execute_lock, 0, __ATOMIC_RELEASE);
 	} else
 		service_runner_do_callback(s, cs, i);
 
@@ -412,11 +422,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 	/* Increment num_mapped_cores to indicate that the service is
 	 * is running on a core.
 	 */
-	rte_atomic32_inc(&s->num_mapped_cores);
+	__atomic_add_fetch(&s->num_mapped_cores, 1, __ATOMIC_ACQUIRE);
 
 	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	rte_atomic32_dec(&s->num_mapped_cores);
+	__atomic_sub_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELEASE);
 
 	return ret;
 }
@@ -549,24 +559,32 @@ service_update(uint32_t sid, uint32_t lcore,
 
 	uint64_t sid_mask = UINT64_C(1) << sid;
 	if (set) {
-		uint64_t lcore_mapped = lcore_states[lcore].service_mask &
-			sid_mask;
+		/* When multiple threads try to update the same lcore
+		 * service concurrently, e.g. set lcore map followed
+		 * by clear lcore map, the unsynchronized service_mask
+		 * values have issues on the num_mapped_cores value
+		 * consistency. So we use ACQUIRE ordering to pair with
+		 * the RELEASE ordering to synchronize the service_mask.
+		 */
+		uint64_t lcore_mapped = __atomic_load_n(
+					&lcore_states[lcore].service_mask,
+					__ATOMIC_ACQUIRE) & sid_mask;
 
 		if (*set && !lcore_mapped) {
 			lcore_states[lcore].service_mask |= sid_mask;
-			rte_atomic32_inc(&rte_services[sid].num_mapped_cores);
+			__atomic_add_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELEASE);
 		}
 		if (!*set && lcore_mapped) {
 			lcore_states[lcore].service_mask &= ~(sid_mask);
-			rte_atomic32_dec(&rte_services[sid].num_mapped_cores);
+			__atomic_sub_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELEASE);
 		}
 	}
 
 	if (enabled)
 		*enabled = !!(lcore_states[lcore].service_mask & (sid_mask));
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -622,7 +640,8 @@ rte_service_lcore_reset_all(void)
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
-		rte_atomic32_set(&rte_services[i].num_mapped_cores, 0);
+		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
+				    __ATOMIC_RELAXED);
 
 	rte_smp_wmb();
 
@@ -705,7 +724,8 @@ rte_service_lcore_stop(uint32_t lcore)
 		int32_t enabled = service_mask & (UINT64_C(1) << i);
 		int32_t service_running = rte_service_runstate_get(i);
 		int32_t only_core = (1 ==
-			rte_atomic32_read(&rte_services[i].num_mapped_cores));
+			__atomic_load_n(&rte_services[i].num_mapped_cores,
+					__ATOMIC_RELAXED));
 
 		/* if the core is mapped, and the service is running, and this
 		 * is the only core that is mapped, the service would cease to
Signature · 1 line
2.7.4
author
#10 lore raw

PATCH 10/10 service: relax barriers with C11 atomic operations

In reply to Phil Yang

To guarantee the inter-threads visibility of the shareable domain, it uses a lot of rte_smp_r/wmb in the service library. This patch relaxed these barriers for service by using c11 atomic one-way barrier operations.

---
 lib/librte_eal/common/rte_service.c | 45 ++++++++++++++++++++-----------------
 1 file changed, 25 insertions(+), 20 deletions(-)
lib/librte_eal/common/rte_service.c +25 -20
index efb3c9f..68542b0 100644
@@ -176,9 +176,11 @@ rte_service_set_stats_enable(uint32_t id, int32_t enabled)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, 0);
 
 	if (enabled)
-		s->internal_flags |= SERVICE_F_STATS_ENABLED;
+		__atomic_or_fetch(&s->internal_flags, SERVICE_F_STATS_ENABLED,
+			__ATOMIC_RELEASE);
 	else
-		s->internal_flags &= ~(SERVICE_F_STATS_ENABLED);
+		__atomic_and_fetch(&s->internal_flags,
+			~(SERVICE_F_STATS_ENABLED), __ATOMIC_RELEASE);
 
 	return 0;
 }
@@ -190,9 +192,11 @@ rte_service_set_runstate_mapped_check(uint32_t id, int32_t enabled)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, 0);
 
 	if (enabled)
-		s->internal_flags |= SERVICE_F_START_CHECK;
+		__atomic_or_fetch(&s->internal_flags, SERVICE_F_START_CHECK,
+			__ATOMIC_RELEASE);
 	else
-		s->internal_flags &= ~(SERVICE_F_START_CHECK);
+		__atomic_and_fetch(&s->internal_flags, ~(SERVICE_F_START_CHECK),
+			__ATOMIC_RELEASE);
 
 	return 0;
 }
@@ -261,8 +265,8 @@ rte_service_component_register(const struct rte_service_spec *spec,
 	s->spec = *spec;
 	s->internal_flags |= SERVICE_F_REGISTERED | SERVICE_F_START_CHECK;
 
-	rte_smp_wmb();
-	rte_service_count++;
+	/* make sure the counter update after the state change. */
+	__atomic_add_fetch(&rte_service_count, 1, __ATOMIC_RELEASE);
 
 	if (id_ptr)
 		*id_ptr = free_slot;
@@ -278,9 +282,10 @@ rte_service_component_unregister(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	rte_service_count--;
-	rte_smp_wmb();
 
-	s->internal_flags &= ~(SERVICE_F_REGISTERED);
+	/* make sure the counter update before the state change. */
+	__atomic_and_fetch(&s->internal_flags, ~(SERVICE_F_REGISTERED),
+			   __ATOMIC_RELEASE);
 
 	/* clear the run-bit in all cores */
 	for (i = 0; i < RTE_MAX_LCORE; i++)
@@ -298,11 +303,12 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	if (runstate)
-		s->comp_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->comp_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -313,11 +319,12 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	if (runstate)
-		s->app_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->app_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -439,7 +446,8 @@ service_runner_func(void *arg)
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (lcore_states[lcore].runstate == RUNSTATE_RUNNING) {
+	while (__atomic_load_n(&cs->runstate,
+		    __ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
@@ -450,8 +458,6 @@ service_runner_func(void *arg)
 		}
 
 		cs->loops++;
-
-		rte_smp_rmb();
 	}
 
 	lcore_config[lcore].state = WAIT;
@@ -660,9 +666,8 @@ rte_service_lcore_add(uint32_t lcore)
 
 	/* ensure that after adding a core the mask and state are defaults */
 	lcore_states[lcore].service_mask = 0;
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
-
-	rte_smp_wmb();
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+			__ATOMIC_RELEASE);
 
 	return rte_eal_wait_lcore(lcore);
 }
Signature · 1 line
2.7.4
author
#11 lore raw

v2 cover generic rte atomic APIs deprecate proposal

In reply to Phil Yang

DPDK provides generic rte_atomic APIs to do several atomic operations. These APIs are using the deprecated __sync built-ins and enforce full memory barriers on aarch64. However, full barriers are not necessary in many use cases. In order to address such use cases, C language offers C11 atomic APIs. The C11 atomic APIs provide finer memory barrier control by making use of the memory ordering parameter provided by the user. Various patches submitted in the past [1] and the patches in this series indicate significant performance gains on multiple aarch64 CPUs and no performance loss on x86.

But the existing rte_atomic API implementations cannot be changed as the APIs do not take the memory ordering parameter. The only choice available is replacing the usage of the rte_atomic APIs with C11 atomic APIs. In order to make this change, the following steps are proposed:

[1] deprecate rte_atomic APIs so that future patches do not use rte_atomic APIs (a script is added to flag the usages).
[2] refactor the code that uses rte_atomic APIs to use c11 atomic APIs.

This patchset contains:
1) the checkpatch script changes to flag rte_atomic API usage in patches.
2) changes to programmer guide describing writing efficient code for aarch64.
3) changes to various libraries to make use of c11 atomic APIs.

We are planning to replicate this idea across all the other libraries, drivers, examples, test applications. In the next phase, we will add changes to the mbuf, the EAL interrupts and the event timer adapter libraries.

v2:
1. fix Clang '-Wincompatible-pointer-types' WARNING.
2. fix typos.

Honnappa Nagarahalli (2):
service: avoid race condition for MT unsafe service
service: identify service running on another core correctly

Phil Yang (8):
doc: add generic atomic deprecation section
devtools: prevent use of rte atomic APIs in future patches
vhost: optimize broadcast rarp sync with c11 atomic
ipsec: optimize with c11 atomic for sa outbound sqn update
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 one-way barrier
service: relax barriers with C11 atomic operations

 devtools/checkpatches.sh                         |   9 ++
 doc/guides/prog_guide/writing_efficient_code.rst |  60 +++++++-
 lib/librte_eal/common/rte_service.c              | 175 ++++++++++++-----------
 lib/librte_ipsec/ipsec_sqn.h                     |   3 +-
 lib/librte_ipsec/sa.h                            |   2 +-
 lib/librte_vhost/vhost.h                         |   2 +-
 lib/librte_vhost/vhost_user.c                    |   7 +-
 lib/librte_vhost/virtio_net.c                    |  16 ++-
 8 files changed, 175 insertions(+), 99 deletions(-)
Signature · 1 line
2.7.4
author
#12 lore raw

PATCH v2 1/10 doc: add generic atomic deprecation section

Add deprecating the generic rte_atomic_xx APIs to c11 atomic built-ins guide and examples.

---
 doc/guides/prog_guide/writing_efficient_code.rst | 60 +++++++++++++++++++++++-
 1 file changed, 59 insertions(+), 1 deletion(-)
doc/guides/prog_guide/writing_efficient_code.rst +59 -1
index 849f63e..b278bc6 100644
@@ -167,7 +167,13 @@ but with the added cost of lower throughput.
 Locks and Atomic Operations
 ---------------------------
 
-Atomic operations imply a lock prefix before the instruction,
+This section describes some key considerations when using locks and atomic
+operations in the DPDK environment.
+
+Locks
+~~~~~
+
+On x86, atomic operations imply a lock prefix before the instruction,
 causing the processor's LOCK# signal to be asserted during execution of the following instruction.
 This has a big impact on performance in a multicore environment.
 
@@ -176,6 +182,58 @@ It can often be replaced by other solutions like per-lcore variables.
 Also, some locking techniques are more efficient than others.
 For instance, the Read-Copy-Update (RCU) algorithm can frequently replace simple rwlocks.
 
+Atomic Operations: Use C11 Atomic Built-ins
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DPDK `generic rte_atomic <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_atomic.h>`_ operations are
+implemented by `__sync built-ins <https://gcc.gnu.org/onlinedocs/gcc/_005f_005fsync-Builtins.html>`_.
+These __sync built-ins result in full barriers on aarch64, which are unnecessary
+in many use cases. They can be replaced by `__atomic built-ins <https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html>`_ that
+conform to the C11 memory model and provide finer memory order control.
+
+So replacing the rte_atomic operations with __atomic built-ins might improve
+performance for aarch64 machines. `More details <https://www.dpdk.org/wp-content/uploads/sites/35/2019/10/StateofC11Code.pdf>`_.
+
+Some typical optimization cases are listed below:
+
+Atomicity
+^^^^^^^^^
+
+Some use cases require atomicity alone, the ordering of the memory operations
+does not matter. For example the packets statistics in the `vhost <https://github.com/DPDK/dpdk/blob/v20.02/examples/vhost/main.c#L796>`_ example application.
+
+It just updates the number of transmitted packets, no subsequent logic depends
+on these counters. So the RELAXED memory ordering is sufficient:
+
+.. code-block:: c
+
+    static __rte_always_inline void
+    virtio_xmit(struct vhost_dev *dst_vdev, struct vhost_dev *src_vdev,
+            struct rte_mbuf *m)
+    {
+        ...
+        ...
+        if (enable_stats) {
+            __atomic_add_fetch(&dst_vdev->stats.rx_total_atomic, 1, __ATOMIC_RELAXED);
+            __atomic_add_fetch(&dst_vdev->stats.rx_atomic, ret, __ATOMIC_RELAXED);
+            ...
+        }
+    }
+
+One-way Barrier
+^^^^^^^^^^^^^^^
+
+Some use cases allow for memory reordering in one way while requiring memory
+ordering in the other direction.
+
+For example, the memory operations before the `lock <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_spinlock.h#L66>`_ can move to the
+critical section, but the memory operations in the critical section cannot move
+above the lock. In this case, the full memory barrier in the CAS operation can
+be replaced to ACQUIRE. On the other hand, the memory operations after the
+`unlock <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_spinlock.h#L88>`_ can move to the critical section, but the memory operations in the
+critical section cannot move below the unlock. So the full barrier in the STORE
+operation can be replaced with RELEASE.
+
 Coding Considerations
 ---------------------
 
Signature · 1 line
2.7.4
author
#13 lore raw

PATCH v2 2/10 devtools: prevent use of rte atomic APIs in future patches

In reply to Phil Yang

In order to deprecate the rte_atomic APIs, prevent the patches from using rte_atomic APIs.

---
 devtools/checkpatches.sh | 9 +++++++++
 1 file changed, 9 insertions(+)
devtools/checkpatches.sh +9 -0
index 1794468..493f48e 100755
@@ -61,6 +61,15 @@ check_forbidden_additions() { # <patch>
 		-f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
 		"$1" || res=1
 
+	# refrain from new additions of 16/32/64 bits rte_atomic_xxx()
+	# multiple folders and expressions are separated by spaces
+	awk -v FOLDERS="lib drivers app examples" \
+		-v EXPRESSIONS="rte_atomic[0-9][0-9]_.*\\\(" \
+		-v RET_ON_FAIL=1 \
+		-v MESSAGE='Using c11 atomic built-ins instead of rte_atomic' \
+		-f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
+		"$1" || res=1
+
 	# svg figures must be included with wildcard extension
 	# because of png conversion for pdf docs
 	awk -v FOLDERS='doc' \
Signature · 1 line
2.7.4
author
#14 lore raw

PATCH v2 3/10 vhost: optimize broadcast rarp sync with c11 atomic

In reply to Phil Yang

The rarp packet broadcast flag is synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.

---
 lib/librte_vhost/vhost.h      |  2 +-
 lib/librte_vhost/vhost_user.c |  7 +++----
 lib/librte_vhost/virtio_net.c | 16 +++++++++-------
 3 files changed, 13 insertions(+), 12 deletions(-)
lib/librte_vhost/vhost.h +1 -1
index 2087d14..0e22125 100644
@@ -350,7 +350,7 @@ struct virtio_net {
 	uint32_t		flags;
 	uint16_t		vhost_hlen;
 	/* to tell if we need broadcast rarp packet */
-	rte_atomic16_t		broadcast_rarp;
+	int16_t			broadcast_rarp;
 	uint32_t		nr_vring;
 	int			dequeue_zero_copy;
 	int			extbuf;
lib/librte_vhost/vhost_user.c +3 -4
index bd1be01..857187d 100644
@@ -2145,11 +2145,10 @@ vhost_user_send_rarp(struct virtio_net **pdev, struct VhostUserMsg *msg,
 	 * Set the flag to inject a RARP broadcast packet at
 	 * rte_vhost_dequeue_burst().
 	 *
-	 * rte_smp_wmb() is for making sure the mac is copied
-	 * before the flag is set.
+	 * __ATOMIC_RELEASE ordering is for making sure the mac is
+	 * copied before the flag is set.
 	 */
-	rte_smp_wmb();
-	rte_atomic16_set(&dev->broadcast_rarp, 1);
+	__atomic_store_n(&dev->broadcast_rarp, 1, __ATOMIC_RELEASE);
 	did = dev->vdpa_dev_id;
 	vdpa_dev = rte_vdpa_get_device(did);
 	if (vdpa_dev && vdpa_dev->ops->migration_done)
lib/librte_vhost/virtio_net.c +9 -7
index 37c47c7..fa10deb 100644
@@ -2203,6 +2203,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
 	struct virtio_net *dev;
 	struct rte_mbuf *rarp_mbuf = NULL;
 	struct vhost_virtqueue *vq;
+	int16_t success = 1;
 
 	dev = get_device(vid);
 	if (!dev)
@@ -2249,16 +2250,17 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
 	 *
 	 * broadcast_rarp shares a cacheline in the virtio_net structure
 	 * with some fields that are accessed during enqueue and
-	 * rte_atomic16_cmpset() causes a write if using cmpxchg. This could
-	 * result in false sharing between enqueue and dequeue.
+	 * __atomic_compare_exchange_n causes a write if performed compare
+	 * and exchange. This could result in false sharing between enqueue
+	 * and dequeue.
 	 *
 	 * Prevent unnecessary false sharing by reading broadcast_rarp first
-	 * and only performing cmpset if the read indicates it is likely to
-	 * be set.
+	 * and only performing compare and exchange if the read indicates it
+	 * is likely to be set.
 	 */
-	if (unlikely(rte_atomic16_read(&dev->broadcast_rarp) &&
-			rte_atomic16_cmpset((volatile uint16_t *)
-				&dev->broadcast_rarp.cnt, 1, 0))) {
+	if (unlikely(__atomic_load_n(&dev->broadcast_rarp, __ATOMIC_ACQUIRE) &&
+			__atomic_compare_exchange_n(&dev->broadcast_rarp,
+			&success, 0, 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED))) {
 
 		rarp_mbuf = rte_net_make_rarp_packet(mbuf_pool, &dev->mac);
 		if (rarp_mbuf == NULL) {
Signature · 1 line
2.7.4
author
#15 lore raw

PATCH v2 4/10 ipsec: optimize with c11 atomic for sa outbound sqn update

In reply to Phil Yang

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/sa.h        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
lib/librte_ipsec/ipsec_sqn.h +2 -1
index 0c2f76a..e884af7 100644
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)
 
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa->sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);
 	else {
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h +1 -1
index d22451b..cab9a2e 100644
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;
 		struct {
Signature · 1 line
2.7.4
author
#16 lore raw

PATCH v2 5/10 service: remove rte prefix from static functions

In reply to Phil Yang
Fixes 3cf5eb1546ed ("service: fix and refactor atomic service accesses")
Fixes 21698354c832 ("service: introduce service cores concept")
---
 lib/librte_eal/common/rte_service.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +9 -9
index 7e537b8..a691f5d 100644
@@ -333,7 +333,7 @@ rte_service_runstate_get(uint32_t id)
 }
 
 static inline void
-rte_service_runner_do_callback(struct rte_service_spec_impl *s,
+service_runner_do_callback(struct rte_service_spec_impl *s,
 			       struct core_state *cs, uint32_t service_idx)
 {
 	void *userdata = s->spec.callback_userdata;
@@ -376,10 +376,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 		rte_atomic32_clear(&s->execute_lock);
 	} else
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 
 	return 0;
 }
@@ -433,7 +433,7 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 }
 
 static int32_t
-rte_service_runner_func(void *arg)
+service_runner_func(void *arg)
 {
 	RTE_SET_USED(arg);
 	uint32_t i;
@@ -703,7 +703,7 @@ rte_service_lcore_start(uint32_t lcore)
 	 */
 	lcore_states[lcore].runstate = RUNSTATE_RUNNING;
 
-	int ret = rte_eal_remote_launch(rte_service_runner_func, 0, lcore);
+	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
 	return ret;
 }
@@ -782,7 +782,7 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id,
 }
 
 static void
-rte_service_dump_one(FILE *f, struct rte_service_spec_impl *s,
+service_dump_one(FILE *f, struct rte_service_spec_impl *s,
 		     uint64_t all_cycles, uint32_t reset)
 {
 	/* avoid divide by zero */
@@ -815,7 +815,7 @@ rte_service_attr_reset_all(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	int reset = 1;
-	rte_service_dump_one(NULL, s, 0, reset);
+	service_dump_one(NULL, s, 0, reset);
 	return 0;
 }
 
@@ -873,7 +873,7 @@ rte_service_dump(FILE *f, uint32_t id)
 		SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 		fprintf(f, "Service %s Summary\n", s->spec.name);
 		uint32_t reset = 0;
-		rte_service_dump_one(f, s, total_cycles, reset);
+		service_dump_one(f, s, total_cycles, reset);
 		return 0;
 	}
 
@@ -883,7 +883,7 @@ rte_service_dump(FILE *f, uint32_t id)
 		if (!service_valid(i))
 			continue;
 		uint32_t reset = 0;
-		rte_service_dump_one(f, &rte_services[i], total_cycles, reset);
+		service_dump_one(f, &rte_services[i], total_cycles, reset);
 	}
 
 	fprintf(f, "Service Cores Summary\n");
Signature · 1 line
2.7.4
author
#17 lore raw

PATCH v2 6/10 service: remove redundant code

In reply to Phil Yang

The service id validation is verified in the calling function, remove the redundant code inside the service_update function.

Fixes 21698354c832 ("service: introduce service cores concept")
---
 lib/librte_eal/common/rte_service.c | 31 ++++++++++++-------------------
 1 file changed, 12 insertions(+), 19 deletions(-)
lib/librte_eal/common/rte_service.c +12 -19
index a691f5d..6990dc2 100644
@@ -549,21 +549,10 @@ rte_service_start_with_defaults(void)
 }
 
 static int32_t
-service_update(struct rte_service_spec *service, uint32_t lcore,
+service_update(uint32_t sid, uint32_t lcore,
 		uint32_t *set, uint32_t *enabled)
 {
-	uint32_t i;
-	int32_t sid = -1;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if ((struct rte_service_spec *)&rte_services[i] == service &&
-				service_valid(i)) {
-			sid = i;
-			break;
-		}
-	}
-
-	if (sid == -1 || lcore >= RTE_MAX_LCORE)
+	if (lcore >= RTE_MAX_LCORE)
 		return -EINVAL;
 
 	if (!lcore_states[lcore].is_service_core)
@@ -595,19 +584,23 @@ service_update(struct rte_service_spec *service, uint32_t lcore,
 int32_t
 rte_service_map_lcore_set(uint32_t id, uint32_t lcore, uint32_t enabled)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
+	/* validate ID, or return error value */
+	if (id >= RTE_SERVICE_NUM_MAX || !service_valid(id))
+		return -EINVAL;
+
 	uint32_t on = enabled > 0;
-	return service_update(&s->spec, lcore, &on, 0);
+	return service_update(id, lcore, &on, 0);
 }
 
 int32_t
 rte_service_map_lcore_get(uint32_t id, uint32_t lcore)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
+	/* validate ID, or return error value */
+	if (id >= RTE_SERVICE_NUM_MAX || !service_valid(id))
+		return -EINVAL;
+
 	uint32_t enabled;
-	int ret = service_update(&s->spec, lcore, 0, &enabled);
+	int ret = service_update(id, lcore, 0, &enabled);
 	if (ret == 0)
 		return enabled;
 	return ret;
Signature · 1 line
2.7.4
author
#18 lore raw

PATCH v2 7/10 service: avoid race condition for MT unsafe service

In reply to Phil Yang

From: Honnappa Nagarahalli <[email protected]>

There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Fixes e9139a32f6e8 ("service: add function to run on app lcore")
---
 lib/librte_eal/common/rte_service.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
index 6990dc2..b37fc56 100644
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {
 	uint8_t internal_flags;
 
 	/* per service statistics */
+	/* Indicates how many cores the service is mapped to run on.
+	 * It does not indicate the number of cores the service is running
+	 * on currently.
+	 */
 	rte_atomic32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
@@ -367,12 +371,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	/* check do we need cmpset, if MT safe or <= 1 core
-	 * mapped, atomic ops are not required.
-	 */
-	const int use_atomics = (service_mt_safe(s) == 0) &&
-				(rte_atomic32_read(&s->num_mapped_cores) > 1);
-	if (use_atomics) {
+	if (service_mt_safe(s) == 0) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
Signature · 1 line
2.7.4
author
#19 lore raw

PATCH v2 8/10 service: identify service running on another core correctly

In reply to Phil Yang

From: Honnappa Nagarahalli <[email protected]>

The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases.

---
 lib/librte_eal/common/rte_service.c | 26 ++++++++------------------
 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
index b37fc56..670f5a9 100644
@@ -357,7 +357,7 @@ service_runner_do_callback(struct rte_service_spec_impl *s,
 /* Expects the service 's' is valid. */
 static int32_t
 service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
-	    struct rte_service_spec_impl *s)
+	    struct rte_service_spec_impl *s, uint32_t serialize_mt_unsafe)
 {
 	if (!s)
 		return -EINVAL;
@@ -371,7 +371,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	if (service_mt_safe(s) == 0) {
+	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
@@ -409,24 +409,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
-	/* Atomically add this core to the mapped cores first, then examine if
-	 * we can run the service. This avoids a race condition between
-	 * checking the value, and atomically adding to the mapped count.
+	/* Increment num_mapped_cores to indicate that the service
+	 * is running on a core.
 	 */
-	if (serialize_mt_unsafe)
-		rte_atomic32_inc(&s->num_mapped_cores);
+	rte_atomic32_inc(&s->num_mapped_cores);
 
-	if (service_mt_safe(s) == 0 &&
-			rte_atomic32_read(&s->num_mapped_cores) > 1) {
-		if (serialize_mt_unsafe)
-			rte_atomic32_dec(&s->num_mapped_cores);
-		return -EBUSY;
-	}
-
-	int ret = service_run(id, cs, UINT64_MAX, s);
+	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	if (serialize_mt_unsafe)
-		rte_atomic32_dec(&s->num_mapped_cores);
+	rte_atomic32_dec(&s->num_mapped_cores);
 
 	return ret;
 }
@@ -446,7 +436,7 @@ service_runner_func(void *arg)
 			if (!service_valid(i))
 				continue;
 			/* return value ignored as no change to code flow */
-			service_run(i, cs, service_mask, service_get(i));
+			service_run(i, cs, service_mask, service_get(i), 1);
 		}
 
 		cs->loops++;
Signature · 1 line
2.7.4
author
#20 lore raw

PATCH v2 9/10 service: optimize with c11 one-way barrier

In reply to Phil Yang

The num_mapped_cores and execute_lock are synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.

---
 lib/librte_eal/common/rte_service.c | 50 ++++++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 15 deletions(-)
lib/librte_eal/common/rte_service.c +35 -15
index 670f5a9..96a59b6 100644
@@ -42,7 +42,7 @@ struct rte_service_spec_impl {
 	 * running this service callback. When not set, a core may take the
 	 * lock and then run the service callback.
 	 */
-	rte_atomic32_t execute_lock;
+	uint32_t execute_lock;
 
 	/* API set/get-able variables */
 	int8_t app_runstate;
@@ -54,7 +54,7 @@ struct rte_service_spec_impl {
 	 * It does not indicate the number of cores the service is running
 	 * on currently.
 	 */
-	rte_atomic32_t num_mapped_cores;
+	int32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
 } __rte_cache_aligned;
@@ -329,7 +329,8 @@ rte_service_runstate_get(uint32_t id)
 	rte_smp_rmb();
 
 	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (rte_atomic32_read(&s->num_mapped_cores) > 0);
+	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+					    __ATOMIC_RELAXED) > 0);
 
 	return (s->app_runstate == RUNSTATE_RUNNING) &&
 		(s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -372,11 +373,20 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 	cs->service_active_on_lcore[i] = 1;
 
 	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
-		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
+		uint32_t expected = 0;
+		/* ACQUIRE ordering here is to prevent the callback
+		 * function from hoisting up before the execute_lock
+		 * setting.
+		 */
+		if (!__atomic_compare_exchange_n(&s->execute_lock, &expected, 1,
+			    0, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED))
 			return -EBUSY;
 
 		service_runner_do_callback(s, cs, i);
-		rte_atomic32_clear(&s->execute_lock);
+		/* RELEASE ordering here is used to pair with ACQUIRE
+		 * above to achieve lock semantic.
+		 */
+		__atomic_store_n(&s->execute_lock, 0, __ATOMIC_RELEASE);
 	} else
 		service_runner_do_callback(s, cs, i);
 
@@ -412,11 +422,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 	/* Increment num_mapped_cores to indicate that the service
 	 * is running on a core.
 	 */
-	rte_atomic32_inc(&s->num_mapped_cores);
+	__atomic_add_fetch(&s->num_mapped_cores, 1, __ATOMIC_ACQUIRE);
 
 	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	rte_atomic32_dec(&s->num_mapped_cores);
+	__atomic_sub_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELEASE);
 
 	return ret;
 }
@@ -549,24 +559,32 @@ service_update(uint32_t sid, uint32_t lcore,
 
 	uint64_t sid_mask = UINT64_C(1) << sid;
 	if (set) {
-		uint64_t lcore_mapped = lcore_states[lcore].service_mask &
-			sid_mask;
+		/* When multiple threads try to update the same lcore
+		 * service concurrently, e.g. set lcore map followed
+		 * by clear lcore map, the unsynchronized service_mask
+		 * values have issues on the num_mapped_cores value
+		 * consistency. So we use ACQUIRE ordering to pair with
+		 * the RELEASE ordering to synchronize the service_mask.
+		 */
+		uint64_t lcore_mapped = __atomic_load_n(
+					&lcore_states[lcore].service_mask,
+					__ATOMIC_ACQUIRE) & sid_mask;
 
 		if (*set && !lcore_mapped) {
 			lcore_states[lcore].service_mask |= sid_mask;
-			rte_atomic32_inc(&rte_services[sid].num_mapped_cores);
+			__atomic_add_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELEASE);
 		}
 		if (!*set && lcore_mapped) {
 			lcore_states[lcore].service_mask &= ~(sid_mask);
-			rte_atomic32_dec(&rte_services[sid].num_mapped_cores);
+			__atomic_sub_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELEASE);
 		}
 	}
 
 	if (enabled)
 		*enabled = !!(lcore_states[lcore].service_mask & (sid_mask));
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -622,7 +640,8 @@ rte_service_lcore_reset_all(void)
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
-		rte_atomic32_set(&rte_services[i].num_mapped_cores, 0);
+		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
+				    __ATOMIC_RELAXED);
 
 	rte_smp_wmb();
 
@@ -705,7 +724,8 @@ rte_service_lcore_stop(uint32_t lcore)
 		int32_t enabled = service_mask & (UINT64_C(1) << i);
 		int32_t service_running = rte_service_runstate_get(i);
 		int32_t only_core = (1 ==
-			rte_atomic32_read(&rte_services[i].num_mapped_cores));
+			__atomic_load_n(&rte_services[i].num_mapped_cores,
+					__ATOMIC_RELAXED));
 
 		/* if the core is mapped, and the service is running, and this
 		 * is the only core that is mapped, the service would cease to
Signature · 1 line
2.7.4
author
#21 lore raw

PATCH v2 10/10 service: relax barriers with C11 atomic operations

In reply to Phil Yang

To guarantee the inter-threads visibility of the shareable domain, it uses a lot of rte_smp_r/wmb in the service library. This patch relaxed these barriers for service by using c11 atomic one-way barrier operations.

---
 lib/librte_eal/common/rte_service.c | 45 ++++++++++++++++++++-----------------
 1 file changed, 25 insertions(+), 20 deletions(-)
lib/librte_eal/common/rte_service.c +25 -20
index 96a59b6..9c02c24 100644
@@ -176,9 +176,11 @@ rte_service_set_stats_enable(uint32_t id, int32_t enabled)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, 0);
 
 	if (enabled)
-		s->internal_flags |= SERVICE_F_STATS_ENABLED;
+		__atomic_or_fetch(&s->internal_flags, SERVICE_F_STATS_ENABLED,
+			__ATOMIC_RELEASE);
 	else
-		s->internal_flags &= ~(SERVICE_F_STATS_ENABLED);
+		__atomic_and_fetch(&s->internal_flags,
+			~(SERVICE_F_STATS_ENABLED), __ATOMIC_RELEASE);
 
 	return 0;
 }
@@ -190,9 +192,11 @@ rte_service_set_runstate_mapped_check(uint32_t id, int32_t enabled)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, 0);
 
 	if (enabled)
-		s->internal_flags |= SERVICE_F_START_CHECK;
+		__atomic_or_fetch(&s->internal_flags, SERVICE_F_START_CHECK,
+			__ATOMIC_RELEASE);
 	else
-		s->internal_flags &= ~(SERVICE_F_START_CHECK);
+		__atomic_and_fetch(&s->internal_flags, ~(SERVICE_F_START_CHECK),
+			__ATOMIC_RELEASE);
 
 	return 0;
 }
@@ -261,8 +265,8 @@ rte_service_component_register(const struct rte_service_spec *spec,
 	s->spec = *spec;
 	s->internal_flags |= SERVICE_F_REGISTERED | SERVICE_F_START_CHECK;
 
-	rte_smp_wmb();
-	rte_service_count++;
+	/* make sure the counter update after the state change. */
+	__atomic_add_fetch(&rte_service_count, 1, __ATOMIC_RELEASE);
 
 	if (id_ptr)
 		*id_ptr = free_slot;
@@ -278,9 +282,10 @@ rte_service_component_unregister(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	rte_service_count--;
-	rte_smp_wmb();
 
-	s->internal_flags &= ~(SERVICE_F_REGISTERED);
+	/* make sure the counter update before the state change. */
+	__atomic_and_fetch(&s->internal_flags, ~(SERVICE_F_REGISTERED),
+			   __ATOMIC_RELEASE);
 
 	/* clear the run-bit in all cores */
 	for (i = 0; i < RTE_MAX_LCORE; i++)
@@ -298,11 +303,12 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	if (runstate)
-		s->comp_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->comp_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -313,11 +319,12 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	if (runstate)
-		s->app_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->app_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -439,7 +446,8 @@ service_runner_func(void *arg)
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (lcore_states[lcore].runstate == RUNSTATE_RUNNING) {
+	while (__atomic_load_n(&cs->runstate,
+		    __ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
@@ -450,8 +458,6 @@ service_runner_func(void *arg)
 		}
 
 		cs->loops++;
-
-		rte_smp_rmb();
 	}
 
 	lcore_config[lcore].state = WAIT;
@@ -660,9 +666,8 @@ rte_service_lcore_add(uint32_t lcore)
 
 	/* ensure that after adding a core the mask and state are defaults */
 	lcore_states[lcore].service_mask = 0;
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
-
-	rte_smp_wmb();
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+			__ATOMIC_RELEASE);
 
 	return rte_eal_wait_lcore(lcore);
 }
Signature · 1 line
2.7.4
author
#22 lore raw

v3 cover generic rte atomic APIs deprecate proposal

In reply to Phil Yang

DPDK provides generic rte_atomic APIs to do several atomic operations. These APIs are using the deprecated __sync built-ins and enforce full memory barriers on aarch64. However, full barriers are not necessary in many use cases. In order to address such use cases, C language offers C11 atomic APIs. The C11 atomic APIs provide finer memory barrier control by making use of the memory ordering parameter provided by the user. Various patches submitted in the past [2] and the patches in this series indicate significant performance gains on multiple aarch64 CPUs and no performance loss on x86.

But the existing rte_atomic API implementations cannot be changed as the APIs do not take the memory ordering parameter. The only choice available is replacing the usage of the rte_atomic APIs with C11 atomic APIs. In order to make this change, the following steps are proposed:

[1] deprecate rte_atomic APIs so that future patches do not use rte_atomic APIs (a script is added to flag the usages).
[2] refactor the code that uses rte_atomic APIs to use c11 atomic APIs.

This patchset contains:
1) the checkpatch script changes to flag rte_atomic API usage in patches.
2) changes to programmer guide describing writing efficient code for aarch64.
3) changes to various libraries to make use of c11 atomic APIs.

We are planning to replicate this idea across all the other libraries, drivers, examples, test applications. In the next phase, we will add changes to the mbuf, the EAL interrupts and the event timer adapter libraries.

v3:
add libatomic dependency for 32-bit clang

v2:
1. fix Clang '-Wincompatible-pointer-types' WARNING.
2. fix typos.

Honnappa Nagarahalli (2):
service: avoid race condition for MT unsafe service
service: identify service running on another core correctly

Phil Yang (10):
doc: add generic atomic deprecation section
devtools: prevent use of rte atomic APIs in future patches eal/build: add libatomic dependency for 32-bit clang
build: remove redundant code
vhost: optimize broadcast rarp sync with c11 atomic
ipsec: optimize with c11 atomic for sa outbound sqn update
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 one-way barrier
service: relax barriers with C11 atomic operations

 devtools/checkpatches.sh                         |   9 ++
 doc/guides/prog_guide/writing_efficient_code.rst |  60 +++++++-
 drivers/event/octeontx/meson.build               |   5 -
 drivers/event/octeontx2/meson.build              |   5 -
 drivers/event/opdl/meson.build                   |   5 -
 lib/librte_eal/common/rte_service.c              | 175 ++++++++++++-----------
 lib/librte_eal/meson.build                       |   6 +
 lib/librte_ipsec/ipsec_sqn.h                     |   3 +-
 lib/librte_ipsec/sa.h                            |   2 +-
 lib/librte_rcu/meson.build                       |   5 -
 lib/librte_vhost/vhost.h                         |   2 +-
 lib/librte_vhost/vhost_user.c                    |   7 +-
 lib/librte_vhost/virtio_net.c                    |  16 ++-
 13 files changed, 181 insertions(+), 119 deletions(-)
Signature · 1 line
2.7.4
author
#23 lore raw

PATCH v3 1/12 doc: add generic atomic deprecation section

Add deprecating the generic rte_atomic_xx APIs to c11 atomic built-ins guide and examples.

---
 doc/guides/prog_guide/writing_efficient_code.rst | 60 +++++++++++++++++++++++-
 1 file changed, 59 insertions(+), 1 deletion(-)
doc/guides/prog_guide/writing_efficient_code.rst +59 -1
index 849f63e..b278bc6 100644
@@ -167,7 +167,13 @@ but with the added cost of lower throughput.
 Locks and Atomic Operations
 ---------------------------
 
-Atomic operations imply a lock prefix before the instruction,
+This section describes some key considerations when using locks and atomic
+operations in the DPDK environment.
+
+Locks
+~~~~~
+
+On x86, atomic operations imply a lock prefix before the instruction,
 causing the processor's LOCK# signal to be asserted during execution of the following instruction.
 This has a big impact on performance in a multicore environment.
 
@@ -176,6 +182,58 @@ It can often be replaced by other solutions like per-lcore variables.
 Also, some locking techniques are more efficient than others.
 For instance, the Read-Copy-Update (RCU) algorithm can frequently replace simple rwlocks.
 
+Atomic Operations: Use C11 Atomic Built-ins
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DPDK `generic rte_atomic <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_atomic.h>`_ operations are
+implemented by `__sync built-ins <https://gcc.gnu.org/onlinedocs/gcc/_005f_005fsync-Builtins.html>`_.
+These __sync built-ins result in full barriers on aarch64, which are unnecessary
+in many use cases. They can be replaced by `__atomic built-ins <https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html>`_ that
+conform to the C11 memory model and provide finer memory order control.
+
+So replacing the rte_atomic operations with __atomic built-ins might improve
+performance for aarch64 machines. `More details <https://www.dpdk.org/wp-content/uploads/sites/35/2019/10/StateofC11Code.pdf>`_.
+
+Some typical optimization cases are listed below:
+
+Atomicity
+^^^^^^^^^
+
+Some use cases require atomicity alone, the ordering of the memory operations
+does not matter. For example the packets statistics in the `vhost <https://github.com/DPDK/dpdk/blob/v20.02/examples/vhost/main.c#L796>`_ example application.
+
+It just updates the number of transmitted packets, no subsequent logic depends
+on these counters. So the RELAXED memory ordering is sufficient:
+
+.. code-block:: c
+
+    static __rte_always_inline void
+    virtio_xmit(struct vhost_dev *dst_vdev, struct vhost_dev *src_vdev,
+            struct rte_mbuf *m)
+    {
+        ...
+        ...
+        if (enable_stats) {
+            __atomic_add_fetch(&dst_vdev->stats.rx_total_atomic, 1, __ATOMIC_RELAXED);
+            __atomic_add_fetch(&dst_vdev->stats.rx_atomic, ret, __ATOMIC_RELAXED);
+            ...
+        }
+    }
+
+One-way Barrier
+^^^^^^^^^^^^^^^
+
+Some use cases allow for memory reordering in one way while requiring memory
+ordering in the other direction.
+
+For example, the memory operations before the `lock <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_spinlock.h#L66>`_ can move to the
+critical section, but the memory operations in the critical section cannot move
+above the lock. In this case, the full memory barrier in the CAS operation can
+be replaced to ACQUIRE. On the other hand, the memory operations after the
+`unlock <https://github.com/DPDK/dpdk/blob/v20.02/lib/librte_eal/common/include/generic/rte_spinlock.h#L88>`_ can move to the critical section, but the memory operations in the
+critical section cannot move below the unlock. So the full barrier in the STORE
+operation can be replaced with RELEASE.
+
 Coding Considerations
 ---------------------
 
Signature · 1 line
2.7.4
author
#24 lore raw

PATCH v3 2/12 devtools: prevent use of rte atomic APIs in future patches

In reply to Phil Yang

In order to deprecate the rte_atomic APIs, prevent the patches from using rte_atomic APIs.

---
 devtools/checkpatches.sh | 9 +++++++++
 1 file changed, 9 insertions(+)
devtools/checkpatches.sh +9 -0
index 1794468..493f48e 100755
@@ -61,6 +61,15 @@ check_forbidden_additions() { # <patch>
 		-f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
 		"$1" || res=1
 
+	# refrain from new additions of 16/32/64 bits rte_atomic_xxx()
+	# multiple folders and expressions are separated by spaces
+	awk -v FOLDERS="lib drivers app examples" \
+		-v EXPRESSIONS="rte_atomic[0-9][0-9]_.*\\\(" \
+		-v RET_ON_FAIL=1 \
+		-v MESSAGE='Using c11 atomic built-ins instead of rte_atomic' \
+		-f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
+		"$1" || res=1
+
 	# svg figures must be included with wildcard extension
 	# because of png conversion for pdf docs
 	awk -v FOLDERS='doc' \
Signature · 1 line
2.7.4
author
#25 lore raw

PATCH v3 3/12 eal/build: add libatomic dependency for 32-bit clang

In reply to Phil Yang

When compiling with clang on 32-bit platforms, we are missing copies of 64-bit atomic functions. We can solve this by linking against libatomic for the drivers and libs which need those atomic ops.

---
 lib/librte_eal/meson.build | 6 ++++++
 1 file changed, 6 insertions(+)
lib/librte_eal/meson.build +6 -0
index 4be5118..3b10eae 100644
@@ -20,6 +20,12 @@ endif
 if cc.has_function('getentropy', prefix : '#include <unistd.h>')
 	cflags += '-DRTE_LIBEAL_USE_GETENTROPY'
 endif
+
+# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
+if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
+    ext_deps += cc.find_library('atomic')
+endif
+
 sources = common_sources + env_sources
 objs = common_objs + env_objs
 headers = common_headers + env_headers
Signature · 1 line
2.7.4
author
#26 lore raw

Re: [dpdk-dev] [PATCH v3 03/12] eal/build: add libatomic dependency for 32-bit clang

Quoted 30 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Phil Yang
Sent: Tuesday, March 17, 2020 9:18 AM
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>
Subject: [dpdk-dev] [PATCH v3 03/12] eal/build: add libatomic dependency for 32-bit clang

When compiling with clang on 32-bit platforms, we are missing copies of 64-bit atomic functions. We can solve this by linking against libatomic for the drivers and libs which need those atomic ops.

---
 lib/librte_eal/meson.build | 6 ++++++
 1 file changed, 6 insertions(+)
lib/librte_eal/meson.build +0 -0
index 4be5118..3b10eae 100644

lib/librte_eal/meson.build endif · line 28

Quoted 9 lines
lib/librte_eal/meson.build +6 -0
@@ -20,6 +20,12 @@ endif
 if cc.has_function('getentropy', prefix : '#include <unistd.h>')
 	cflags += '-DRTE_LIBEAL_USE_GETENTROPY'
 endif
+
+# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
+if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
+    ext_deps += cc.find_library('atomic')
+endif
+

This should be unneeded since:
https://git.dpdk.org/dpdk/commit/?id=da4eae278b56e698c64d0c39939a7a55c5b6abdd

Thanks,
Phil Yang

Quoted 5 lines
lib/librte_eal/meson.build +0 -1
 sources = common_sources + env_sources
 objs = common_objs + env_objs
 headers = common_headers + env_headers
--

2.7.4

author
#27 lore raw

PATCH v3 4/12 build: remove redundant code

In reply to Phil Yang

All these libs and drivers are built upon the eal lib. So when compiling with clang on 32-bit platforms linking against libatomic for the eal lib is sufficient. Remove the redundant code.

---
 drivers/event/octeontx/meson.build  | 5 -----
 drivers/event/octeontx2/meson.build | 5 -----
 drivers/event/opdl/meson.build      | 5 -----
 lib/librte_rcu/meson.build          | 5 -----
 4 files changed, 20 deletions(-)
drivers/event/octeontx/meson.build +0 -5
index 73118a4..2b74bb6 100644
@@ -11,8 +11,3 @@ sources = files('ssovf_worker.c',
 )
 
 deps += ['common_octeontx', 'mempool_octeontx', 'bus_vdev', 'pmd_octeontx']
-
-# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
-if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
-	ext_deps += cc.find_library('atomic')
-endif
drivers/event/octeontx2/meson.build +0 -5
index 56febb8..dfe8fc4 100644
@@ -20,11 +20,6 @@ if not dpdk_conf.get('RTE_ARCH_64')
 	extra_flags += ['-Wno-int-to-pointer-cast', '-Wno-pointer-to-int-cast']
 endif
 
-# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
-if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
-	ext_deps += cc.find_library('atomic')
-endif
-
 foreach flag: extra_flags
 	if cc.has_argument(flag)
 		cflags += flag
drivers/event/opdl/meson.build +0 -5
index e67b164..566462f 100644
@@ -10,8 +10,3 @@ sources = files(
 	'opdl_test.c',
 )
 deps += ['bus_vdev']
-
-# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
-if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
-	ext_deps += cc.find_library('atomic')
-endif
lib/librte_rcu/meson.build +0 -5
index 62920ba..0c2d5a2 100644
@@ -5,8 +5,3 @@ allow_experimental_apis = true
 
 sources = files('rte_rcu_qsbr.c')
 headers = files('rte_rcu_qsbr.h')
-
-# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
-if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
-	ext_deps += cc.find_library('atomic')
-endif
Signature · 1 line
2.7.4
author
#28 lore raw

Re: [dpdk-dev] [PATCH v3 04/12] build: remove redundant code

Quoted 20 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Phil Yang
Sent: Tuesday, March 17, 2020 9:18 AM
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>
Subject: [dpdk-dev] [PATCH v3 04/12] build: remove redundant code

All these libs and drivers are built upon the eal lib. So when compiling with clang on 32-bit platforms linking against libatomic for the eal lib is sufficient. Remove the redundant code.

Quoted 6 lines
---
 drivers/event/octeontx/meson.build  | 5 -----
 drivers/event/octeontx2/meson.build | 5 -----
 drivers/event/opdl/meson.build      | 5 -----
 lib/librte_rcu/meson.build          | 5 -----
 4 files changed, 20 deletions(-)

This should be unneeded since:
https://git.dpdk.org/dpdk/commit/?id=da4eae278b56e698c64d0c39939a7a55c5b6abdd

Abandon this patch.

Thanks,
Phil Yang

<snip>

author
#29 lore raw

PATCH v3 5/12 vhost: optimize broadcast rarp sync with c11 atomic

In reply to Phil Yang

The rarp packet broadcast flag is synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.

---
 lib/librte_vhost/vhost.h      |  2 +-
 lib/librte_vhost/vhost_user.c |  7 +++----
 lib/librte_vhost/virtio_net.c | 16 +++++++++-------
 3 files changed, 13 insertions(+), 12 deletions(-)
lib/librte_vhost/vhost.h +1 -1
index 2087d14..0e22125 100644
@@ -350,7 +350,7 @@ struct virtio_net {
 	uint32_t		flags;
 	uint16_t		vhost_hlen;
 	/* to tell if we need broadcast rarp packet */
-	rte_atomic16_t		broadcast_rarp;
+	int16_t			broadcast_rarp;
 	uint32_t		nr_vring;
 	int			dequeue_zero_copy;
 	int			extbuf;
lib/librte_vhost/vhost_user.c +3 -4
index bd1be01..857187d 100644
@@ -2145,11 +2145,10 @@ vhost_user_send_rarp(struct virtio_net **pdev, struct VhostUserMsg *msg,
 	 * Set the flag to inject a RARP broadcast packet at
 	 * rte_vhost_dequeue_burst().
 	 *
-	 * rte_smp_wmb() is for making sure the mac is copied
-	 * before the flag is set.
+	 * __ATOMIC_RELEASE ordering is for making sure the mac is
+	 * copied before the flag is set.
 	 */
-	rte_smp_wmb();
-	rte_atomic16_set(&dev->broadcast_rarp, 1);
+	__atomic_store_n(&dev->broadcast_rarp, 1, __ATOMIC_RELEASE);
 	did = dev->vdpa_dev_id;
 	vdpa_dev = rte_vdpa_get_device(did);
 	if (vdpa_dev && vdpa_dev->ops->migration_done)
lib/librte_vhost/virtio_net.c +9 -7
index 37c47c7..fa10deb 100644
@@ -2203,6 +2203,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
 	struct virtio_net *dev;
 	struct rte_mbuf *rarp_mbuf = NULL;
 	struct vhost_virtqueue *vq;
+	int16_t success = 1;
 
 	dev = get_device(vid);
 	if (!dev)
@@ -2249,16 +2250,17 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
 	 *
 	 * broadcast_rarp shares a cacheline in the virtio_net structure
 	 * with some fields that are accessed during enqueue and
-	 * rte_atomic16_cmpset() causes a write if using cmpxchg. This could
-	 * result in false sharing between enqueue and dequeue.
+	 * __atomic_compare_exchange_n causes a write if performed compare
+	 * and exchange. This could result in false sharing between enqueue
+	 * and dequeue.
 	 *
 	 * Prevent unnecessary false sharing by reading broadcast_rarp first
-	 * and only performing cmpset if the read indicates it is likely to
-	 * be set.
+	 * and only performing compare and exchange if the read indicates it
+	 * is likely to be set.
 	 */
-	if (unlikely(rte_atomic16_read(&dev->broadcast_rarp) &&
-			rte_atomic16_cmpset((volatile uint16_t *)
-				&dev->broadcast_rarp.cnt, 1, 0))) {
+	if (unlikely(__atomic_load_n(&dev->broadcast_rarp, __ATOMIC_ACQUIRE) &&
+			__atomic_compare_exchange_n(&dev->broadcast_rarp,
+			&success, 0, 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED))) {
 
 		rarp_mbuf = rte_net_make_rarp_packet(mbuf_pool, &dev->mac);
 		if (rarp_mbuf == NULL) {
Signature · 1 line
2.7.4
author
#30 lore raw

PATCH v2 1/1 vhost: optimize broadcast rarp sync with c11 atomic

The rarp packet broadcast flag is synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.

---
v2:
split from the 'generic rte atomic APIs deprecate proposal' patchset.

 lib/librte_vhost/vhost.h      |  2 +-
 lib/librte_vhost/vhost_user.c |  7 +++----
 lib/librte_vhost/virtio_net.c | 16 +++++++++-------
 3 files changed, 13 insertions(+), 12 deletions(-)
lib/librte_vhost/vhost.h +1 -1
index 2087d14..0e22125 100644
@@ -350,7 +350,7 @@ struct virtio_net {
 	uint32_t		flags;
 	uint16_t		vhost_hlen;
 	/* to tell if we need broadcast rarp packet */
-	rte_atomic16_t		broadcast_rarp;
+	int16_t			broadcast_rarp;
 	uint32_t		nr_vring;
 	int			dequeue_zero_copy;
 	int			extbuf;
lib/librte_vhost/vhost_user.c +3 -4
index bd1be01..857187d 100644
@@ -2145,11 +2145,10 @@ vhost_user_send_rarp(struct virtio_net **pdev, struct VhostUserMsg *msg,
 	 * Set the flag to inject a RARP broadcast packet at
 	 * rte_vhost_dequeue_burst().
 	 *
-	 * rte_smp_wmb() is for making sure the mac is copied
-	 * before the flag is set.
+	 * __ATOMIC_RELEASE ordering is for making sure the mac is
+	 * copied before the flag is set.
 	 */
-	rte_smp_wmb();
-	rte_atomic16_set(&dev->broadcast_rarp, 1);
+	__atomic_store_n(&dev->broadcast_rarp, 1, __ATOMIC_RELEASE);
 	did = dev->vdpa_dev_id;
 	vdpa_dev = rte_vdpa_get_device(did);
 	if (vdpa_dev && vdpa_dev->ops->migration_done)
lib/librte_vhost/virtio_net.c +9 -7
index 37c47c7..fa10deb 100644
@@ -2203,6 +2203,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
 	struct virtio_net *dev;
 	struct rte_mbuf *rarp_mbuf = NULL;
 	struct vhost_virtqueue *vq;
+	int16_t success = 1;
 
 	dev = get_device(vid);
 	if (!dev)
@@ -2249,16 +2250,17 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
 	 *
 	 * broadcast_rarp shares a cacheline in the virtio_net structure
 	 * with some fields that are accessed during enqueue and
-	 * rte_atomic16_cmpset() causes a write if using cmpxchg. This could
-	 * result in false sharing between enqueue and dequeue.
+	 * __atomic_compare_exchange_n causes a write if performed compare
+	 * and exchange. This could result in false sharing between enqueue
+	 * and dequeue.
 	 *
 	 * Prevent unnecessary false sharing by reading broadcast_rarp first
-	 * and only performing cmpset if the read indicates it is likely to
-	 * be set.
+	 * and only performing compare and exchange if the read indicates it
+	 * is likely to be set.
 	 */
-	if (unlikely(rte_atomic16_read(&dev->broadcast_rarp) &&
-			rte_atomic16_cmpset((volatile uint16_t *)
-				&dev->broadcast_rarp.cnt, 1, 0))) {
+	if (unlikely(__atomic_load_n(&dev->broadcast_rarp, __ATOMIC_ACQUIRE) &&
+			__atomic_compare_exchange_n(&dev->broadcast_rarp,
+			&success, 0, 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED))) {
 
 		rarp_mbuf = rte_net_make_rarp_packet(mbuf_pool, &dev->mac);
 		if (rarp_mbuf == NULL) {
Signature · 1 line
2.7.4
Reviewed-by on v2 1/1
#31 lore raw

Re: [dpdk-dev] [PATCH v2] vhost: optimize broadcast rarp sync with c11 atomic

Quoted 8 lines · Phil Yang

The rarp packet broadcast flag is synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.

Quoted 8 lines

---
v2:
split from the 'generic rte atomic APIs deprecate proposal' patchset.

 lib/librte_vhost/vhost.h      |  2 +-
 lib/librte_vhost/vhost_user.c |  7 +++----
 lib/librte_vhost/virtio_net.c | 16 +++++++++-------
 3 files changed, 13 insertions(+), 12 deletions(-)

Thanks,
Maxime

#32 lore raw

Re: [dpdk-dev] [PATCH v2] vhost: optimize broadcast rarp sync with c11 atomic

In reply to Phil Yang
Quoted 8 lines · Phil Yang

The rarp packet broadcast flag is synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.

Quoted 8 lines

---
v2:
split from the 'generic rte atomic APIs deprecate proposal' patchset.

 lib/librte_vhost/vhost.h      |  2 +-
 lib/librte_vhost/vhost_user.c |  7 +++----
 lib/librte_vhost/virtio_net.c | 16 +++++++++-------
 3 files changed, 13 insertions(+), 12 deletions(-)

Applied to dpdk-next-virtio/master

Thanks,
Maxime

author
#33 lore raw

PATCH v3 6/12 ipsec: optimize with c11 atomic for sa outbound sqn update

In reply to Phil Yang

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/sa.h        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
lib/librte_ipsec/ipsec_sqn.h +2 -1
index 0c2f76a..e884af7 100644
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)
 
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa->sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);
 	else {
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h +1 -1
index d22451b..cab9a2e 100644
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;
 		struct {
Signature · 1 line
2.7.4
#34 lore raw

Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

Hi Phil,

Quoted 20 lines

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/sa.h        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
lib/librte_ipsec/ipsec_sqn.h +0 -0
index 0c2f76a..e884af7 100644

lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132

Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2 -1
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)
 
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa->sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);

One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit __atomic builtins (gcc seems to always generate cmpxchng8b for such cases). Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.

Quoted 8 lines
lib/librte_ipsec/ipsec_sqn.h +0 -0
 	else {
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h +0 -0
index d22451b..cab9a2e 100644

lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125

Quoted 7 lines
lib/librte_ipsec/sa.h +1 -1
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;

If we don't need rte_atomic64 here anymore,
then I think we can collapse the union to just:
uint64_t outb;

Quoted 3 lines
lib/librte_ipsec/sa.h +0 -1
 		struct {
--

2.7.4

#35 lore raw

Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

<snip>

Quoted 27 lines

Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

Hi Phil,

Quoted 22 lines

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/sa.h        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/lib/librte_ipsec/ipsec_sqn.h index 0c2f76a..e884af7 100644

lib/librte_ipsec/ipsec_sqn.h +0 -0
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa,
 uint32_t *num)
 
Quoted 12 lines

lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132

Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2 -1
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa-
 sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);

One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit __atomic builtins (gcc seems to always generate cmpxchng8b for such cases). Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.

I had looked at this some time back for GCC. The function call is generated only if the underlying platform does not support the atomic instructions for the operand size. Otherwise, gcc generates the instructions directly. I would think the behavior would be the same for clang.

Quoted 24 lines
Quoted 9 lines
lib/librte_ipsec/ipsec_sqn.h +0 -0
 	else {
lib/librte_ipsec/ipsec_sqn.h +0 -0
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0 -0

d22451b..cab9a2e 100644

lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125

Quoted 7 lines
lib/librte_ipsec/sa.h +1 -1
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;

If we don't need rte_atomic64 here anymore, then I think we can collapse the union to just:
uint64_t outb;

Quoted 3 lines
lib/librte_ipsec/sa.h +0 -1
 		struct {
--

2.7.4

#36 lore raw

Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

Quoted 54 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Monday, March 23, 2020 7:08 PM
To: Ananyev, Konstantin <[email protected]>; Phil Yang <[email protected]>; [email protected]; Van Haaren, Harry <[email protected]>; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

<snip>

Quoted 27 lines

Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

Hi Phil,

Quoted 22 lines

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/sa.h        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/lib/librte_ipsec/ipsec_sqn.h index 0c2f76a..e884af7 100644

lib/librte_ipsec/ipsec_sqn.h +0 -0
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa,
 uint32_t *num)
 
Quoted 12 lines

lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132

Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2 -1
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa-
 sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);

One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit __atomic builtins (gcc seems to always generate cmpxchng8b for such cases). Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.

Quoted 3 lines

I had looked at this some time back for GCC. The function call is generated only if the underlying platform does not support the atomic instructions for the operand size. Otherwise, gcc generates the instructions directly.
I would think the behavior would be the same for clang.

From what I see not really.
As an example:

$ cat tatm11.c
#include <stdint.h>
struct x {
        uint64_t v __attribute__((aligned(8)));
};

uint64_t
ffxadd1(struct x *x, uint32_t n, uint32_t m)

{
        return __atomic_add_fetch(&x->v, n, __ATOMIC_RELAXED);
}

uint64_t
ffxadd11(uint64_t *v, uint32_t n, uint32_t m)

{
        return __atomic_add_fetch(v, n, __ATOMIC_RELAXED);
}

gcc for i686 will generate code with cmpxchng8b for both cases.
clang will generate cmpxchng8b for ffxadd1() - when data is explicitly 8B aligned, but will emit a function call for ffxadd11().

Quoted 25 lines
Quoted 25 lines
Quoted 10 lines
lib/librte_ipsec/ipsec_sqn.h +0 -0
 	else {
lib/librte_ipsec/ipsec_sqn.h +0 -0
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0 -0

d22451b..cab9a2e 100644

lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125

Quoted 7 lines
lib/librte_ipsec/sa.h +1 -1
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;

If we don't need rte_atomic64 here anymore, then I think we can collapse the union to just:
uint64_t outb;

Quoted 3 lines
lib/librte_ipsec/sa.h +0 -1
 		struct {
--

2.7.4

#37 lore raw

Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

<snip>

Quoted 59 lines
Quoted 27 lines
Quoted 27 lines

Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

Hi Phil,

Quoted 22 lines

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/sa.h        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/lib/librte_ipsec/ipsec_sqn.h index 0c2f76a..e884af7 100644

lib/librte_ipsec/ipsec_sqn.h +0 -0
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa,
 uint32_t *num)
 
Quoted 10 lines
Quoted 10 lines

lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132

Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2 -1
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa-
 sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);

One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit __atomic builtins (gcc seems to always generate cmpxchng8b for

such cases).

Quoted 4 lines
Quoted 2 lines

Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.

I had looked at this some time back for GCC. The function call is generated only if the underlying platform does not support the atomic

instructions for the operand size. Otherwise, gcc generates the instructions directly.

Quoted 1 line

I would think the behavior would be the same for clang.

From what I see not really.
As an example:

$ cat tatm11.c
#include <stdint.h>
struct x {
        uint64_t v __attribute__((aligned(8))); };

uint64_t

ffxadd1(struct x *x, uint32_t n, uint32_t m) {
        return __atomic_add_fetch(&x->v, n, __ATOMIC_RELAXED); }
Quoted 7 lines

uint64_t

ffxadd11(uint64_t *v, uint32_t n, uint32_t m) {
        return __atomic_add_fetch(v, n, __ATOMIC_RELAXED); }

gcc for i686 will generate code with cmpxchng8b for both cases. clang will generate cmpxchng8b for ffxadd1() - when data is explicitly 8B aligned, but will emit a function call for ffxadd11().

Does it require libatomic to be linked in this case? Clang documentation calls out unaligned case where it would generate the function call [1]. On aarch64, the atomic instructions need the address to be aligned.

[1] https://clang.llvm.org/docs/Toolchain.html#atomics-library

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 11 lines
lib/librte_ipsec/ipsec_sqn.h +0 -0
 	else {
lib/librte_ipsec/ipsec_sqn.h +0 -0
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0 -0

d22451b..cab9a2e 100644

lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125

Quoted 7 lines
lib/librte_ipsec/sa.h +1 -1
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;

If we don't need rte_atomic64 here anymore, then I think we can collapse the union to just:
uint64_t outb;

Quoted 3 lines
lib/librte_ipsec/sa.h +0 -1
 		struct {
--

2.7.4

#38 lore raw

Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

Quoted 54 lines
Quoted 54 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/sa.h        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/lib/librte_ipsec/ipsec_sqn.h index 0c2f76a..e884af7 100644

lib/librte_ipsec/ipsec_sqn.h +0 -0
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa,
 uint32_t *num)
 
Quoted 10 lines
Quoted 10 lines

lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132

Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2 -1
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa-
 sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);

One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit __atomic builtins (gcc seems to always generate cmpxchng8b for

such cases).

Quoted 4 lines
Quoted 2 lines

Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.

I had looked at this some time back for GCC. The function call is generated only if the underlying platform does not support the atomic

instructions for the operand size. Otherwise, gcc generates the instructions directly.

Quoted 1 line

I would think the behavior would be the same for clang.

From what I see not really.
As an example:

$ cat tatm11.c
#include <stdint.h>
struct x {
        uint64_t v __attribute__((aligned(8))); };

uint64_t

ffxadd1(struct x *x, uint32_t n, uint32_t m) {
        return __atomic_add_fetch(&x->v, n, __ATOMIC_RELAXED); }
Quoted 8 lines
Quoted 7 lines

uint64_t

ffxadd11(uint64_t *v, uint32_t n, uint32_t m) {
        return __atomic_add_fetch(v, n, __ATOMIC_RELAXED); }

gcc for i686 will generate code with cmpxchng8b for both cases. clang will generate cmpxchng8b for ffxadd1() - when data is explicitly 8B aligned, but will emit a function call for ffxadd11().

Does it require libatomic to be linked in this case?

Yes, it does.
In fact same story even with current dpdk.org master. To make i686-native-linuxapp-clang successfully, I have to explicitly add EXTRA_LDFLAGS="-latomic".
To be more specific:

$ for i in i686-native-linuxapp-clang/lib/*.a; do x=`nm $i | grep __atomic_`; if [[ -n "${x}" ]]; then echo $i; echo $x; fi; done
i686-native-linuxapp-clang/lib/librte_distributor.a
U __atomic_load_8 U __atomic_store_8
i686-native-linuxapp-clang/lib/librte_pmd_opdl_event.a
U __atomic_load_8 U __atomic_store_8
i686-native-linuxapp-clang/lib/librte_rcu.a
U __atomic_compare_exchange_8 U __atomic_load_8

As there were no complains so far, it makes me think that probably no-one using clang for IA-32 builds.

Quoted 2 lines

Clang documentation calls out unaligned case where it would generate the function call [1].

Seems so, and it treats uin64_t as 4B aligned for IA.

Quoted 1 line

On aarch64, the atomic instructions need the address to be aligned.

For that particular case (cmpxchng8b) there is no such restrictions for IA-32. Again, as I said before, gcc manages to emit code without function calls for exactly the same source.

Quoted 29 lines

[1] https://clang.llvm.org/docs/Toolchain.html#atomics-library

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 11 lines
lib/librte_ipsec/ipsec_sqn.h +0 -0
 	else {
lib/librte_ipsec/ipsec_sqn.h +0 -0
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0 -0

d22451b..cab9a2e 100644

lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125

Quoted 7 lines
lib/librte_ipsec/sa.h +1 -1
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;

If we don't need rte_atomic64 here anymore, then I think we can collapse the union to just:
uint64_t outb;

Quoted 3 lines
lib/librte_ipsec/sa.h +0 -1
 		struct {
--

2.7.4

#39 lore raw

Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

Quoted 88 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Ananyev, Konstantin
Sent: Tuesday, March 24, 2020 1:10 PM
To: Honnappa Nagarahalli <[email protected]>; Phil Yang <[email protected]>; [email protected]; Van Haaren, Harry <[email protected]>; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; nd <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

Quoted 53 lines
Quoted 53 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/sa.h        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/lib/librte_ipsec/ipsec_sqn.h index 0c2f76a..e884af7 100644

lib/librte_ipsec/ipsec_sqn.h +0 -0
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa,
 uint32_t *num)
 
Quoted 10 lines
Quoted 10 lines

lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132

Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2 -1
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa-
 sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);

One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit __atomic builtins (gcc seems to always generate cmpxchng8b for

such cases).

Quoted 4 lines
Quoted 2 lines

Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.

I had looked at this some time back for GCC. The function call is generated only if the underlying platform does not support the atomic

instructions for the operand size. Otherwise, gcc generates the instructions directly.

Quoted 1 line

I would think the behavior would be the same for clang.

From what I see not really.
As an example:

$ cat tatm11.c
#include <stdint.h>
struct x {
        uint64_t v __attribute__((aligned(8))); };

uint64_t

ffxadd1(struct x *x, uint32_t n, uint32_t m) {
        return __atomic_add_fetch(&x->v, n, __ATOMIC_RELAXED); }
Quoted 8 lines
Quoted 7 lines

uint64_t

ffxadd11(uint64_t *v, uint32_t n, uint32_t m) {
        return __atomic_add_fetch(v, n, __ATOMIC_RELAXED); }

gcc for i686 will generate code with cmpxchng8b for both cases. clang will generate cmpxchng8b for ffxadd1() - when data is explicitly 8B aligned, but will emit a function call for ffxadd11().

Does it require libatomic to be linked in this case?

Yes, it does.
In fact same story even with current dpdk.org master. To make i686-native-linuxapp-clang successfully, I have to explicitly add EXTRA_LDFLAGS="-latomic".
To be more specific:

$ for i in i686-native-linuxapp-clang/lib/*.a; do x=`nm $i | grep __atomic_`; if [[ -n "${x}" ]]; then echo $i; echo $x; fi; done
i686-native-linuxapp-clang/lib/librte_distributor.a
U __atomic_load_8 U __atomic_store_8
i686-native-linuxapp-clang/lib/librte_pmd_opdl_event.a
U __atomic_load_8 U __atomic_store_8
i686-native-linuxapp-clang/lib/librte_rcu.a
U __atomic_compare_exchange_8 U __atomic_load_8
Quoted 7 lines

As there were no complains so far, it makes me think that probably no-one using clang for IA-32 builds.

Quoted 2 lines

Clang documentation calls out unaligned case where it would generate the function call [1].

Seems so, and it treats uin64_t as 4B aligned for IA.

correction: for IA-32

Quoted 36 lines
Quoted 2 lines

On aarch64, the atomic instructions need the address to be aligned.

For that particular case (cmpxchng8b) there is no such restrictions for IA-32. Again, as I said before, gcc manages to emit code without function calls for exactly the same source.

Quoted 29 lines

[1] https://clang.llvm.org/docs/Toolchain.html#atomics-library

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 11 lines
lib/librte_ipsec/ipsec_sqn.h +0 -0
 	else {
lib/librte_ipsec/ipsec_sqn.h +0 -0
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0 -0

d22451b..cab9a2e 100644

lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125

Quoted 7 lines
lib/librte_ipsec/sa.h +1 -1
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;

If we don't need rte_atomic64 here anymore, then I think we can collapse the union to just:
uint64_t outb;

Quoted 3 lines
lib/librte_ipsec/sa.h +0 -1
 		struct {
--

2.7.4

author
#40 lore raw

Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

In reply to Ananyev, Konstantin

Hi Konstantin,

<snip>

Quoted 67 lines
Quoted 2 lines
Quoted 2 lines

Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa

outbound

Quoted 6 lines
Quoted 6 lines

sqn update

Hi Phil,

Quoted 2 lines

For SA outbound packets, rte_atomic64_add_return is used to

generate

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/sa.h        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/lib/librte_ipsec/ipsec_sqn.h index 0c2f76a..e884af7 100644

lib/librte_ipsec/ipsec_sqn.h +0 -0
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa,
 uint32_t *num)
 
Quoted 10 lines
Quoted 10 lines

lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132

Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2 -1
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa-
 sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);

One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit __atomic builtins (gcc seems to always generate cmpxchng8b for such

cases).

Quoted 3 lines
Quoted 2 lines

Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.

I had looked at this some time back for GCC. The function call is generated

only if the underlying platform does not support the atomic

Quoted 1 line

instructions for the operand size. Otherwise, gcc generates the instructions

directly.

Quoted 1 line

I would think the behavior would be the same for clang.

From what I see not really.
As an example:

$ cat tatm11.c
#include <stdint.h>
struct x {
        uint64_t v __attribute__((aligned(8)));
};

uint64_t
ffxadd1(struct x *x, uint32_t n, uint32_t m)

{
        return __atomic_add_fetch(&x->v, n, __ATOMIC_RELAXED);
}

uint64_t
ffxadd11(uint64_t *v, uint32_t n, uint32_t m)

Quoted 8 lines
{
        return __atomic_add_fetch(v, n, __ATOMIC_RELAXED);
}

gcc for i686 will generate code with cmpxchng8b for both cases. clang will generate cmpxchng8b for ffxadd1() - when data is explicitly 8B aligned,
but will emit a function call for ffxadd11().

I guess your testbed is an i386 platform. However, what I see here is different.

Testbed i686: Ubuntu 18.04.4 LTS/GCC 8.3/ Clang 9.0.0-2 Both Clang and GCC for i686 generate code with xadd for these two cases.

Testbed i386: Ubuntu 16.04 LTS (Installed libatomic)/GCC 5.4.0/ Clang 4.0.0 GCC will generate code with cmpxchng8b for both cases.
Clang generated code emits a function call for both cases.

Thanks,
Phil

Quoted 27 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
lib/librte_ipsec/ipsec_sqn.h +0 -0
 	else {
lib/librte_ipsec/ipsec_sqn.h +0 -0
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0 -0

d22451b..cab9a2e 100644

lib/librte_ipsec/sa.h +0 -0
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
Quoted 8 lines
Quoted 8 lines

lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125

Quoted 6 lines
lib/librte_ipsec/sa.h +1 -1
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;

If we don't need rte_atomic64 here anymore, then I think we can

collapse the

Quoted 6 lines
Quoted 6 lines

union to just:
uint64_t outb;

Quoted 3 lines
lib/librte_ipsec/sa.h +0 -1
 		struct {
--

2.7.4

#41 lore raw

Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

Hi Phil,

Quoted 77 lines

<snip>

Quoted 67 lines
Quoted 2 lines
Quoted 2 lines

Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa

outbound

Quoted 6 lines
Quoted 6 lines

sqn update

Hi Phil,

Quoted 2 lines

For SA outbound packets, rte_atomic64_add_return is used to

generate

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/sa.h        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/lib/librte_ipsec/ipsec_sqn.h index 0c2f76a..e884af7 100644

lib/librte_ipsec/ipsec_sqn.h +0 -0
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa,
 uint32_t *num)
 
Quoted 10 lines
Quoted 10 lines

lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132

Quoted 6 lines
lib/librte_ipsec/ipsec_sqn.h +2 -1
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa-
 sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);

One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-bit __atomic builtins (gcc seems to always generate cmpxchng8b for such

cases).

Quoted 3 lines
Quoted 2 lines

Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.

I had looked at this some time back for GCC. The function call is generated

only if the underlying platform does not support the atomic

Quoted 1 line

instructions for the operand size. Otherwise, gcc generates the instructions

directly.

Quoted 1 line

I would think the behavior would be the same for clang.

From what I see not really.
As an example:

$ cat tatm11.c
#include <stdint.h>
struct x {
        uint64_t v __attribute__((aligned(8)));
};

uint64_t
ffxadd1(struct x *x, uint32_t n, uint32_t m)

{
        return __atomic_add_fetch(&x->v, n, __ATOMIC_RELAXED);
}

uint64_t
ffxadd11(uint64_t *v, uint32_t n, uint32_t m)

Quoted 8 lines
{
        return __atomic_add_fetch(v, n, __ATOMIC_RELAXED);
}

gcc for i686 will generate code with cmpxchng8b for both cases. clang will generate cmpxchng8b for ffxadd1() - when data is explicitly 8B aligned,
but will emit a function call for ffxadd11().

Quoted 4 lines

I guess your testbed is an i386 platform. However, what I see here is different.

Testbed i686: Ubuntu 18.04.4 LTS/GCC 8.3/ Clang 9.0.0-2 Both Clang and GCC for i686 generate code with xadd for these two cases.

I suppose you meant x86_64 here (-m64), right?

Quoted 4 lines

Testbed i386: Ubuntu 16.04 LTS (Installed libatomic)/GCC 5.4.0/ Clang 4.0.0 GCC will generate code with cmpxchng8b for both cases.
Clang generated code emits a function call for both cases.

That's exactly what I am talking about above.
X86_64 (64 bit binary) - no function calls for both gcc and clang i686 (32 bit binary) - no function calls with gcc, functions calls with clang when explicit alignment is not specified.

As I said in my initial email, that's probably not a big deal - from what I was told so far we don't officially support clang for IA-32 and I don't know does anyone uses it at all right now. Though if someone thinks it is a potential problem here -
it is better to flag it at early stage.
So once again my questions to the community:
1/ Does anyone builds/uses DPDK with i686-clang?
2/ If there are anyone, can these persons try to evaluate how big perf drop it would cause for them?
3/ Is there an option to switch to i686-gcc (supported one)? Konstantin

Quoted 27 lines
Quoted 27 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
lib/librte_ipsec/ipsec_sqn.h +0 -0
 	else {
lib/librte_ipsec/ipsec_sqn.h +0 -0
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0 -0

d22451b..cab9a2e 100644

lib/librte_ipsec/sa.h +0 -0
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
Quoted 8 lines
Quoted 8 lines

lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125

Quoted 6 lines
lib/librte_ipsec/sa.h +1 -1
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;

If we don't need rte_atomic64 here anymore, then I think we can

collapse the

Quoted 6 lines
Quoted 6 lines

union to just:
uint64_t outb;

Quoted 3 lines
lib/librte_ipsec/sa.h +0 -1
 		struct {
--

2.7.4

author
#42 lore raw

Re: [dpdk-dev] [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

Quoted 100 lines

-----Original Message-----
From: Ananyev, Konstantin <[email protected]>
Sent: Tuesday, March 24, 2020 7:04 PM
To: Phil Yang <[email protected]>; Honnappa Nagarahalli <[email protected]>; [email protected]; Van Haaren, Harry <[email protected]>; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa outbound sqn update

Hi Phil,

Quoted 6 lines

<snip>

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

Subject: RE: [PATCH v3 06/12] ipsec: optimize with c11 atomic for sa

outbound

Quoted 2 lines
Quoted 2 lines

sqn update

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

Hi Phil,

Quoted 2 lines

For SA outbound packets, rte_atomic64_add_return is used to

generate

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64.

This

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/sa.h        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa

Quoted 5 lines
(quoted diff) +0 -0

b/lib/librte_ipsec/ipsec_sqn.h index 0c2f76a..e884af7 100644

lib/librte_ipsec/ipsec_sqn.h +0 -0
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa

*sa,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) · line 132

Quoted 8 lines
lib/librte_ipsec/ipsec_sqn.h +2 -1
 uint32_t *num)
 
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa-
 sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);

One generic thing to note:
clang for i686 in some cases will generate a proper function call for 64-

bit

Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line

__atomic builtins (gcc seems to always generate cmpxchng8b for such

cases).

Quoted 3 lines
Quoted 2 lines

Does anyone consider it as a potential problem?
It probably not a big deal, but would like to know broader opinion.

I had looked at this some time back for GCC. The function call is

generated

Quoted 2 lines
Quoted 2 lines

only if the underlying platform does not support the atomic

Quoted 1 line

instructions for the operand size. Otherwise, gcc generates the

instructions

Quoted 30 lines
Quoted 29 lines

directly.

Quoted 1 line

I would think the behavior would be the same for clang.

From what I see not really.
As an example:

$ cat tatm11.c
#include <stdint.h>
struct x {
        uint64_t v __attribute__((aligned(8)));
};

uint64_t
ffxadd1(struct x *x, uint32_t n, uint32_t m)

{
        return __atomic_add_fetch(&x->v, n, __ATOMIC_RELAXED);
}

uint64_t
ffxadd11(uint64_t *v, uint32_t n, uint32_t m)

{
        return __atomic_add_fetch(v, n, __ATOMIC_RELAXED);
}

gcc for i686 will generate code with cmpxchng8b for both cases. clang will generate cmpxchng8b for ffxadd1() - when data is explicitly 8B aligned,
but will emit a function call for ffxadd11().

Quoted 7 lines
Quoted 1 line

I guess your testbed is an i386 platform. However, what I see here is

different.

Quoted 3 lines

Testbed i686: Ubuntu 18.04.4 LTS/GCC 8.3/ Clang 9.0.0-2 Both Clang and GCC for i686 generate code with xadd for these two cases.

I suppose you meant x86_64 here (-m64), right?

Yes. It is x86_64 here.

Quoted 51 lines
Quoted 6 lines

Testbed i386: Ubuntu 16.04 LTS (Installed libatomic)/GCC 5.4.0/ Clang 4.0.0 GCC will generate code with cmpxchng8b for both cases.
Clang generated code emits a function call for both cases.

That's exactly what I am talking about above.
X86_64 (64 bit binary) - no function calls for both gcc and clang i686 (32 bit binary) - no function calls with gcc, functions calls with clang when explicit alignment is not specified.

As I said in my initial email, that's probably not a big deal - from what I was told so far we don't officially support clang for IA-32 and I don't know does anyone uses it at all right now. Though if someone thinks it is a potential problem here -
it is better to flag it at early stage.
So once again my questions to the community:
1/ Does anyone builds/uses DPDK with i686-clang?
2/ If there are anyone, can these persons try to evaluate how big perf drop it would cause for them?
3/ Is there an option to switch to i686-gcc (supported one)? Konstantin

Quoted 27 lines
Quoted 27 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
lib/librte_ipsec/ipsec_sqn.h +0 -0
 	else {
lib/librte_ipsec/ipsec_sqn.h +0 -0
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/sa.h index renamed from lib/librte_ipsec/sa.h +0 -0

d22451b..cab9a2e 100644

lib/librte_ipsec/sa.h +0 -0
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
Quoted 8 lines
Quoted 8 lines

lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 125

Quoted 6 lines
lib/librte_ipsec/sa.h +1 -1
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;

If we don't need rte_atomic64 here anymore, then I think we can

collapse the

Quoted 6 lines
Quoted 6 lines

union to just:
uint64_t outb;

Quoted 3 lines
lib/librte_ipsec/sa.h +0 -1
 		struct {
--

2.7.4

author
#43 lore raw

PATCH v2 1/1 ipsec: optimize with c11 atomic for sa outbound sqn update

In reply to Phil Yang

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
v2:
split from the "generic rte atomic APIs deprecate proposal" patchset.

 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/meson.build | 5 +++++
 lib/librte_ipsec/sa.h        | 2 +-
 3 files changed, 8 insertions(+), 2 deletions(-)
lib/librte_ipsec/ipsec_sqn.h +2 -1
index 0c2f76a..e884af7 100644
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)
 
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa->sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);
 	else {
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/meson.build +5 -0
index fc69970..9335f28 100644
@@ -6,3 +6,8 @@ sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c', 'ipsec_sad.c')
 headers = files('rte_ipsec.h', 'rte_ipsec_group.h', 'rte_ipsec_sa.h', 'rte_ipsec_sad.h')
 
 deps += ['mbuf', 'net', 'cryptodev', 'security', 'hash']
+
+# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
+if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
+    ext_deps += cc.find_library('atomic')
+endif
lib/librte_ipsec/sa.h +1 -1
index d22451b..cab9a2e 100644
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;
 		struct {
Signature · 1 line
2.7.4
Acked-by Bruce Richardson on v2 1/1
#44 lore raw

Re: [dpdk-dev] [PATCH v2] ipsec: optimize with c11 atomic for sa outbound sqn update

Quoted 16 lines · Phil Yang

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

lib/librte_ipsec/meson.build +0 -0
index fc69970..9335f28 100644

lib/librte_ipsec/meson.build sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c', 'ipsec_sad.c') · line 13

Quoted 8 lines
lib/librte_ipsec/meson.build +5 -0
@@ -6,3 +6,8 @@ sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c', 'ipsec_sad.c')
 headers = files('rte_ipsec.h', 'rte_ipsec_group.h', 'rte_ipsec_sa.h', 'rte_ipsec_sad.h')
 
 deps += ['mbuf', 'net', 'cryptodev', 'security', 'hash']
+
+# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
+if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
+    ext_deps += cc.find_library('atomic')
+endif

The following patch has been merged in master now. You don't need this anymore.

commit da4eae278b56e698c64d0c39939a7a55c5b6abdd

Author: Pavan Nikhilesh <[email protected]>
Date:   Sun Apr 19 15:31:01 2020 +0530

build: add global libatomic dependency for 32-bit clang

Add libatomic as a global dependency when compiling for 32-bit using clang. As we need libatomic for 64-bit atomic ops.

author
#45 lore raw

Re: [dpdk-dev] [PATCH v2] ipsec: optimize with c11 atomic for sa outbound sqn update

Quoted 49 lines

-----Original Message-----
From: Jerin Jacob <[email protected]>
Sent: Friday, April 24, 2020 1:45 AM
To: Phil Yang <[email protected]>
Cc: Ananyev, Konstantin <[email protected]>; dpdk-dev <[email protected]>; [email protected]; Bernard Iremonger <[email protected]>; Vladimir Medvedkin <[email protected]>; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v2] ipsec: optimize with c11 atomic for sa outbound sqn update

lib/librte_ipsec/meson.build sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c',

Quoted 16 lines · Phil Yang

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

lib/librte_ipsec/meson.build +0 -0
index fc69970..9335f28 100644
@@ -6,3 +6,8 @@ sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c',

'ipsec_sad.c')

lib/librte_ipsec/meson.build sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c', · line 6

Quoted 1 line
lib/librte_ipsec/meson.build +0 -0
 headers = files('rte_ipsec.h', 'rte_ipsec_group.h', 'rte_ipsec_sa.h',

'rte_ipsec_sad.h')

lib/librte_ipsec/meson.build sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c', 'ipsec_sad.c') · line 13

Quoted 7 lines
lib/librte_ipsec/meson.build +5 -0
 deps += ['mbuf', 'net', 'cryptodev', 'security', 'hash']
+
+# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
+if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
+    ext_deps += cc.find_library('atomic')
+endif

The following patch has been merged in master now. You don't need this anymore.

commit da4eae278b56e698c64d0c39939a7a55c5b6abdd

Author: Pavan Nikhilesh <[email protected]>
Date:   Sun Apr 19 15:31:01 2020 +0530
Quoted 7 lines

build: add global libatomic dependency for 32-bit clang

Add libatomic as a global dependency when compiling for 32-bit using clang. As we need libatomic for 64-bit atomic ops.

Great, we don't need to add it module by module anymore. Updated in v3. Thank you very much.

Thanks,
Phil

#46 lore raw

Re: [dpdk-dev] [PATCH v2] ipsec: optimize with c11 atomic for sa outbound sqn update

In reply to Phil Yang
Quoted 52 lines

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
v2:
split from the "generic rte atomic APIs deprecate proposal" patchset.

 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/meson.build | 5 +++++
 lib/librte_ipsec/sa.h        | 2 +-
 3 files changed, 8 insertions(+), 2 deletions(-)
lib/librte_ipsec/ipsec_sqn.h +2 -1
index 0c2f76a..e884af7 100644
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)
 
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa->sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);
 	else {
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/meson.build +5 -0
index fc69970..9335f28 100644
@@ -6,3 +6,8 @@ sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c', 'ipsec_sad.c')
 headers = files('rte_ipsec.h', 'rte_ipsec_group.h', 'rte_ipsec_sa.h', 'rte_ipsec_sad.h')
 
 deps += ['mbuf', 'net', 'cryptodev', 'security', 'hash']
+
+# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
+if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
+    ext_deps += cc.find_library('atomic')
+endif
lib/librte_ipsec/sa.h +0 -0
index d22451b..cab9a2e 100644

lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 126

Quoted 8 lines
lib/librte_ipsec/sa.h +1 -1
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;
 		struct {

Seems you missed my comments for previous version, so I put here:

If we don't need rte_atomic64 here anymore,
then I think we can collapse the union to just:
uint64_t outb;

Konstantin

author
#47 lore raw

Re: [dpdk-dev] [PATCH v2] ipsec: optimize with c11 atomic for sa outbound sqn update

Quoted 79 lines

-----Original Message-----
From: Ananyev, Konstantin <[email protected]>
Sent: Friday, April 24, 2020 2:11 AM
To: Phil Yang <[email protected]>; [email protected]
Cc: [email protected]; Iremonger, Bernard <[email protected]>; Medvedkin, Vladimir <[email protected]>; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v2] ipsec: optimize with c11 atomic for sa outbound sqn update

lib/librte_ipsec/ipsec_sqn.h esn_outb_update_sqn(struct rte_ipsec_sa *sa,

Quoted 25 lines

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64.

---
v2:
split from the "generic rte atomic APIs deprecate proposal" patchset.

 lib/librte_ipsec/ipsec_sqn.h | 3 ++-
 lib/librte_ipsec/meson.build | 5 +++++
 lib/librte_ipsec/sa.h        | 2 +-
 3 files changed, 8 insertions(+), 2 deletions(-)
lib/librte_ipsec/ipsec_sqn.h +0 -0
index 0c2f76a..e884af7 100644
@@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa,

uint32_t *num)

lib/librte_ipsec/meson.build sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c',

Quoted 15 lines
lib/librte_ipsec/ipsec_sqn.h +2 -1
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa-
 sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n,
+			__ATOMIC_RELAXED);
 	else {
 		sqn = sa->sqn.outb.raw + n;
 		sa->sqn.outb.raw = sqn;
lib/librte_ipsec/meson.build +0 -0
index fc69970..9335f28 100644
@@ -6,3 +6,8 @@ sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c',

'ipsec_sad.c')

lib/librte_ipsec/meson.build sources = files('esp_inb.c', 'esp_outb.c', 'sa.c', 'ses.c', · line 6

Quoted 1 line
lib/librte_ipsec/meson.build +0 -0
 headers = files('rte_ipsec.h', 'rte_ipsec_group.h', 'rte_ipsec_sa.h',

'rte_ipsec_sad.h')

Quoted 12 lines
lib/librte_ipsec/meson.build +5 -0
 deps += ['mbuf', 'net', 'cryptodev', 'security', 'hash']
+
+# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
+if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
+    ext_deps += cc.find_library('atomic')
+endif
lib/librte_ipsec/sa.h +0 -0
index d22451b..cab9a2e 100644

lib/librte_ipsec/sa.h struct rte_ipsec_sa { · line 126

Quoted 8 lines
lib/librte_ipsec/sa.h +1 -1
@@ -120,7 +120,7 @@ struct rte_ipsec_sa {
 	 */
 	union {
 		union {
-			rte_atomic64_t atom;
+			uint64_t atom;
 			uint64_t raw;
 		} outb;
 		struct {

Seems you missed my comments for previous version, so I put here:

Quoted 3 lines

If we don't need rte_atomic64 here anymore, then I think we can collapse the union to just:
uint64_t outb;

My bad, I missed this comment.
Updated in v3. Please review it.

Thanks,
Phil

Quoted 2 lines

Konstantin

author
#48 lore raw

PATCH v3 1/1 ipsec: optimize with c11 atomic for sa outbound sqn update

In reply to Phil Yang

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. Use c11 atomics with RELAXED ordering for outbound SQN update instead of rte_atomic ops which enforce unnecessary barriers on aarch64.

---
v3:
1. since libatomic dependency for 32-bit clang added globally, so remove the redundant code.
2. collapse union outb to unint64_t outb as the rte_atomic is no needed.

v2:
split from the "generic rte atomic APIs deprecate proposal" patchset.

 lib/librte_ipsec/ipsec_sqn.h | 6 +++---
 lib/librte_ipsec/sa.c        | 2 +-
 lib/librte_ipsec/sa.h        | 5 +----
 3 files changed, 5 insertions(+), 8 deletions(-)
lib/librte_ipsec/ipsec_sqn.h +3 -3
index 0c2f76a..2636cb1 100644
@@ -128,10 +128,10 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)
 
 	n = *num;
 	if (SQN_ATOMIC(sa))
-		sqn = (uint64_t)rte_atomic64_add_return(&sa->sqn.outb.atom, n);
+		sqn = __atomic_add_fetch(&sa->sqn.outb, n, __ATOMIC_RELAXED);
 	else {
-		sqn = sa->sqn.outb.raw + n;
-		sa->sqn.outb.raw = sqn;
+		sqn = sa->sqn.outb + n;
+		sa->sqn.outb = sqn;
 	}
 
 	/* overflow */
lib/librte_ipsec/sa.c +1 -1
index ada195c..e59189d 100644
@@ -283,7 +283,7 @@ esp_outb_init(struct rte_ipsec_sa *sa, uint32_t hlen)
 {
 	uint8_t algo_type;
 
-	sa->sqn.outb.raw = 1;
+	sa->sqn.outb = 1;
 
 	algo_type = sa->algo_type;
 
lib/librte_ipsec/sa.h +1 -4
index d22451b..508dd2b 100644
@@ -119,10 +119,7 @@ struct rte_ipsec_sa {
 	 * place from other frequently accesed data.
 	 */
 	union {
-		union {
-			rte_atomic64_t atom;
-			uint64_t raw;
-		} outb;
+		uint64_t outb;
 		struct {
 			uint32_t rdidx; /* read index */
 			uint32_t wridx; /* write index */
Signature · 1 line
2.7.4
Acked-by on v3 1/1
#49 lore raw

Re: [dpdk-dev] [PATCH v3] ipsec: optimize with c11 atomic for sa outbound sqn update

Quoted 10 lines

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. Use c11 atomics with RELAXED ordering for outbound SQN update instead of rte_atomic ops which enforce unnecessary barriers on aarch64.

---

Quoted 1 line

2.7.4

Applied
#50 lore raw

Re: [dpdk-dev] [PATCH v3] ipsec: optimize with c11 atomic for sa outbound sqn update

Quoted 7 lines
Quoted 7 lines

For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. Use c11 atomics with RELAXED ordering for outbound SQN update instead of rte_atomic ops which enforce unnecessary barriers on aarch64.

Quoted 7 lines
Quoted 4 lines

---

Applied to dpdk-next-crypto

Thanks.

author
#51 lore raw

PATCH v3 7/12 service: remove rte prefix from static functions

In reply to Phil Yang
Fixes 3cf5eb1546ed ("service: fix and refactor atomic service accesses")
Fixes 21698354c832 ("service: introduce service cores concept")
---
 lib/librte_eal/common/rte_service.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +9 -9
index b0b78ba..2117726 100644
@@ -336,7 +336,7 @@ rte_service_runstate_get(uint32_t id)
 }
 
 static inline void
-rte_service_runner_do_callback(struct rte_service_spec_impl *s,
+service_runner_do_callback(struct rte_service_spec_impl *s,
 			       struct core_state *cs, uint32_t service_idx)
 {
 	void *userdata = s->spec.callback_userdata;
@@ -379,10 +379,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 		rte_atomic32_clear(&s->execute_lock);
 	} else
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 
 	return 0;
 }
@@ -436,7 +436,7 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 }
 
 static int32_t
-rte_service_runner_func(void *arg)
+service_runner_func(void *arg)
 {
 	RTE_SET_USED(arg);
 	uint32_t i;
@@ -706,7 +706,7 @@ rte_service_lcore_start(uint32_t lcore)
 	 */
 	lcore_states[lcore].runstate = RUNSTATE_RUNNING;
 
-	int ret = rte_eal_remote_launch(rte_service_runner_func, 0, lcore);
+	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
 	return ret;
 }
@@ -785,7 +785,7 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id,
 }
 
 static void
-rte_service_dump_one(FILE *f, struct rte_service_spec_impl *s,
+service_dump_one(FILE *f, struct rte_service_spec_impl *s,
 		     uint64_t all_cycles, uint32_t reset)
 {
 	/* avoid divide by zero */
@@ -818,7 +818,7 @@ rte_service_attr_reset_all(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	int reset = 1;
-	rte_service_dump_one(NULL, s, 0, reset);
+	service_dump_one(NULL, s, 0, reset);
 	return 0;
 }
 
@@ -876,7 +876,7 @@ rte_service_dump(FILE *f, uint32_t id)
 		SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 		fprintf(f, "Service %s Summary\n", s->spec.name);
 		uint32_t reset = 0;
-		rte_service_dump_one(f, s, total_cycles, reset);
+		service_dump_one(f, s, total_cycles, reset);
 		return 0;
 	}
 
@@ -886,7 +886,7 @@ rte_service_dump(FILE *f, uint32_t id)
 		if (!service_valid(i))
 			continue;
 		uint32_t reset = 0;
-		rte_service_dump_one(f, &rte_services[i], total_cycles, reset);
+		service_dump_one(f, &rte_services[i], total_cycles, reset);
 	}
 
 	fprintf(f, "Service Cores Summary\n");
Signature · 1 line
2.7.4
#52 lore raw

Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions

Quoted 10 lines

From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 07/12] service: remove rte prefix from static functions

Quoted 6 lines
Fixes 3cf5eb1546ed ("service: fix and refactor atomic service accesses")
Fixes 21698354c832 ("service: introduce service cores concept")

This patchset needs a rebase since the EAL file movement got merged, however I'll review here so we can include some Acks etc and make progress.

Is this really a "Fix"? The internal function names were not exported in the .map file, so are not part of public ABI. This is an internal naming improvement (thanks for doing cleanup), but I don't think the
Fixes: tags make sense?

Also I'm not sure if we want to port this patch back to stable? Changing (internal) function names seems like unnecessary churn, and hence risk to a stable release, without any benefit?

---

<snip patch diff>

author
#53 lore raw

Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions

Quoted 40 lines

-----Original Message-----
From: Van Haaren, Harry <[email protected]>
Sent: Friday, April 3, 2020 7:58 PM
To: Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]
Subject: RE: [PATCH v3 07/12] service: remove rte prefix from static functions

Quoted 3 lines

From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry

<[email protected]>;

Quoted 2 lines

Ananyev, Konstantin <[email protected]>; [email protected]; [email protected];

[email protected]

Quoted 1 line

[email protected];

Quoted 1 line

[email protected]; [email protected];

[email protected];

Quoted 9 lines

[email protected]; [email protected]; [email protected]
Subject: [PATCH v3 07/12] service: remove rte prefix from static functions

Fixes 3cf5eb1546ed ("service: fix and refactor atomic service accesses")
Fixes 21698354c832 ("service: introduce service cores concept")

This patchset needs a rebase since the EAL file movement got merged, however I'll review here so we can include some Acks etc and make progress.

Quoted 8 lines

Is this really a "Fix"? The internal function names were not exported in the .map file, so are not part of public ABI. This is an internal naming improvement (thanks for doing cleanup), but I don't think the
Fixes: tags make sense?

Also I'm not sure if we want to port this patch back to stable? Changing (internal) function names seems like unnecessary churn, and hence risk to a stable release, without any benefit?

OK.
I will remove these tags in the next version and split the service core patches from the original series into a series by itself.

Thanks,
Phil

Quoted 4 lines

---

<snip patch diff>

#54 lore raw

Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions

Quoted 11 lines

-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Wednesday, April 8, 2020 11:15 AM
To: Van Haaren, Harry <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v3 07/12] service: remove rte prefix from static functions

<snip>

Quoted 11 lines
Quoted 8 lines

Is this really a "Fix"? The internal function names were not exported in the .map file, so are not part of public ABI. This is an internal naming improvement (thanks for doing cleanup), but I don't think the
Fixes: tags make sense?

Also I'm not sure if we want to port this patch back to stable? Changing (internal) function names seems like unnecessary churn, and hence risk to a stable release, without any benefit?

OK.
I will remove these tags in the next version and split the service core patches from the original series into a series by itself.

Cool - good idea to split.

Perhaps we should focus on getting bugfixes in for the existing code, before doing cleanup? It would make backports easier if churn is minimal.

Suggesting patches order (first to last)
1. bugfixes/things to backport
2. cleanups
3. C11 atomic optimizations

Quoted 2 lines

Thanks,
Phil

Thanks, and I'll get to reading/reviewing your and Honnappa's feedback later today.

-H

author
#55 lore raw

Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions

Quoted 48 lines

-----Original Message-----
From: Van Haaren, Harry <[email protected]>
Sent: Wednesday, April 8, 2020 6:37 PM
To: Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v3 07/12] service: remove rte prefix from static functions

Quoted 4 lines

-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Wednesday, April 8, 2020 11:15 AM
To: Van Haaren, Harry <[email protected]>;

[email protected];

Quoted 2 lines

Ananyev, Konstantin <[email protected]>; [email protected]; [email protected];

[email protected]

Quoted 1 line

[email protected];

Quoted 2 lines

Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce

Kong

Quoted 1 line

<[email protected]>; nd <[email protected]>; [email protected]; nd

<[email protected]>

Quoted 1 line

Subject: RE: [PATCH v3 07/12] service: remove rte prefix from static

functions
<snip>

Quoted 7 lines
Quoted 7 lines

Is this really a "Fix"? The internal function names were not exported in the .map file, so are not part of public ABI. This is an internal naming improvement (thanks for doing cleanup), but I don't think the
Fixes: tags make sense?

Also I'm not sure if we want to port this patch back to stable? Changing (internal) function names seems like unnecessary churn, and hence risk

to a

Quoted 4 lines
Quoted 1 line

stable release, without any benefit?

OK.
I will remove these tags in the next version and split the service core patches from the original series into a series by itself.

Cool - good idea to split.

Quoted 7 lines

Perhaps we should focus on getting bugfixes in for the existing code, before doing cleanup? It would make backports easier if churn is minimal.

Suggesting patches order (first to last)
1. bugfixes/things to backport
2. cleanups
3. C11 atomic optimizations

That is a good idea. I will follow this order.

Quoted 9 lines
Quoted 4 lines

Thanks,
Phil

Thanks, and I'll get to reading/reviewing your and Honnappa's feedback later today.

-H

#56 lore raw

Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions

In reply to Phil Yang

<snip>

Quoted 38 lines

Subject: [PATCH v3 07/12] service: remove rte prefix from static functions

Fixes 3cf5eb1546ed ("service: fix and refactor atomic service accesses")
Fixes 21698354c832 ("service: introduce service cores concept")
---
 lib/librte_eal/common/rte_service.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +1 -1
index b0b78ba..2117726 100644
@@ -336,7 +336,7 @@ rte_service_runstate_get(uint32_t id)  }
 
 static inline void
-rte_service_runner_do_callback(struct rte_service_spec_impl *s,
+service_runner_do_callback(struct rte_service_spec_impl *s,
 			       struct core_state *cs, uint32_t service_idx)  {
 	void *userdata = s->spec.callback_userdata; @@ -379,10 +379,10
 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
lib/librte_eal/common/rte_service.c +2 -2
 			return -EBUSY;
 
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 		rte_atomic32_clear(&s->execute_lock);
 	} else
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 
 	return 0;
 }

lib/librte_eal/common/rte_service.c rte_service_run_iter_on_app_lcore(uint32_t id, · line 442

Quoted 8 lines
lib/librte_eal/common/rte_service.c +1 -1
@@ -436,7 +436,7 @@ rte_service_run_iter_on_app_lcore(uint32_t id,
 uint32_t serialize_mt_unsafe)  }
 
 static int32_t
-rte_service_runner_func(void *arg)
+service_runner_func(void *arg)
 {
 	RTE_SET_USED(arg);
 	uint32_t i;

This is a minor comment.
Since you are touching 'service_runner_func', please consider doing the below improvement:

struct core_state *cs = &lcore_states[lcore]; while (lcore_states[lcore].runstate == RUNSTATE_RUNNING) {

The while above can be changed as follows to make it more readable

while (cs->runstate == RUNSTATE_RUNNING) {

Quoted 12 lines
lib/librte_eal/common/rte_service.c +1 -1
@@ -706,7 +706,7 @@ rte_service_lcore_start(uint32_t lcore)
 	 */
 	lcore_states[lcore].runstate = RUNSTATE_RUNNING;
 
-	int ret = rte_eal_remote_launch(rte_service_runner_func, 0, lcore);
+	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
 	return ret;
 }
@@ -785,7 +785,7 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t
 attr_id,  }
 

<snip>

author
#57 lore raw

Re: [dpdk-dev] [PATCH v3 07/12] service: remove rte prefix from static functions

Quoted 67 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Monday, April 6, 2020 5:35 AM
To: Phil Yang <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 07/12] service: remove rte prefix from static functions

<snip>

Quoted 18 lines

Subject: [PATCH v3 07/12] service: remove rte prefix from static functions

Fixes 3cf5eb1546ed ("service: fix and refactor atomic service accesses")
Fixes 21698354c832 ("service: introduce service cores concept")
---
 lib/librte_eal/common/rte_service.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +0 -0
index b0b78ba..2117726 100644

lib/librte_eal/common/rte_service.c rte_service_runstate_get(uint32_t id) } · line 342

Quoted 8 lines
lib/librte_eal/common/rte_service.c +1 -1
@@ -336,7 +336,7 @@ rte_service_runstate_get(uint32_t id)  }
 
 static inline void
-rte_service_runner_do_callback(struct rte_service_spec_impl *s,
+service_runner_do_callback(struct rte_service_spec_impl *s,
 			       struct core_state *cs, uint32_t service_idx)  {
 	void *userdata = s->spec.callback_userdata; @@ -379,10 +379,10
 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0,

1))

Quoted 12 lines
lib/librte_eal/common/rte_service.c +2 -2
 			return -EBUSY;
 
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 		rte_atomic32_clear(&s->execute_lock);
 	} else
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 
 	return 0;
 }

lib/librte_eal/common/rte_service.c rte_service_run_iter_on_app_lcore(uint32_t id, · line 442

Quoted 8 lines
lib/librte_eal/common/rte_service.c +1 -1
@@ -436,7 +436,7 @@ rte_service_run_iter_on_app_lcore(uint32_t id,
 uint32_t serialize_mt_unsafe)  }
 
 static int32_t
-rte_service_runner_func(void *arg)
+service_runner_func(void *arg)
 {
 	RTE_SET_USED(arg);
 	uint32_t i;

This is a minor comment.
Since you are touching 'service_runner_func', please consider doing the below improvement:

Quoted 6 lines

struct core_state *cs = &lcore_states[lcore]; while (lcore_states[lcore].runstate == RUNSTATE_RUNNING) {

The while above can be changed as follows to make it more readable

while (cs->runstate == RUNSTATE_RUNNING) {

OK. I will clean it up in the next version.

Thanks,
Phil

Quoted 14 lines
Quoted 13 lines
lib/librte_eal/common/rte_service.c +1 -1
@@ -706,7 +706,7 @@ rte_service_lcore_start(uint32_t lcore)
 	 */
 	lcore_states[lcore].runstate = RUNSTATE_RUNNING;
 
-	int ret = rte_eal_remote_launch(rte_service_runner_func, 0, lcore);
+	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
 	return ret;
 }
@@ -785,7 +785,7 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t
 attr_id,  }
 

<snip>

author
#58 lore raw

v2 cover use c11 atomics for service core lib

In reply to Phil Yang

The rte_atomic ops and rte_smp barriers enforce DMB barriers on aarch64. Using c11 atomics with explicit memory ordering instead of the rte_atomic ops and rte_smp barriers for inter-threads synchronization can uplift the performance on aarch64 and no performance loss on x86.

This patchset contains:
1) fix race condition for MT unsafe service.
2) clean up redundant code.
3) use c11 atomics for service core lib to avoid unnecessary barriers.

v2:
Still waiting on Harry for the final solution on the MT unsafe race condition issue. But I have incorporated the comments so far.
1. add 'Fixes' tag for bug-fix patches.
2. remove 'Fixes' tag for code cleanup patches.
3. remove unused parameter for service_dump_one function.
4. replace the execute_lock atomic CAS operation to spinlock_try_lock.
5. use c11 atomics with RELAXED memory ordering for num_mapped_cores.
6. relax barriers for guard variables runstate, comp_runstate and app_runstate with c11 one-way barriers.

Honnappa Nagarahalli (2):
service: fix race condition for MT unsafe service
service: identify service running on another core correctly

Phil Yang (4):
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 atomics
service: relax barriers with C11 atomics

 lib/librte_eal/common/rte_service.c | 234 +++++++++++++++++++-----------------
 lib/librte_eal/meson.build          |   4 +
 2 files changed, 130 insertions(+), 108 deletions(-)
Signature · 1 line
2.7.4
author
#59 lore raw

PATCH v2 1/6 service: fix race condition for MT unsafe service

From: Honnappa Nagarahalli <[email protected]>

The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Fixes e9139a32f6e8 ("service: add function to run on app lcore")
---
 lib/librte_eal/common/rte_service.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
index 70d17a5..b8c465e 100644
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {
 	uint8_t internal_flags;
 
 	/* per service statistics */
+	/* Indicates how many cores the service is mapped to run on.
+	 * It does not indicate the number of cores the service is running
+	 * on currently.
+	 */
 	rte_atomic32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	/* check do we need cmpset, if MT safe or <= 1 core
-	 * mapped, atomic ops are not required.
-	 */
-	const int use_atomics = (service_mt_safe(s) == 0) &&
-				(rte_atomic32_read(&s->num_mapped_cores) > 1);
-	if (use_atomics) {
+	if (service_mt_safe(s) == 0) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
Signature · 1 line
2.7.4
#60 lore raw

Re: [dpdk-dev] [PATCH v2 1/6] service: fix race condition for MT unsafe service

Quoted 18 lines

-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Thursday, April 23, 2020 5:31 PM
To: Van Haaren, Harry <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: [PATCH v2 1/6] service: fix race condition for MT unsafe service

From: Honnappa Nagarahalli <[email protected]>

The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")

---

Thanks for spinning a new revision - based on ML discussion previously, it seems like the "use service-run-count" to avoid this race would be a complex solution.

Suggesting the following;
1) Take the approach as per this patch, to always take the atomic, fixing the race condition.
2) Add an API to service-cores, which allows "committing" of mappings. Committing the mapping would imply that the mappings will not be changed in future. With runtime-remapping being removed from the equation, the existing branch-over-atomic optimization is valid again.

So this would offer applications two situations
A) No application change: possible performance regression due to atomic always taken.
B) Call "commit" API, and regain the performance as per previous DPDK versions.

Thoughts/opinions on the above? I've flagged the rest of the patchset for review ASAP. Regards, -Harry

Quoted 36 lines
 lib/librte_eal/common/rte_service.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
lib/librte_eal/common/rte_service.c +5 -7
index 70d17a5..b8c465e 100644
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {
 	uint8_t internal_flags;
 
 	/* per service statistics */
+	/* Indicates how many cores the service is mapped to run on.
+	 * It does not indicate the number of cores the service is running
+	 * on currently.
+	 */
 	rte_atomic32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t
 service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	/* check do we need cmpset, if MT safe or <= 1 core
-	 * mapped, atomic ops are not required.
-	 */
-	const int use_atomics = (service_mt_safe(s) == 0) &&
-				(rte_atomic32_read(&s->num_mapped_cores) > 1);
-	if (use_atomics) {
+	if (service_mt_safe(s) == 0) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
--

2.7.4

#61 lore raw

Re: [dpdk-dev] [PATCH v2 1/6] service: fix race condition for MT unsafe service

Hi Harry,
Thanks for getting back on this.

<snip>

Quoted 17 lines
Quoted 10 lines

Subject: [PATCH v2 1/6] service: fix race condition for MT unsafe service

From: Honnappa Nagarahalli <[email protected]>

The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")

---

Quoted 7 lines

Thanks for spinning a new revision - based on ML discussion previously, it seems like the "use service-run-count" to avoid this race would be a complex solution.

Suggesting the following;
1) Take the approach as per this patch, to always take the atomic, fixing the race condition.

Ok

Quoted 4 lines

2) Add an API to service-cores, which allows "committing" of mappings. Committing the mapping would imply that the mappings will not be changed in future. With runtime-remapping being removed from the equation, the existing branch-over-atomic optimization is valid again.

Ok. Just to make sure I understand this:
a) on the data plane, if commit API is called (probably a new state variable) and num_mapped_cores is set to 1, there is no need to take the lock.
b) possible implementation of the commit API would check if num_mapped_cores for the service is set to 1 and set a variable to indicate that the lock is not required.

What do you think about asking the application to set the service capability to MT_SAFE if it knows that the service will run on a single core? This would require us to change the documentation and does not require additional code.

Quoted 47 lines

So this would offer applications two situations
A) No application change: possible performance regression due to atomic always taken.
B) Call "commit" API, and regain the performance as per previous DPDK versions.

Thoughts/opinions on the above? I've flagged the rest of the patchset for review ASAP. Regards, -Harry

Quoted 37 lines
 lib/librte_eal/common/rte_service.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
index 70d17a5..b8c465e 100644
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {
 	uint8_t internal_flags;
 
 	/* per service statistics */
+	/* Indicates how many cores the service is mapped to run on.
+	 * It does not indicate the number of cores the service is running
+	 * on currently.
+	 */
 	rte_atomic32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs,
 uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	/* check do we need cmpset, if MT safe or <= 1 core
-	 * mapped, atomic ops are not required.
-	 */
-	const int use_atomics = (service_mt_safe(s) == 0) &&
-				(rte_atomic32_read(&s-
 num_mapped_cores) > 1);
-	if (use_atomics) {
+	if (service_mt_safe(s) == 0) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))

return -EBUSY;

--
2.7.4

#62 lore raw

Re: [dpdk-dev] [PATCH v2 1/6] service: fix race condition for MT unsafe service

Quoted 35 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Wednesday, April 29, 2020 11:49 PM
To: Van Haaren, Harry <[email protected]>; Phil Yang <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Gavin Hu <[email protected]>; nd <[email protected]>; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v2 1/6] service: fix race condition for MT unsafe service

Hi Harry,
Thanks for getting back on this.

<snip>

Quoted 17 lines
Quoted 16 lines

Subject: [PATCH v2 1/6] service: fix race condition for MT unsafe service

From: Honnappa Nagarahalli <[email protected]>

The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Fixes e9139a32f6e8 ("service: add function to run on app lcore")

---

Quoted 8 lines
Quoted 7 lines

Thanks for spinning a new revision - based on ML discussion previously, it seems like the "use service-run-count" to avoid this race would be a complex solution.

Suggesting the following;
1) Take the approach as per this patch, to always take the atomic, fixing the race condition.

Ok

I've micro-benchmarked this code change inside the service cores autotest, and it introduces around 35 cycles of overhead per service call. This is not ideal, but given it's a bugfix, and by far the simplest method to fix this race-condition. Having discussed and investigated multiple other solutions, I believe this is the right solution. Thanks Honnappa and Phil for identifying and driving a solution.

I suggest to post the benchmarking unit-test addition patch, and integrate that *before* the series under review here gets merged? This makes benchmarking of the "before bugfix" performance in future easier should it be required.

Quoted 11 lines
Quoted 4 lines

2) Add an API to service-cores, which allows "committing" of mappings. Committing the mapping would imply that the mappings will not be changed in future. With runtime-remapping being removed from the equation, the existing branch-over-atomic optimization is valid again.

Ok. Just to make sure I understand this:
a) on the data plane, if commit API is called (probably a new state variable) and num_mapped_cores is set to 1, there is no need to take the lock.
b) possible implementation of the commit API would check if num_mapped_cores for the service is set to 1 and set a variable to indicate that the lock is not required.

Quoted 3 lines

What do you think about asking the application to set the service capability to MT_SAFE if it knows that the service will run on a single core? This would require us to change the documentation and does not require additional code.

That's a nice idea - I like that if applications want to micro-optimize around the atomic, that they have a workaround/solution to do so, particularly that it doesn't require code-changes and backporting.

Will send review and send feedback on the patches themselves. Regards, -Harry

Quoted 11 lines
Quoted 11 lines

So this would offer applications two situations
A) No application change: possible performance regression due to atomic always taken.
B) Call "commit" API, and regain the performance as per previous DPDK versions.

Thoughts/opinions on the above? I've flagged the rest of the patchset for review ASAP. Regards, -Harry

Quoted 2 lines
 lib/librte_eal/common/rte_service.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

<snip patch changes>

#63 lore raw

Re: [dpdk-dev] [PATCH v2 1/6] service: fix race condition for MT unsafe service

<snip>

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 11 lines

Subject: [PATCH v2 1/6] service: fix race condition for MT unsafe service

From: Honnappa Nagarahalli <[email protected]>

The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")

---

Thanks for spinning a new revision - based on ML discussion previously, it seems like the "use service-run-count" to avoid this race would be a complex solution.

Suggesting the following;
1) Take the approach as per this patch, to always take the atomic, fixing the race condition.

Quoted 8 lines
Quoted 1 line

Ok

I've micro-benchmarked this code change inside the service cores autotest, and it introduces around 35 cycles of overhead per service call. This is not ideal, but given it's a bugfix, and by far the simplest method to fix this race- condition. Having discussed and investigated multiple other solutions, I believe this is the right solution.
Thanks Honnappa and Phil for identifying and driving a solution.

You are welcome. Thank you for your timely responses.

Quoted 6 lines

I suggest to post the benchmarking unit-test addition patch, and integrate that
*before* the series under review here gets merged? This makes benchmarking of the "before bugfix" performance in future easier should it be required.

I do not see any issues, would be happy to review. I think we still have time to catch up with RC2 (May 8th). You had also mentioned about calling out that, the control plane APIs are not MT safe. Should I add that to this patch?

Quoted 38 lines
Quoted 9 lines
Quoted 6 lines

2) Add an API to service-cores, which allows "committing" of mappings. Committing the mapping would imply that the mappings will not be changed in future. With runtime-remapping being removed from the equation, the existing branch-over-atomic optimization is valid again.

Ok. Just to make sure I understand this:
a) on the data plane, if commit API is called (probably a new state variable) and num_mapped_cores is set to 1, there is no need to take the

lock.

Quoted 7 lines

b) possible implementation of the commit API would check if num_mapped_cores for the service is set to 1 and set a variable to indicate that the lock is not required.

What do you think about asking the application to set the service capability to MT_SAFE if it knows that the service will run on a single core? This would require us to change the documentation and does

not require additional code.

That's a nice idea - I like that if applications want to micro-optimize around the atomic, that they have a workaround/solution to do so, particularly that it doesn't require code-changes and backporting.

Will send review and send feedback on the patches themselves. Regards, -Harry

Quoted 11 lines
Quoted 11 lines

So this would offer applications two situations
A) No application change: possible performance regression due to atomic always taken.
B) Call "commit" API, and regain the performance as per previous DPDK versions.

Thoughts/opinions on the above? I've flagged the rest of the patchset for review ASAP. Regards, -Harry

Quoted 2 lines
 lib/librte_eal/common/rte_service.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

<snip patch changes>

#64 lore raw

Re: [dpdk-dev] [PATCH v2 1/6] service: fix race condition for MT unsafe service

Quoted 41 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Friday, May 1, 2020 3:56 PM
To: Van Haaren, Harry <[email protected]>; Phil Yang <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Gavin Hu <[email protected]>; nd <[email protected]>; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v2 1/6] service: fix race condition for MT unsafe service

<snip>

Quoted 27 lines
Quoted 26 lines
Quoted 18 lines
Quoted 11 lines

Subject: [PATCH v2 1/6] service: fix race condition for MT unsafe service

From: Honnappa Nagarahalli <[email protected]>

The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")

---

Quoted 7 lines

Thanks for spinning a new revision - based on ML discussion previously, it seems like the "use service-run-count" to avoid this race would be a complex solution.

Suggesting the following;
1) Take the approach as per this patch, to always take the atomic, fixing the race condition.

Ok

Quoted 7 lines
Quoted 6 lines

I've micro-benchmarked this code change inside the service cores autotest, and it introduces around 35 cycles of overhead per service call. This is not ideal, but given it's a bugfix, and by far the simplest method to fix this race- condition. Having discussed and investigated multiple other solutions, I believe this is the right solution.
Thanks Honnappa and Phil for identifying and driving a solution.

You are welcome. Thank you for your timely responses.

Perhaps not so timely after all ... I'll review C11 patches Tuesday morning, it's a long weekend in Ireland!

Quoted 6 lines
Quoted 5 lines

I suggest to post the benchmarking unit-test addition patch, and integrate that
*before* the series under review here gets merged? This makes benchmarking of the "before bugfix" performance in future easier should it be required.

I do not see any issues, would be happy to review.

Thanks for volunteering, you're on CC when sent, for convenience:
http://patches.dpdk.org/patch/69651/

Quoted 1 line

I think we still have time to catch up with RC2 (May 8th).

Agree, merge into RC2 would be great.

Quoted 2 lines

You had also mentioned about calling out that, the control plane APIs are not MT safe. Should I add that to this patch?

Yes, that'd be great.

<snip discussion details>

Cheers, -Harry

author
#65 lore raw

PATCH v2 2/6 service: identify service running on another core correctly

In reply to Phil Yang

From: Honnappa Nagarahalli <[email protected]>

The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases.

Fixes 8d39d3e237c2 ("service: fix race in service on app lcore function")
---
 lib/librte_eal/common/rte_service.c | 26 ++++++++------------------
 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
index b8c465e..c89472b 100644
@@ -360,7 +360,7 @@ rte_service_runner_do_callback(struct rte_service_spec_impl *s,
 /* Expects the service 's' is valid. */
 static int32_t
 service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
-	    struct rte_service_spec_impl *s)
+	    struct rte_service_spec_impl *s, uint32_t serialize_mt_unsafe)
 {
 	if (!s)
 		return -EINVAL;
@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	if (service_mt_safe(s) == 0) {
+	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
-	/* Atomically add this core to the mapped cores first, then examine if
-	 * we can run the service. This avoids a race condition between
-	 * checking the value, and atomically adding to the mapped count.
+	/* Increment num_mapped_cores to indicate that the service
+	 * is running on a core.
 	 */
-	if (serialize_mt_unsafe)
-		rte_atomic32_inc(&s->num_mapped_cores);
+	rte_atomic32_inc(&s->num_mapped_cores);
 
-	if (service_mt_safe(s) == 0 &&
-			rte_atomic32_read(&s->num_mapped_cores) > 1) {
-		if (serialize_mt_unsafe)
-			rte_atomic32_dec(&s->num_mapped_cores);
-		return -EBUSY;
-	}
-
-	int ret = service_run(id, cs, UINT64_MAX, s);
+	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	if (serialize_mt_unsafe)
-		rte_atomic32_dec(&s->num_mapped_cores);
+	rte_atomic32_dec(&s->num_mapped_cores);
 
 	return ret;
 }
@@ -449,7 +439,7 @@ rte_service_runner_func(void *arg)
 			if (!service_valid(i))
 				continue;
 			/* return value ignored as no change to code flow */
-			service_run(i, cs, service_mask, service_get(i));
+			service_run(i, cs, service_mask, service_get(i), 1);
 		}
 
 		cs->loops++;
Signature · 1 line
2.7.4
author
#66 lore raw

PATCH v2 3/6 service: remove rte prefix from static functions

In reply to Phil Yang

clean up rte prefix from static functions. remove unused parameter for service_dump_one function.

---
 lib/librte_eal/common/rte_service.c | 34 +++++++++++-----------------------
 1 file changed, 11 insertions(+), 23 deletions(-)
lib/librte_eal/common/rte_service.c +11 -23
index c89472b..ed20702 100644
@@ -340,7 +340,7 @@ rte_service_runstate_get(uint32_t id)
 }
 
 static inline void
-rte_service_runner_do_callback(struct rte_service_spec_impl *s,
+service_runner_do_callback(struct rte_service_spec_impl *s,
 			       struct core_state *cs, uint32_t service_idx)
 {
 	void *userdata = s->spec.callback_userdata;
@@ -378,10 +378,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 		rte_atomic32_clear(&s->execute_lock);
 	} else
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 
 	return 0;
 }
@@ -425,14 +425,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 }
 
 static int32_t
-rte_service_runner_func(void *arg)
+service_runner_func(void *arg)
 {
 	RTE_SET_USED(arg);
 	uint32_t i;
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (lcore_states[lcore].runstate == RUNSTATE_RUNNING) {
+	while (cs->runstate == RUNSTATE_RUNNING) {
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
@@ -693,9 +693,9 @@ rte_service_lcore_start(uint32_t lcore)
 	/* set core to run state first, and then launch otherwise it will
 	 * return immediately as runstate keeps it in the service poll loop
 	 */
-	lcore_states[lcore].runstate = RUNSTATE_RUNNING;
+	cs->runstate = RUNSTATE_RUNNING;
 
-	int ret = rte_eal_remote_launch(rte_service_runner_func, 0, lcore);
+	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
 	return ret;
 }
@@ -774,13 +774,9 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id,
 }
 
 static void
-rte_service_dump_one(FILE *f, struct rte_service_spec_impl *s,
-		     uint64_t all_cycles, uint32_t reset)
+service_dump_one(FILE *f, struct rte_service_spec_impl *s, uint32_t reset)
 {
 	/* avoid divide by zero */
-	if (all_cycles == 0)
-		all_cycles = 1;
-
 	int calls = 1;
 	if (s->calls != 0)
 		calls = s->calls;
@@ -807,7 +803,7 @@ rte_service_attr_reset_all(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	int reset = 1;
-	rte_service_dump_one(NULL, s, 0, reset);
+	service_dump_one(NULL, s, reset);
 	return 0;
 }
 
@@ -851,21 +847,13 @@ rte_service_dump(FILE *f, uint32_t id)
 	uint32_t i;
 	int print_one = (id != UINT32_MAX);
 
-	uint64_t total_cycles = 0;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if (!service_valid(i))
-			continue;
-		total_cycles += rte_services[i].cycles_spent;
-	}
-
 	/* print only the specified service */
 	if (print_one) {
 		struct rte_service_spec_impl *s;
 		SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 		fprintf(f, "Service %s Summary\n", s->spec.name);
 		uint32_t reset = 0;
-		rte_service_dump_one(f, s, total_cycles, reset);
+		service_dump_one(f, s, reset);
 		return 0;
 	}
 
@@ -875,7 +863,7 @@ rte_service_dump(FILE *f, uint32_t id)
 		if (!service_valid(i))
 			continue;
 		uint32_t reset = 0;
-		rte_service_dump_one(f, &rte_services[i], total_cycles, reset);
+		service_dump_one(f, &rte_services[i], reset);
 	}
 
 	fprintf(f, "Service Cores Summary\n");
Signature · 1 line
2.7.4
author
#67 lore raw

PATCH v2 4/6 service: remove redundant code

In reply to Phil Yang

The service id validation is duplicated, remove the redundant code in the calling functions.

---
 lib/librte_eal/common/rte_service.c | 28 ++++++----------------------
 1 file changed, 6 insertions(+), 22 deletions(-)
lib/librte_eal/common/rte_service.c +6 -22
index ed20702..9c1a1d5 100644
@@ -541,24 +541,12 @@ rte_service_start_with_defaults(void)
 }
 
 static int32_t
-service_update(struct rte_service_spec *service, uint32_t lcore,
+service_update(uint32_t sid, uint32_t lcore,
 		uint32_t *set, uint32_t *enabled)
 {
-	uint32_t i;
-	int32_t sid = -1;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if ((struct rte_service_spec *)&rte_services[i] == service &&
-				service_valid(i)) {
-			sid = i;
-			break;
-		}
-	}
-
-	if (sid == -1 || lcore >= RTE_MAX_LCORE)
-		return -EINVAL;
-
-	if (!lcore_states[lcore].is_service_core)
+	/* validate ID, or return error value */
+	if (sid >= RTE_SERVICE_NUM_MAX || !service_valid(sid) ||
+	    lcore >= RTE_MAX_LCORE || !lcore_states[lcore].is_service_core)
 		return -EINVAL;
 
 	uint64_t sid_mask = UINT64_C(1) << sid;
@@ -587,19 +575,15 @@ service_update(struct rte_service_spec *service, uint32_t lcore,
 int32_t
 rte_service_map_lcore_set(uint32_t id, uint32_t lcore, uint32_t enabled)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 	uint32_t on = enabled > 0;
-	return service_update(&s->spec, lcore, &on, 0);
+	return service_update(id, lcore, &on, 0);
 }
 
 int32_t
 rte_service_map_lcore_get(uint32_t id, uint32_t lcore)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 	uint32_t enabled;
-	int ret = service_update(&s->spec, lcore, 0, &enabled);
+	int ret = service_update(id, lcore, 0, &enabled);
 	if (ret == 0)
 		return enabled;
 	return ret;
Signature · 1 line
2.7.4
author
#68 lore raw

PATCH v2 5/6 service: optimize with c11 atomics

In reply to Phil Yang

The num_mapped_cores is used as a statistics. Use c11 atomics with RELAXED ordering for num_mapped_cores instead of rte_atomic ops which enforce unnessary barriers on aarch64.

Replace execute_lock operations to spinlock_try_lock to avoid duplicate code.

---
 lib/librte_eal/common/rte_service.c | 32 ++++++++++++++++++--------------
 lib/librte_eal/meson.build          |  4 ++++
 2 files changed, 22 insertions(+), 14 deletions(-)
lib/librte_eal/common/rte_service.c +18 -14
index 9c1a1d5..8cac265 100644
@@ -20,6 +20,7 @@
 #include <rte_atomic.h>
 #include <rte_memory.h>
 #include <rte_malloc.h>
+#include <rte_spinlock.h>
 
 #include "eal_private.h"
 
@@ -38,11 +39,11 @@ struct rte_service_spec_impl {
 	/* public part of the struct */
 	struct rte_service_spec spec;
 
-	/* atomic lock that when set indicates a service core is currently
+	/* spin lock that when set indicates a service core is currently
 	 * running this service callback. When not set, a core may take the
 	 * lock and then run the service callback.
 	 */
-	rte_atomic32_t execute_lock;
+	rte_spinlock_t execute_lock;
 
 	/* API set/get-able variables */
 	int8_t app_runstate;
@@ -54,7 +55,7 @@ struct rte_service_spec_impl {
 	 * It does not indicate the number of cores the service is running
 	 * on currently.
 	 */
-	rte_atomic32_t num_mapped_cores;
+	uint32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
 } __rte_cache_aligned;
@@ -332,7 +333,8 @@ rte_service_runstate_get(uint32_t id)
 	rte_smp_rmb();
 
 	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (rte_atomic32_read(&s->num_mapped_cores) > 0);
+	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+					    __ATOMIC_RELAXED) > 0);
 
 	return (s->app_runstate == RUNSTATE_RUNNING) &&
 		(s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -375,11 +377,11 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 	cs->service_active_on_lcore[i] = 1;
 
 	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
-		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
+		if (!rte_spinlock_trylock(&s->execute_lock))
 			return -EBUSY;
 
 		service_runner_do_callback(s, cs, i);
-		rte_atomic32_clear(&s->execute_lock);
+		rte_spinlock_unlock(&s->execute_lock);
 	} else
 		service_runner_do_callback(s, cs, i);
 
@@ -415,11 +417,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 	/* Increment num_mapped_cores to indicate that the service
 	 * is running on a core.
 	 */
-	rte_atomic32_inc(&s->num_mapped_cores);
+	__atomic_add_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELAXED);
 
 	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	rte_atomic32_dec(&s->num_mapped_cores);
+	__atomic_sub_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELAXED);
 
 	return ret;
 }
@@ -556,19 +558,19 @@ service_update(uint32_t sid, uint32_t lcore,
 
 		if (*set && !lcore_mapped) {
 			lcore_states[lcore].service_mask |= sid_mask;
-			rte_atomic32_inc(&rte_services[sid].num_mapped_cores);
+			__atomic_add_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELAXED);
 		}
 		if (!*set && lcore_mapped) {
 			lcore_states[lcore].service_mask &= ~(sid_mask);
-			rte_atomic32_dec(&rte_services[sid].num_mapped_cores);
+			__atomic_sub_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELAXED);
 		}
 	}
 
 	if (enabled)
 		*enabled = !!(lcore_states[lcore].service_mask & (sid_mask));
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -616,7 +618,8 @@ rte_service_lcore_reset_all(void)
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
-		rte_atomic32_set(&rte_services[i].num_mapped_cores, 0);
+		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
+				    __ATOMIC_RELAXED);
 
 	rte_smp_wmb();
 
@@ -699,7 +702,8 @@ rte_service_lcore_stop(uint32_t lcore)
 		int32_t enabled = service_mask & (UINT64_C(1) << i);
 		int32_t service_running = rte_service_runstate_get(i);
 		int32_t only_core = (1 ==
-			rte_atomic32_read(&rte_services[i].num_mapped_cores));
+			__atomic_load_n(&rte_services[i].num_mapped_cores,
+					__ATOMIC_RELAXED));
 
 		/* if the core is mapped, and the service is running, and this
 		 * is the only core that is mapped, the service would cease to
lib/librte_eal/meson.build +4 -0
index 0267c3b..c2d7a69 100644
@@ -21,3 +21,7 @@ endif
 if cc.has_header('getopt.h')
 	cflags += ['-DHAVE_GETOPT_H', '-DHAVE_GETOPT', '-DHAVE_GETOPT_LONG']
 endif
+# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
+if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
+    ext_deps += cc.find_library('atomic')
+endif
Signature · 1 line
2.7.4
author
#69 lore raw

PATCH v2 6/6 service: relax barriers with C11 atomics

In reply to Phil Yang

The runstate, comp_runstate and app_runstate are used as guard variables in the service core lib. To guarantee the inter-threads visibility of these guard variables, it uses rte_smp_r/wmb. This patch use c11 atomic built-ins to relax these barriers.

---
 lib/librte_eal/common/rte_service.c | 115 ++++++++++++++++++++++++++----------
 1 file changed, 84 insertions(+), 31 deletions(-)
lib/librte_eal/common/rte_service.c +84 -31
index 8cac265..dbb8211 100644
@@ -265,7 +265,6 @@ rte_service_component_register(const struct rte_service_spec *spec,
 	s->spec = *spec;
 	s->internal_flags |= SERVICE_F_REGISTERED | SERVICE_F_START_CHECK;
 
-	rte_smp_wmb();
 	rte_service_count++;
 
 	if (id_ptr)
@@ -282,7 +281,6 @@ rte_service_component_unregister(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	rte_service_count--;
-	rte_smp_wmb();
 
 	s->internal_flags &= ~(SERVICE_F_REGISTERED);
 
@@ -301,12 +299,17 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
+	/* comp_runstate act as the guard variable. Use store-release
+	 * memory order. This synchronizes with load-acquire in
+	 * service_run and service_runstate_get function.
+	 */
 	if (runstate)
-		s->comp_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->comp_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -316,12 +319,17 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
+	/* app_runstate act as the guard variable. Use store-release
+	 * memory order. This synchronizes with load-acquire in
+	 * service_run runstate_get function.
+	 */
 	if (runstate)
-		s->app_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->app_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -330,15 +338,24 @@ rte_service_runstate_get(uint32_t id)
 {
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
-	rte_smp_rmb();
 
-	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+	/* comp_runstate and app_runstate act as the guard variables.
+	 * Use load-acquire memory order. This synchronizes with
+	 * store-release in service state set functions.
+	 */
+	if (__atomic_load_n(&s->comp_runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING &&
+		 __atomic_load_n(&s->app_runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {
+		int check_disabled = !(s->internal_flags &
+					SERVICE_F_START_CHECK);
+		int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
 					    __ATOMIC_RELAXED) > 0);
 
-	return (s->app_runstate == RUNSTATE_RUNNING) &&
-		(s->comp_runstate == RUNSTATE_RUNNING) &&
-		(check_disabled | lcore_mapped);
+		return (check_disabled | lcore_mapped);
+	} else
+		return 0;
+
 }
 
 static inline void
@@ -367,9 +384,15 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 	if (!s)
 		return -EINVAL;
 
-	if (s->comp_runstate != RUNSTATE_RUNNING ||
-			s->app_runstate != RUNSTATE_RUNNING ||
-			!(service_mask & (UINT64_C(1) << i))) {
+	/* comp_runstate and app_runstate act as the guard variables.
+	 * Use load-acquire memory order. This synchronizes with
+	 * store-release in service state set functions.
+	 */
+	if (__atomic_load_n(&s->comp_runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_RUNNING ||
+		 __atomic_load_n(&s->app_runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_RUNNING ||
+		!(service_mask & (UINT64_C(1) << i))) {
 		cs->service_active_on_lcore[i] = 0;
 		return -ENOEXEC;
 	}
@@ -434,7 +457,12 @@ service_runner_func(void *arg)
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (cs->runstate == RUNSTATE_RUNNING) {
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	while (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
@@ -445,8 +473,6 @@ service_runner_func(void *arg)
 		}
 
 		cs->loops++;
-
-		rte_smp_rmb();
 	}
 
 	lcore_config[lcore].state = WAIT;
@@ -614,15 +640,18 @@ rte_service_lcore_reset_all(void)
 		if (lcore_states[i].is_service_core) {
 			lcore_states[i].service_mask = 0;
 			set_lcore_state(i, ROLE_RTE);
-			lcore_states[i].runstate = RUNSTATE_STOPPED;
+			/* runstate act as guard variable Use
+			 * store-release memory order here to synchronize
+			 * with load-acquire in runstate read functions.
+			 */
+			__atomic_store_n(&lcore_states[i].runstate,
+				RUNSTATE_STOPPED, __ATOMIC_RELEASE);
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
 		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
 				    __ATOMIC_RELAXED);
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -638,9 +667,11 @@ rte_service_lcore_add(uint32_t lcore)
 
 	/* ensure that after adding a core the mask and state are defaults */
 	lcore_states[lcore].service_mask = 0;
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
-
-	rte_smp_wmb();
+	/* Use store-release memory order here to synchronize with
+	 * load-acquire in runstate read functions.
+	 */
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+		__ATOMIC_RELEASE);
 
 	return rte_eal_wait_lcore(lcore);
 }
@@ -655,7 +686,12 @@ rte_service_lcore_del(uint32_t lcore)
 	if (!cs->is_service_core)
 		return -EINVAL;
 
-	if (cs->runstate != RUNSTATE_STOPPED)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_STOPPED)
 		return -EBUSY;
 
 	set_lcore_state(lcore, ROLE_RTE);
@@ -674,13 +710,21 @@ rte_service_lcore_start(uint32_t lcore)
 	if (!cs->is_service_core)
 		return -EINVAL;
 
-	if (cs->runstate == RUNSTATE_RUNNING)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING)
 		return -EALREADY;
 
 	/* set core to run state first, and then launch otherwise it will
 	 * return immediately as runstate keeps it in the service poll loop
 	 */
-	cs->runstate = RUNSTATE_RUNNING;
+	/* Use load-acquire memory order here to synchronize with
+	 * store-release in runstate update functions.
+	 */
+	__atomic_store_n(&cs->runstate, RUNSTATE_RUNNING, __ATOMIC_RELEASE);
 
 	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
@@ -693,7 +737,12 @@ rte_service_lcore_stop(uint32_t lcore)
 	if (lcore >= RTE_MAX_LCORE)
 		return -EINVAL;
 
-	if (lcore_states[lcore].runstate == RUNSTATE_STOPPED)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&lcore_states[lcore].runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_STOPPED)
 		return -EALREADY;
 
 	uint32_t i;
@@ -713,7 +762,11 @@ rte_service_lcore_stop(uint32_t lcore)
 			return -EBUSY;
 	}
 
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
+	/* Use store-release memory order here to synchronize with
+	 * load-acquire in runstate read functions.
+	 */
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+		__ATOMIC_RELEASE);
 
 	return 0;
 }
Signature · 1 line
2.7.4
#70 lore raw

v3 cover use c11 atomics for service core lib

In reply to Phil Yang

The rte_atomic ops and rte_smp barriers enforce DMB barriers on aarch64. Using c11 atomics with explicit memory ordering instead of the rte_atomic ops and rte_smp barriers for inter-threads synchronization can uplift the performance on aarch64 and no performance loss on x86.

This patchset contains:
1) fix race condition for MT unsafe service.
2) clean up redundant code.
3) use c11 atomics for service core lib to avoid unnecessary barriers.

v2:
Still waiting on Harry for the final solution on the MT unsafe race condition issue. But I have incorporated the comments so far.
1. add 'Fixes' tag for bug-fix patches.
2. remove 'Fixes' tag for code cleanup patches.
3. remove unused parameter for service_dump_one function.
4. replace the execute_lock atomic CAS operation to spinlock_try_lock.
5. use c11 atomics with RELAXED memory ordering for num_mapped_cores.
6. relax barriers for guard variables runstate, comp_runstate and app_runstate with c11 one-way barriers.

v3:
Sending this version since Phil is on holiday.
1. Updated the API documentation to indicate how the locking can be avoided.

Honnappa Nagarahalli (2):
service: fix race condition for MT unsafe service
service: identify service running on another core correctly

Phil Yang (4):
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 atomics
service: relax barriers with C11 atomics

 lib/librte_eal/common/rte_service.c           | 234 ++++++++++--------
 lib/librte_eal/include/rte_service.h          |   8 +-
 .../include/rte_service_component.h           |   6 +-
 lib/librte_eal/meson.build                    |   4 +
 4 files changed, 141 insertions(+), 111 deletions(-)
Signature · 1 line
2.17.1
#71 lore raw

PATCH v3 1/6 service: fix race condition for MT unsafe service

The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

If the service is known to be mmapped on a single lcore, setting the service capability to MT safe will avoid taking the lock and improve the performance.

Fixes e9139a32f6e8 ("service: add function to run on app lcore")
---
 lib/librte_eal/common/rte_service.c            | 11 +++++------
 lib/librte_eal/include/rte_service.h           |  8 ++++++--
 lib/librte_eal/include/rte_service_component.h |  6 +++++-
 3 files changed, 16 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
index 70d17a5d7..b8c465eb9 100644
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {
 	uint8_t internal_flags;
 
 	/* per service statistics */
+	/* Indicates how many cores the service is mapped to run on.
+	 * It does not indicate the number of cores the service is running
+	 * on currently.
+	 */
 	rte_atomic32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	/* check do we need cmpset, if MT safe or <= 1 core
-	 * mapped, atomic ops are not required.
-	 */
-	const int use_atomics = (service_mt_safe(s) == 0) &&
-				(rte_atomic32_read(&s->num_mapped_cores) > 1);
-	if (use_atomics) {
+	if (service_mt_safe(s) == 0) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
lib/librte_eal/include/rte_service.h +6 -2
index d8701dd4c..3a1c735c5 100644
@@ -104,12 +104,16 @@ int32_t rte_service_probe_capability(uint32_t id, uint32_t capability);
  * Each core can be added or removed from running a specific service. This
  * function enables or disables *lcore* to run *service_id*.
  *
- * If multiple cores are enabled on a service, an atomic is used to ensure that
- * only one cores runs the service at a time. The exception to this is when
+ * If multiple cores are enabled on a service, a lock is used to ensure that
+ * only one core runs the service at a time. The exception to this is when
  * a service indicates that it is multi-thread safe by setting the capability
  * called RTE_SERVICE_CAP_MT_SAFE. With the multi-thread safe capability set,
  * the service function can be run on multiple threads at the same time.
  *
+ * If the service is known to be mapped to a single lcore, setting the
+ * capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve
+ * better performance by avoiding the use of lock.
+ *
  * @param service_id the service to apply the lcore to
  * @param lcore The lcore that will be mapped to service
  * @param enable Zero to unmap or disable the core, non-zero to enable
lib/librte_eal/include/rte_service_component.h +5 -1
index 16eab79ee..b75aba11b 100644
@@ -43,7 +43,7 @@ struct rte_service_spec {
 /**
  * Register a new service.
  *
- * A service represents a component that the requires CPU time periodically to
+ * A service represents a component that requires CPU time periodically to
  * achieve its purpose.
  *
  * For example the eventdev SW PMD requires CPU cycles to perform its
@@ -56,6 +56,10 @@ struct rte_service_spec {
  * *rte_service_component_runstate_set*, which indicates that the service
  * component is ready to be executed.
  *
+ * If the service is known to be mapped to a single lcore, setting the
+ * capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve
+ * better performance.
+ *
  * @param spec The specification of the service to register
  * @param[out] service_id A pointer to a uint32_t, which will be filled in
  *             during registration of the service. It is set to the integers
Signature · 1 line
2.17.1
Acked-by on v3 1/6
#72 lore raw

Re: [dpdk-dev] [PATCH v3 1/6] service: fix race condition for MT unsafe service

Quoted 12 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, May 2, 2020 1:03 AM
To: [email protected]; [email protected]; Van Haaren, Harry <[email protected]>
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 1/6] service: fix race condition for MT unsafe service

Quoted 6 lines

The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

If the service is known to be mmapped on a single lcore,

mmapped is a typo? Just mapped.

Quoted 8 lines

setting the service capability to MT safe will avoid taking the lock and improve the performance.

Fixes e9139a32f6e8 ("service: add function to run on app lcore")

<snip diff>

#73 lore raw

PATCH v3 2/6 service: identify service running on another core correctly

In reply to Honnappa Nagarahalli

The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases.

Fixes 8d39d3e237c2 ("service: fix race in service on app lcore function")
---
 lib/librte_eal/common/rte_service.c | 26 ++++++++------------------
 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
index b8c465eb9..c89472b83 100644
@@ -360,7 +360,7 @@ rte_service_runner_do_callback(struct rte_service_spec_impl *s,
 /* Expects the service 's' is valid. */
 static int32_t
 service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
-	    struct rte_service_spec_impl *s)
+	    struct rte_service_spec_impl *s, uint32_t serialize_mt_unsafe)
 {
 	if (!s)
 		return -EINVAL;
@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	if (service_mt_safe(s) == 0) {
+	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
-	/* Atomically add this core to the mapped cores first, then examine if
-	 * we can run the service. This avoids a race condition between
-	 * checking the value, and atomically adding to the mapped count.
+	/* Increment num_mapped_cores to indicate that the service
+	 * is running on a core.
 	 */
-	if (serialize_mt_unsafe)
-		rte_atomic32_inc(&s->num_mapped_cores);
+	rte_atomic32_inc(&s->num_mapped_cores);
 
-	if (service_mt_safe(s) == 0 &&
-			rte_atomic32_read(&s->num_mapped_cores) > 1) {
-		if (serialize_mt_unsafe)
-			rte_atomic32_dec(&s->num_mapped_cores);
-		return -EBUSY;
-	}
-
-	int ret = service_run(id, cs, UINT64_MAX, s);
+	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	if (serialize_mt_unsafe)
-		rte_atomic32_dec(&s->num_mapped_cores);
+	rte_atomic32_dec(&s->num_mapped_cores);
 
 	return ret;
 }
@@ -449,7 +439,7 @@ rte_service_runner_func(void *arg)
 			if (!service_valid(i))
 				continue;
 			/* return value ignored as no change to code flow */
-			service_run(i, cs, service_mask, service_get(i));
+			service_run(i, cs, service_mask, service_get(i), 1);
 		}
 
 		cs->loops++;
Signature · 1 line
2.17.1
Acked-by on v3 2/6
#74 lore raw

Re: [dpdk-dev] [PATCH v3 2/6] service: identify service running on another core correctly

Quoted 13 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, May 2, 2020 1:03 AM
To: [email protected]; [email protected]; Van Haaren, Harry <[email protected]>
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 2/6] service: identify service running on another core correctly

Quoted 8 lines

The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases.

Fixes 8d39d3e237c2 ("service: fix race in service on app lcore function")

Add "fix" to the title, suggestion:
service: fix identification of service running on other lcore

Quoted 2 lines

I believe there may be some optimizations we can apply after this patchset as the "num_mapped_cores" variable is no longer used in a significant way for the atomic selection, however lets leave that optimization outside of 20.05 scope.

With title (see above) & comment (see below) updated.

Quoted 1 line

---

<snip some diff>

Quoted 4 lines
lib/librte_eal/common/rte_service.c +0 -0
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id,
 uint32_t serialize_mt_unsafe)
 
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);

lib/librte_eal/common/rte_service.c rte_service_run_iter_on_app_lcore(uint32_t id, · line 419

Quoted 10 lines
lib/librte_eal/common/rte_service.c +3 -5
 
-	/* Atomically add this core to the mapped cores first, then examine if
-	 * we can run the service. This avoids a race condition between
-	 * checking the value, and atomically adding to the mapped count.
+	/* Increment num_mapped_cores to indicate that the service
+	 * is running on a core.
 	 */
-	if (serialize_mt_unsafe)
-		rte_atomic32_inc(&s->num_mapped_cores);
+	rte_atomic32_inc(&s->num_mapped_cores);

The comment for the added lines here are a little confusing to me, the "num_mapped_cores" does not indicate that the service "is running on a core", it indicates the number of mapped lcores to that service. Suggestion below?

/* Increment num_mapped_cores to reflect that this core is
* now mapped capable of running the service.
*/

lib/librte_eal/common/rte_service.c rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) · line 420

Quoted 9 lines
lib/librte_eal/common/rte_service.c +1 -8
-	if (service_mt_safe(s) == 0 &&
-			rte_atomic32_read(&s->num_mapped_cores) > 1) {
-		if (serialize_mt_unsafe)
-			rte_atomic32_dec(&s->num_mapped_cores);
-		return -EBUSY;
-	}
-
-	int ret = service_run(id, cs, UINT64_MAX, s);
+	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);

<snip rest of diff>

#75 lore raw

PATCH v3 3/6 service: remove rte prefix from static functions

In reply to Honnappa Nagarahalli

From: Phil Yang <[email protected]>

clean up rte prefix from static functions. remove unused parameter for service_dump_one function.

---
 lib/librte_eal/common/rte_service.c | 34 ++++++++++-------------------
 1 file changed, 11 insertions(+), 23 deletions(-)
lib/librte_eal/common/rte_service.c +11 -23
index c89472b83..ed2070267 100644
@@ -340,7 +340,7 @@ rte_service_runstate_get(uint32_t id)
 }
 
 static inline void
-rte_service_runner_do_callback(struct rte_service_spec_impl *s,
+service_runner_do_callback(struct rte_service_spec_impl *s,
 			       struct core_state *cs, uint32_t service_idx)
 {
 	void *userdata = s->spec.callback_userdata;
@@ -378,10 +378,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 		rte_atomic32_clear(&s->execute_lock);
 	} else
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 
 	return 0;
 }
@@ -425,14 +425,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 }
 
 static int32_t
-rte_service_runner_func(void *arg)
+service_runner_func(void *arg)
 {
 	RTE_SET_USED(arg);
 	uint32_t i;
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (lcore_states[lcore].runstate == RUNSTATE_RUNNING) {
+	while (cs->runstate == RUNSTATE_RUNNING) {
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
@@ -693,9 +693,9 @@ rte_service_lcore_start(uint32_t lcore)
 	/* set core to run state first, and then launch otherwise it will
 	 * return immediately as runstate keeps it in the service poll loop
 	 */
-	lcore_states[lcore].runstate = RUNSTATE_RUNNING;
+	cs->runstate = RUNSTATE_RUNNING;
 
-	int ret = rte_eal_remote_launch(rte_service_runner_func, 0, lcore);
+	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
 	return ret;
 }
@@ -774,13 +774,9 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id,
 }
 
 static void
-rte_service_dump_one(FILE *f, struct rte_service_spec_impl *s,
-		     uint64_t all_cycles, uint32_t reset)
+service_dump_one(FILE *f, struct rte_service_spec_impl *s, uint32_t reset)
 {
 	/* avoid divide by zero */
-	if (all_cycles == 0)
-		all_cycles = 1;
-
 	int calls = 1;
 	if (s->calls != 0)
 		calls = s->calls;
@@ -807,7 +803,7 @@ rte_service_attr_reset_all(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	int reset = 1;
-	rte_service_dump_one(NULL, s, 0, reset);
+	service_dump_one(NULL, s, reset);
 	return 0;
 }
 
@@ -851,21 +847,13 @@ rte_service_dump(FILE *f, uint32_t id)
 	uint32_t i;
 	int print_one = (id != UINT32_MAX);
 
-	uint64_t total_cycles = 0;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if (!service_valid(i))
-			continue;
-		total_cycles += rte_services[i].cycles_spent;
-	}
-
 	/* print only the specified service */
 	if (print_one) {
 		struct rte_service_spec_impl *s;
 		SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 		fprintf(f, "Service %s Summary\n", s->spec.name);
 		uint32_t reset = 0;
-		rte_service_dump_one(f, s, total_cycles, reset);
+		service_dump_one(f, s, reset);
 		return 0;
 	}
 
@@ -875,7 +863,7 @@ rte_service_dump(FILE *f, uint32_t id)
 		if (!service_valid(i))
 			continue;
 		uint32_t reset = 0;
-		rte_service_dump_one(f, &rte_services[i], total_cycles, reset);
+		service_dump_one(f, &rte_services[i], reset);
 	}
 
 	fprintf(f, "Service Cores Summary\n");
Signature · 1 line
2.17.1
Acked-by on v3 3/6
#76 lore raw

Re: [dpdk-dev] [PATCH v3 3/6] service: remove rte prefix from static functions

Quoted 12 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, May 2, 2020 1:03 AM
To: [email protected]; [email protected]; Van Haaren, Harry <[email protected]>
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v3 3/6] service: remove rte prefix from static functions

Quoted 7 lines

From: Phil Yang <[email protected]>

clean up rte prefix from static functions. remove unused parameter for service_dump_one function.

#77 lore raw

PATCH v3 4/6 service: remove redundant code

In reply to Honnappa Nagarahalli

From: Phil Yang <[email protected]>

The service id validation is duplicated, remove the redundant code in the calling functions.

---
 lib/librte_eal/common/rte_service.c | 28 ++++++----------------------
 1 file changed, 6 insertions(+), 22 deletions(-)
lib/librte_eal/common/rte_service.c +6 -22
index ed2070267..9c1a1d5cd 100644
@@ -541,24 +541,12 @@ rte_service_start_with_defaults(void)
 }
 
 static int32_t
-service_update(struct rte_service_spec *service, uint32_t lcore,
+service_update(uint32_t sid, uint32_t lcore,
 		uint32_t *set, uint32_t *enabled)
 {
-	uint32_t i;
-	int32_t sid = -1;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if ((struct rte_service_spec *)&rte_services[i] == service &&
-				service_valid(i)) {
-			sid = i;
-			break;
-		}
-	}
-
-	if (sid == -1 || lcore >= RTE_MAX_LCORE)
-		return -EINVAL;
-
-	if (!lcore_states[lcore].is_service_core)
+	/* validate ID, or return error value */
+	if (sid >= RTE_SERVICE_NUM_MAX || !service_valid(sid) ||
+	    lcore >= RTE_MAX_LCORE || !lcore_states[lcore].is_service_core)
 		return -EINVAL;
 
 	uint64_t sid_mask = UINT64_C(1) << sid;
@@ -587,19 +575,15 @@ service_update(struct rte_service_spec *service, uint32_t lcore,
 int32_t
 rte_service_map_lcore_set(uint32_t id, uint32_t lcore, uint32_t enabled)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 	uint32_t on = enabled > 0;
-	return service_update(&s->spec, lcore, &on, 0);
+	return service_update(id, lcore, &on, 0);
 }
 
 int32_t
 rte_service_map_lcore_get(uint32_t id, uint32_t lcore)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 	uint32_t enabled;
-	int ret = service_update(&s->spec, lcore, 0, &enabled);
+	int ret = service_update(id, lcore, 0, &enabled);
 	if (ret == 0)
 		return enabled;
 	return ret;
Signature · 1 line
2.17.1
Acked-by on v3 4/6
#78 lore raw

Re: [dpdk-dev] [PATCH v3 4/6] service: remove redundant code

Quoted 12 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, May 2, 2020 1:03 AM
To: [email protected]; [email protected]; Van Haaren, Harry <[email protected]>
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v3 4/6] service: remove redundant code

Quoted 7 lines

From: Phil Yang <[email protected]>

The service id validation is duplicated, remove the redundant code in the calling functions.

<snip diff>

#79 lore raw

PATCH v3 5/6 service: optimize with c11 atomics

In reply to Honnappa Nagarahalli

From: Phil Yang <[email protected]>

The num_mapped_cores is used as a statistics. Use c11 atomics with RELAXED ordering for num_mapped_cores instead of rte_atomic ops which enforce unnessary barriers on aarch64.

Replace execute_lock operations to spinlock_try_lock to avoid duplicate code.

---
 lib/librte_eal/common/rte_service.c | 32 ++++++++++++++++-------------
 lib/librte_eal/meson.build          |  4 ++++
 2 files changed, 22 insertions(+), 14 deletions(-)
lib/librte_eal/common/rte_service.c +18 -14
index 9c1a1d5cd..8cac265c9 100644
@@ -20,6 +20,7 @@
 #include <rte_atomic.h>
 #include <rte_memory.h>
 #include <rte_malloc.h>
+#include <rte_spinlock.h>
 
 #include "eal_private.h"
 
@@ -38,11 +39,11 @@ struct rte_service_spec_impl {
 	/* public part of the struct */
 	struct rte_service_spec spec;
 
-	/* atomic lock that when set indicates a service core is currently
+	/* spin lock that when set indicates a service core is currently
 	 * running this service callback. When not set, a core may take the
 	 * lock and then run the service callback.
 	 */
-	rte_atomic32_t execute_lock;
+	rte_spinlock_t execute_lock;
 
 	/* API set/get-able variables */
 	int8_t app_runstate;
@@ -54,7 +55,7 @@ struct rte_service_spec_impl {
 	 * It does not indicate the number of cores the service is running
 	 * on currently.
 	 */
-	rte_atomic32_t num_mapped_cores;
+	uint32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
 } __rte_cache_aligned;
@@ -332,7 +333,8 @@ rte_service_runstate_get(uint32_t id)
 	rte_smp_rmb();
 
 	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (rte_atomic32_read(&s->num_mapped_cores) > 0);
+	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+					    __ATOMIC_RELAXED) > 0);
 
 	return (s->app_runstate == RUNSTATE_RUNNING) &&
 		(s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -375,11 +377,11 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 	cs->service_active_on_lcore[i] = 1;
 
 	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
-		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
+		if (!rte_spinlock_trylock(&s->execute_lock))
 			return -EBUSY;
 
 		service_runner_do_callback(s, cs, i);
-		rte_atomic32_clear(&s->execute_lock);
+		rte_spinlock_unlock(&s->execute_lock);
 	} else
 		service_runner_do_callback(s, cs, i);
 
@@ -415,11 +417,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 	/* Increment num_mapped_cores to indicate that the service
 	 * is running on a core.
 	 */
-	rte_atomic32_inc(&s->num_mapped_cores);
+	__atomic_add_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELAXED);
 
 	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	rte_atomic32_dec(&s->num_mapped_cores);
+	__atomic_sub_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELAXED);
 
 	return ret;
 }
@@ -556,19 +558,19 @@ service_update(uint32_t sid, uint32_t lcore,
 
 		if (*set && !lcore_mapped) {
 			lcore_states[lcore].service_mask |= sid_mask;
-			rte_atomic32_inc(&rte_services[sid].num_mapped_cores);
+			__atomic_add_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELAXED);
 		}
 		if (!*set && lcore_mapped) {
 			lcore_states[lcore].service_mask &= ~(sid_mask);
-			rte_atomic32_dec(&rte_services[sid].num_mapped_cores);
+			__atomic_sub_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELAXED);
 		}
 	}
 
 	if (enabled)
 		*enabled = !!(lcore_states[lcore].service_mask & (sid_mask));
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -616,7 +618,8 @@ rte_service_lcore_reset_all(void)
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
-		rte_atomic32_set(&rte_services[i].num_mapped_cores, 0);
+		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
+				    __ATOMIC_RELAXED);
 
 	rte_smp_wmb();
 
@@ -699,7 +702,8 @@ rte_service_lcore_stop(uint32_t lcore)
 		int32_t enabled = service_mask & (UINT64_C(1) << i);
 		int32_t service_running = rte_service_runstate_get(i);
 		int32_t only_core = (1 ==
-			rte_atomic32_read(&rte_services[i].num_mapped_cores));
+			__atomic_load_n(&rte_services[i].num_mapped_cores,
+					__ATOMIC_RELAXED));
 
 		/* if the core is mapped, and the service is running, and this
 		 * is the only core that is mapped, the service would cease to
lib/librte_eal/meson.build +4 -0
index 0267c3b9d..c2d7a6954 100644
@@ -21,3 +21,7 @@ endif
 if cc.has_header('getopt.h')
 	cflags += ['-DHAVE_GETOPT_H', '-DHAVE_GETOPT', '-DHAVE_GETOPT_LONG']
 endif
+# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
+if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
+    ext_deps += cc.find_library('atomic')
+endif
Signature · 1 line
2.17.1
Acked-by on v3 5/6
#80 lore raw

Re: [dpdk-dev] [PATCH v3 5/6] service: optimize with c11 atomics

Quoted 18 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, May 2, 2020 1:03 AM
To: [email protected]; [email protected]; Van Haaren, Harry <[email protected]>
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v3 5/6] service: optimize with c11 atomics

From: Phil Yang <[email protected]>

The num_mapped_cores is used as a statistics. Use c11 atomics with RELAXED ordering for num_mapped_cores instead of rte_atomic ops which enforce unnessary barriers on aarch64.

Quoted 5 lines

Replace execute_lock operations to spinlock_try_lock to avoid duplicate code.

<snip diff>

#81 lore raw

PATCH v3 6/6 service: relax barriers with C11 atomics

In reply to Honnappa Nagarahalli

From: Phil Yang <[email protected]>

The runstate, comp_runstate and app_runstate are used as guard variables in the service core lib. To guarantee the inter-threads visibility of these guard variables, it uses rte_smp_r/wmb. This patch use c11 atomic built-ins to relax these barriers.

---
 lib/librte_eal/common/rte_service.c | 115 ++++++++++++++++++++--------
 1 file changed, 84 insertions(+), 31 deletions(-)
lib/librte_eal/common/rte_service.c +84 -31
index 8cac265c9..dbb821139 100644
@@ -265,7 +265,6 @@ rte_service_component_register(const struct rte_service_spec *spec,
 	s->spec = *spec;
 	s->internal_flags |= SERVICE_F_REGISTERED | SERVICE_F_START_CHECK;
 
-	rte_smp_wmb();
 	rte_service_count++;
 
 	if (id_ptr)
@@ -282,7 +281,6 @@ rte_service_component_unregister(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	rte_service_count--;
-	rte_smp_wmb();
 
 	s->internal_flags &= ~(SERVICE_F_REGISTERED);
 
@@ -301,12 +299,17 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
+	/* comp_runstate act as the guard variable. Use store-release
+	 * memory order. This synchronizes with load-acquire in
+	 * service_run and service_runstate_get function.
+	 */
 	if (runstate)
-		s->comp_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->comp_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -316,12 +319,17 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
+	/* app_runstate act as the guard variable. Use store-release
+	 * memory order. This synchronizes with load-acquire in
+	 * service_run runstate_get function.
+	 */
 	if (runstate)
-		s->app_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->app_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -330,15 +338,24 @@ rte_service_runstate_get(uint32_t id)
 {
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
-	rte_smp_rmb();
 
-	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+	/* comp_runstate and app_runstate act as the guard variables.
+	 * Use load-acquire memory order. This synchronizes with
+	 * store-release in service state set functions.
+	 */
+	if (__atomic_load_n(&s->comp_runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING &&
+		 __atomic_load_n(&s->app_runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {
+		int check_disabled = !(s->internal_flags &
+					SERVICE_F_START_CHECK);
+		int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
 					    __ATOMIC_RELAXED) > 0);
 
-	return (s->app_runstate == RUNSTATE_RUNNING) &&
-		(s->comp_runstate == RUNSTATE_RUNNING) &&
-		(check_disabled | lcore_mapped);
+		return (check_disabled | lcore_mapped);
+	} else
+		return 0;
+
 }
 
 static inline void
@@ -367,9 +384,15 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 	if (!s)
 		return -EINVAL;
 
-	if (s->comp_runstate != RUNSTATE_RUNNING ||
-			s->app_runstate != RUNSTATE_RUNNING ||
-			!(service_mask & (UINT64_C(1) << i))) {
+	/* comp_runstate and app_runstate act as the guard variables.
+	 * Use load-acquire memory order. This synchronizes with
+	 * store-release in service state set functions.
+	 */
+	if (__atomic_load_n(&s->comp_runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_RUNNING ||
+		 __atomic_load_n(&s->app_runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_RUNNING ||
+		!(service_mask & (UINT64_C(1) << i))) {
 		cs->service_active_on_lcore[i] = 0;
 		return -ENOEXEC;
 	}
@@ -434,7 +457,12 @@ service_runner_func(void *arg)
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (cs->runstate == RUNSTATE_RUNNING) {
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	while (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
@@ -445,8 +473,6 @@ service_runner_func(void *arg)
 		}
 
 		cs->loops++;
-
-		rte_smp_rmb();
 	}
 
 	lcore_config[lcore].state = WAIT;
@@ -614,15 +640,18 @@ rte_service_lcore_reset_all(void)
 		if (lcore_states[i].is_service_core) {
 			lcore_states[i].service_mask = 0;
 			set_lcore_state(i, ROLE_RTE);
-			lcore_states[i].runstate = RUNSTATE_STOPPED;
+			/* runstate act as guard variable Use
+			 * store-release memory order here to synchronize
+			 * with load-acquire in runstate read functions.
+			 */
+			__atomic_store_n(&lcore_states[i].runstate,
+				RUNSTATE_STOPPED, __ATOMIC_RELEASE);
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
 		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
 				    __ATOMIC_RELAXED);
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -638,9 +667,11 @@ rte_service_lcore_add(uint32_t lcore)
 
 	/* ensure that after adding a core the mask and state are defaults */
 	lcore_states[lcore].service_mask = 0;
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
-
-	rte_smp_wmb();
+	/* Use store-release memory order here to synchronize with
+	 * load-acquire in runstate read functions.
+	 */
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+		__ATOMIC_RELEASE);
 
 	return rte_eal_wait_lcore(lcore);
 }
@@ -655,7 +686,12 @@ rte_service_lcore_del(uint32_t lcore)
 	if (!cs->is_service_core)
 		return -EINVAL;
 
-	if (cs->runstate != RUNSTATE_STOPPED)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_STOPPED)
 		return -EBUSY;
 
 	set_lcore_state(lcore, ROLE_RTE);
@@ -674,13 +710,21 @@ rte_service_lcore_start(uint32_t lcore)
 	if (!cs->is_service_core)
 		return -EINVAL;
 
-	if (cs->runstate == RUNSTATE_RUNNING)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING)
 		return -EALREADY;
 
 	/* set core to run state first, and then launch otherwise it will
 	 * return immediately as runstate keeps it in the service poll loop
 	 */
-	cs->runstate = RUNSTATE_RUNNING;
+	/* Use load-acquire memory order here to synchronize with
+	 * store-release in runstate update functions.
+	 */
+	__atomic_store_n(&cs->runstate, RUNSTATE_RUNNING, __ATOMIC_RELEASE);
 
 	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
@@ -693,7 +737,12 @@ rte_service_lcore_stop(uint32_t lcore)
 	if (lcore >= RTE_MAX_LCORE)
 		return -EINVAL;
 
-	if (lcore_states[lcore].runstate == RUNSTATE_STOPPED)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&lcore_states[lcore].runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_STOPPED)
 		return -EALREADY;
 
 	uint32_t i;
@@ -713,7 +762,11 @@ rte_service_lcore_stop(uint32_t lcore)
 			return -EBUSY;
 	}
 
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
+	/* Use store-release memory order here to synchronize with
+	 * load-acquire in runstate read functions.
+	 */
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+		__ATOMIC_RELEASE);
 
 	return 0;
 }
Signature · 1 line
2.17.1
Acked-by on v3 6/6
#82 lore raw

Re: [dpdk-dev] [PATCH v3 6/6] service: relax barriers with C11 atomics

Quoted 14 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, May 2, 2020 1:03 AM
To: [email protected]; [email protected]; Van Haaren, Harry <[email protected]>
Cc: [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v3 6/6] service: relax barriers with C11 atomics

From: Phil Yang <[email protected]>

Quoted 7 lines

The runstate, comp_runstate and app_runstate are used as guard variables in the service core lib. To guarantee the inter-threads visibility of these guard variables, it uses rte_smp_r/wmb. This patch use c11 atomic built-ins to relax these barriers.

<snip diff>

#83 lore raw

v4 cover use c11 atomics for service core lib

In reply to Phil Yang

The rte_atomic ops and rte_smp barriers enforce DMB barriers on aarch64. Using c11 atomics with explicit memory ordering instead of the rte_atomic ops and rte_smp barriers for inter-threads synchronization can uplift the performance on aarch64 and no performance loss on x86.

This patchset contains:
1) fix race condition for MT unsafe service.
2) clean up redundant code.
3) use c11 atomics for service core lib to avoid unnecessary barriers.

v2:
Still waiting on Harry for the final solution on the MT unsafe race condition issue. But I have incorporated the comments so far.
1. add 'Fixes' tag for bug-fix patches.
2. remove 'Fixes' tag for code cleanup patches.
3. remove unused parameter for service_dump_one function.
4. replace the execute_lock atomic CAS operation to spinlock_try_lock.
5. use c11 atomics with RELAXED memory ordering for num_mapped_cores.
6. relax barriers for guard variables runstate, comp_runstate and app_runstate with c11 one-way barriers.

v3:
Sending this version since Phil is on holiday.
1. Updated the API documentation to indicate how the locking can be avoided.

v4:
1. Fix the nits in 2/6 commit message and comments in code.

Honnappa Nagarahalli (2):
service: fix race condition for MT unsafe service
service: fix identification of service running on other lcore

Phil Yang (4):
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 atomics
service: relax barriers with C11 atomics

 lib/librte_eal/common/rte_service.c           | 234 ++++++++++--------
 lib/librte_eal/include/rte_service.h          |   8 +-
 .../include/rte_service_component.h           |   6 +-
 lib/librte_eal/meson.build                    |   4 +
 4 files changed, 141 insertions(+), 111 deletions(-)
Signature · 1 line
2.17.1
#84 lore raw

PATCH v4 1/6 service: fix race condition for MT unsafe service

The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

If the service is known to be mmapped on a single lcore, setting the service capability to MT safe will avoid taking the lock and improve the performance.

Fixes e9139a32f6e8 ("service: add function to run on app lcore")
---
 lib/librte_eal/common/rte_service.c            | 11 +++++------
 lib/librte_eal/include/rte_service.h           |  8 ++++++--
 lib/librte_eal/include/rte_service_component.h |  6 +++++-
 3 files changed, 16 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
index 70d17a5d7..b8c465eb9 100644
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {
 	uint8_t internal_flags;
 
 	/* per service statistics */
+	/* Indicates how many cores the service is mapped to run on.
+	 * It does not indicate the number of cores the service is running
+	 * on currently.
+	 */
 	rte_atomic32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	/* check do we need cmpset, if MT safe or <= 1 core
-	 * mapped, atomic ops are not required.
-	 */
-	const int use_atomics = (service_mt_safe(s) == 0) &&
-				(rte_atomic32_read(&s->num_mapped_cores) > 1);
-	if (use_atomics) {
+	if (service_mt_safe(s) == 0) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
lib/librte_eal/include/rte_service.h +6 -2
index d8701dd4c..3a1c735c5 100644
@@ -104,12 +104,16 @@ int32_t rte_service_probe_capability(uint32_t id, uint32_t capability);
  * Each core can be added or removed from running a specific service. This
  * function enables or disables *lcore* to run *service_id*.
  *
- * If multiple cores are enabled on a service, an atomic is used to ensure that
- * only one cores runs the service at a time. The exception to this is when
+ * If multiple cores are enabled on a service, a lock is used to ensure that
+ * only one core runs the service at a time. The exception to this is when
  * a service indicates that it is multi-thread safe by setting the capability
  * called RTE_SERVICE_CAP_MT_SAFE. With the multi-thread safe capability set,
  * the service function can be run on multiple threads at the same time.
  *
+ * If the service is known to be mapped to a single lcore, setting the
+ * capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve
+ * better performance by avoiding the use of lock.
+ *
  * @param service_id the service to apply the lcore to
  * @param lcore The lcore that will be mapped to service
  * @param enable Zero to unmap or disable the core, non-zero to enable
lib/librte_eal/include/rte_service_component.h +5 -1
index 16eab79ee..b75aba11b 100644
@@ -43,7 +43,7 @@ struct rte_service_spec {
 /**
  * Register a new service.
  *
- * A service represents a component that the requires CPU time periodically to
+ * A service represents a component that requires CPU time periodically to
  * achieve its purpose.
  *
  * For example the eventdev SW PMD requires CPU cycles to perform its
@@ -56,6 +56,10 @@ struct rte_service_spec {
  * *rte_service_component_runstate_set*, which indicates that the service
  * component is ready to be executed.
  *
+ * If the service is known to be mapped to a single lcore, setting the
+ * capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve
+ * better performance.
+ *
  * @param spec The specification of the service to register
  * @param[out] service_id A pointer to a uint32_t, which will be filled in
  *             during registration of the service. It is set to the integers
Signature · 1 line
2.17.1
#85 lore raw

PATCH v4 2/6 service: fix identification of service running on other lcore

In reply to Honnappa Nagarahalli

The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduce the number of instructions for all cases.

Fixes 8d39d3e237c2 ("service: fix race in service on app lcore function")
---
 lib/librte_eal/common/rte_service.c | 26 ++++++++------------------
 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
index b8c465eb9..c283408cf 100644
@@ -360,7 +360,7 @@ rte_service_runner_do_callback(struct rte_service_spec_impl *s,
 /* Expects the service 's' is valid. */
 static int32_t
 service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
-	    struct rte_service_spec_impl *s)
+	    struct rte_service_spec_impl *s, uint32_t serialize_mt_unsafe)
 {
 	if (!s)
 		return -EINVAL;
@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	if (service_mt_safe(s) == 0) {
+	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
-	/* Atomically add this core to the mapped cores first, then examine if
-	 * we can run the service. This avoids a race condition between
-	 * checking the value, and atomically adding to the mapped count.
+	/* Increment num_mapped_cores to reflect that this core is
+	 * now mapped capable of running the service.
 	 */
-	if (serialize_mt_unsafe)
-		rte_atomic32_inc(&s->num_mapped_cores);
+	rte_atomic32_inc(&s->num_mapped_cores);
 
-	if (service_mt_safe(s) == 0 &&
-			rte_atomic32_read(&s->num_mapped_cores) > 1) {
-		if (serialize_mt_unsafe)
-			rte_atomic32_dec(&s->num_mapped_cores);
-		return -EBUSY;
-	}
-
-	int ret = service_run(id, cs, UINT64_MAX, s);
+	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	if (serialize_mt_unsafe)
-		rte_atomic32_dec(&s->num_mapped_cores);
+	rte_atomic32_dec(&s->num_mapped_cores);
 
 	return ret;
 }
@@ -449,7 +439,7 @@ rte_service_runner_func(void *arg)
 			if (!service_valid(i))
 				continue;
 			/* return value ignored as no change to code flow */
-			service_run(i, cs, service_mask, service_get(i));
+			service_run(i, cs, service_mask, service_get(i), 1);
 		}
 
 		cs->loops++;
Signature · 1 line
2.17.1
#86 lore raw

PATCH v4 3/6 service: remove rte prefix from static functions

In reply to Honnappa Nagarahalli

From: Phil Yang <[email protected]>

clean up rte prefix from static functions. remove unused parameter for service_dump_one function.

---
 lib/librte_eal/common/rte_service.c | 34 ++++++++++-------------------
 1 file changed, 11 insertions(+), 23 deletions(-)
lib/librte_eal/common/rte_service.c +11 -23
index c283408cf..62ea9cbd6 100644
@@ -340,7 +340,7 @@ rte_service_runstate_get(uint32_t id)
 }
 
 static inline void
-rte_service_runner_do_callback(struct rte_service_spec_impl *s,
+service_runner_do_callback(struct rte_service_spec_impl *s,
 			       struct core_state *cs, uint32_t service_idx)
 {
 	void *userdata = s->spec.callback_userdata;
@@ -378,10 +378,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 		rte_atomic32_clear(&s->execute_lock);
 	} else
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 
 	return 0;
 }
@@ -425,14 +425,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 }
 
 static int32_t
-rte_service_runner_func(void *arg)
+service_runner_func(void *arg)
 {
 	RTE_SET_USED(arg);
 	uint32_t i;
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (lcore_states[lcore].runstate == RUNSTATE_RUNNING) {
+	while (cs->runstate == RUNSTATE_RUNNING) {
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
@@ -693,9 +693,9 @@ rte_service_lcore_start(uint32_t lcore)
 	/* set core to run state first, and then launch otherwise it will
 	 * return immediately as runstate keeps it in the service poll loop
 	 */
-	lcore_states[lcore].runstate = RUNSTATE_RUNNING;
+	cs->runstate = RUNSTATE_RUNNING;
 
-	int ret = rte_eal_remote_launch(rte_service_runner_func, 0, lcore);
+	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
 	return ret;
 }
@@ -774,13 +774,9 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id,
 }
 
 static void
-rte_service_dump_one(FILE *f, struct rte_service_spec_impl *s,
-		     uint64_t all_cycles, uint32_t reset)
+service_dump_one(FILE *f, struct rte_service_spec_impl *s, uint32_t reset)
 {
 	/* avoid divide by zero */
-	if (all_cycles == 0)
-		all_cycles = 1;
-
 	int calls = 1;
 	if (s->calls != 0)
 		calls = s->calls;
@@ -807,7 +803,7 @@ rte_service_attr_reset_all(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	int reset = 1;
-	rte_service_dump_one(NULL, s, 0, reset);
+	service_dump_one(NULL, s, reset);
 	return 0;
 }
 
@@ -851,21 +847,13 @@ rte_service_dump(FILE *f, uint32_t id)
 	uint32_t i;
 	int print_one = (id != UINT32_MAX);
 
-	uint64_t total_cycles = 0;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if (!service_valid(i))
-			continue;
-		total_cycles += rte_services[i].cycles_spent;
-	}
-
 	/* print only the specified service */
 	if (print_one) {
 		struct rte_service_spec_impl *s;
 		SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 		fprintf(f, "Service %s Summary\n", s->spec.name);
 		uint32_t reset = 0;
-		rte_service_dump_one(f, s, total_cycles, reset);
+		service_dump_one(f, s, reset);
 		return 0;
 	}
 
@@ -875,7 +863,7 @@ rte_service_dump(FILE *f, uint32_t id)
 		if (!service_valid(i))
 			continue;
 		uint32_t reset = 0;
-		rte_service_dump_one(f, &rte_services[i], total_cycles, reset);
+		service_dump_one(f, &rte_services[i], reset);
 	}
 
 	fprintf(f, "Service Cores Summary\n");
Signature · 1 line
2.17.1
#87 lore raw

PATCH v4 4/6 service: remove redundant code

In reply to Honnappa Nagarahalli

From: Phil Yang <[email protected]>

The service id validation is duplicated, remove the redundant code in the calling functions.

---
 lib/librte_eal/common/rte_service.c | 28 ++++++----------------------
 1 file changed, 6 insertions(+), 22 deletions(-)
lib/librte_eal/common/rte_service.c +6 -22
index 62ea9cbd6..37c16c4bc 100644
@@ -541,24 +541,12 @@ rte_service_start_with_defaults(void)
 }
 
 static int32_t
-service_update(struct rte_service_spec *service, uint32_t lcore,
+service_update(uint32_t sid, uint32_t lcore,
 		uint32_t *set, uint32_t *enabled)
 {
-	uint32_t i;
-	int32_t sid = -1;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if ((struct rte_service_spec *)&rte_services[i] == service &&
-				service_valid(i)) {
-			sid = i;
-			break;
-		}
-	}
-
-	if (sid == -1 || lcore >= RTE_MAX_LCORE)
-		return -EINVAL;
-
-	if (!lcore_states[lcore].is_service_core)
+	/* validate ID, or return error value */
+	if (sid >= RTE_SERVICE_NUM_MAX || !service_valid(sid) ||
+	    lcore >= RTE_MAX_LCORE || !lcore_states[lcore].is_service_core)
 		return -EINVAL;
 
 	uint64_t sid_mask = UINT64_C(1) << sid;
@@ -587,19 +575,15 @@ service_update(struct rte_service_spec *service, uint32_t lcore,
 int32_t
 rte_service_map_lcore_set(uint32_t id, uint32_t lcore, uint32_t enabled)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 	uint32_t on = enabled > 0;
-	return service_update(&s->spec, lcore, &on, 0);
+	return service_update(id, lcore, &on, 0);
 }
 
 int32_t
 rte_service_map_lcore_get(uint32_t id, uint32_t lcore)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 	uint32_t enabled;
-	int ret = service_update(&s->spec, lcore, 0, &enabled);
+	int ret = service_update(id, lcore, 0, &enabled);
 	if (ret == 0)
 		return enabled;
 	return ret;
Signature · 1 line
2.17.1
#88 lore raw

PATCH v4 5/6 service: optimize with c11 atomics

In reply to Honnappa Nagarahalli

From: Phil Yang <[email protected]>

The num_mapped_cores is used as a statistics. Use c11 atomics with RELAXED ordering for num_mapped_cores instead of rte_atomic ops which enforce unnessary barriers on aarch64.

Replace execute_lock operations to spinlock_try_lock to avoid duplicate code.

---
 lib/librte_eal/common/rte_service.c | 32 ++++++++++++++++-------------
 lib/librte_eal/meson.build          |  4 ++++
 2 files changed, 22 insertions(+), 14 deletions(-)
lib/librte_eal/common/rte_service.c +18 -14
index 37c16c4bc..5d35f8a8d 100644
@@ -20,6 +20,7 @@
 #include <rte_atomic.h>
 #include <rte_memory.h>
 #include <rte_malloc.h>
+#include <rte_spinlock.h>
 
 #include "eal_private.h"
 
@@ -38,11 +39,11 @@ struct rte_service_spec_impl {
 	/* public part of the struct */
 	struct rte_service_spec spec;
 
-	/* atomic lock that when set indicates a service core is currently
+	/* spin lock that when set indicates a service core is currently
 	 * running this service callback. When not set, a core may take the
 	 * lock and then run the service callback.
 	 */
-	rte_atomic32_t execute_lock;
+	rte_spinlock_t execute_lock;
 
 	/* API set/get-able variables */
 	int8_t app_runstate;
@@ -54,7 +55,7 @@ struct rte_service_spec_impl {
 	 * It does not indicate the number of cores the service is running
 	 * on currently.
 	 */
-	rte_atomic32_t num_mapped_cores;
+	uint32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
 } __rte_cache_aligned;
@@ -332,7 +333,8 @@ rte_service_runstate_get(uint32_t id)
 	rte_smp_rmb();
 
 	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (rte_atomic32_read(&s->num_mapped_cores) > 0);
+	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+					    __ATOMIC_RELAXED) > 0);
 
 	return (s->app_runstate == RUNSTATE_RUNNING) &&
 		(s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -375,11 +377,11 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 	cs->service_active_on_lcore[i] = 1;
 
 	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
-		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
+		if (!rte_spinlock_trylock(&s->execute_lock))
 			return -EBUSY;
 
 		service_runner_do_callback(s, cs, i);
-		rte_atomic32_clear(&s->execute_lock);
+		rte_spinlock_unlock(&s->execute_lock);
 	} else
 		service_runner_do_callback(s, cs, i);
 
@@ -415,11 +417,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 	/* Increment num_mapped_cores to reflect that this core is
 	 * now mapped capable of running the service.
 	 */
-	rte_atomic32_inc(&s->num_mapped_cores);
+	__atomic_add_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELAXED);
 
 	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	rte_atomic32_dec(&s->num_mapped_cores);
+	__atomic_sub_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELAXED);
 
 	return ret;
 }
@@ -556,19 +558,19 @@ service_update(uint32_t sid, uint32_t lcore,
 
 		if (*set && !lcore_mapped) {
 			lcore_states[lcore].service_mask |= sid_mask;
-			rte_atomic32_inc(&rte_services[sid].num_mapped_cores);
+			__atomic_add_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELAXED);
 		}
 		if (!*set && lcore_mapped) {
 			lcore_states[lcore].service_mask &= ~(sid_mask);
-			rte_atomic32_dec(&rte_services[sid].num_mapped_cores);
+			__atomic_sub_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELAXED);
 		}
 	}
 
 	if (enabled)
 		*enabled = !!(lcore_states[lcore].service_mask & (sid_mask));
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -616,7 +618,8 @@ rte_service_lcore_reset_all(void)
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
-		rte_atomic32_set(&rte_services[i].num_mapped_cores, 0);
+		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
+				    __ATOMIC_RELAXED);
 
 	rte_smp_wmb();
 
@@ -699,7 +702,8 @@ rte_service_lcore_stop(uint32_t lcore)
 		int32_t enabled = service_mask & (UINT64_C(1) << i);
 		int32_t service_running = rte_service_runstate_get(i);
 		int32_t only_core = (1 ==
-			rte_atomic32_read(&rte_services[i].num_mapped_cores));
+			__atomic_load_n(&rte_services[i].num_mapped_cores,
+					__ATOMIC_RELAXED));
 
 		/* if the core is mapped, and the service is running, and this
 		 * is the only core that is mapped, the service would cease to
lib/librte_eal/meson.build +4 -0
index 0267c3b9d..c2d7a6954 100644
@@ -21,3 +21,7 @@ endif
 if cc.has_header('getopt.h')
 	cflags += ['-DHAVE_GETOPT_H', '-DHAVE_GETOPT', '-DHAVE_GETOPT_LONG']
 endif
+# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
+if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
+    ext_deps += cc.find_library('atomic')
+endif
Signature · 1 line
2.17.1
author
#89 lore raw

Re: [dpdk-dev] [PATCH v4 5/6] service: optimize with c11 atomics

Quoted 29 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Wednesday, May 6, 2020 5:18 AM
To: [email protected]; Phil Yang <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: [PATCH v4 5/6] service: optimize with c11 atomics

From: Phil Yang <[email protected]>

The num_mapped_cores is used as a statistics. Use c11 atomics with RELAXED ordering for num_mapped_cores instead of rte_atomic ops which enforce unnessary barriers on aarch64.

Replace execute_lock operations to spinlock_try_lock to avoid duplicate code.

---
 lib/librte_eal/common/rte_service.c | 32 ++++++++++++++++-------------
 lib/librte_eal/meson.build          |  4 ++++
 2 files changed, 22 insertions(+), 14 deletions(-)

<snip>

lib/librte_eal/meson.build endif · line 27

Quoted 13 lines
lib/librte_eal/meson.build +0 -0
index 0267c3b9d..c2d7a6954 100644
@@ -21,3 +21,7 @@ endif
 if cc.has_header('getopt.h')
 	cflags += ['-DHAVE_GETOPT_H', '-DHAVE_GETOPT', '-

DHAVE_GETOPT_LONG']
endif

lib/librte_eal/meson.build +4 -0
+# for clang 32-bit compiles we need libatomic for 64-bit atomic ops
+if cc.get_id() == 'clang' and dpdk_conf.get('RTE_ARCH_64') == false
+    ext_deps += cc.find_library('atomic')
+endif

We can remove this as it has been added in global. "da4eae278b56 - build: add global libatomic dependency for 32-bit clang"

I've updated it in v5.

Thanks,
Phil

Quoted 2 lines

--
2.17.1

#90 lore raw

PATCH v4 6/6 service: relax barriers with C11 atomics

In reply to Honnappa Nagarahalli

From: Phil Yang <[email protected]>

The runstate, comp_runstate and app_runstate are used as guard variables in the service core lib. To guarantee the inter-threads visibility of these guard variables, it uses rte_smp_r/wmb. This patch use c11 atomic built-ins to relax these barriers.

---
 lib/librte_eal/common/rte_service.c | 115 ++++++++++++++++++++--------
 1 file changed, 84 insertions(+), 31 deletions(-)
lib/librte_eal/common/rte_service.c +84 -31
index 5d35f8a8d..3bae7d66d 100644
@@ -265,7 +265,6 @@ rte_service_component_register(const struct rte_service_spec *spec,
 	s->spec = *spec;
 	s->internal_flags |= SERVICE_F_REGISTERED | SERVICE_F_START_CHECK;
 
-	rte_smp_wmb();
 	rte_service_count++;
 
 	if (id_ptr)
@@ -282,7 +281,6 @@ rte_service_component_unregister(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	rte_service_count--;
-	rte_smp_wmb();
 
 	s->internal_flags &= ~(SERVICE_F_REGISTERED);
 
@@ -301,12 +299,17 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
+	/* comp_runstate act as the guard variable. Use store-release
+	 * memory order. This synchronizes with load-acquire in
+	 * service_run and service_runstate_get function.
+	 */
 	if (runstate)
-		s->comp_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->comp_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -316,12 +319,17 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
+	/* app_runstate act as the guard variable. Use store-release
+	 * memory order. This synchronizes with load-acquire in
+	 * service_run runstate_get function.
+	 */
 	if (runstate)
-		s->app_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->app_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -330,15 +338,24 @@ rte_service_runstate_get(uint32_t id)
 {
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
-	rte_smp_rmb();
 
-	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+	/* comp_runstate and app_runstate act as the guard variables.
+	 * Use load-acquire memory order. This synchronizes with
+	 * store-release in service state set functions.
+	 */
+	if (__atomic_load_n(&s->comp_runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING &&
+		 __atomic_load_n(&s->app_runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {
+		int check_disabled = !(s->internal_flags &
+					SERVICE_F_START_CHECK);
+		int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
 					    __ATOMIC_RELAXED) > 0);
 
-	return (s->app_runstate == RUNSTATE_RUNNING) &&
-		(s->comp_runstate == RUNSTATE_RUNNING) &&
-		(check_disabled | lcore_mapped);
+		return (check_disabled | lcore_mapped);
+	} else
+		return 0;
+
 }
 
 static inline void
@@ -367,9 +384,15 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 	if (!s)
 		return -EINVAL;
 
-	if (s->comp_runstate != RUNSTATE_RUNNING ||
-			s->app_runstate != RUNSTATE_RUNNING ||
-			!(service_mask & (UINT64_C(1) << i))) {
+	/* comp_runstate and app_runstate act as the guard variables.
+	 * Use load-acquire memory order. This synchronizes with
+	 * store-release in service state set functions.
+	 */
+	if (__atomic_load_n(&s->comp_runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_RUNNING ||
+		 __atomic_load_n(&s->app_runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_RUNNING ||
+		!(service_mask & (UINT64_C(1) << i))) {
 		cs->service_active_on_lcore[i] = 0;
 		return -ENOEXEC;
 	}
@@ -434,7 +457,12 @@ service_runner_func(void *arg)
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (cs->runstate == RUNSTATE_RUNNING) {
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	while (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
@@ -445,8 +473,6 @@ service_runner_func(void *arg)
 		}
 
 		cs->loops++;
-
-		rte_smp_rmb();
 	}
 
 	lcore_config[lcore].state = WAIT;
@@ -614,15 +640,18 @@ rte_service_lcore_reset_all(void)
 		if (lcore_states[i].is_service_core) {
 			lcore_states[i].service_mask = 0;
 			set_lcore_state(i, ROLE_RTE);
-			lcore_states[i].runstate = RUNSTATE_STOPPED;
+			/* runstate act as guard variable Use
+			 * store-release memory order here to synchronize
+			 * with load-acquire in runstate read functions.
+			 */
+			__atomic_store_n(&lcore_states[i].runstate,
+				RUNSTATE_STOPPED, __ATOMIC_RELEASE);
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
 		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
 				    __ATOMIC_RELAXED);
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -638,9 +667,11 @@ rte_service_lcore_add(uint32_t lcore)
 
 	/* ensure that after adding a core the mask and state are defaults */
 	lcore_states[lcore].service_mask = 0;
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
-
-	rte_smp_wmb();
+	/* Use store-release memory order here to synchronize with
+	 * load-acquire in runstate read functions.
+	 */
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+		__ATOMIC_RELEASE);
 
 	return rte_eal_wait_lcore(lcore);
 }
@@ -655,7 +686,12 @@ rte_service_lcore_del(uint32_t lcore)
 	if (!cs->is_service_core)
 		return -EINVAL;
 
-	if (cs->runstate != RUNSTATE_STOPPED)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_STOPPED)
 		return -EBUSY;
 
 	set_lcore_state(lcore, ROLE_RTE);
@@ -674,13 +710,21 @@ rte_service_lcore_start(uint32_t lcore)
 	if (!cs->is_service_core)
 		return -EINVAL;
 
-	if (cs->runstate == RUNSTATE_RUNNING)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING)
 		return -EALREADY;
 
 	/* set core to run state first, and then launch otherwise it will
 	 * return immediately as runstate keeps it in the service poll loop
 	 */
-	cs->runstate = RUNSTATE_RUNNING;
+	/* Use load-acquire memory order here to synchronize with
+	 * store-release in runstate update functions.
+	 */
+	__atomic_store_n(&cs->runstate, RUNSTATE_RUNNING, __ATOMIC_RELEASE);
 
 	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
@@ -693,7 +737,12 @@ rte_service_lcore_stop(uint32_t lcore)
 	if (lcore >= RTE_MAX_LCORE)
 		return -EINVAL;
 
-	if (lcore_states[lcore].runstate == RUNSTATE_STOPPED)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&lcore_states[lcore].runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_STOPPED)
 		return -EALREADY;
 
 	uint32_t i;
@@ -713,7 +762,11 @@ rte_service_lcore_stop(uint32_t lcore)
 			return -EBUSY;
 	}
 
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
+	/* Use store-release memory order here to synchronize with
+	 * load-acquire in runstate read functions.
+	 */
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+		__ATOMIC_RELEASE);
 
 	return 0;
 }
Signature · 1 line
2.17.1
author
#91 lore raw

v5 cover use c11 atomics for service core lib

In reply to Honnappa Nagarahalli

The rte_atomic ops and rte_smp barriers enforce DMB barriers on aarch64. Using c11 atomics with explicit memory ordering instead of the rte_atomic ops and rte_smp barriers for inter-threads synchronization can uplift the performance on aarch64 and no performance loss on x86.

This patchset contains:
1) fix race condition for MT unsafe service.
2) clean up redundant code.
3) use c11 atomics for service core lib to avoid unnecessary barriers.

v2:
Still waiting on Harry for the final solution on the MT unsafe race condition issue. But I have incorporated the comments so far.
1. add 'Fixes' tag for bug-fix patches.
2. remove 'Fixes' tag for code cleanup patches.
3. remove unused parameter for service_dump_one function.
4. replace the execute_lock atomic CAS operation to spinlock_try_lock.
5. use c11 atomics with RELAXED memory ordering for num_mapped_cores.
6. relax barriers for guard variables runstate, comp_runstate and app_runstate with c11 one-way barriers.

v3:
Sending this version since Phil is on holiday.
1. Updated the API documentation to indicate how the locking can be avoided.

v4:
1. Fix the nits in 2/6 commit message and comments in code.

v5:
1. Remove redundant libatomic clang dependency claim code as it has been added in global. (The commit da4eae278b56)

Honnappa Nagarahalli (2):
service: fix race condition for MT unsafe service
service: fix identification of service running on other lcore

Phil Yang (4):
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 atomics
service: relax barriers with C11 atomics

 lib/librte_eal/common/rte_service.c            | 234 +++++++++++++------------
 lib/librte_eal/include/rte_service.h           |   8 +-
 lib/librte_eal/include/rte_service_component.h |   6 +-
 3 files changed, 137 insertions(+), 111 deletions(-)
Signature · 1 line
2.7.4
author
#92 lore raw

PATCH v5 1/6 service: fix race condition for MT unsafe service

From: Honnappa Nagarahalli <[email protected]>

The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

If the service is known to be mmapped on a single lcore, setting the service capability to MT safe will avoid taking the lock and improve the performance.

Fixes e9139a32f6e8 ("service: add function to run on app lcore")
---
 lib/librte_eal/common/rte_service.c            | 11 +++++------
 lib/librte_eal/include/rte_service.h           |  8 ++++++--
 lib/librte_eal/include/rte_service_component.h |  6 +++++-
 3 files changed, 16 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
index 70d17a5..b8c465e 100644
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {
 	uint8_t internal_flags;
 
 	/* per service statistics */
+	/* Indicates how many cores the service is mapped to run on.
+	 * It does not indicate the number of cores the service is running
+	 * on currently.
+	 */
 	rte_atomic32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	/* check do we need cmpset, if MT safe or <= 1 core
-	 * mapped, atomic ops are not required.
-	 */
-	const int use_atomics = (service_mt_safe(s) == 0) &&
-				(rte_atomic32_read(&s->num_mapped_cores) > 1);
-	if (use_atomics) {
+	if (service_mt_safe(s) == 0) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
lib/librte_eal/include/rte_service.h +6 -2
index d8701dd..3a1c735 100644
@@ -104,12 +104,16 @@ int32_t rte_service_probe_capability(uint32_t id, uint32_t capability);
  * Each core can be added or removed from running a specific service. This
  * function enables or disables *lcore* to run *service_id*.
  *
- * If multiple cores are enabled on a service, an atomic is used to ensure that
- * only one cores runs the service at a time. The exception to this is when
+ * If multiple cores are enabled on a service, a lock is used to ensure that
+ * only one core runs the service at a time. The exception to this is when
  * a service indicates that it is multi-thread safe by setting the capability
  * called RTE_SERVICE_CAP_MT_SAFE. With the multi-thread safe capability set,
  * the service function can be run on multiple threads at the same time.
  *
+ * If the service is known to be mapped to a single lcore, setting the
+ * capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve
+ * better performance by avoiding the use of lock.
+ *
  * @param service_id the service to apply the lcore to
  * @param lcore The lcore that will be mapped to service
  * @param enable Zero to unmap or disable the core, non-zero to enable
lib/librte_eal/include/rte_service_component.h +5 -1
index 16eab79..b75aba1 100644
@@ -43,7 +43,7 @@ struct rte_service_spec {
 /**
  * Register a new service.
  *
- * A service represents a component that the requires CPU time periodically to
+ * A service represents a component that requires CPU time periodically to
  * achieve its purpose.
  *
  * For example the eventdev SW PMD requires CPU cycles to perform its
@@ -56,6 +56,10 @@ struct rte_service_spec {
  * *rte_service_component_runstate_set*, which indicates that the service
  * component is ready to be executed.
  *
+ * If the service is known to be mapped to a single lcore, setting the
+ * capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve
+ * better performance.
+ *
  * @param spec The specification of the service to register
  * @param[out] service_id A pointer to a uint32_t, which will be filled in
  *             during registration of the service. It is set to the integers
Signature · 1 line
2.7.4
author
#93 lore raw

PATCH v5 2/6 service: fix identification of service running on other lcore

In reply to Phil Yang

From: Honnappa Nagarahalli <[email protected]>

The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduce the number of instructions for all cases.

Fixes 8d39d3e237c2 ("service: fix race in service on app lcore function")
---
 lib/librte_eal/common/rte_service.c | 26 ++++++++------------------
 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
index b8c465e..c283408 100644
@@ -360,7 +360,7 @@ rte_service_runner_do_callback(struct rte_service_spec_impl *s,
 /* Expects the service 's' is valid. */
 static int32_t
 service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
-	    struct rte_service_spec_impl *s)
+	    struct rte_service_spec_impl *s, uint32_t serialize_mt_unsafe)
 {
 	if (!s)
 		return -EINVAL;
@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	if (service_mt_safe(s) == 0) {
+	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
-	/* Atomically add this core to the mapped cores first, then examine if
-	 * we can run the service. This avoids a race condition between
-	 * checking the value, and atomically adding to the mapped count.
+	/* Increment num_mapped_cores to reflect that this core is
+	 * now mapped capable of running the service.
 	 */
-	if (serialize_mt_unsafe)
-		rte_atomic32_inc(&s->num_mapped_cores);
+	rte_atomic32_inc(&s->num_mapped_cores);
 
-	if (service_mt_safe(s) == 0 &&
-			rte_atomic32_read(&s->num_mapped_cores) > 1) {
-		if (serialize_mt_unsafe)
-			rte_atomic32_dec(&s->num_mapped_cores);
-		return -EBUSY;
-	}
-
-	int ret = service_run(id, cs, UINT64_MAX, s);
+	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	if (serialize_mt_unsafe)
-		rte_atomic32_dec(&s->num_mapped_cores);
+	rte_atomic32_dec(&s->num_mapped_cores);
 
 	return ret;
 }
@@ -449,7 +439,7 @@ rte_service_runner_func(void *arg)
 			if (!service_valid(i))
 				continue;
 			/* return value ignored as no change to code flow */
-			service_run(i, cs, service_mask, service_get(i));
+			service_run(i, cs, service_mask, service_get(i), 1);
 		}
 
 		cs->loops++;
Signature · 1 line
2.7.4
author
#94 lore raw

PATCH v5 3/6 service: remove rte prefix from static functions

In reply to Phil Yang

clean up rte prefix from static functions. remove unused parameter for service_dump_one function.

---
 lib/librte_eal/common/rte_service.c | 34 +++++++++++-----------------------
 1 file changed, 11 insertions(+), 23 deletions(-)
lib/librte_eal/common/rte_service.c +11 -23
index c283408..62ea9cb 100644
@@ -340,7 +340,7 @@ rte_service_runstate_get(uint32_t id)
 }
 
 static inline void
-rte_service_runner_do_callback(struct rte_service_spec_impl *s,
+service_runner_do_callback(struct rte_service_spec_impl *s,
 			       struct core_state *cs, uint32_t service_idx)
 {
 	void *userdata = s->spec.callback_userdata;
@@ -378,10 +378,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 		rte_atomic32_clear(&s->execute_lock);
 	} else
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 
 	return 0;
 }
@@ -425,14 +425,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 }
 
 static int32_t
-rte_service_runner_func(void *arg)
+service_runner_func(void *arg)
 {
 	RTE_SET_USED(arg);
 	uint32_t i;
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (lcore_states[lcore].runstate == RUNSTATE_RUNNING) {
+	while (cs->runstate == RUNSTATE_RUNNING) {
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
@@ -693,9 +693,9 @@ rte_service_lcore_start(uint32_t lcore)
 	/* set core to run state first, and then launch otherwise it will
 	 * return immediately as runstate keeps it in the service poll loop
 	 */
-	lcore_states[lcore].runstate = RUNSTATE_RUNNING;
+	cs->runstate = RUNSTATE_RUNNING;
 
-	int ret = rte_eal_remote_launch(rte_service_runner_func, 0, lcore);
+	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
 	return ret;
 }
@@ -774,13 +774,9 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id,
 }
 
 static void
-rte_service_dump_one(FILE *f, struct rte_service_spec_impl *s,
-		     uint64_t all_cycles, uint32_t reset)
+service_dump_one(FILE *f, struct rte_service_spec_impl *s, uint32_t reset)
 {
 	/* avoid divide by zero */
-	if (all_cycles == 0)
-		all_cycles = 1;
-
 	int calls = 1;
 	if (s->calls != 0)
 		calls = s->calls;
@@ -807,7 +803,7 @@ rte_service_attr_reset_all(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	int reset = 1;
-	rte_service_dump_one(NULL, s, 0, reset);
+	service_dump_one(NULL, s, reset);
 	return 0;
 }
 
@@ -851,21 +847,13 @@ rte_service_dump(FILE *f, uint32_t id)
 	uint32_t i;
 	int print_one = (id != UINT32_MAX);
 
-	uint64_t total_cycles = 0;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if (!service_valid(i))
-			continue;
-		total_cycles += rte_services[i].cycles_spent;
-	}
-
 	/* print only the specified service */
 	if (print_one) {
 		struct rte_service_spec_impl *s;
 		SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 		fprintf(f, "Service %s Summary\n", s->spec.name);
 		uint32_t reset = 0;
-		rte_service_dump_one(f, s, total_cycles, reset);
+		service_dump_one(f, s, reset);
 		return 0;
 	}
 
@@ -875,7 +863,7 @@ rte_service_dump(FILE *f, uint32_t id)
 		if (!service_valid(i))
 			continue;
 		uint32_t reset = 0;
-		rte_service_dump_one(f, &rte_services[i], total_cycles, reset);
+		service_dump_one(f, &rte_services[i], reset);
 	}
 
 	fprintf(f, "Service Cores Summary\n");
Signature · 1 line
2.7.4
author
#95 lore raw

PATCH v5 4/6 service: remove redundant code

In reply to Phil Yang

The service id validation is duplicated, remove the redundant code in the calling functions.

---
 lib/librte_eal/common/rte_service.c | 28 ++++++----------------------
 1 file changed, 6 insertions(+), 22 deletions(-)
lib/librte_eal/common/rte_service.c +6 -22
index 62ea9cb..37c16c4 100644
@@ -541,24 +541,12 @@ rte_service_start_with_defaults(void)
 }
 
 static int32_t
-service_update(struct rte_service_spec *service, uint32_t lcore,
+service_update(uint32_t sid, uint32_t lcore,
 		uint32_t *set, uint32_t *enabled)
 {
-	uint32_t i;
-	int32_t sid = -1;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if ((struct rte_service_spec *)&rte_services[i] == service &&
-				service_valid(i)) {
-			sid = i;
-			break;
-		}
-	}
-
-	if (sid == -1 || lcore >= RTE_MAX_LCORE)
-		return -EINVAL;
-
-	if (!lcore_states[lcore].is_service_core)
+	/* validate ID, or return error value */
+	if (sid >= RTE_SERVICE_NUM_MAX || !service_valid(sid) ||
+	    lcore >= RTE_MAX_LCORE || !lcore_states[lcore].is_service_core)
 		return -EINVAL;
 
 	uint64_t sid_mask = UINT64_C(1) << sid;
@@ -587,19 +575,15 @@ service_update(struct rte_service_spec *service, uint32_t lcore,
 int32_t
 rte_service_map_lcore_set(uint32_t id, uint32_t lcore, uint32_t enabled)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 	uint32_t on = enabled > 0;
-	return service_update(&s->spec, lcore, &on, 0);
+	return service_update(id, lcore, &on, 0);
 }
 
 int32_t
 rte_service_map_lcore_get(uint32_t id, uint32_t lcore)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 	uint32_t enabled;
-	int ret = service_update(&s->spec, lcore, 0, &enabled);
+	int ret = service_update(id, lcore, 0, &enabled);
 	if (ret == 0)
 		return enabled;
 	return ret;
Signature · 1 line
2.7.4
author
#96 lore raw

PATCH v5 5/6 service: optimize with c11 atomics

In reply to Phil Yang

The num_mapped_cores is used as a statistics. Use c11 atomics with RELAXED ordering for num_mapped_cores instead of rte_atomic ops which enforce unnessary barriers on aarch64.

Replace execute_lock operations to spinlock_try_lock to avoid duplicate code.

Change-Id I2edf1feb64c9192fc7577be741865ea61c8680cd
---
 lib/librte_eal/common/rte_service.c | 32 ++++++++++++++++++--------------
 1 file changed, 18 insertions(+), 14 deletions(-)
lib/librte_eal/common/rte_service.c +18 -14
index 37c16c4..5d35f8a 100644
@@ -20,6 +20,7 @@
 #include <rte_atomic.h>
 #include <rte_memory.h>
 #include <rte_malloc.h>
+#include <rte_spinlock.h>
 
 #include "eal_private.h"
 
@@ -38,11 +39,11 @@ struct rte_service_spec_impl {
 	/* public part of the struct */
 	struct rte_service_spec spec;
 
-	/* atomic lock that when set indicates a service core is currently
+	/* spin lock that when set indicates a service core is currently
 	 * running this service callback. When not set, a core may take the
 	 * lock and then run the service callback.
 	 */
-	rte_atomic32_t execute_lock;
+	rte_spinlock_t execute_lock;
 
 	/* API set/get-able variables */
 	int8_t app_runstate;
@@ -54,7 +55,7 @@ struct rte_service_spec_impl {
 	 * It does not indicate the number of cores the service is running
 	 * on currently.
 	 */
-	rte_atomic32_t num_mapped_cores;
+	uint32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
 } __rte_cache_aligned;
@@ -332,7 +333,8 @@ rte_service_runstate_get(uint32_t id)
 	rte_smp_rmb();
 
 	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (rte_atomic32_read(&s->num_mapped_cores) > 0);
+	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+					    __ATOMIC_RELAXED) > 0);
 
 	return (s->app_runstate == RUNSTATE_RUNNING) &&
 		(s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -375,11 +377,11 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 	cs->service_active_on_lcore[i] = 1;
 
 	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
-		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
+		if (!rte_spinlock_trylock(&s->execute_lock))
 			return -EBUSY;
 
 		service_runner_do_callback(s, cs, i);
-		rte_atomic32_clear(&s->execute_lock);
+		rte_spinlock_unlock(&s->execute_lock);
 	} else
 		service_runner_do_callback(s, cs, i);
 
@@ -415,11 +417,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 	/* Increment num_mapped_cores to reflect that this core is
 	 * now mapped capable of running the service.
 	 */
-	rte_atomic32_inc(&s->num_mapped_cores);
+	__atomic_add_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELAXED);
 
 	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	rte_atomic32_dec(&s->num_mapped_cores);
+	__atomic_sub_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELAXED);
 
 	return ret;
 }
@@ -556,19 +558,19 @@ service_update(uint32_t sid, uint32_t lcore,
 
 		if (*set && !lcore_mapped) {
 			lcore_states[lcore].service_mask |= sid_mask;
-			rte_atomic32_inc(&rte_services[sid].num_mapped_cores);
+			__atomic_add_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELAXED);
 		}
 		if (!*set && lcore_mapped) {
 			lcore_states[lcore].service_mask &= ~(sid_mask);
-			rte_atomic32_dec(&rte_services[sid].num_mapped_cores);
+			__atomic_sub_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELAXED);
 		}
 	}
 
 	if (enabled)
 		*enabled = !!(lcore_states[lcore].service_mask & (sid_mask));
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -616,7 +618,8 @@ rte_service_lcore_reset_all(void)
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
-		rte_atomic32_set(&rte_services[i].num_mapped_cores, 0);
+		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
+				    __ATOMIC_RELAXED);
 
 	rte_smp_wmb();
 
@@ -699,7 +702,8 @@ rte_service_lcore_stop(uint32_t lcore)
 		int32_t enabled = service_mask & (UINT64_C(1) << i);
 		int32_t service_running = rte_service_runstate_get(i);
 		int32_t only_core = (1 ==
-			rte_atomic32_read(&rte_services[i].num_mapped_cores));
+			__atomic_load_n(&rte_services[i].num_mapped_cores,
+					__ATOMIC_RELAXED));
 
 		/* if the core is mapped, and the service is running, and this
 		 * is the only core that is mapped, the service would cease to
Signature · 1 line
2.7.4
author
#97 lore raw

PATCH v5 6/6 service: relax barriers with C11 atomics

In reply to Phil Yang

The runstate, comp_runstate and app_runstate are used as guard variables in the service core lib. To guarantee the inter-threads visibility of these guard variables, it uses rte_smp_r/wmb. This patch use c11 atomic built-ins to relax these barriers.

---
 lib/librte_eal/common/rte_service.c | 115 ++++++++++++++++++++++++++----------
 1 file changed, 84 insertions(+), 31 deletions(-)
lib/librte_eal/common/rte_service.c +84 -31
index 5d35f8a..3bae7d6 100644
@@ -265,7 +265,6 @@ rte_service_component_register(const struct rte_service_spec *spec,
 	s->spec = *spec;
 	s->internal_flags |= SERVICE_F_REGISTERED | SERVICE_F_START_CHECK;
 
-	rte_smp_wmb();
 	rte_service_count++;
 
 	if (id_ptr)
@@ -282,7 +281,6 @@ rte_service_component_unregister(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	rte_service_count--;
-	rte_smp_wmb();
 
 	s->internal_flags &= ~(SERVICE_F_REGISTERED);
 
@@ -301,12 +299,17 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
+	/* comp_runstate act as the guard variable. Use store-release
+	 * memory order. This synchronizes with load-acquire in
+	 * service_run and service_runstate_get function.
+	 */
 	if (runstate)
-		s->comp_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->comp_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -316,12 +319,17 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
+	/* app_runstate act as the guard variable. Use store-release
+	 * memory order. This synchronizes with load-acquire in
+	 * service_run runstate_get function.
+	 */
 	if (runstate)
-		s->app_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->app_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -330,15 +338,24 @@ rte_service_runstate_get(uint32_t id)
 {
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
-	rte_smp_rmb();
 
-	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+	/* comp_runstate and app_runstate act as the guard variables.
+	 * Use load-acquire memory order. This synchronizes with
+	 * store-release in service state set functions.
+	 */
+	if (__atomic_load_n(&s->comp_runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING &&
+		 __atomic_load_n(&s->app_runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {
+		int check_disabled = !(s->internal_flags &
+					SERVICE_F_START_CHECK);
+		int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
 					    __ATOMIC_RELAXED) > 0);
 
-	return (s->app_runstate == RUNSTATE_RUNNING) &&
-		(s->comp_runstate == RUNSTATE_RUNNING) &&
-		(check_disabled | lcore_mapped);
+		return (check_disabled | lcore_mapped);
+	} else
+		return 0;
+
 }
 
 static inline void
@@ -367,9 +384,15 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 	if (!s)
 		return -EINVAL;
 
-	if (s->comp_runstate != RUNSTATE_RUNNING ||
-			s->app_runstate != RUNSTATE_RUNNING ||
-			!(service_mask & (UINT64_C(1) << i))) {
+	/* comp_runstate and app_runstate act as the guard variables.
+	 * Use load-acquire memory order. This synchronizes with
+	 * store-release in service state set functions.
+	 */
+	if (__atomic_load_n(&s->comp_runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_RUNNING ||
+		 __atomic_load_n(&s->app_runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_RUNNING ||
+		!(service_mask & (UINT64_C(1) << i))) {
 		cs->service_active_on_lcore[i] = 0;
 		return -ENOEXEC;
 	}
@@ -434,7 +457,12 @@ service_runner_func(void *arg)
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (cs->runstate == RUNSTATE_RUNNING) {
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	while (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
@@ -445,8 +473,6 @@ service_runner_func(void *arg)
 		}
 
 		cs->loops++;
-
-		rte_smp_rmb();
 	}
 
 	lcore_config[lcore].state = WAIT;
@@ -614,15 +640,18 @@ rte_service_lcore_reset_all(void)
 		if (lcore_states[i].is_service_core) {
 			lcore_states[i].service_mask = 0;
 			set_lcore_state(i, ROLE_RTE);
-			lcore_states[i].runstate = RUNSTATE_STOPPED;
+			/* runstate act as guard variable Use
+			 * store-release memory order here to synchronize
+			 * with load-acquire in runstate read functions.
+			 */
+			__atomic_store_n(&lcore_states[i].runstate,
+				RUNSTATE_STOPPED, __ATOMIC_RELEASE);
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
 		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
 				    __ATOMIC_RELAXED);
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -638,9 +667,11 @@ rte_service_lcore_add(uint32_t lcore)
 
 	/* ensure that after adding a core the mask and state are defaults */
 	lcore_states[lcore].service_mask = 0;
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
-
-	rte_smp_wmb();
+	/* Use store-release memory order here to synchronize with
+	 * load-acquire in runstate read functions.
+	 */
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+		__ATOMIC_RELEASE);
 
 	return rte_eal_wait_lcore(lcore);
 }
@@ -655,7 +686,12 @@ rte_service_lcore_del(uint32_t lcore)
 	if (!cs->is_service_core)
 		return -EINVAL;
 
-	if (cs->runstate != RUNSTATE_STOPPED)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_STOPPED)
 		return -EBUSY;
 
 	set_lcore_state(lcore, ROLE_RTE);
@@ -674,13 +710,21 @@ rte_service_lcore_start(uint32_t lcore)
 	if (!cs->is_service_core)
 		return -EINVAL;
 
-	if (cs->runstate == RUNSTATE_RUNNING)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING)
 		return -EALREADY;
 
 	/* set core to run state first, and then launch otherwise it will
 	 * return immediately as runstate keeps it in the service poll loop
 	 */
-	cs->runstate = RUNSTATE_RUNNING;
+	/* Use load-acquire memory order here to synchronize with
+	 * store-release in runstate update functions.
+	 */
+	__atomic_store_n(&cs->runstate, RUNSTATE_RUNNING, __ATOMIC_RELEASE);
 
 	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
@@ -693,7 +737,12 @@ rte_service_lcore_stop(uint32_t lcore)
 	if (lcore >= RTE_MAX_LCORE)
 		return -EINVAL;
 
-	if (lcore_states[lcore].runstate == RUNSTATE_STOPPED)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&lcore_states[lcore].runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_STOPPED)
 		return -EALREADY;
 
 	uint32_t i;
@@ -713,7 +762,11 @@ rte_service_lcore_stop(uint32_t lcore)
 			return -EBUSY;
 	}
 
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
+	/* Use store-release memory order here to synchronize with
+	 * load-acquire in runstate read functions.
+	 */
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+		__ATOMIC_RELEASE);
 
 	return 0;
 }
Signature · 1 line
2.7.4
author
#98 lore raw

v6 cover use c11 atomics for service core lib

In reply to Phil Yang

The rte_atomic ops and rte_smp barriers enforce DMB barriers on aarch64. Using c11 atomics with explicit memory ordering instead of the rte_atomic ops and rte_smp barriers for inter-threads synchronization can uplift the performance on aarch64 and no performance loss on x86.

This patchset contains:
1) fix race condition for MT unsafe service.
2) clean up redundant code.
3) use c11 atomics for service core lib to avoid unnecessary barriers.

v2:
Still waiting on Harry for the final solution on the MT unsafe race condition issue. But I have incorporated the comments so far.
1. add 'Fixes' tag for bug-fix patches.
2. remove 'Fixes' tag for code cleanup patches.
3. remove unused parameter for service_dump_one function.
4. replace the execute_lock atomic CAS operation to spinlock_try_lock.
5. use c11 atomics with RELAXED memory ordering for num_mapped_cores.
6. relax barriers for guard variables runstate, comp_runstate and app_runstate with c11 one-way barriers.

v3:
Sending this version since Phil is on holiday.
1. Updated the API documentation to indicate how the locking can be avoided.

v4:
1. Fix the nits in 2/6 commit message and comments in code.

v5:
1. Remove redundant libatomic clang dependency claim code as it has been added in global. (The commit da4eae278b56)

v6:
1. Fix coding style issue. Remove illegal Change-ID tag in patch 5/6.

Honnappa Nagarahalli (2):
service: fix race condition for MT unsafe service
service: fix identification of service running on other lcore

Phil Yang (4):
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 atomics
service: relax barriers with C11 atomics

 lib/librte_eal/common/rte_service.c            | 234 +++++++++++++------------
 lib/librte_eal/include/rte_service.h           |   8 +-
 lib/librte_eal/include/rte_service_component.h |   6 +-
 3 files changed, 137 insertions(+), 111 deletions(-)
Signature · 1 line
2.7.4
author
#99 lore raw

PATCH v6 1/6 service: fix race condition for MT unsafe service

From: Honnappa Nagarahalli <[email protected]>

The MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

If the service is known to be mmapped on a single lcore, setting the service capability to MT safe will avoid taking the lock and improve the performance.

Fixes e9139a32f6e8 ("service: add function to run on app lcore")
---
 lib/librte_eal/common/rte_service.c            | 11 +++++------
 lib/librte_eal/include/rte_service.h           |  8 ++++++--
 lib/librte_eal/include/rte_service_component.h |  6 +++++-
 3 files changed, 16 insertions(+), 9 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
index 70d17a5..b8c465e 100644
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {
 	uint8_t internal_flags;
 
 	/* per service statistics */
+	/* Indicates how many cores the service is mapped to run on.
+	 * It does not indicate the number of cores the service is running
+	 * on currently.
+	 */
 	rte_atomic32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	/* check do we need cmpset, if MT safe or <= 1 core
-	 * mapped, atomic ops are not required.
-	 */
-	const int use_atomics = (service_mt_safe(s) == 0) &&
-				(rte_atomic32_read(&s->num_mapped_cores) > 1);
-	if (use_atomics) {
+	if (service_mt_safe(s) == 0) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
lib/librte_eal/include/rte_service.h +6 -2
index d8701dd..3a1c735 100644
@@ -104,12 +104,16 @@ int32_t rte_service_probe_capability(uint32_t id, uint32_t capability);
  * Each core can be added or removed from running a specific service. This
  * function enables or disables *lcore* to run *service_id*.
  *
- * If multiple cores are enabled on a service, an atomic is used to ensure that
- * only one cores runs the service at a time. The exception to this is when
+ * If multiple cores are enabled on a service, a lock is used to ensure that
+ * only one core runs the service at a time. The exception to this is when
  * a service indicates that it is multi-thread safe by setting the capability
  * called RTE_SERVICE_CAP_MT_SAFE. With the multi-thread safe capability set,
  * the service function can be run on multiple threads at the same time.
  *
+ * If the service is known to be mapped to a single lcore, setting the
+ * capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve
+ * better performance by avoiding the use of lock.
+ *
  * @param service_id the service to apply the lcore to
  * @param lcore The lcore that will be mapped to service
  * @param enable Zero to unmap or disable the core, non-zero to enable
lib/librte_eal/include/rte_service_component.h +5 -1
index 16eab79..b75aba1 100644
@@ -43,7 +43,7 @@ struct rte_service_spec {
 /**
  * Register a new service.
  *
- * A service represents a component that the requires CPU time periodically to
+ * A service represents a component that requires CPU time periodically to
  * achieve its purpose.
  *
  * For example the eventdev SW PMD requires CPU cycles to perform its
@@ -56,6 +56,10 @@ struct rte_service_spec {
  * *rte_service_component_runstate_set*, which indicates that the service
  * component is ready to be executed.
  *
+ * If the service is known to be mapped to a single lcore, setting the
+ * capability of the service to RTE_SERVICE_CAP_MT_SAFE can achieve
+ * better performance.
+ *
  * @param spec The specification of the service to register
  * @param[out] service_id A pointer to a uint32_t, which will be filled in
  *             during registration of the service. It is set to the integers
Signature · 1 line
2.7.4
author
#100 lore raw

PATCH v6 2/6 service: fix identification of service running on other lcore

In reply to Phil Yang

From: Honnappa Nagarahalli <[email protected]>

The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduce the number of instructions for all cases.

Fixes 8d39d3e237c2 ("service: fix race in service on app lcore function")
---
 lib/librte_eal/common/rte_service.c | 26 ++++++++------------------
 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
index b8c465e..c283408 100644
@@ -360,7 +360,7 @@ rte_service_runner_do_callback(struct rte_service_spec_impl *s,
 /* Expects the service 's' is valid. */
 static int32_t
 service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
-	    struct rte_service_spec_impl *s)
+	    struct rte_service_spec_impl *s, uint32_t serialize_mt_unsafe)
 {
 	if (!s)
 		return -EINVAL;
@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	if (service_mt_safe(s) == 0) {
+	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
-	/* Atomically add this core to the mapped cores first, then examine if
-	 * we can run the service. This avoids a race condition between
-	 * checking the value, and atomically adding to the mapped count.
+	/* Increment num_mapped_cores to reflect that this core is
+	 * now mapped capable of running the service.
 	 */
-	if (serialize_mt_unsafe)
-		rte_atomic32_inc(&s->num_mapped_cores);
+	rte_atomic32_inc(&s->num_mapped_cores);
 
-	if (service_mt_safe(s) == 0 &&
-			rte_atomic32_read(&s->num_mapped_cores) > 1) {
-		if (serialize_mt_unsafe)
-			rte_atomic32_dec(&s->num_mapped_cores);
-		return -EBUSY;
-	}
-
-	int ret = service_run(id, cs, UINT64_MAX, s);
+	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	if (serialize_mt_unsafe)
-		rte_atomic32_dec(&s->num_mapped_cores);
+	rte_atomic32_dec(&s->num_mapped_cores);
 
 	return ret;
 }
@@ -449,7 +439,7 @@ rte_service_runner_func(void *arg)
 			if (!service_valid(i))
 				continue;
 			/* return value ignored as no change to code flow */
-			service_run(i, cs, service_mask, service_get(i));
+			service_run(i, cs, service_mask, service_get(i), 1);
 		}
 
 		cs->loops++;
Signature · 1 line
2.7.4
author
#101 lore raw

PATCH v6 3/6 service: remove rte prefix from static functions

In reply to Phil Yang

clean up rte prefix from static functions. remove unused parameter for service_dump_one function.

---
 lib/librte_eal/common/rte_service.c | 34 +++++++++++-----------------------
 1 file changed, 11 insertions(+), 23 deletions(-)
lib/librte_eal/common/rte_service.c +11 -23
index c283408..62ea9cb 100644
@@ -340,7 +340,7 @@ rte_service_runstate_get(uint32_t id)
 }
 
 static inline void
-rte_service_runner_do_callback(struct rte_service_spec_impl *s,
+service_runner_do_callback(struct rte_service_spec_impl *s,
 			       struct core_state *cs, uint32_t service_idx)
 {
 	void *userdata = s->spec.callback_userdata;
@@ -378,10 +378,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 		rte_atomic32_clear(&s->execute_lock);
 	} else
-		rte_service_runner_do_callback(s, cs, i);
+		service_runner_do_callback(s, cs, i);
 
 	return 0;
 }
@@ -425,14 +425,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 }
 
 static int32_t
-rte_service_runner_func(void *arg)
+service_runner_func(void *arg)
 {
 	RTE_SET_USED(arg);
 	uint32_t i;
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (lcore_states[lcore].runstate == RUNSTATE_RUNNING) {
+	while (cs->runstate == RUNSTATE_RUNNING) {
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
@@ -693,9 +693,9 @@ rte_service_lcore_start(uint32_t lcore)
 	/* set core to run state first, and then launch otherwise it will
 	 * return immediately as runstate keeps it in the service poll loop
 	 */
-	lcore_states[lcore].runstate = RUNSTATE_RUNNING;
+	cs->runstate = RUNSTATE_RUNNING;
 
-	int ret = rte_eal_remote_launch(rte_service_runner_func, 0, lcore);
+	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
 	return ret;
 }
@@ -774,13 +774,9 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id,
 }
 
 static void
-rte_service_dump_one(FILE *f, struct rte_service_spec_impl *s,
-		     uint64_t all_cycles, uint32_t reset)
+service_dump_one(FILE *f, struct rte_service_spec_impl *s, uint32_t reset)
 {
 	/* avoid divide by zero */
-	if (all_cycles == 0)
-		all_cycles = 1;
-
 	int calls = 1;
 	if (s->calls != 0)
 		calls = s->calls;
@@ -807,7 +803,7 @@ rte_service_attr_reset_all(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	int reset = 1;
-	rte_service_dump_one(NULL, s, 0, reset);
+	service_dump_one(NULL, s, reset);
 	return 0;
 }
 
@@ -851,21 +847,13 @@ rte_service_dump(FILE *f, uint32_t id)
 	uint32_t i;
 	int print_one = (id != UINT32_MAX);
 
-	uint64_t total_cycles = 0;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if (!service_valid(i))
-			continue;
-		total_cycles += rte_services[i].cycles_spent;
-	}
-
 	/* print only the specified service */
 	if (print_one) {
 		struct rte_service_spec_impl *s;
 		SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 		fprintf(f, "Service %s Summary\n", s->spec.name);
 		uint32_t reset = 0;
-		rte_service_dump_one(f, s, total_cycles, reset);
+		service_dump_one(f, s, reset);
 		return 0;
 	}
 
@@ -875,7 +863,7 @@ rte_service_dump(FILE *f, uint32_t id)
 		if (!service_valid(i))
 			continue;
 		uint32_t reset = 0;
-		rte_service_dump_one(f, &rte_services[i], total_cycles, reset);
+		service_dump_one(f, &rte_services[i], reset);
 	}
 
 	fprintf(f, "Service Cores Summary\n");
Signature · 1 line
2.7.4
author
#102 lore raw

PATCH v6 4/6 service: remove redundant code

In reply to Phil Yang

The service id validation is duplicated, remove the redundant code in the calling functions.

---
 lib/librte_eal/common/rte_service.c | 28 ++++++----------------------
 1 file changed, 6 insertions(+), 22 deletions(-)
lib/librte_eal/common/rte_service.c +6 -22
index 62ea9cb..37c16c4 100644
@@ -541,24 +541,12 @@ rte_service_start_with_defaults(void)
 }
 
 static int32_t
-service_update(struct rte_service_spec *service, uint32_t lcore,
+service_update(uint32_t sid, uint32_t lcore,
 		uint32_t *set, uint32_t *enabled)
 {
-	uint32_t i;
-	int32_t sid = -1;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if ((struct rte_service_spec *)&rte_services[i] == service &&
-				service_valid(i)) {
-			sid = i;
-			break;
-		}
-	}
-
-	if (sid == -1 || lcore >= RTE_MAX_LCORE)
-		return -EINVAL;
-
-	if (!lcore_states[lcore].is_service_core)
+	/* validate ID, or return error value */
+	if (sid >= RTE_SERVICE_NUM_MAX || !service_valid(sid) ||
+	    lcore >= RTE_MAX_LCORE || !lcore_states[lcore].is_service_core)
 		return -EINVAL;
 
 	uint64_t sid_mask = UINT64_C(1) << sid;
@@ -587,19 +575,15 @@ service_update(struct rte_service_spec *service, uint32_t lcore,
 int32_t
 rte_service_map_lcore_set(uint32_t id, uint32_t lcore, uint32_t enabled)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 	uint32_t on = enabled > 0;
-	return service_update(&s->spec, lcore, &on, 0);
+	return service_update(id, lcore, &on, 0);
 }
 
 int32_t
 rte_service_map_lcore_get(uint32_t id, uint32_t lcore)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 	uint32_t enabled;
-	int ret = service_update(&s->spec, lcore, 0, &enabled);
+	int ret = service_update(id, lcore, 0, &enabled);
 	if (ret == 0)
 		return enabled;
 	return ret;
Signature · 1 line
2.7.4
author
#103 lore raw

PATCH v6 5/6 service: optimize with c11 atomics

In reply to Phil Yang

The num_mapped_cores is used as a statistics. Use c11 atomics with RELAXED ordering for num_mapped_cores instead of rte_atomic ops which enforce unnessary barriers on aarch64.

Replace execute_lock operations to spinlock_try_lock to avoid duplicate code.

---
 lib/librte_eal/common/rte_service.c | 32 ++++++++++++++++++--------------
 1 file changed, 18 insertions(+), 14 deletions(-)
lib/librte_eal/common/rte_service.c +18 -14
index 37c16c4..5d35f8a 100644
@@ -20,6 +20,7 @@
 #include <rte_atomic.h>
 #include <rte_memory.h>
 #include <rte_malloc.h>
+#include <rte_spinlock.h>
 
 #include "eal_private.h"
 
@@ -38,11 +39,11 @@ struct rte_service_spec_impl {
 	/* public part of the struct */
 	struct rte_service_spec spec;
 
-	/* atomic lock that when set indicates a service core is currently
+	/* spin lock that when set indicates a service core is currently
 	 * running this service callback. When not set, a core may take the
 	 * lock and then run the service callback.
 	 */
-	rte_atomic32_t execute_lock;
+	rte_spinlock_t execute_lock;
 
 	/* API set/get-able variables */
 	int8_t app_runstate;
@@ -54,7 +55,7 @@ struct rte_service_spec_impl {
 	 * It does not indicate the number of cores the service is running
 	 * on currently.
 	 */
-	rte_atomic32_t num_mapped_cores;
+	uint32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
 } __rte_cache_aligned;
@@ -332,7 +333,8 @@ rte_service_runstate_get(uint32_t id)
 	rte_smp_rmb();
 
 	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (rte_atomic32_read(&s->num_mapped_cores) > 0);
+	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+					    __ATOMIC_RELAXED) > 0);
 
 	return (s->app_runstate == RUNSTATE_RUNNING) &&
 		(s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -375,11 +377,11 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 	cs->service_active_on_lcore[i] = 1;
 
 	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
-		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
+		if (!rte_spinlock_trylock(&s->execute_lock))
 			return -EBUSY;
 
 		service_runner_do_callback(s, cs, i);
-		rte_atomic32_clear(&s->execute_lock);
+		rte_spinlock_unlock(&s->execute_lock);
 	} else
 		service_runner_do_callback(s, cs, i);
 
@@ -415,11 +417,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 	/* Increment num_mapped_cores to reflect that this core is
 	 * now mapped capable of running the service.
 	 */
-	rte_atomic32_inc(&s->num_mapped_cores);
+	__atomic_add_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELAXED);
 
 	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	rte_atomic32_dec(&s->num_mapped_cores);
+	__atomic_sub_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELAXED);
 
 	return ret;
 }
@@ -556,19 +558,19 @@ service_update(uint32_t sid, uint32_t lcore,
 
 		if (*set && !lcore_mapped) {
 			lcore_states[lcore].service_mask |= sid_mask;
-			rte_atomic32_inc(&rte_services[sid].num_mapped_cores);
+			__atomic_add_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELAXED);
 		}
 		if (!*set && lcore_mapped) {
 			lcore_states[lcore].service_mask &= ~(sid_mask);
-			rte_atomic32_dec(&rte_services[sid].num_mapped_cores);
+			__atomic_sub_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELAXED);
 		}
 	}
 
 	if (enabled)
 		*enabled = !!(lcore_states[lcore].service_mask & (sid_mask));
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -616,7 +618,8 @@ rte_service_lcore_reset_all(void)
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
-		rte_atomic32_set(&rte_services[i].num_mapped_cores, 0);
+		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
+				    __ATOMIC_RELAXED);
 
 	rte_smp_wmb();
 
@@ -699,7 +702,8 @@ rte_service_lcore_stop(uint32_t lcore)
 		int32_t enabled = service_mask & (UINT64_C(1) << i);
 		int32_t service_running = rte_service_runstate_get(i);
 		int32_t only_core = (1 ==
-			rte_atomic32_read(&rte_services[i].num_mapped_cores));
+			__atomic_load_n(&rte_services[i].num_mapped_cores,
+					__ATOMIC_RELAXED));
 
 		/* if the core is mapped, and the service is running, and this
 		 * is the only core that is mapped, the service would cease to
Signature · 1 line
2.7.4
author
#104 lore raw

PATCH v6 6/6 service: relax barriers with C11 atomics

In reply to Phil Yang

The runstate, comp_runstate and app_runstate are used as guard variables in the service core lib. To guarantee the inter-threads visibility of these guard variables, it uses rte_smp_r/wmb. This patch use c11 atomic built-ins to relax these barriers.

---
 lib/librte_eal/common/rte_service.c | 115 ++++++++++++++++++++++++++----------
 1 file changed, 84 insertions(+), 31 deletions(-)
lib/librte_eal/common/rte_service.c +84 -31
index 5d35f8a..3bae7d6 100644
@@ -265,7 +265,6 @@ rte_service_component_register(const struct rte_service_spec *spec,
 	s->spec = *spec;
 	s->internal_flags |= SERVICE_F_REGISTERED | SERVICE_F_START_CHECK;
 
-	rte_smp_wmb();
 	rte_service_count++;
 
 	if (id_ptr)
@@ -282,7 +281,6 @@ rte_service_component_unregister(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	rte_service_count--;
-	rte_smp_wmb();
 
 	s->internal_flags &= ~(SERVICE_F_REGISTERED);
 
@@ -301,12 +299,17 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
+	/* comp_runstate act as the guard variable. Use store-release
+	 * memory order. This synchronizes with load-acquire in
+	 * service_run and service_runstate_get function.
+	 */
 	if (runstate)
-		s->comp_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->comp_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -316,12 +319,17 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
+	/* app_runstate act as the guard variable. Use store-release
+	 * memory order. This synchronizes with load-acquire in
+	 * service_run runstate_get function.
+	 */
 	if (runstate)
-		s->app_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->app_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -330,15 +338,24 @@ rte_service_runstate_get(uint32_t id)
 {
 	struct rte_service_spec_impl *s;
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
-	rte_smp_rmb();
 
-	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+	/* comp_runstate and app_runstate act as the guard variables.
+	 * Use load-acquire memory order. This synchronizes with
+	 * store-release in service state set functions.
+	 */
+	if (__atomic_load_n(&s->comp_runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING &&
+		 __atomic_load_n(&s->app_runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {
+		int check_disabled = !(s->internal_flags &
+					SERVICE_F_START_CHECK);
+		int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
 					    __ATOMIC_RELAXED) > 0);
 
-	return (s->app_runstate == RUNSTATE_RUNNING) &&
-		(s->comp_runstate == RUNSTATE_RUNNING) &&
-		(check_disabled | lcore_mapped);
+		return (check_disabled | lcore_mapped);
+	} else
+		return 0;
+
 }
 
 static inline void
@@ -367,9 +384,15 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 	if (!s)
 		return -EINVAL;
 
-	if (s->comp_runstate != RUNSTATE_RUNNING ||
-			s->app_runstate != RUNSTATE_RUNNING ||
-			!(service_mask & (UINT64_C(1) << i))) {
+	/* comp_runstate and app_runstate act as the guard variables.
+	 * Use load-acquire memory order. This synchronizes with
+	 * store-release in service state set functions.
+	 */
+	if (__atomic_load_n(&s->comp_runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_RUNNING ||
+		 __atomic_load_n(&s->app_runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_RUNNING ||
+		!(service_mask & (UINT64_C(1) << i))) {
 		cs->service_active_on_lcore[i] = 0;
 		return -ENOEXEC;
 	}
@@ -434,7 +457,12 @@ service_runner_func(void *arg)
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (cs->runstate == RUNSTATE_RUNNING) {
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	while (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
@@ -445,8 +473,6 @@ service_runner_func(void *arg)
 		}
 
 		cs->loops++;
-
-		rte_smp_rmb();
 	}
 
 	lcore_config[lcore].state = WAIT;
@@ -614,15 +640,18 @@ rte_service_lcore_reset_all(void)
 		if (lcore_states[i].is_service_core) {
 			lcore_states[i].service_mask = 0;
 			set_lcore_state(i, ROLE_RTE);
-			lcore_states[i].runstate = RUNSTATE_STOPPED;
+			/* runstate act as guard variable Use
+			 * store-release memory order here to synchronize
+			 * with load-acquire in runstate read functions.
+			 */
+			__atomic_store_n(&lcore_states[i].runstate,
+				RUNSTATE_STOPPED, __ATOMIC_RELEASE);
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
 		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
 				    __ATOMIC_RELAXED);
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -638,9 +667,11 @@ rte_service_lcore_add(uint32_t lcore)
 
 	/* ensure that after adding a core the mask and state are defaults */
 	lcore_states[lcore].service_mask = 0;
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
-
-	rte_smp_wmb();
+	/* Use store-release memory order here to synchronize with
+	 * load-acquire in runstate read functions.
+	 */
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+		__ATOMIC_RELEASE);
 
 	return rte_eal_wait_lcore(lcore);
 }
@@ -655,7 +686,12 @@ rte_service_lcore_del(uint32_t lcore)
 	if (!cs->is_service_core)
 		return -EINVAL;
 
-	if (cs->runstate != RUNSTATE_STOPPED)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) != RUNSTATE_STOPPED)
 		return -EBUSY;
 
 	set_lcore_state(lcore, ROLE_RTE);
@@ -674,13 +710,21 @@ rte_service_lcore_start(uint32_t lcore)
 	if (!cs->is_service_core)
 		return -EINVAL;
 
-	if (cs->runstate == RUNSTATE_RUNNING)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&cs->runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_RUNNING)
 		return -EALREADY;
 
 	/* set core to run state first, and then launch otherwise it will
 	 * return immediately as runstate keeps it in the service poll loop
 	 */
-	cs->runstate = RUNSTATE_RUNNING;
+	/* Use load-acquire memory order here to synchronize with
+	 * store-release in runstate update functions.
+	 */
+	__atomic_store_n(&cs->runstate, RUNSTATE_RUNNING, __ATOMIC_RELEASE);
 
 	int ret = rte_eal_remote_launch(service_runner_func, 0, lcore);
 	/* returns -EBUSY if the core is already launched, 0 on success */
@@ -693,7 +737,12 @@ rte_service_lcore_stop(uint32_t lcore)
 	if (lcore >= RTE_MAX_LCORE)
 		return -EINVAL;
 
-	if (lcore_states[lcore].runstate == RUNSTATE_STOPPED)
+	/* runstate act as the guard variable. Use load-acquire
+	 * memory order here to synchronize with store-release
+	 * in runstate update functions.
+	 */
+	if (__atomic_load_n(&lcore_states[lcore].runstate,
+			__ATOMIC_ACQUIRE) == RUNSTATE_STOPPED)
 		return -EALREADY;
 
 	uint32_t i;
@@ -713,7 +762,11 @@ rte_service_lcore_stop(uint32_t lcore)
 			return -EBUSY;
 	}
 
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
+	/* Use store-release memory order here to synchronize with
+	 * load-acquire in runstate read functions.
+	 */
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+		__ATOMIC_RELEASE);
 
 	return 0;
 }
Signature · 1 line
2.7.4
#105 lore raw

Re: [dpdk-dev] [PATCH v6 0/6] use c11 atomics for service core lib

In reply to Phil Yang
Quoted 10 lines · Phil Yang

The rte_atomic ops and rte_smp barriers enforce DMB barriers on aarch64. Using c11 atomics with explicit memory ordering instead of the rte_atomic ops and rte_smp barriers for inter-threads synchronization can uplift the performance on aarch64 and no performance loss on x86.

This patchset contains:
1) fix race condition for MT unsafe service.
2) clean up redundant code.
3) use c11 atomics for service core lib to avoid unnecessary barriers.

Series applied, thanks for the cleanup and fixes.

Signature · 1 line
David Marchand
author
#106 lore raw

PATCH v3 8/12 service: remove redundant code

In reply to Phil Yang

The service id validation is verified in the calling function, remove the redundant code inside the service_update function.

Fixes 21698354c832 ("service: introduce service cores concept")
---
 lib/librte_eal/common/rte_service.c | 31 ++++++++++++-------------------
 1 file changed, 12 insertions(+), 19 deletions(-)
lib/librte_eal/common/rte_service.c +12 -19
index 2117726..557b5a9 100644
@@ -552,21 +552,10 @@ rte_service_start_with_defaults(void)
 }
 
 static int32_t
-service_update(struct rte_service_spec *service, uint32_t lcore,
+service_update(uint32_t sid, uint32_t lcore,
 		uint32_t *set, uint32_t *enabled)
 {
-	uint32_t i;
-	int32_t sid = -1;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if ((struct rte_service_spec *)&rte_services[i] == service &&
-				service_valid(i)) {
-			sid = i;
-			break;
-		}
-	}
-
-	if (sid == -1 || lcore >= RTE_MAX_LCORE)
+	if (lcore >= RTE_MAX_LCORE)
 		return -EINVAL;
 
 	if (!lcore_states[lcore].is_service_core)
@@ -598,19 +587,23 @@ service_update(struct rte_service_spec *service, uint32_t lcore,
 int32_t
 rte_service_map_lcore_set(uint32_t id, uint32_t lcore, uint32_t enabled)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
+	/* validate ID, or return error value */
+	if (id >= RTE_SERVICE_NUM_MAX || !service_valid(id))
+		return -EINVAL;
+
 	uint32_t on = enabled > 0;
-	return service_update(&s->spec, lcore, &on, 0);
+	return service_update(id, lcore, &on, 0);
 }
 
 int32_t
 rte_service_map_lcore_get(uint32_t id, uint32_t lcore)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
+	/* validate ID, or return error value */
+	if (id >= RTE_SERVICE_NUM_MAX || !service_valid(id))
+		return -EINVAL;
+
 	uint32_t enabled;
-	int ret = service_update(&s->spec, lcore, 0, &enabled);
+	int ret = service_update(id, lcore, 0, &enabled);
 	if (ret == 0)
 		return enabled;
 	return ret;
Signature · 1 line
2.7.4
#107 lore raw

Re: [dpdk-dev] [PATCH v3 08/12] service: remove redundant code

Quoted 10 lines

From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 08/12] service: remove redundant code

Quoted 8 lines

The service id validation is verified in the calling function, remove the redundant code inside the service_update function.

Fixes 21698354c832 ("service: introduce service cores concept")

Same comment as patch 7/12, is this really a "Fix"? This functionality is not "broken" in the current code? And is there value in porting to stable? I'd see this as unnecessary churn.

As before, it is a valid cleanup (thanks), and I'd like to take it for new DPDK releases.

Happy to Ack without Fixes or Cc Stable, if that's acceptable to you?

Quoted 66 lines
---
 lib/librte_eal/common/rte_service.c | 31 ++++++++++++-------------------
 1 file changed, 12 insertions(+), 19 deletions(-)
lib/librte_eal/common/rte_service.c +12 -19
index 2117726..557b5a9 100644
@@ -552,21 +552,10 @@ rte_service_start_with_defaults(void)
 }
 
 static int32_t
-service_update(struct rte_service_spec *service, uint32_t lcore,
+service_update(uint32_t sid, uint32_t lcore,
 		uint32_t *set, uint32_t *enabled)
 {
-	uint32_t i;
-	int32_t sid = -1;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if ((struct rte_service_spec *)&rte_services[i] == service &&
-				service_valid(i)) {
-			sid = i;
-			break;
-		}
-	}
-
-	if (sid == -1 || lcore >= RTE_MAX_LCORE)
+	if (lcore >= RTE_MAX_LCORE)
 		return -EINVAL;
 
 	if (!lcore_states[lcore].is_service_core)
@@ -598,19 +587,23 @@ service_update(struct rte_service_spec *service,
 uint32_t lcore,
 int32_t
 rte_service_map_lcore_set(uint32_t id, uint32_t lcore, uint32_t enabled)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
+	/* validate ID, or return error value */
+	if (id >= RTE_SERVICE_NUM_MAX || !service_valid(id))
+		return -EINVAL;
+
 	uint32_t on = enabled > 0;
-	return service_update(&s->spec, lcore, &on, 0);
+	return service_update(id, lcore, &on, 0);
 }
 
 int32_t
 rte_service_map_lcore_get(uint32_t id, uint32_t lcore)
 {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
+	/* validate ID, or return error value */
+	if (id >= RTE_SERVICE_NUM_MAX || !service_valid(id))
+		return -EINVAL;
+
 	uint32_t enabled;
-	int ret = service_update(&s->spec, lcore, 0, &enabled);
+	int ret = service_update(id, lcore, 0, &enabled);
 	if (ret == 0)
 		return enabled;

return ret;
--
2.7.4

#108 lore raw

Re: [dpdk-dev] [PATCH v3 08/12] service: remove redundant code

<snip>

Quoted 11 lines
Quoted 9 lines

The service id validation is verified in the calling function, remove the redundant code inside the service_update function.

Fixes 21698354c832 ("service: introduce service cores concept")
Quoted 8 lines

Same comment as patch 7/12, is this really a "Fix"? This functionality is not "broken" in the current code? And is there value in porting to stable? I'd see this as unnecessary churn.

As before, it is a valid cleanup (thanks), and I'd like to take it for new DPDK releases.

Happy to Ack without Fixes or Cc Stable, if that's acceptable to you?

Agreed.

Quoted 16 lines
Quoted 16 lines


---
 lib/librte_eal/common/rte_service.c | 31
++++++++++++-------------------
 1 file changed, 12 insertions(+), 19 deletions(-)
lib/librte_eal/common/rte_service.c +0 -0
index 2117726..557b5a9 100644
@@ -552,21 +552,10 @@ rte_service_start_with_defaults(void)
 }
 
Quoted 4 lines

lib/librte_eal/common/rte_service.c rte_service_start_with_defaults(void) · line 556

Quoted 4 lines
lib/librte_eal/common/rte_service.c +1 -1
 static int32_t
-service_update(struct rte_service_spec *service, uint32_t lcore,
+service_update(uint32_t sid, uint32_t lcore,
 		uint32_t *set, uint32_t *enabled)

'set' parameter does not need be passed by reference, pass by value is enough.

Quoted 7 lines
Quoted 7 lines
lib/librte_eal/common/rte_service.c +0 -6
 {
-	uint32_t i;
-	int32_t sid = -1;
-
-	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-		if ((struct rte_service_spec *)&rte_services[i] == service &&
-				service_valid(i)) {
Quoted 8 lines

lib/librte_eal/common/rte_service.c rte_service_start_with_defaults(void) · line 559

Quoted 8 lines
lib/librte_eal/common/rte_service.c +1 -6
-			sid = i;
-			break;
-		}
-	}
-
-	if (sid == -1 || lcore >= RTE_MAX_LCORE)
+	if (lcore >= RTE_MAX_LCORE)
 		return -EINVAL;

The validations look somewhat inconsistent in service_update function, we are validating some parameters and not some. Suggest bringing the validation of the service id also into this function and remove it from the calling functions.

Quoted 12 lines

lib/librte_eal/common/rte_service.c service_update(struct rte_service_spec *service, · line 593

Quoted 12 lines
lib/librte_eal/common/rte_service.c +4 -2
 	if (!lcore_states[lcore].is_service_core)
@@ -598,19 +587,23 @@ service_update(struct rte_service_spec *service,
 uint32_t lcore,  int32_t  rte_service_map_lcore_set(uint32_t id,
 uint32_t lcore, uint32_t enabled)  {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
+	/* validate ID, or return error value */
+	if (id >= RTE_SERVICE_NUM_MAX || !service_valid(id))
+		return -EINVAL;
+
 	uint32_t on = enabled > 0;

We do not need the above line. 'enabled' can be passed directly to 'service_update'.

Quoted 20 lines
Quoted 20 lines
lib/librte_eal/common/rte_service.c +6 -5
-	return service_update(&s->spec, lcore, &on, 0);
+	return service_update(id, lcore, &on, 0);
 }
 
 int32_t
 rte_service_map_lcore_get(uint32_t id, uint32_t lcore)  {
-	struct rte_service_spec_impl *s;
-	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
+	/* validate ID, or return error value */
+	if (id >= RTE_SERVICE_NUM_MAX || !service_valid(id))
+		return -EINVAL;
+
 	uint32_t enabled;
-	int ret = service_update(&s->spec, lcore, 0, &enabled);
+	int ret = service_update(id, lcore, 0, &enabled);
 	if (ret == 0)
 		return enabled;
 	return ret;
--

2.7.4

author
#109 lore raw

Re: [dpdk-dev] [PATCH v3 08/12] service: remove redundant code

Quoted 29 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Monday, April 6, 2020 2:35 AM
To: Van Haaren, Harry <[email protected]>; Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 08/12] service: remove redundant code

<snip>

Quoted 7 lines
Quoted 7 lines

The service id validation is verified in the calling function, remove the redundant code inside the service_update function.

Fixes 21698354c832 ("service: introduce service cores concept")
Quoted 6 lines
Quoted 2 lines

Same comment as patch 7/12, is this really a "Fix"? This functionality is not "broken" in the current code? And is there value in porting to stable? I'd

Quoted 8 lines

see

Quoted 6 lines

this as unnecessary churn.

As before, it is a valid cleanup (thanks), and I'd like to take it for new DPDK releases.

Happy to Ack without Fixes or Cc Stable, if that's acceptable to you?

Agreed.

Agreed.

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines



---
 lib/librte_eal/common/rte_service.c | 31
++++++++++++-------------------
 1 file changed, 12 insertions(+), 19 deletions(-)
lib/librte_eal/common/rte_service.c +0 -0
index 2117726..557b5a9 100644
@@ -552,21 +552,10 @@ rte_service_start_with_defaults(void)
 }
 
Quoted 6 lines
Quoted 4 lines

lib/librte_eal/common/rte_service.c rte_service_start_with_defaults(void) · line 556

Quoted 4 lines
lib/librte_eal/common/rte_service.c +1 -1
 static int32_t
-service_update(struct rte_service_spec *service, uint32_t lcore,
+service_update(uint32_t sid, uint32_t lcore,
 uint32_t *set, uint32_t *enabled)

'set' parameter does not need be passed by reference, pass by value is enough.

Agreed.

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
lib/librte_eal/common/rte_service.c +0 -10
 {
-uint32_t i;
-int32_t sid = -1;
-
-for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
-if ((struct rte_service_spec *)&rte_services[i] == service &&
-service_valid(i)) {
-sid = i;
-break;
-}
-}
Quoted 8 lines
Quoted 4 lines

lib/librte_eal/common/rte_service.c rte_service_start_with_defaults(void) · line 559

Quoted 4 lines
lib/librte_eal/common/rte_service.c +1 -2
-
-if (sid == -1 || lcore >= RTE_MAX_LCORE)
+if (lcore >= RTE_MAX_LCORE)
 return -EINVAL;

The validations look somewhat inconsistent in service_update function, we are validating some parameters and not some.
Suggest bringing the validation of the service id also into this function and remove it from the calling functions.

Agreed. I will update it in the next version.

Quoted 5 lines
Quoted 4 lines

lib/librte_eal/common/rte_service.c service_update(struct rte_service_spec

Quoted 4 lines
lib/librte_eal/common/rte_service.c +0 -0
 if (!lcore_states[lcore].is_service_core)
@@ -598,19 +587,23 @@ service_update(struct rte_service_spec

*service,

Quoted 11 lines
Quoted 9 lines

lib/librte_eal/common/rte_service.c service_update(struct rte_service_spec *service, uint32_t lcore, · line 594

Quoted 9 lines
lib/librte_eal/common/rte_service.c +4 -2
 uint32_t lcore,  int32_t  rte_service_map_lcore_set(uint32_t id,
 uint32_t lcore, uint32_t enabled)  {
-struct rte_service_spec_impl *s;
-SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
+/* validate ID, or return error value */
+if (id >= RTE_SERVICE_NUM_MAX || !service_valid(id))
+return -EINVAL;
+
 uint32_t on = enabled > 0;

We do not need the above line. 'enabled' can be passed directly to 'service_update'.

Agreed.

Quoted 22 lines
Quoted 21 lines
Quoted 21 lines
lib/librte_eal/common/rte_service.c +6 -5
-return service_update(&s->spec, lcore, &on, 0);
+return service_update(id, lcore, &on, 0);
 }
 
 int32_t
 rte_service_map_lcore_get(uint32_t id, uint32_t lcore)  {
-struct rte_service_spec_impl *s;
-SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
+/* validate ID, or return error value */
+if (id >= RTE_SERVICE_NUM_MAX || !service_valid(id))
+return -EINVAL;
+
 uint32_t enabled;
-int ret = service_update(&s->spec, lcore, 0, &enabled);
+int ret = service_update(id, lcore, 0, &enabled);
 if (ret == 0)
 return enabled;
 return ret;
--

2.7.4

author
#110 lore raw

PATCH v3 9/12 service: avoid race condition for MT unsafe service

In reply to Phil Yang

From: Honnappa Nagarahalli <[email protected]>

There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Fixes e9139a32f6e8 ("service: add function to run on app lcore")
---
 lib/librte_eal/common/rte_service.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
index 557b5a9..32a2f8a 100644
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {
 	uint8_t internal_flags;
 
 	/* per service statistics */
+	/* Indicates how many cores the service is mapped to run on.
+	 * It does not indicate the number of cores the service is running
+	 * on currently.
+	 */
 	rte_atomic32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	/* check do we need cmpset, if MT safe or <= 1 core
-	 * mapped, atomic ops are not required.
-	 */
-	const int use_atomics = (service_mt_safe(s) == 0) &&
-				(rte_atomic32_read(&s->num_mapped_cores) > 1);
-	if (use_atomics) {
+	if (service_mt_safe(s) == 0) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
Signature · 1 line
2.7.4
#111 lore raw

Re: [dpdk-dev] [PATCH v3 09/12] service: avoid race condition for MT unsafe service

Quoted 19 lines

From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: [PATCH v3 09/12] service: avoid race condition for MT unsafe service

From: Honnappa Nagarahalli <[email protected]>

There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")

We should put "fix" in the title, once converged on an implementation.

Regarding Fixes and stable backport, we should consider if fixing this in stable with a performance degradation, fixing with more complex solution, or documenting a known issue a better solution.

This fix (always taking the atomic lock) will have a negative performance impact on existing code using services. We should investigate a way to fix it without causing datapath performance degradation.

I think there is a way to achieve this by moving more checks/time to the control path (lcore updating the map), and not forcing the datapath lcore to always take an atomic.

In this particular case, we have a counter for number of iterations that a service has done. If this increments we know that the lcore running the service has re-entered the critical section, so would see an updated "needs atomic" flag.

This approach may introduce a predictable branch on the datapath, however the cost of a predictable branch vs always taking an atomic is order(s?) of magnitude, so a branch is much preferred.

It must be possible to avoid the datapath overhead using a scheme like this. It will likely be more complex than your proposed change below, however if it avoids datapath performance drops I feel that a more complex solution is worth investigating at least.

A unit test is required to validate a fix like this - although perhaps found by inspection/review, a real-world test to validate would give confidence.

Thoughts on such an approach?

Quoted 37 lines
---
 lib/librte_eal/common/rte_service.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
lib/librte_eal/common/rte_service.c +5 -7
index 557b5a9..32a2f8a 100644
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {
 	uint8_t internal_flags;
 
 	/* per service statistics */
+	/* Indicates how many cores the service is mapped to run on.
+	 * It does not indicate the number of cores the service is running
+	 * on currently.
+	 */
 	rte_atomic32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t
 service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	/* check do we need cmpset, if MT safe or <= 1 core
-	 * mapped, atomic ops are not required.
-	 */
-	const int use_atomics = (service_mt_safe(s) == 0) &&
-				(rte_atomic32_read(&s->num_mapped_cores) > 1);
-	if (use_atomics) {
+	if (service_mt_safe(s) == 0) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
--

2.7.4

#112 lore raw

Re: [dpdk-dev] [PATCH v3 09/12] service: avoid race condition for MT unsafe service

<snip>

Quoted 11 lines
Quoted 11 lines

Subject: [PATCH v3 09/12] service: avoid race condition for MT unsafe service

From: Honnappa Nagarahalli <[email protected]>

There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Quoted 8 lines
Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")

We should put "fix" in the title, once converged on an implementation.

Ok, will replace 'avoid' with 'fix' (once we agree on the solution)

Quoted 9 lines

Regarding Fixes and stable backport, we should consider if fixing this in stable with a performance degradation, fixing with more complex solution, or documenting a known issue a better solution.

This fix (always taking the atomic lock) will have a negative performance impact on existing code using services. We should investigate a way to fix it without causing datapath performance degradation.

Trying to gauge the impact on the existing applications...
The documentation does not explicitly disallow run time mapping of cores to service.
1) If the applications are mapping the cores to services at run time, they are running with a bug. IMO, bug fix resulting in a performance drop should be acceptable.
2) If the service is configured to run on single core (num_mapped_cores == 1), but service is set to MT unsafe - this will have a (possible) performance impact.
a) This can be solved by setting the service to MT safe and can be documented. This might be a reasonable solution for applications which are compiling with
future DPDK releases.
b) We can also solve this using symbol versioning - the old version of this function will use the old code, the new version of this function will use the code in
this patch. So, if the application is run with future DPDK releases without recompiling, it will continue to use the old version. If the application is compiled
with future releases, they can use solution in 2a. We also should think if this is an appropriate solution as this would force 1) to recompile to get the fix.
3) If the service is configured to run on multiple cores (num_mapped_cores > 1), then for those applications, the lock is being taken already. These applications might see some improvements as this patch removes few instructions.

Quoted 4 lines

I think there is a way to achieve this by moving more checks/time to the control path (lcore updating the map), and not forcing the datapath lcore to always take an atomic.

I think 2a above is the solution.

Quoted 2 lines

In this particular case, we have a counter for number of iterations that a

Which counter are you thinking about?
All the counters I checked are not atomic operations currently. If we are going to use counters they have to be atomic, which means additional cycles in the data path.

Quoted 4 lines

service has done. If this increments we know that the lcore running the service has re-entered the critical section, so would see an updated "needs atomic" flag.

Quoted 8 lines

This approach may introduce a predictable branch on the datapath, however the cost of a predictable branch vs always taking an atomic is order(s?) of magnitude, so a branch is much preferred.

It must be possible to avoid the datapath overhead using a scheme like this. It will likely be more complex than your proposed change below, however if it avoids datapath performance drops I feel that a more complex solution is worth investigating at least.

I do not completely understand the approach you are proposing, may be you can elaborate more. But, it seems to be based on a counter approach. Following is my assessment on what happens if we use a counter. Let us say we kept track of how many cores are running the service currently. We need an atomic counter other than 'num_mapped_cores'. Let us call that counter 'num_current_cores'. The code to call the service would look like below.

1) rte_atomic32_inc(&num_current_cores); /* this results in a full memory barrier */
2) if (__atomic_load_n(&num_current_cores, __ATOMIC_ACQUIRE) == 1) { /* rte_atomic_read is not enough here as it does not provide the required memory barrier for any architecture */
3) run_service(); /* Call the service */
4) }
5) rte_atomic32_sub(&num_current_cores); /* Calling rte_atomic32_clear is not enough as it is not an atomic operation and does not provide the required memory barrier */

But the above code has race conditions in lines 1 and 2. It is possible that none of the cores will ever get to run the service as they all could simultaneously increment the counter. Hence lines 1 and 2 together need to be atomic, which is nothing but 'compare-exchange' operation.

BTW, the current code has a bug where it calls 'rte_atomic_clear(&s->execute_lock)', it is missing memory barriers which results in clearing the execute_lock before the service has completed running. I suggest changing the 'execute_lock' to rte_spinlock_t and using rte_spinlock_try_lock and rte_spinlock_unlock APIs.

Quoted 3 lines

A unit test is required to validate a fix like this - although perhaps found by inspection/review, a real-world test to validate would give confidence.

Agree, need to have a test case.

Quoted 44 lines

Thoughts on such an approach?

Quoted 38 lines
---
 lib/librte_eal/common/rte_service.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
lib/librte_eal/common/rte_service.c +5 -6
index 557b5a9..32a2f8a 100644
@@ -50,6 +50,10 @@ struct rte_service_spec_impl {
 	uint8_t internal_flags;
 
 	/* per service statistics */
+	/* Indicates how many cores the service is mapped to run on.
+	 * It does not indicate the number of cores the service is running
+	 * on currently.
+	 */
 	rte_atomic32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
@@ -370,12 +374,7 @@ service_run(uint32_t i, struct core_state *cs,
 uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	/* check do we need cmpset, if MT safe or <= 1 core
-	 * mapped, atomic ops are not required.
-	 */
-	const int use_atomics = (service_mt_safe(s) == 0) &&
-				(rte_atomic32_read(&s-
 num_mapped_cores) > 1);
-	if (use_atomics) {
+	if (service_mt_safe(s) == 0) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))

return -EBUSY;

--
2.7.4

#113 lore raw

Re: [dpdk-dev] [PATCH v3 09/12] service: avoid race condition for MT unsafe service

Quoted 76 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, April 4, 2020 7:03 PM
To: Van Haaren, Harry <[email protected]>; Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 09/12] service: avoid race condition for MT unsafe service

<snip>

Quoted 11 lines
Quoted 11 lines

Subject: [PATCH v3 09/12] service: avoid race condition for MT unsafe service

From: Honnappa Nagarahalli <[email protected]>

There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Quoted 8 lines
Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")

We should put "fix" in the title, once converged on an implementation.

Ok, will replace 'avoid' with 'fix' (once we agree on the solution)

Quoted 2 lines

Regarding Fixes and stable backport, we should consider if fixing this in

stable

Quoted 6 lines

with a performance degradation, fixing with more complex solution, or documenting a known issue a better solution.

This fix (always taking the atomic lock) will have a negative performance impact on existing code using services. We should investigate a way to fix

it

Quoted 1 line

without causing datapath performance degradation.

Trying to gauge the impact on the existing applications...
The documentation does not explicitly disallow run time mapping of cores to service.
1) If the applications are mapping the cores to services at run time, they are running with a bug. IMO, bug fix resulting in a performance drop should be acceptable.
2) If the service is configured to run on single core (num_mapped_cores == 1), but service is set to MT unsafe - this will have a (possible) performance impact.
a) This can be solved by setting the service to MT safe and can be
documented. This might be a reasonable solution for applications which are compiling with
future DPDK releases.
b) We can also solve this using symbol versioning - the old version of
this function will use the old code, the new version of this function will use the code in
this patch. So, if the application is run with future DPDK
releases without recompiling, it will continue to use the old version. If the application is compiled
with future releases, they can use solution in 2a. We also
should think if this is an appropriate solution as this would force 1) to recompile to get the fix.
3) If the service is configured to run on multiple cores (num_mapped_cores > 1), then for those applications, the lock is being taken already. These applications might see some improvements as this patch removes few instructions.

Quoted 1 line
Quoted 4 lines
Quoted 3 lines

I think there is a way to achieve this by moving more checks/time to the control path (lcore updating the map), and not forcing the datapath lcore to always take an atomic.

I think 2a above is the solution.

2a above is e.g. the Eventdev SW routines like Rx/Tx scheduler services. We should strive to not reduce datapath performance at all here.

Quoted 5 lines
Quoted 1 line

In this particular case, we have a counter for number of iterations that a

Which counter are you thinking about?
All the counters I checked are not atomic operations currently. If we are going to use counters they have to be atomic, which means additional cycles in the data path.

I'll try to explain the concept better, take this example:
- One service core is mapped to a MT_UNSAFE service, like event/sw pmd
- Application wants to map a 2nd lcore to the same service
- You point out that today there is a race over the lock
-- the lock is not taken if (num_mapped_lcores == 1)
-- this avoids an atomic lock/unlock on the datapath

To achieve our desired goal;
- control thread doing mapping performs the following operations
-- write service->num_mapped_lcores++ (atomic not required, only single-writer allowed by APIs) -- MFENCE (full st-ld barrier) to flush write, and force later loads to issue after -- read the "calls_per_service" counter for each lcores, add them up.
---- Wait :)
-- re-read the "calls_per_service", and ensure the count has changed.
---- The fact that the calls_per_service has changed ensures the service-lcore
has seen the new "num_mapped_cores" value, and has now taken the lock!
-- *now* it is safe to map the 2nd lcore to the service

There is a caveat here that the increments to the "calls_per_service" variable must become globally-observable. To force this immediately would require a write memory barrier, which could impact datapath performance. Given the service is now taking a lock, the unlock() thereof would ensure the "calls_per_service" is flushed to memory.

Note: we could use calls_per_service, or add a new variable to the service struct. Writes to this do not need to be atomic, as it is either mapped to a single core, or else there's a lock around it.

Quoted 8 lines
Quoted 8 lines

service has done. If this increments we know that the lcore running the service has re-entered the critical section, so would see an updated "needs atomic" flag.

This approach may introduce a predictable branch on the datapath, however the cost of a predictable branch vs always taking an atomic is order(s?) of magnitude, so a branch is much preferred.

Quoted 8 lines
Quoted 1 line

It must be possible to avoid the datapath overhead using a scheme like this.

It

Quoted 3 lines

will likely be more complex than your proposed change below, however if it avoids datapath performance drops I feel that a more complex solution is worth investigating at least.

I do not completely understand the approach you are proposing, may be you can elaborate more.

Expanded above, showing a possible solution that does not require additional atomics on the datapath.

Quoted 22 lines

But, it seems to be based on a counter approach. Following is
my assessment on what happens if we use a counter. Let us say we kept track of how many cores are running the service currently. We need an atomic counter other than 'num_mapped_cores'. Let us call that counter 'num_current_cores'. The code to call the service would look like below.

1) rte_atomic32_inc(&num_current_cores); /* this results in a full memory barrier */
2) if (__atomic_load_n(&num_current_cores, __ATOMIC_ACQUIRE) == 1) { /* rte_atomic_read is not enough here as it does not provide the required memory barrier for any architecture */
3) run_service(); /* Call the service */
4) }
5) rte_atomic32_sub(&num_current_cores); /* Calling rte_atomic32_clear is not enough as it is not an atomic operation and does not provide the required memory barrier */

But the above code has race conditions in lines 1 and 2. It is possible that none of the cores will ever get to run the service as they all could simultaneously increment the counter. Hence lines 1 and 2 together need to be atomic, which is nothing but 'compare-exchange' operation.

Quoted 5 lines

BTW, the current code has a bug where it calls 'rte_atomic_clear(&s-

Quoted 1 line

execute_lock)', it is missing memory barriers which results in clearing the

execute_lock before the service has completed running. I suggest changing the 'execute_lock' to rte_spinlock_t and using rte_spinlock_try_lock and rte_spinlock_unlock APIs.

I don't think a spinlock is what we want here:

The idea is that a service-lcore can be mapped to multiple services. If one service is already being run (by another thread), we do not want to spin here waiting for it to become "free" to run by this thread, it should continue to the next service that it is mapped to.

Quoted 10 lines
Quoted 2 lines

A unit test is required to validate a fix like this - although perhaps found

by

Quoted 1 line

inspection/review, a real-world test to validate would give confidence.

Agree, need to have a test case.

Quoted 4 lines

Thoughts on such an approach?

<snip patch contents>

#114 lore raw

Re: [dpdk-dev] [PATCH v3 09/12] service: avoid race condition for MT unsafe service

<snip>

Quoted 63 lines
Quoted 63 lines
Quoted 12 lines
Quoted 12 lines

Subject: [PATCH v3 09/12] service: avoid race condition for MT unsafe service

From: Honnappa Nagarahalli <[email protected]>

There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Quoted 8 lines
Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")

We should put "fix" in the title, once converged on an implementation.

Ok, will replace 'avoid' with 'fix' (once we agree on the solution)

Quoted 3 lines

Regarding Fixes and stable backport, we should consider if fixing this in

stable

Quoted 7 lines

with a performance degradation, fixing with more complex solution, or documenting a known issue a better solution.

This fix (always taking the atomic lock) will have a negative performance impact on existing code using services. We should investigate a way to fix

it

Quoted 1 line

without causing datapath performance degradation.

Trying to gauge the impact on the existing applications...
The documentation does not explicitly disallow run time mapping of cores to service.
1) If the applications are mapping the cores to services at run time, they are running with a bug. IMO, bug fix resulting in a performance drop should be acceptable.
2) If the service is configured to run on single core (num_mapped_cores == 1), but service is set to MT unsafe - this will have a (possible) performance impact.
a) This can be solved by setting the service to MT safe and can be
documented. This might be a reasonable solution for applications which are compiling with
future DPDK releases.
b) We can also solve this using symbol versioning - the old version
of this function will use the old code, the new version of this function will use the code in
this patch. So, if the application is run with
future DPDK releases without recompiling, it will continue to use the old version. If the application is compiled
with future releases, they can use solution in 2a.
We also should think if this is an appropriate solution as this would force 1) to recompile to get the fix.
3) If the service is configured to run on multiple cores (num_mapped_cores > 1), then for those applications, the lock is being taken already. These applications might see some improvements as this patch removes few instructions.

Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

I think there is a way to achieve this by moving more checks/time to the control path (lcore updating the map), and not forcing the datapath lcore to always take an atomic.

I think 2a above is the solution.

2a above is e.g. the Eventdev SW routines like Rx/Tx scheduler services.

I scanned through the code briefly
I see that Eth RX/TX, Crypto adapters are setting the MT_SAFE capabilities, can be ignored.
Timer adaptor and some others do not set MT_SAFE. Seems like the cores to run on are mapped during run time. But it is not clear to me if it can get mapped to run on multiple cores. If they are, they are running with the bug. But, these are all internal to DPDK and can be fixed.
Are there no performance tests in these components that we can run?

Quoted 17 lines

We should strive to not reduce datapath performance at all here.

Quoted 6 lines
Quoted 2 lines

In this particular case, we have a counter for number of iterations that a

Which counter are you thinking about?
All the counters I checked are not atomic operations currently. If we are going to use counters they have to be atomic, which means additional cycles in the data path.

I'll try to explain the concept better, take this example:
- One service core is mapped to a MT_UNSAFE service, like event/sw pmd
- Application wants to map a 2nd lcore to the same service
- You point out that today there is a race over the lock
-- the lock is not taken if (num_mapped_lcores == 1)
-- this avoids an atomic lock/unlock on the datapath

Quoted 4 lines

To achieve our desired goal;
- control thread doing mapping performs the following operations

    -- write service->num_mapped_lcores++ (atomic not required, only single-

writer allowed by APIs)

This has to be atomic because of rte_service_run_iter_on_app_lcore API. Performance should be fine as this API is not called frequently. But need to consider the implications of more than one thread updating num_mapped_cores.

Quoted 2 lines

-- MFENCE (full st-ld barrier) to flush write, and force later loads to issue
after

I am not exactly sure what MFENCE on x86 does. On Arm platforms, the full barrier (DMB ISH) just makes sure that memory operations are not re-ordered around it. It does not say anything about when that store is visible to other cores. It will be visible at some point in time to cores. But, I do not think we need to be worried about flushing to memory.

Quoted 1 line

-- read the "calls_per_service" counter for each lcores, add them up.

This can be trimmed down to the single core the service is mapped to currently, no need to add all the counters.

Quoted 2 lines

---- Wait :)
-- re-read the "calls_per_service", and ensure the count has changed.

Basically, polling. This causes more traffic on the interconnect between the cores. But might be ok since this API might not be called frequently.

Quoted 6 lines

---- The fact that the calls_per_service has changed ensures the service-
lcore
has seen the new "num_mapped_cores" value, and has now taken the
lock!
-- *now* it is safe to map the 2nd lcore to the service

Quoted 6 lines

There is a caveat here that the increments to the "calls_per_service" variable must become globally-observable. To force this immediately would require a write memory barrier, which could impact datapath performance. Given the service is now taking a lock, the unlock() thereof would ensure the "calls_per_service"
is flushed to memory.

If we increment this variable only when the lock is held, we should be fine. We could have a separate variable.

Quoted 5 lines

Note: we could use calls_per_service, or add a new variable to the service struct.
Writes to this do not need to be atomic, as it is either mapped to a single core, or else there's a lock around it.

I think it is better to have a separate variable that is updated only when the lock is held.
I do not see any change in API sequence. We do this hand-shake only if the service is running (which is all controlled in the writer thread), correct?

This does not solve the problem with rte_service_run_iter_on_app_lcore getting called on multiple cores concurrently for the same service.

Quoted 55 lines
Quoted 11 lines
Quoted 11 lines

service has done. If this increments we know that the lcore running the service has re-entered the critical section, so would see an updated "needs atomic" flag.

This approach may introduce a predictable branch on the datapath, however the cost of a predictable branch vs always taking an atomic is order(s?) of magnitude, so a branch is much preferred.

It must be possible to avoid the datapath overhead using a scheme like

this.

Quoted 7 lines

It

Quoted 3 lines

will likely be more complex than your proposed change below, however if it avoids datapath performance drops I feel that a more complex solution is worth investigating at least.

I do not completely understand the approach you are proposing, may be you can elaborate more.

Expanded above, showing a possible solution that does not require additional atomics on the datapath.

Quoted 4 lines

But, it seems to be based on a counter approach. Following is my assessment on what happens if we use a counter. Let us say we kept track of how many cores are running the service currently. We need an atomic counter other than 'num_mapped_cores'. Let us call that counter

'num_current_cores'.

Quoted 13 lines

The code to call the service would look like below.

1) rte_atomic32_inc(&num_current_cores); /* this results in a full memory barrier */
2) if (__atomic_load_n(&num_current_cores, __ATOMIC_ACQUIRE) == 1) { /* rte_atomic_read is not enough here as it does not provide the required memory barrier for any architecture */
3) run_service(); /* Call the service */
4) }
5) rte_atomic32_sub(&num_current_cores); /* Calling rte_atomic32_clear is not enough as it is not an atomic operation and does not provide the required memory barrier */

Quoted 4 lines

But the above code has race conditions in lines 1 and 2. It is possible that none of the cores will ever get to run the service as they all could simultaneously increment the counter. Hence lines 1 and 2 together need to be atomic, which is nothing but 'compare-exchange'

operation.

Quoted 7 lines

BTW, the current code has a bug where it calls 'rte_atomic_clear(&s-

Quoted 2 lines

execute_lock)', it is missing memory barriers which results in clearing the

execute_lock before the service has completed running. I suggest changing the 'execute_lock' to rte_spinlock_t and using rte_spinlock_try_lock and rte_spinlock_unlock APIs.

Quoted 6 lines

I don't think a spinlock is what we want here:

The idea is that a service-lcore can be mapped to multiple services. If one service is already being run (by another thread), we do not want to spin here waiting for it to become "free" to run by this thread, it should continue to the next service that it is mapped to.

Agree. I am suggesting to use 'rte_spinlock_try_lock' (does not spin) which is nothing but 'compare-exchange'. Since the API is available, we should make use of it instead of repeating the code.

Quoted 14 lines
Quoted 13 lines
Quoted 5 lines

A unit test is required to validate a fix like this - although perhaps found

by

Quoted 1 line

inspection/review, a real-world test to validate would give confidence.

Agree, need to have a test case.

Quoted 4 lines

Thoughts on such an approach?

<snip patch contents>

#115 lore raw

Re: [dpdk-dev] [PATCH v3 09/12] service: avoid race condition for MT unsafe service

Quoted 84 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Thursday, April 9, 2020 2:32 AM
To: Van Haaren, Harry <[email protected]>; Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 09/12] service: avoid race condition for MT unsafe service

<snip>

Quoted 68 lines
Quoted 67 lines
Quoted 12 lines
Quoted 12 lines

Subject: [PATCH v3 09/12] service: avoid race condition for MT unsafe service

From: Honnappa Nagarahalli <[email protected]>

There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Quoted 8 lines
Quoted 6 lines
Fixes e9139a32f6e8 ("service: add function to run on app lcore")

We should put "fix" in the title, once converged on an implementation.

Ok, will replace 'avoid' with 'fix' (once we agree on the solution)

Quoted 3 lines

Regarding Fixes and stable backport, we should consider if fixing this in

stable

Quoted 7 lines

with a performance degradation, fixing with more complex solution, or documenting a known issue a better solution.

This fix (always taking the atomic lock) will have a negative performance impact on existing code using services. We should investigate a way to fix

it

Quoted 1 line

without causing datapath performance degradation.

Trying to gauge the impact on the existing applications...
The documentation does not explicitly disallow run time mapping of cores to service.
1) If the applications are mapping the cores to services at run time, they are running with a bug. IMO, bug fix resulting in a performance drop should be acceptable.
2) If the service is configured to run on single core (num_mapped_cores == 1), but service is set to MT unsafe - this will have a (possible) performance impact.
a) This can be solved by setting the service to MT safe and can be
documented. This might be a reasonable solution for applications which are compiling with
future DPDK releases.
b) We can also solve this using symbol versioning - the old version
of this function will use the old code, the new version of this function will use the code in
this patch. So, if the application is run with
future DPDK releases without recompiling, it will continue to use the old version. If the application is compiled
with future releases, they can use solution in 2a.
We also should think if this is an appropriate solution as this would force 1) to recompile to get the fix.
3) If the service is configured to run on multiple cores (num_mapped_cores > 1), then for those applications, the lock is being taken already. These applications might see some improvements as this patch removes few instructions.

Quoted 4 lines

I think there is a way to achieve this by moving more checks/time to the control path (lcore updating the map), and not forcing the datapath lcore to always take an atomic.

I think 2a above is the solution.

Quoted 7 lines
Quoted 1 line

2a above is e.g. the Eventdev SW routines like Rx/Tx scheduler services.

I scanned through the code briefly
I see that Eth RX/TX, Crypto adapters are setting the MT_SAFE capabilities, can be ignored.
Timer adaptor and some others do not set MT_SAFE. Seems like the cores to run on are mapped during run time. But it is not clear to me if it can get mapped to run on multiple cores. If they are, they are running with the bug.

EAL will map each service to a single lcore. It will "round-robin" if there are more services than service-lcores to run them on. So agree that DPDK's default mappings will not suffer this issue.

Quoted 37 lines

But, these are all internal to DPDK and can be fixed. Are there no performance tests in these components that we can run?

Quoted 17 lines

We should strive to not reduce datapath performance at all here.

Quoted 6 lines
Quoted 2 lines

In this particular case, we have a counter for number of iterations that a

Which counter are you thinking about?
All the counters I checked are not atomic operations currently. If we are going to use counters they have to be atomic, which means additional cycles in the data path.

I'll try to explain the concept better, take this example:
- One service core is mapped to a MT_UNSAFE service, like event/sw pmd
- Application wants to map a 2nd lcore to the same service
- You point out that today there is a race over the lock
-- the lock is not taken if (num_mapped_lcores == 1)
-- this avoids an atomic lock/unlock on the datapath

Quoted 4 lines

To achieve our desired goal;
- control thread doing mapping performs the following operations

    -- write service->num_mapped_lcores++ (atomic not required, only single-

writer allowed by APIs)

This has to be atomic because of rte_service_run_iter_on_app_lcore API. Performance should be fine as this API is not called frequently. But need to consider the implications of more than one thread updating num_mapped_cores.

Quoted 1 line

-- MFENCE (full st-ld barrier) to flush write, and force later loads to

issue

Quoted 1 line

after

I am not exactly sure what MFENCE on x86 does. On Arm platforms, the full barrier (DMB ISH) just makes sure that memory operations are not re-ordered around it. It does not say anything about when that store is visible to other cores. It will be visible at some point in time to cores.
But, I do not think we need to be worried about flushing to memory.

Quoted 3 lines
Quoted 1 line

-- read the "calls_per_service" counter for each lcores, add them up.

This can be trimmed down to the single core the service is mapped to currently, no need to add all the counters.

Correct - however that requires figuring out which lcore is running the service. Anyway, agree - it's an implementation detail as to exactly how we detect it.

Quoted 5 lines
Quoted 3 lines

---- Wait :)
-- re-read the "calls_per_service", and ensure the count has changed.

Basically, polling. This causes more traffic on the interconnect between the cores. But might be ok since this API might not be called frequently.

Agree this will not be called frequently, and that some polling here will not be a problem.

Quoted 8 lines
Quoted 1 line

---- The fact that the calls_per_service has changed ensures the

service-

Quoted 6 lines

lcore
has seen the new "num_mapped_cores" value, and has now taken the
lock!
-- *now* it is safe to map the 2nd lcore to the service

There is a caveat here that the increments to the "calls_per_service"

Quoted 8 lines

variable

Quoted 5 lines

must become globally-observable. To force this immediately would require a write memory barrier, which could impact datapath performance. Given the service is now taking a lock, the unlock() thereof would ensure the "calls_per_service"
is flushed to memory.

If we increment this variable only when the lock is held, we should be fine. We could have a separate variable.

Sure, if a separate variable is preferred that's fine with me.

Quoted 9 lines
Quoted 3 lines

Note: we could use calls_per_service, or add a new variable to the service struct.
Writes to this do not need to be atomic, as it is either mapped to a single

core,

Quoted 1 line

or else there's a lock around it.

I think it is better to have a separate variable that is updated only when the lock is held.
I do not see any change in API sequence. We do this hand-shake only if the service is running (which is all controlled in the writer thread), correct?

Yes this increment can be localized to just the branch when the unlock() occurs, as that is the only time it could make a difference.

Quoted 2 lines

This does not solve the problem with rte_service_run_iter_on_app_lcore getting called on multiple cores concurrently for the same service.

Agreed. This "on_app_lcore" API was an addition required to enable unit-testing in a sane way, to run iterations of eg Eventdev PMD.

I am in favor of documenting that the application is responsible to ensure the service being run on a specific application lcore is not concurrently running on another application lcore.

Quoted 57 lines
Quoted 49 lines
Quoted 9 lines
Quoted 9 lines

service has done. If this increments we know that the lcore running the service has re-entered the critical section, so would see an updated "needs atomic" flag.

This approach may introduce a predictable branch on the datapath, however the cost of a predictable branch vs always taking an atomic is order(s?) of magnitude, so a branch is much preferred.

It must be possible to avoid the datapath overhead using a scheme like

this.

Quoted 7 lines

It

Quoted 3 lines

will likely be more complex than your proposed change below, however if it avoids datapath performance drops I feel that a more complex solution is worth investigating at least.

I do not completely understand the approach you are proposing, may be you can elaborate more.

Expanded above, showing a possible solution that does not require additional atomics on the datapath.

Quoted 4 lines

But, it seems to be based on a counter approach. Following is my assessment on what happens if we use a counter. Let us say we kept track of how many cores are running the service currently. We need an atomic counter other than 'num_mapped_cores'. Let us call that counter

'num_current_cores'.

Quoted 13 lines

The code to call the service would look like below.

1) rte_atomic32_inc(&num_current_cores); /* this results in a full memory barrier */
2) if (__atomic_load_n(&num_current_cores, __ATOMIC_ACQUIRE) == 1) { /* rte_atomic_read is not enough here as it does not provide the required memory barrier for any architecture */
3) run_service(); /* Call the service */
4) }
5) rte_atomic32_sub(&num_current_cores); /* Calling rte_atomic32_clear is not enough as it is not an atomic operation and does not provide the required memory barrier */

Quoted 4 lines

But the above code has race conditions in lines 1 and 2. It is possible that none of the cores will ever get to run the service as they all could simultaneously increment the counter. Hence lines 1 and 2 together need to be atomic, which is nothing but 'compare-exchange'

operation.

Quoted 4 lines

BTW, the current code has a bug where it calls 'rte_atomic_clear(&s-

Quoted 2 lines

execute_lock)', it is missing memory barriers which results in clearing the

Quoted 8 lines
Quoted 3 lines

execute_lock before the service has completed running. I suggest changing the 'execute_lock' to rte_spinlock_t and using rte_spinlock_try_lock and rte_spinlock_unlock APIs.

I don't think a spinlock is what we want here:

The idea is that a service-lcore can be mapped to multiple services. If one service is already being run (by another thread), we do not want to

Quoted 7 lines

spin

Quoted 1 line

here waiting for it to become "free" to run by this thread, it should

continue

Quoted 1 line

to the next service that it is mapped to.

Agree. I am suggesting to use 'rte_spinlock_try_lock' (does not spin) which is nothing but 'compare-exchange'. Since the API is available, we should make use of it instead of repeating the code.

Ah apologies, I misread the spinlock usage. Sure if the spinlock_t code is preferred I'm ok with a change. It would be clean to have a separate patch in the patchset to make this change, and have it later in the set than the changes for backporting to ease integration with stable branch.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 2 lines

A unit test is required to validate a fix like this - although perhaps found

by

Quoted 1 line

inspection/review, a real-world test to validate would give confidence.

Agree, need to have a test case.

Quoted 4 lines

Thoughts on such an approach?

<snip patch contents>

#116 lore raw

Re: [dpdk-dev] [PATCH v3 09/12] service: avoid race condition for MT unsafe service

<snip>

Quoted 79 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Subject: [PATCH v3 09/12] service: avoid race condition for MT unsafe service

From: Honnappa Nagarahalli <[email protected]>

There has possible that a MT unsafe service might get configured to run on another core while the service is running

currently.

Quoted 63 lines
Quoted 63 lines
Quoted 62 lines
Quoted 8 lines
Quoted 8 lines

This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe.

Fixes e9139a32f6e8 ("service: add function to run on app

lcore")
Cc: [email protected]

Quoted 6 lines
Quoted 4 lines

Signed-off-by: Honnappa Nagarahalli
<[email protected]>

We should put "fix" in the title, once converged on an implementation.

Ok, will replace 'avoid' with 'fix' (once we agree on the solution)

Quoted 3 lines

Regarding Fixes and stable backport, we should consider if fixing this in

stable

Quoted 7 lines

with a performance degradation, fixing with more complex solution, or documenting a known issue a better solution.

This fix (always taking the atomic lock) will have a negative performance impact on existing code using services. We should investigate a way to fix

it

Quoted 1 line

without causing datapath performance degradation.

Trying to gauge the impact on the existing applications... The documentation does not explicitly disallow run time mapping of cores to service.
1) If the applications are mapping the cores to services at run time, they are running with a bug. IMO, bug fix resulting in a performance drop should be acceptable.
2) If the service is configured to run on single core (num_mapped_cores == 1), but service is set to MT unsafe - this will have a (possible) performance impact.
a) This can be solved by setting the service to MT safe and can
be documented. This might be a reasonable solution for applications which are compiling with
future DPDK releases.
b) We can also solve this using symbol versioning - the old
version of this function will use the old code, the new version of this function will use the code in
this patch. So, if the application is run with
future DPDK releases without recompiling, it will continue to use the old version. If the application is compiled
with future releases, they can use solution in 2a.
We also should think if this is an appropriate solution as this would force 1) to recompile to get the fix.
3) If the service is configured to run on multiple cores (num_mapped_cores > 1), then for those applications, the lock is being taken already. These applications might see some improvements as this patch removes few instructions.

Quoted 4 lines

I think there is a way to achieve this by moving more checks/time to the control path (lcore updating the map), and not forcing the datapath lcore to always take an atomic.

I think 2a above is the solution.

Quoted 7 lines
Quoted 1 line

2a above is e.g. the Eventdev SW routines like Rx/Tx scheduler services.

I scanned through the code briefly
I see that Eth RX/TX, Crypto adapters are setting the MT_SAFE capabilities, can be ignored.
Timer adaptor and some others do not set MT_SAFE. Seems like the cores to run on are mapped during run time. But it is not clear to me if it can get mapped to run on multiple cores. If they are, they are running with

Quoted 21 lines

the bug.

EAL will map each service to a single lcore. It will "round-robin" if there are more services than service-lcores to run them on. So agree that DPDK's default mappings will not suffer this issue.

Quoted 14 lines

But, these are all internal to DPDK and can be fixed. Are there no performance tests in these components that we can run?

Quoted 11 lines

We should strive to not reduce datapath performance at all here.

Quoted 6 lines
Quoted 2 lines

In this particular case, we have a counter for number of iterations that a

Which counter are you thinking about?
All the counters I checked are not atomic operations currently. If we are going to use counters they have to be atomic, which means additional cycles in the data path.

I'll try to explain the concept better, take this example:

I tried to implement this algorithm, but there are few issues, please see below.

Quoted 26 lines
Quoted 7 lines
Quoted 7 lines

- One service core is mapped to a MT_UNSAFE service, like event/sw
pmd
- Application wants to map a 2nd lcore to the same service
- You point out that today there is a race over the lock -- the lock is not taken if (num_mapped_lcores == 1) -- this avoids an atomic lock/unlock on the datapath

Quoted 7 lines
Quoted 4 lines

To achieve our desired goal;
- control thread doing mapping performs the following operations

    -- write service->num_mapped_lcores++ (atomic not required, only

single- writer allowed by APIs)

This has to be atomic because of rte_service_run_iter_on_app_lcore API. Performance should be fine as this API is not called frequently. But need to consider the implications of more than one thread updating

num_mapped_cores.

Quoted 11 lines
Quoted 3 lines

-- MFENCE (full st-ld barrier) to flush write, and force later
loads to

issue

Quoted 1 line

after

I am not exactly sure what MFENCE on x86 does. On Arm platforms, the full barrier (DMB ISH) just makes sure that memory operations are not re-ordered around it. It does not say anything about when that store is visible to other cores. It will be visible at some point in time to cores. But, I do not think we need to be worried about flushing to memory.

Quoted 10 lines
Quoted 3 lines
Quoted 1 line

-- read the "calls_per_service" counter for each lcores, add them up.

This can be trimmed down to the single core the service is mapped to currently, no need to add all the counters.

Correct - however that requires figuring out which lcore is running the service. Anyway, agree - it's an implementation detail as to exactly how we detect it.

Quoted 3 lines
Quoted 3 lines

---- Wait :)
-- re-read the "calls_per_service", and ensure the count has changed.

Here, there is an assumption that the service core function is running on the service core. If the service core is not running, the code will be stuck in this polling loop.

I could not come up with a good way to check if the service core is running. Checking the app_runstate and comp_runstate is not enough as they just indicate that the service is ready to run. Using the counter 'calls_per_service' introduces race conditions.

Only way I can think of is asking the user to follow a specific sequence of APIs to ensure the service core is running before calling rte_service_map_lcore_set.

Quoted 143 lines
Quoted 2 lines

Basically, polling. This causes more traffic on the interconnect between the cores. But might be ok since this API might not be called

frequently.

Agree this will not be called frequently, and that some polling here will not be a problem.

Quoted 8 lines
Quoted 1 line

---- The fact that the calls_per_service has changed ensures the

service-

Quoted 6 lines

lcore
has seen the new "num_mapped_cores" value, and has now
taken the lock!
-- *now* it is safe to map the 2nd lcore to the service

There is a caveat here that the increments to the "calls_per_service"

Quoted 8 lines

variable

Quoted 5 lines

must become globally-observable. To force this immediately would require a write memory barrier, which could impact datapath performance. Given the service is now taking a lock, the unlock() thereof would ensure the "calls_per_service"
is flushed to memory.

If we increment this variable only when the lock is held, we should be fine. We could have a separate variable.

Sure, if a separate variable is preferred that's fine with me.

Quoted 10 lines
Quoted 4 lines

Note: we could use calls_per_service, or add a new variable to the service struct.
Writes to this do not need to be atomic, as it is either mapped to a single

core,

Quoted 1 line

or else there's a lock around it.

I think it is better to have a separate variable that is updated only when the lock is held.
I do not see any change in API sequence. We do this hand-shake only if the service is running (which is all controlled in the writer thread), correct?

Yes this increment can be localized to just the branch when the unlock() occurs, as that is the only time it could make a difference.

Quoted 2 lines

This does not solve the problem with rte_service_run_iter_on_app_lcore getting called on multiple cores concurrently for the same service.

Agreed. This "on_app_lcore" API was an addition required to enable unit- testing in a sane way, to run iterations of eg Eventdev PMD.

I am in favor of documenting that the application is responsible to ensure the service being run on a specific application lcore is not concurrently running on another application lcore.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

service has done. If this increments we know that the lcore running the service has re-entered the critical section, so would see an updated "needs atomic" flag.

This approach may introduce a predictable branch on the datapath, however the cost of a predictable branch vs always taking an atomic is order(s?) of magnitude, so a branch is much

preferred.

Quoted 56 lines
Quoted 51 lines
Quoted 3 lines
Quoted 3 lines

It must be possible to avoid the datapath overhead using a scheme like

this.

Quoted 7 lines

It

Quoted 3 lines

will likely be more complex than your proposed change below, however if it avoids datapath performance drops I feel that a more complex solution is worth investigating at least.

I do not completely understand the approach you are proposing, may be you can elaborate more.

Expanded above, showing a possible solution that does not require additional atomics on the datapath.

Quoted 5 lines

But, it seems to be based on a counter approach. Following is my assessment on what happens if we use a counter. Let us say we kept track of how many cores are running the service currently. We need an atomic counter other than 'num_mapped_cores'. Let us call that counter

'num_current_cores'.

Quoted 14 lines

The code to call the service would look like below.

1) rte_atomic32_inc(&num_current_cores); /* this results in a full memory barrier */
2) if (__atomic_load_n(&num_current_cores, __ATOMIC_ACQUIRE) == 1) {
/* rte_atomic_read is not enough here as it does not provide the required memory barrier for any architecture */
3) run_service(); /* Call the service */
4) }
5) rte_atomic32_sub(&num_current_cores); /* Calling rte_atomic32_clear is not enough as it is not an atomic operation and does not provide the required memory barrier */

Quoted 5 lines

But the above code has race conditions in lines 1 and 2. It is possible that none of the cores will ever get to run the service as they all could simultaneously increment the counter. Hence lines 1 and
2 together need to be atomic, which is nothing but 'compare-exchange'

operation.

Quoted 8 lines

BTW, the current code has a bug where it calls
'rte_atomic_clear(&s-

Quoted 2 lines

execute_lock)', it is missing memory barriers which results in clearing the

execute_lock before the service has completed running. I suggest changing the 'execute_lock' to rte_spinlock_t and using rte_spinlock_try_lock and rte_spinlock_unlock APIs.

Quoted 5 lines

I don't think a spinlock is what we want here:

The idea is that a service-lcore can be mapped to multiple services. If one service is already being run (by another thread), we do not want to

Quoted 8 lines

spin

Quoted 2 lines

here waiting for it to become "free" to run by this thread, it should

continue

Quoted 1 line

to the next service that it is mapped to.

Agree. I am suggesting to use 'rte_spinlock_try_lock' (does not spin) which is nothing but 'compare-exchange'. Since the API is available, we should make use of it instead of repeating the code.

Ah apologies, I misread the spinlock usage. Sure if the spinlock_t code is preferred I'm ok with a change. It would be clean to have a separate patch in the patchset to make this change, and have it later in the set than the changes for backporting to ease integration with stable branch.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 2 lines

A unit test is required to validate a fix like this - although perhaps found

by

Quoted 1 line

inspection/review, a real-world test to validate would give confidence.

Agree, need to have a test case.

Quoted 4 lines

Thoughts on such an approach?

<snip patch contents>

#117 lore raw

Re: [dpdk-dev] [PATCH v3 09/12] service: avoid race condition for MT unsafe service

Quoted 14 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Saturday, April 18, 2020 7:22 AM
To: Van Haaren, Harry <[email protected]>; Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 09/12] service: avoid race condition for MT unsafe service

Quoted 1 line

<snip>

<snip snip>

Quoted 27 lines
Quoted 27 lines
Quoted 7 lines
Quoted 4 lines

To achieve our desired goal;
- control thread doing mapping performs the following operations

    -- write service->num_mapped_lcores++ (atomic not required, only

single- writer allowed by APIs)

This has to be atomic because of rte_service_run_iter_on_app_lcore API. Performance should be fine as this API is not called frequently. But need to consider the implications of more than one thread updating

num_mapped_cores.

Quoted 11 lines
Quoted 3 lines

-- MFENCE (full st-ld barrier) to flush write, and force later
loads to

issue

Quoted 1 line

after

I am not exactly sure what MFENCE on x86 does. On Arm platforms, the full barrier (DMB ISH) just makes sure that memory operations are not re-ordered around it. It does not say anything about when that store is visible to other cores. It will be visible at some point in time to cores. But, I do not think we need to be worried about flushing to memory.

Quoted 3 lines
Quoted 1 line

-- read the "calls_per_service" counter for each lcores, add them up.

This can be trimmed down to the single core the service is mapped to currently, no need to add all the counters.

Correct - however that requires figuring out which lcore is running the service. Anyway, agree - it's an implementation detail as to exactly how we detect it.

Quoted 1 line
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

---- Wait :)
-- re-read the "calls_per_service", and ensure the count has changed.

Here, there is an assumption that the service core function is running on the service core. If the service core is not running, the code will be stuck in this polling loop.

Right - we could add a timeout ehre, but that just moves the problem somewhere else (the application) which now needs to handle error rets, and possibly retries. It could be a possible solution.. I'm not in favour of it at the moment, but it needs some more time to think.

Quoted 7 lines

I could not come up with a good way to check if the service core is running. Checking the app_runstate and comp_runstate is not enough as they just indicate that the service is ready to run. Using the counter 'calls_per_service' introduces race conditions.

Only way I can think of is asking the user to follow a specific sequence of APIs to ensure the service core is running before calling rte_service_map_lcore_set.

Good point - I'm thinking about this - but haven't come to an obvious conclusion yet. I'm considering other ways to detect the core is/isn't running, and also considering just "high-jacking" the service function pointer temporarily with a CAS, which gives some new options on avoiding threads entering the critical section.

As above, I don't have a good solution yet.

<snip irrelevant to above discussion stuff>

author
#118 lore raw

PATCH v3 10/12 service: identify service running on another core correctly

In reply to Phil Yang

From: Honnappa Nagarahalli <[email protected]>

The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases.

---
 lib/librte_eal/common/rte_service.c | 26 ++++++++------------------
 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +8 -18
index 32a2f8a..0843c3c 100644
@@ -360,7 +360,7 @@ service_runner_do_callback(struct rte_service_spec_impl *s,
 /* Expects the service 's' is valid. */
 static int32_t
 service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
-	    struct rte_service_spec_impl *s)
+	    struct rte_service_spec_impl *s, uint32_t serialize_mt_unsafe)
 {
 	if (!s)
 		return -EINVAL;
@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	if (service_mt_safe(s) == 0) {
+	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
 
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
-	/* Atomically add this core to the mapped cores first, then examine if
-	 * we can run the service. This avoids a race condition between
-	 * checking the value, and atomically adding to the mapped count.
+	/* Increment num_mapped_cores to indicate that the service
+	 * is running on a core.
 	 */
-	if (serialize_mt_unsafe)
-		rte_atomic32_inc(&s->num_mapped_cores);
+	rte_atomic32_inc(&s->num_mapped_cores);
 
-	if (service_mt_safe(s) == 0 &&
-			rte_atomic32_read(&s->num_mapped_cores) > 1) {
-		if (serialize_mt_unsafe)
-			rte_atomic32_dec(&s->num_mapped_cores);
-		return -EBUSY;
-	}
-
-	int ret = service_run(id, cs, UINT64_MAX, s);
+	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	if (serialize_mt_unsafe)
-		rte_atomic32_dec(&s->num_mapped_cores);
+	rte_atomic32_dec(&s->num_mapped_cores);
 
 	return ret;
 }
@@ -449,7 +439,7 @@ service_runner_func(void *arg)
 			if (!service_valid(i))
 				continue;
 			/* return value ignored as no change to code flow */
-			service_run(i, cs, service_mask, service_get(i));
+			service_run(i, cs, service_mask, service_get(i), 1);
 		}
 
 		cs->loops++;
Signature · 1 line
2.7.4
#119 lore raw

Re: [dpdk-dev] [PATCH v3 10/12] service: identify service running on another core correctly

Quoted 20 lines

From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: [PATCH v3 10/12] service: identify service running on another core correctly

From: Honnappa Nagarahalli <[email protected]>

The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases.

Quoted 5 lines

Is this fixing broken functionality, or does it aim to only "optimize"? Lack of "fixes" tag suggests optimization.

I'm cautious about the commit phrase "Assume that the application ...", if the code was previously checking things, we must not stop checking them now, this may introduce race-conditions in existing applications?

It seems like the "serialize_mt_unsafe" branch is being pushed further down the callgraph, and instead of branching over atomics this patch forces always executing 2 atomics?

This feels like too specific an optimization/tradeoff, without data to backup that there are no regressions on any DPDK supported platforms.

DPDK today doesn't have a micro-benchmark to gather such perf data, but I would welcome one and we can have a data-driven decision.

Hope this point-of-view makes sense, -Harry

Quoted 71 lines
---
 lib/librte_eal/common/rte_service.c | 26 ++++++++------------------
 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +1 -1
index 32a2f8a..0843c3c 100644
@@ -360,7 +360,7 @@ service_runner_do_callback(struct rte_service_spec_impl
 *s,
 /* Expects the service 's' is valid. */
 static int32_t
 service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
-	    struct rte_service_spec_impl *s)
+	    struct rte_service_spec_impl *s, uint32_t serialize_mt_unsafe)
 {
 	if (!s)
lib/librte_eal/common/rte_service.c +6 -16
 		return -EINVAL;
@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t
 service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	if (service_mt_safe(s) == 0) {
+	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t
 serialize_mt_unsafe)
 
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
-	/* Atomically add this core to the mapped cores first, then examine if
-	 * we can run the service. This avoids a race condition between
-	 * checking the value, and atomically adding to the mapped count.
+	/* Increment num_mapped_cores to indicate that the service
+	 * is running on a core.
 	 */
-	if (serialize_mt_unsafe)
-		rte_atomic32_inc(&s->num_mapped_cores);
+	rte_atomic32_inc(&s->num_mapped_cores);
 
-	if (service_mt_safe(s) == 0 &&
-			rte_atomic32_read(&s->num_mapped_cores) > 1) {
-		if (serialize_mt_unsafe)
-			rte_atomic32_dec(&s->num_mapped_cores);
-		return -EBUSY;
-	}
-
-	int ret = service_run(id, cs, UINT64_MAX, s);
+	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	if (serialize_mt_unsafe)
-		rte_atomic32_dec(&s->num_mapped_cores);
+	rte_atomic32_dec(&s->num_mapped_cores);
 
 	return ret;

}

lib/librte_eal/common/rte_service.c +1 -2
@@ -449,7 +439,7 @@ service_runner_func(void *arg)
 			if (!service_valid(i))
 				continue;
 			/* return value ignored as no change to code flow */
-			service_run(i, cs, service_mask, service_get(i));
+			service_run(i, cs, service_mask, service_get(i), 1);
 		}
 
 		cs->loops++;
--

2.7.4

#120 lore raw

Re: [dpdk-dev] [PATCH v3 10/12] service: identify service running on another core correctly

<snip>

Quoted 11 lines
Quoted 11 lines

Subject: [PATCH v3 10/12] service: identify service running on another core correctly

From: Honnappa Nagarahalli <[email protected]>

The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases.

Quoted 8 lines
Quoted 5 lines

Is this fixing broken functionality, or does it aim to only "optimize"? Lack of "fixes" tag suggests optimization.

Good point, it is missing the 'Fixes' tag. Relooking at the code, following are few problems that I observe, please correct me if I am wrong:
1) Looking at the API rte_service_map_lcore_set, 'num_mapped_cores' keeps track of the number of cores the service is mapped to. However, in the function 'rte_service_run_iter_on_app_lcore', the 'num_mapped_cores' is incremented only if 'serialize_mt_unsafe' is set. This will return incorrect result in 'rte_service_runstate_get' API when 'serialize_mt_unsafe' is not set.

2) Since incrementing 'num_mapped_cores' and checking its value is not an atomic operation, it introduces race conditions. Consider the current code snippet from the function 'rte_service_run_iter_on_app_lcore'.

1) if (serialize_mt_unsafe)
2) rte_atomic32_inc(&s->num_mapped_cores);
3) if (service_mt_safe(s) == 0 && rte_atomic32_read(&s->num_mapped_cores) > 1) {
4) if (serialize_mt_unsafe)
5) rte_atomic32_dec(&s->num_mapped_cores);
6) return -EBUSY;
7) }

It is possible that more than 1 thread is executing line 3) concurrently, in which case all of them will hit line 6). Due to this, it is possible that the service might never run or it might take multiple attempts to run the service wasting cycles.

If you agree these are bugs, I can add 'fixes' tag.

Quoted 4 lines

I'm cautious about the commit phrase "Assume that the application ...", if the code was previously checking things, we must not stop checking them now, this may introduce race-conditions in existing applications?

What I meant here is, let us believe in what the application says. i.e. if the applications sets 'serialize_mt_unsafe' to 1, let us assume that the service is infact 'unsafe'.

Quoted 4 lines

It seems like the "serialize_mt_unsafe" branch is being pushed further down the callgraph, and instead of branching over atomics this patch forces always executing 2 atomics?

Yes, that is correct. Please see explanation in 1) above.

Quoted 3 lines

This feels like too specific an optimization/tradeoff, without data to backup that there are no regressions on any DPDK supported platforms.

Apologies for missing the 'Fixes' tag, this patch is a bug fix.

Quoted 3 lines

DPDK today doesn't have a micro-benchmark to gather such perf data, but I would welcome one and we can have a data-driven decision.

When this was developed, was the logic to avoid taking the lock measured to provide any improvements?

Quoted 74 lines

Hope this point-of-view makes sense, -Harry

Quoted 71 lines
---
 lib/librte_eal/common/rte_service.c | 26 ++++++++------------------
 1 file changed, 8 insertions(+), 18 deletions(-)
lib/librte_eal/common/rte_service.c +2 -2
index 32a2f8a..0843c3c 100644
@@ -360,7 +360,7 @@ service_runner_do_callback(struct
 rte_service_spec_impl *s,
 /* Expects the service 's' is valid. */  static int32_t
 service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
-	    struct rte_service_spec_impl *s)
+	    struct rte_service_spec_impl *s, uint32_t serialize_mt_unsafe)
 {
 	if (!s)
 		return -EINVAL;
@@ -374,7 +374,7 @@ service_run(uint32_t i, struct core_state *cs,
 uint64_t service_mask,
 
 	cs->service_active_on_lcore[i] = 1;
 
-	if (service_mt_safe(s) == 0) {
+	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
 		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
 			return -EBUSY;
@@ -412,24 +412,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id,

uint32_t
serialize_mt_unsafe)

lib/librte_eal/common/rte_service.c +6 -17
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
-	/* Atomically add this core to the mapped cores first, then examine if
-	 * we can run the service. This avoids a race condition between
-	 * checking the value, and atomically adding to the mapped count.
+	/* Increment num_mapped_cores to indicate that the service
+	 * is running on a core.
 	 */
-	if (serialize_mt_unsafe)
-		rte_atomic32_inc(&s->num_mapped_cores);
+	rte_atomic32_inc(&s->num_mapped_cores);
 
-	if (service_mt_safe(s) == 0 &&
-			rte_atomic32_read(&s->num_mapped_cores) > 1) {
-		if (serialize_mt_unsafe)
-			rte_atomic32_dec(&s->num_mapped_cores);
-		return -EBUSY;
-	}
-
-	int ret = service_run(id, cs, UINT64_MAX, s);
+	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	if (serialize_mt_unsafe)
-		rte_atomic32_dec(&s->num_mapped_cores);
+	rte_atomic32_dec(&s->num_mapped_cores);
 
 	return ret;
 }
@@ -449,7 +439,7 @@ service_runner_func(void *arg)
 			if (!service_valid(i))
 				continue;
 			/* return value ignored as no change to code flow */
-			service_run(i, cs, service_mask, service_get(i));
+			service_run(i, cs, service_mask, service_get(i), 1);
 		}
 
 		cs->loops++;
--

2.7.4

author
#121 lore raw

PATCH v3 11/12 service: optimize with c11 one-way barrier

In reply to Phil Yang

The num_mapped_cores and execute_lock are synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.

---
 lib/librte_eal/common/rte_service.c | 50 ++++++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 15 deletions(-)
lib/librte_eal/common/rte_service.c +35 -15
index 0843c3c..c033224 100644
@@ -42,7 +42,7 @@ struct rte_service_spec_impl {
 	 * running this service callback. When not set, a core may take the
 	 * lock and then run the service callback.
 	 */
-	rte_atomic32_t execute_lock;
+	uint32_t execute_lock;
 
 	/* API set/get-able variables */
 	int8_t app_runstate;
@@ -54,7 +54,7 @@ struct rte_service_spec_impl {
 	 * It does not indicate the number of cores the service is running
 	 * on currently.
 	 */
-	rte_atomic32_t num_mapped_cores;
+	int32_t num_mapped_cores;
 	uint64_t calls;
 	uint64_t cycles_spent;
 } __rte_cache_aligned;
@@ -332,7 +332,8 @@ rte_service_runstate_get(uint32_t id)
 	rte_smp_rmb();
 
 	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (rte_atomic32_read(&s->num_mapped_cores) > 0);
+	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+					    __ATOMIC_RELAXED) > 0);
 
 	return (s->app_runstate == RUNSTATE_RUNNING) &&
 		(s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -375,11 +376,20 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,
 	cs->service_active_on_lcore[i] = 1;
 
 	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
-		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
+		uint32_t expected = 0;
+		/* ACQUIRE ordering here is to prevent the callback
+		 * function from hoisting up before the execute_lock
+		 * setting.
+		 */
+		if (!__atomic_compare_exchange_n(&s->execute_lock, &expected, 1,
+			    0, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED))
 			return -EBUSY;
 
 		service_runner_do_callback(s, cs, i);
-		rte_atomic32_clear(&s->execute_lock);
+		/* RELEASE ordering here is used to pair with ACQUIRE
+		 * above to achieve lock semantic.
+		 */
+		__atomic_store_n(&s->execute_lock, 0, __ATOMIC_RELEASE);
 	} else
 		service_runner_do_callback(s, cs, i);
 
@@ -415,11 +425,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)
 	/* Increment num_mapped_cores to indicate that the service
 	 * is running on a core.
 	 */
-	rte_atomic32_inc(&s->num_mapped_cores);
+	__atomic_add_fetch(&s->num_mapped_cores, 1, __ATOMIC_ACQUIRE);
 
 	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	rte_atomic32_dec(&s->num_mapped_cores);
+	__atomic_sub_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELEASE);
 
 	return ret;
 }
@@ -552,24 +562,32 @@ service_update(uint32_t sid, uint32_t lcore,
 
 	uint64_t sid_mask = UINT64_C(1) << sid;
 	if (set) {
-		uint64_t lcore_mapped = lcore_states[lcore].service_mask &
-			sid_mask;
+		/* When multiple threads try to update the same lcore
+		 * service concurrently, e.g. set lcore map followed
+		 * by clear lcore map, the unsynchronized service_mask
+		 * values have issues on the num_mapped_cores value
+		 * consistency. So we use ACQUIRE ordering to pair with
+		 * the RELEASE ordering to synchronize the service_mask.
+		 */
+		uint64_t lcore_mapped = __atomic_load_n(
+					&lcore_states[lcore].service_mask,
+					__ATOMIC_ACQUIRE) & sid_mask;
 
 		if (*set && !lcore_mapped) {
 			lcore_states[lcore].service_mask |= sid_mask;
-			rte_atomic32_inc(&rte_services[sid].num_mapped_cores);
+			__atomic_add_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELEASE);
 		}
 		if (!*set && lcore_mapped) {
 			lcore_states[lcore].service_mask &= ~(sid_mask);
-			rte_atomic32_dec(&rte_services[sid].num_mapped_cores);
+			__atomic_sub_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELEASE);
 		}
 	}
 
 	if (enabled)
 		*enabled = !!(lcore_states[lcore].service_mask & (sid_mask));
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -625,7 +643,8 @@ rte_service_lcore_reset_all(void)
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
-		rte_atomic32_set(&rte_services[i].num_mapped_cores, 0);
+		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
+				    __ATOMIC_RELAXED);
 
 	rte_smp_wmb();
 
@@ -708,7 +727,8 @@ rte_service_lcore_stop(uint32_t lcore)
 		int32_t enabled = service_mask & (UINT64_C(1) << i);
 		int32_t service_running = rte_service_runstate_get(i);
 		int32_t only_core = (1 ==
-			rte_atomic32_read(&rte_services[i].num_mapped_cores));
+			__atomic_load_n(&rte_services[i].num_mapped_cores,
+					__ATOMIC_RELAXED));
 
 		/* if the core is mapped, and the service is running, and this
 		 * is the only core that is mapped, the service would cease to
Signature · 1 line
2.7.4
#122 lore raw

Re: [dpdk-dev] [PATCH v3 11/12] service: optimize with c11 one-way barrier

Quoted 11 lines

-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 11/12] service: optimize with c11 one-way barrier

Quoted 8 lines

The num_mapped_cores and execute_lock are synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.

Based on discussion on-list, it seems the consensus is to not use GCC builtins, but instead use C11 APIs "proper"? If my conclusion is correct, the v+1 of this patchset would require updates to that style API.

Inline comments for context below, -Harry

Quoted 20 lines
---
 lib/librte_eal/common/rte_service.c | 50 ++++++++++++++++++++++++++----------
(quoted diff) +0 -1
-
 1 file changed, 35 insertions(+), 15 deletions(-)
lib/librte_eal/common/rte_service.c +1 -1
index 0843c3c..c033224 100644
@@ -42,7 +42,7 @@ struct rte_service_spec_impl {
 	 * running this service callback. When not set, a core may take the
 	 * lock and then run the service callback.
 	 */
-	rte_atomic32_t execute_lock;
+	uint32_t execute_lock;
 
 	/* API set/get-able variables */
 	int8_t app_runstate;

lib/librte_eal/common/rte_service.c struct rte_service_spec_impl { · line 57

Quoted 5 lines
lib/librte_eal/common/rte_service.c +1 -1
@@ -54,7 +54,7 @@ struct rte_service_spec_impl {
 	 * It does not indicate the number of cores the service is running
 	 * on currently.
 	 */
-	rte_atomic32_t num_mapped_cores;
+	int32_t num_mapped_cores;

Any reason why "int32_t" or "uint32_t" is used over another? execute_lock is a uint32_t above, num_mapped_cores is an int32_t?

Quoted 17 lines
lib/librte_eal/common/rte_service.c +2 -1
 	uint64_t calls;
 	uint64_t cycles_spent;
 } __rte_cache_aligned;
@@ -332,7 +332,8 @@ rte_service_runstate_get(uint32_t id)
 	rte_smp_rmb();
 
 	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (rte_atomic32_read(&s->num_mapped_cores) > 0);
+	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+					    __ATOMIC_RELAXED) > 0);
 
 	return (s->app_runstate == RUNSTATE_RUNNING) &&
 		(s->comp_runstate == RUNSTATE_RUNNING) &&
@@ -375,11 +376,20 @@ service_run(uint32_t i, struct core_state *cs, uint64_t
 service_mask,
 	cs->service_active_on_lcore[i] = 1;
 

lib/librte_eal/common/rte_service.c service_run(uint32_t i, struct core_state *cs, uint64_t · line 387

Quoted 10 lines
lib/librte_eal/common/rte_service.c +7 -1
 	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
-		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
+		uint32_t expected = 0;
+		/* ACQUIRE ordering here is to prevent the callback
+		 * function from hoisting up before the execute_lock
+		 * setting.
+		 */
+		if (!__atomic_compare_exchange_n(&s->execute_lock, &expected, 1,
+			    0, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED))
 			return -EBUSY;

Let's try improve the magic "1" and "0" constants, I believe the "1" here
is the desired "new value on success", and the 0 is "bool weak", where our 0/false constant implies a strongly ordered compare exchange?

"Weak is true for weak compare_exchange, which may fail spuriously, and false for the strong variation, which never fails spuriously.", from https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html

const uint32_t on_success_value = 1;
const bool weak = 0;
__atomic_compare_exchange_n(&s->execute_lock, &expected, on_success_value, weak, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED);

Although a bit more verbose, I feel this documents usage a lot better, particularly for those who aren't as familiar with the C11 function arguments order.

Admittedly with the API change to not use __builtins, perhaps this comment is moot.

Quoted 31 lines
lib/librte_eal/common/rte_service.c +6 -3
 		service_runner_do_callback(s, cs, i);
-		rte_atomic32_clear(&s->execute_lock);
+		/* RELEASE ordering here is used to pair with ACQUIRE
+		 * above to achieve lock semantic.
+		 */
+		__atomic_store_n(&s->execute_lock, 0, __ATOMIC_RELEASE);
 	} else
 		service_runner_do_callback(s, cs, i);
 
@@ -415,11 +425,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t
 serialize_mt_unsafe)
 	/* Increment num_mapped_cores to indicate that the service
 	 * is running on a core.
 	 */
-	rte_atomic32_inc(&s->num_mapped_cores);
+	__atomic_add_fetch(&s->num_mapped_cores, 1, __ATOMIC_ACQUIRE);
 
 	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	rte_atomic32_dec(&s->num_mapped_cores);
+	__atomic_sub_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELEASE);
 
 	return ret;

}

lib/librte_eal/common/rte_service.c +0 -2
@@ -552,24 +562,32 @@ service_update(uint32_t sid, uint32_t lcore,
 
 	uint64_t sid_mask = UINT64_C(1) << sid;
 	if (set) {
-		uint64_t lcore_mapped = lcore_states[lcore].service_mask &
-			sid_mask;

lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 574

Quoted 10 lines
lib/librte_eal/common/rte_service.c +10 -0
+		/* When multiple threads try to update the same lcore
+		 * service concurrently, e.g. set lcore map followed
+		 * by clear lcore map, the unsynchronized service_mask
+		 * values have issues on the num_mapped_cores value
+		 * consistency. So we use ACQUIRE ordering to pair with
+		 * the RELEASE ordering to synchronize the service_mask.
+		 */
+		uint64_t lcore_mapped = __atomic_load_n(
+					&lcore_states[lcore].service_mask,
+					__ATOMIC_ACQUIRE) & sid_mask;

Thanks for the comment - it helps me understand things a bit better. Some questions/theories to validate;
1) The service_mask ACQUIRE avoids other loads being hoisted above it, correct?

2) There are non-atomic stores to service_mask. Is it correct that the stores themselves aren't the issue, but relative visibility of service_mask stores vs num_mapped_cores? (Detail in (3) below)

lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 585

Quoted 12 lines
lib/librte_eal/common/rte_service.c +4 -2
 		if (*set && !lcore_mapped) {
 			lcore_states[lcore].service_mask |= sid_mask;
-			rte_atomic32_inc(&rte_services[sid].num_mapped_cores);
+			__atomic_add_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELEASE);
 		}
 		if (!*set && lcore_mapped) {
 			lcore_states[lcore].service_mask &= ~(sid_mask);
-			rte_atomic32_dec(&rte_services[sid].num_mapped_cores);
+			__atomic_sub_fetch(&rte_services[sid].num_mapped_cores,
+					    1, __ATOMIC_RELEASE);
 		}

3) Here we update the core-local service_mask, and then update the num_mapped_cores with an ATOMIC_RELEASE. The RELEASE here ensures that the previous store to service_mask is guaranteed to be visible on all cores if this store is visible. Why do we care about this property? The service_mask is core local anway.

4) Even with the load ACQ service_mask, and REL num_mapped_cores store, is there not still a race-condition possible where 2 lcores simultaneously load-ACQ the service_mask, and then both do atomic add/sub_fetch with REL?

5) Assuming 4 above race is true, it raises the real question - the service-cores control APIs are not designed to be multi-thread-safe. Orchestration of service/lcore mappings is not meant to be done by multiple threads at the same time. Documenting this loudly may help, I'm happy to send a patch to do so if we're agreed on the above?

Quoted 32 lines
lib/librte_eal/common/rte_service.c +4 -5
 	}
 
 	if (enabled)
 		*enabled = !!(lcore_states[lcore].service_mask & (sid_mask));
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -625,7 +643,8 @@ rte_service_lcore_reset_all(void)
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
-		rte_atomic32_set(&rte_services[i].num_mapped_cores, 0);
+		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
+				    __ATOMIC_RELAXED);
 
 	rte_smp_wmb();
 
@@ -708,7 +727,8 @@ rte_service_lcore_stop(uint32_t lcore)
 		int32_t enabled = service_mask & (UINT64_C(1) << i);
 		int32_t service_running = rte_service_runstate_get(i);
 		int32_t only_core = (1 ==
-			rte_atomic32_read(&rte_services[i].num_mapped_cores));
+			__atomic_load_n(&rte_services[i].num_mapped_cores,
+					__ATOMIC_RELAXED));
 
 		/* if the core is mapped, and the service is running, and this
 		 * is the only core that is mapped, the service would cease to
--

2.7.4

#123 lore raw

Re: [dpdk-dev] [PATCH v3 11/12] service: optimize with c11 one-way barrier

<snip>

Quoted 94 lines
Quoted 11 lines

Subject: [PATCH v3 11/12] service: optimize with c11 one-way barrier

The num_mapped_cores and execute_lock are synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with
c11 atomic one-way barrier.

Based on discussion on-list, it seems the consensus is to not use GCC builtins, but instead use C11 APIs "proper"? If my conclusion is correct, the v+1 of this patchset would require updates to that style API.

Inline comments for context below, -Harry

Quoted 21 lines
---
 lib/librte_eal/common/rte_service.c | 50

++++++++++++++++++++++++++----------

(quoted diff) +0 -1
-
 1 file changed, 35 insertions(+), 15 deletions(-)
lib/librte_eal/common/rte_service.c +1 -1
index 0843c3c..c033224 100644
@@ -42,7 +42,7 @@ struct rte_service_spec_impl {
 	 * running this service callback. When not set, a core may take the
 	 * lock and then run the service callback.
 	 */
-	rte_atomic32_t execute_lock;
+	uint32_t execute_lock;
 
 	/* API set/get-able variables */
 	int8_t app_runstate;

lib/librte_eal/common/rte_service.c struct rte_service_spec_impl { · line 57

Quoted 5 lines
lib/librte_eal/common/rte_service.c +1 -1
@@ -54,7 +54,7 @@ struct rte_service_spec_impl {
 	 * It does not indicate the number of cores the service is running
 	 * on currently.
 	 */
-	rte_atomic32_t num_mapped_cores;
+	int32_t num_mapped_cores;

Any reason why "int32_t" or "uint32_t" is used over another? execute_lock is a uint32_t above, num_mapped_cores is an int32_t?

lib/librte_eal/common/rte_service.c rte_service_runstate_get(uint32_t id) · line 335

Quoted 8 lines
lib/librte_eal/common/rte_service.c +0 -1
 	uint64_t calls;
 	uint64_t cycles_spent;
 } __rte_cache_aligned;
@@ -332,7 +332,8 @@ rte_service_runstate_get(uint32_t id)
 	rte_smp_rmb();
 
 	int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK);
-	int lcore_mapped = (rte_atomic32_read(&s->num_mapped_cores) >

0);

lib/librte_eal/common/rte_service.c rte_service_runstate_get(uint32_t id)

Quoted 5 lines
lib/librte_eal/common/rte_service.c +2 -0
+	int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores,
+					    __ATOMIC_RELAXED) > 0);
 
 	return (s->app_runstate == RUNSTATE_RUNNING) &&
 		(s->comp_runstate == RUNSTATE_RUNNING) && @@ -375,11

+376,20 @@

lib/librte_eal/common/rte_service.c service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,

Quoted 11 lines

service_run(uint32_t i, struct core_state *cs, uint64_t service_mask,

lib/librte_eal/common/rte_service.c +6 -1
 	cs->service_active_on_lcore[i] = 1;
 
 	if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) {
-		if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1))
+		uint32_t expected = 0;
+		/* ACQUIRE ordering here is to prevent the callback
+		 * function from hoisting up before the execute_lock
+		 * setting.
+		 */
+		if (!__atomic_compare_exchange_n(&s->execute_lock,

&expected, 1,

lib/librte_eal/common/rte_service.c service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, · line 386

Quoted 2 lines
lib/librte_eal/common/rte_service.c +1 -0
+			    0, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED))
 			return -EBUSY;

Let's try improve the magic "1" and "0" constants, I believe the "1" here is the desired "new value on success", and the 0 is "bool weak", where our 0/false constant implies a strongly ordered compare exchange?

"Weak is true for weak compare_exchange, which may fail spuriously, and false for the strong variation, which never fails spuriously.", from
https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html

const uint32_t on_success_value = 1;
const bool weak = 0; __atomic_compare_exchange_n(&s->execute_lock, &expected, on_success_value, weak, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED);

Quoted 6 lines

Although a bit more verbose, I feel this documents usage a lot better, particularly for those who aren't as familiar with the C11 function arguments order.

Admittedly with the API change to not use __builtins, perhaps this comment is moot.

Suggest changing the execute_lock to rte_spinlock_t and use rte_spinlock_trylock API.

Quoted 71 lines
Quoted 15 lines
lib/librte_eal/common/rte_service.c +4 -1
 		service_runner_do_callback(s, cs, i);
-		rte_atomic32_clear(&s->execute_lock);
+		/* RELEASE ordering here is used to pair with ACQUIRE
+		 * above to achieve lock semantic.
+		 */
+		__atomic_store_n(&s->execute_lock, 0, __ATOMIC_RELEASE);
 	} else
 		service_runner_do_callback(s, cs, i);
@@ -415,11 +425,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id,

uint32_t
serialize_mt_unsafe)

lib/librte_eal/common/rte_service.c rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)

Quoted 5 lines
lib/librte_eal/common/rte_service.c +1 -1
 	/* Increment num_mapped_cores to indicate that the service
 	 * is running on a core.
 	 */
-	rte_atomic32_inc(&s->num_mapped_cores);
+	__atomic_add_fetch(&s->num_mapped_cores, 1,

__ATOMIC_ACQUIRE);

lib/librte_eal/common/rte_service.c rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe)

Quoted 5 lines
lib/librte_eal/common/rte_service.c +1 -1
 	int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe);
 
-	rte_atomic32_dec(&s->num_mapped_cores);
+	__atomic_sub_fetch(&s->num_mapped_cores, 1,

__ATOMIC_RELEASE);

Quoted 9 lines
lib/librte_eal/common/rte_service.c +0 -2
 	return ret;
 }
@@ -552,24 +562,32 @@ service_update(uint32_t sid, uint32_t lcore,
 
 	uint64_t sid_mask = UINT64_C(1) << sid;
 	if (set) {
-		uint64_t lcore_mapped = lcore_states[lcore].service_mask &
-			sid_mask;

lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 574

Quoted 10 lines
lib/librte_eal/common/rte_service.c +10 -0
+		/* When multiple threads try to update the same lcore
+		 * service concurrently, e.g. set lcore map followed
+		 * by clear lcore map, the unsynchronized service_mask
+		 * values have issues on the num_mapped_cores value
+		 * consistency. So we use ACQUIRE ordering to pair with
+		 * the RELEASE ordering to synchronize the service_mask.
+		 */
+		uint64_t lcore_mapped = __atomic_load_n(
+					&lcore_states[lcore].service_mask,
+					__ATOMIC_ACQUIRE) & sid_mask;

Thanks for the comment - it helps me understand things a bit better. Some questions/theories to validate;
1) The service_mask ACQUIRE avoids other loads being hoisted above it, correct?

2) There are non-atomic stores to service_mask. Is it correct that the stores themselves aren't the issue, but relative visibility of service_mask stores vs num_mapped_cores? (Detail in (3) below)

lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 572

Quoted 3 lines
lib/librte_eal/common/rte_service.c +0 -1
 		if (*set && !lcore_mapped) {
 			lcore_states[lcore].service_mask |= sid_mask;
-
	rte_atomic32_inc(&rte_services[sid].num_mapped_cores);

lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 577

Quoted 1 line
lib/librte_eal/common/rte_service.c +1 -0
+

__atomic_add_fetch(&rte_services[sid].num_mapped_cores,

lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 572

Quoted 5 lines
lib/librte_eal/common/rte_service.c +1 -1
+					    1, __ATOMIC_RELEASE);
 		}
 		if (!*set && lcore_mapped) {
 			lcore_states[lcore].service_mask &= ~(sid_mask);
-
	rte_atomic32_dec(&rte_services[sid].num_mapped_cores);

lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 582

Quoted 1 line
lib/librte_eal/common/rte_service.c +1 -0
+

__atomic_sub_fetch(&rte_services[sid].num_mapped_cores,

Quoted 8 lines

lib/librte_eal/common/rte_service.c service_update(uint32_t sid, uint32_t lcore, · line 584

Quoted 2 lines
lib/librte_eal/common/rte_service.c +1 -0
+					    1, __ATOMIC_RELEASE);
 		}

3) Here we update the core-local service_mask, and then update the num_mapped_cores with an ATOMIC_RELEASE. The RELEASE here ensures that the previous store to service_mask is guaranteed to be visible on all cores if this store is visible. Why do we care about this property?
The service_mask is core local anway.

We are working on concurrency between the reader and writer. The service_mask is local to the core, but it is accessed by a reader and writer.
I think we should wait to conclude on the meaning of 'num_mapped_cores', that will dictate what the order should be. For ex: if it is just for statistics purpose, then we could use just RELAXED memory order and then the order for service_mask will also change.

Quoted 10 lines

4) Even with the load ACQ service_mask, and REL num_mapped_cores store, is there not still a race-condition possible where 2 lcores simultaneously load- ACQ the service_mask, and then both do atomic add/sub_fetch with REL?

5) Assuming 4 above race is true, it raises the real question - the service-cores control APIs are not designed to be multi-thread-safe. Orchestration of service/lcore mappings is not meant to be done by multiple threads at the same time. Documenting this loudly may help, I'm happy to send a patch to do so if we're agreed on the above?

I completely agree here. writer-writer concurrency is another topic and we should (for now at least) say that the control plane APIs are not thread safe.

Quoted 38 lines
Quoted 25 lines
lib/librte_eal/common/rte_service.c +2 -3
 	}
 
 	if (enabled)
 		*enabled = !!(lcore_states[lcore].service_mask & (sid_mask));
 
-	rte_smp_wmb();
-
 	return 0;
 }
 
@@ -625,7 +643,8 @@ rte_service_lcore_reset_all(void)
 		}
 	}
 	for (i = 0; i < RTE_SERVICE_NUM_MAX; i++)
-		rte_atomic32_set(&rte_services[i].num_mapped_cores, 0);
+		__atomic_store_n(&rte_services[i].num_mapped_cores, 0,
+				    __ATOMIC_RELAXED);
 
 	rte_smp_wmb();
 

lib/librte_eal/common/rte_service.c rte_service_lcore_stop(uint32_t lcore) · line 711

Quoted 4 lines
lib/librte_eal/common/rte_service.c +0 -1
@@ -708,7 +727,8 @@ rte_service_lcore_stop(uint32_t lcore)
 		int32_t enabled = service_mask & (UINT64_C(1) << i);
 		int32_t service_running = rte_service_runstate_get(i);
 		int32_t only_core = (1 ==
-
	rte_atomic32_read(&rte_services[i].num_mapped_cores));

lib/librte_eal/common/rte_service.c rte_service_lcore_stop(uint32_t lcore) · line 730

Quoted 1 line
lib/librte_eal/common/rte_service.c +1 -0
+
	__atomic_load_n(&rte_services[i].num_mapped_cores,
Quoted 6 lines
lib/librte_eal/common/rte_service.c +1 -1
+					__ATOMIC_RELAXED));
 
 		/* if the core is mapped, and the service is running, and this
 		 * is the only core that is mapped, the service would cease to
--

2.7.4

author
#124 lore raw

Re: [dpdk-dev] [PATCH v3 11/12] service: optimize with c11 one-way barrier

In reply to Van Haaren, Harry
Quoted 64 lines

-----Original Message-----
From: Van Haaren, Harry <[email protected]>
Sent: Friday, April 3, 2020 7:58 PM
To: Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 11/12] service: optimize with c11 one-way barrier

Quoted 4 lines

-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry

<[email protected]>;

Quoted 2 lines

Ananyev, Konstantin <[email protected]>; [email protected]; [email protected];

[email protected]

Quoted 1 line

[email protected];

Quoted 1 line

[email protected]; [email protected];

[email protected];

Quoted 4 lines

[email protected]; [email protected]
Subject: [PATCH v3 11/12] service: optimize with c11 one-way barrier

The num_mapped_cores and execute_lock are synchronized with

rte_atomic_XX

Quoted 7 lines

APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier.

Based on discussion on-list, it seems the consensus is to not use GCC builtins, but instead use C11 APIs "proper"? If my conclusion is correct, the v+1 of this patchset would require updates to that style API.

Inline comments for context below, -Harry

Quoted 2 lines
---
 lib/librte_eal/common/rte_service.c | 50

++++++++++++++++++++++++++----------

Quoted 15 lines
(quoted diff) +0 -1
-
 1 file changed, 35 insertions(+), 15 deletions(-)
lib/librte_eal/common/rte_service.c +1 -1
index 0843c3c..c033224 100644
@@ -42,7 +42,7 @@ struct rte_service_spec_impl {
 	 * running this service callback. When not set, a core may take the
 	 * lock and then run the service callback.
 	 */
-	rte_atomic32_t execute_lock;
+	uint32_t execute_lock;
 
Quoted 11 lines

lib/librte_eal/common/rte_service.c struct rte_service_spec_impl { · line 57

Quoted 8 lines
lib/librte_eal/common/rte_service.c +1 -1
 	/* API set/get-able variables */
 	int8_t app_runstate;
@@ -54,7 +54,7 @@ struct rte_service_spec_impl {
 	 * It does not indicate the number of cores the service is running
 	 * on currently.
 	 */
-	rte_atomic32_t num_mapped_cores;
+	int32_t num_mapped_cores;

Any reason why "int32_t" or "uint32_t" is used over another? execute_lock is a uint32_t above, num_mapped_cores is an int32_t?

It should be uint32_t for num_mapped_cores.
This value will not be negative after __atomic_sub_fetch operation, because of the sequence of writer and reader accesses are guaranteed by the memory ordering. I will update it in the next version.

Thanks,
Phil

<snip>

author
#125 lore raw

PATCH v3 12/12 service: relax barriers with C11 atomic operations

In reply to Phil Yang

To guarantee the inter-threads visibility of the shareable domain, it uses a lot of rte_smp_r/wmb in the service library. This patch relaxed these barriers for service by using c11 atomic one-way barrier operations.

---
 lib/librte_eal/common/rte_service.c | 45 ++++++++++++++++++++-----------------
 1 file changed, 25 insertions(+), 20 deletions(-)
lib/librte_eal/common/rte_service.c +25 -20
index c033224..d31663e 100644
@@ -179,9 +179,11 @@ rte_service_set_stats_enable(uint32_t id, int32_t enabled)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, 0);
 
 	if (enabled)
-		s->internal_flags |= SERVICE_F_STATS_ENABLED;
+		__atomic_or_fetch(&s->internal_flags, SERVICE_F_STATS_ENABLED,
+			__ATOMIC_RELEASE);
 	else
-		s->internal_flags &= ~(SERVICE_F_STATS_ENABLED);
+		__atomic_and_fetch(&s->internal_flags,
+			~(SERVICE_F_STATS_ENABLED), __ATOMIC_RELEASE);
 
 	return 0;
 }
@@ -193,9 +195,11 @@ rte_service_set_runstate_mapped_check(uint32_t id, int32_t enabled)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, 0);
 
 	if (enabled)
-		s->internal_flags |= SERVICE_F_START_CHECK;
+		__atomic_or_fetch(&s->internal_flags, SERVICE_F_START_CHECK,
+			__ATOMIC_RELEASE);
 	else
-		s->internal_flags &= ~(SERVICE_F_START_CHECK);
+		__atomic_and_fetch(&s->internal_flags, ~(SERVICE_F_START_CHECK),
+			__ATOMIC_RELEASE);
 
 	return 0;
 }
@@ -264,8 +268,8 @@ rte_service_component_register(const struct rte_service_spec *spec,
 	s->spec = *spec;
 	s->internal_flags |= SERVICE_F_REGISTERED | SERVICE_F_START_CHECK;
 
-	rte_smp_wmb();
-	rte_service_count++;
+	/* make sure the counter update after the state change. */
+	__atomic_add_fetch(&rte_service_count, 1, __ATOMIC_RELEASE);
 
 	if (id_ptr)
 		*id_ptr = free_slot;
@@ -281,9 +285,10 @@ rte_service_component_unregister(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	rte_service_count--;
-	rte_smp_wmb();
 
-	s->internal_flags &= ~(SERVICE_F_REGISTERED);
+	/* make sure the counter update before the state change. */
+	__atomic_and_fetch(&s->internal_flags, ~(SERVICE_F_REGISTERED),
+			   __ATOMIC_RELEASE);
 
 	/* clear the run-bit in all cores */
 	for (i = 0; i < RTE_MAX_LCORE; i++)
@@ -301,11 +306,12 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	if (runstate)
-		s->comp_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->comp_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -316,11 +322,12 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	if (runstate)
-		s->app_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->app_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -442,7 +449,8 @@ service_runner_func(void *arg)
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (lcore_states[lcore].runstate == RUNSTATE_RUNNING) {
+	while (__atomic_load_n(&cs->runstate,
+		    __ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
@@ -453,8 +461,6 @@ service_runner_func(void *arg)
 		}
 
 		cs->loops++;
-
-		rte_smp_rmb();
 	}
 
 	lcore_config[lcore].state = WAIT;
@@ -663,9 +669,8 @@ rte_service_lcore_add(uint32_t lcore)
 
 	/* ensure that after adding a core the mask and state are defaults */
 	lcore_states[lcore].service_mask = 0;
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
-
-	rte_smp_wmb();
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+			__ATOMIC_RELEASE);
 
 	return rte_eal_wait_lcore(lcore);
 }
Signature · 1 line
2.7.4
#126 lore raw

Re: [dpdk-dev] [PATCH v3 12/12] service: relax barriers with C11 atomic operations

Quoted 29 lines

From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 12/12] service: relax barriers with C11 atomic operations

To guarantee the inter-threads visibility of the shareable domain, it uses a lot of rte_smp_r/wmb in the service library. This patch relaxed these barriers for service by using c11 atomic one-way barrier operations.

---
 lib/librte_eal/common/rte_service.c | 45 ++++++++++++++++++++----------------
(quoted diff) +0 -1
-
 1 file changed, 25 insertions(+), 20 deletions(-)
lib/librte_eal/common/rte_service.c +0 -0
index c033224..d31663e 100644
@@ -179,9 +179,11 @@ rte_service_set_stats_enable(uint32_t id, int32_t
 enabled)

lib/librte_eal/common/rte_service.c rte_service_set_stats_enable(uint32_t id, int32_t · line 187

Quoted 10 lines
lib/librte_eal/common/rte_service.c +4 -2
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, 0);
 
 	if (enabled)
-		s->internal_flags |= SERVICE_F_STATS_ENABLED;
+		__atomic_or_fetch(&s->internal_flags, SERVICE_F_STATS_ENABLED,
+			__ATOMIC_RELEASE);
 	else
-		s->internal_flags &= ~(SERVICE_F_STATS_ENABLED);
+		__atomic_and_fetch(&s->internal_flags,
+			~(SERVICE_F_STATS_ENABLED), __ATOMIC_RELEASE);

Not sure why these have to become stores with RELEASE memory ordering? (More occurances of same Q below, just answer here?)

Quoted 4 lines
lib/librte_eal/common/rte_service.c +0 -0
 	return 0;
 }
@@ -193,9 +195,11 @@ rte_service_set_runstate_mapped_check(uint32_t id,
 int32_t enabled)

lib/librte_eal/common/rte_service.c rte_service_set_runstate_mapped_check(uint32_t id, · line 203

Quoted 10 lines
lib/librte_eal/common/rte_service.c +4 -2
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, 0);
 
 	if (enabled)
-		s->internal_flags |= SERVICE_F_START_CHECK;
+		__atomic_or_fetch(&s->internal_flags, SERVICE_F_START_CHECK,
+			__ATOMIC_RELEASE);
 	else
-		s->internal_flags &= ~(SERVICE_F_START_CHECK);
+		__atomic_and_fetch(&s->internal_flags, ~(SERVICE_F_START_CHECK),
+			__ATOMIC_RELEASE);

Same as above, why do these require RELEASE?

Remainder of patch below seems to make sense - there's a wmb() involved hence RELEASE m/o.

lib/librte_eal/common/rte_service.c rte_service_component_register(const struct · line 273

Quoted 11 lines
lib/librte_eal/common/rte_service.c +2 -2
 	return 0;
 }
@@ -264,8 +268,8 @@ rte_service_component_register(const struct
 rte_service_spec *spec,
 	s->spec = *spec;
 	s->internal_flags |= SERVICE_F_REGISTERED | SERVICE_F_START_CHECK;
 
-	rte_smp_wmb();
-	rte_service_count++;
+	/* make sure the counter update after the state change. */
+	__atomic_add_fetch(&rte_service_count, 1, __ATOMIC_RELEASE);

This makes sense to me - the RELEASE ensures that previous stores to the s->internal_flags are visible to other cores before rte_service_count increments atomically.

Quoted 82 lines
lib/librte_eal/common/rte_service.c +15 -15
 	if (id_ptr)
 		*id_ptr = free_slot;
@@ -281,9 +285,10 @@ rte_service_component_unregister(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	rte_service_count--;
-	rte_smp_wmb();
 
-	s->internal_flags &= ~(SERVICE_F_REGISTERED);
+	/* make sure the counter update before the state change. */
+	__atomic_and_fetch(&s->internal_flags, ~(SERVICE_F_REGISTERED),
+			   __ATOMIC_RELEASE);
 
 	/* clear the run-bit in all cores */
 	for (i = 0; i < RTE_MAX_LCORE; i++)
@@ -301,11 +306,12 @@ rte_service_component_runstate_set(uint32_t id, uint32_t
 runstate)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	if (runstate)
-		s->comp_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->comp_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
@@ -316,11 +322,12 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	if (runstate)
-		s->app_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->app_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
@@ -442,7 +449,8 @@ service_runner_func(void *arg)
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (lcore_states[lcore].runstate == RUNSTATE_RUNNING) {
+	while (__atomic_load_n(&cs->runstate,
+		    __ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) {
@@ -453,8 +461,6 @@ service_runner_func(void *arg)
 		}
 
 		cs->loops++;
-
-		rte_smp_rmb();
 	}
 
 	lcore_config[lcore].state = WAIT;
@@ -663,9 +669,8 @@ rte_service_lcore_add(uint32_t lcore)
 
 	/* ensure that after adding a core the mask and state are defaults */
 	lcore_states[lcore].service_mask = 0;
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
-
-	rte_smp_wmb();
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+			__ATOMIC_RELEASE);
 
 	return rte_eal_wait_lcore(lcore);
 }
--

2.7.4

#127 lore raw

Re: [dpdk-dev] [PATCH v3 12/12] service: relax barriers with C11 atomic operations

<snip>
Just to get us on the same page on 'techniques to communicate data from writer to reader' (apologies if it is too trivial)

Let us say that the writer has 512B (key point is - cannot be written atomically) that needs to be communicated to the reader.

Since the data cannot be written atomically, we need a guard variable (which can be written atomically, can be a flag or pointer to data). So, the writer will store 512B in non-atomic way and write to guard variable with release memory order. So, if the guard variable is valid (set in the case of flag or not null in the case of pointer), it guarantees that 512B is written.

The reader will read the guard variable with acquire memory order and read the 512B data only if the guard variable is valid. So, the acquire memory order on the guard variable guarantees that the load of 512B does not happen before the guard variable is read. The validity check on the guard variable guarantees that 512B was written before it was read.

The store(guard_variable, RELEASE) on the writer and the load(guard_variable, ACQUIRE) can be said as synchronizing with each other.

(the guard variable technique applies even if we are not using C11 atomics)

Let us say that the writer has 4B (key point is - can be written atomically) that needs to be communicated to the reader. The writer is free to write this atomically with no constraints on memory ordering as long as this data is not acting as a guard variable for any other data.

In my understanding, the sequence of APIs to call to start a service (writer) are as follows:
1) rte_service_init
2) rte_service_component_register
3) <possible configuration of the service>
4) rte_service_component_runstate_set (the reader is allowed at this point to read the information about the service - written by rte_service_component_register API. This API should not be called before rte_service_component_register)
5) <possible configuration of the service>
6) rte_service_runstate_set (the reader is allowed at this point to read the information about the service - written by rte_service_component_register API and run the service. This API can be called anytime. But, the reader should not attempt to run the service before this API is called)
7) rte_lcore_service_add (multiple of these probably, can be called before this, can't be called later)
8) rte_service_map_lcore_set (this can be called anytime. Can be called even if the service is not registered)
9) rte_service_lcore_start (again, this can be called anytime, even before the service is registered)

So, there are 2 guard variables - 'comp_runstate' and 'app_runstate'. Only these 2 need to have RELEASE ordering in writer and ACQUIRE ordering in reader.

We can write test cases with different orders of these API calls to prove that the memory orders we use are sufficient.

Few comments are inline based on this assessment.

Quoted 42 lines

Subject: RE: [PATCH v3 12/12] service: relax barriers with C11 atomic operations

Quoted 9 lines

From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected]

Quoted 22 lines

Subject: [PATCH v3 12/12] service: relax barriers with C11 atomic operations

To guarantee the inter-threads visibility of the shareable domain, it uses a lot of rte_smp_r/wmb in the service library. This patch relaxed these barriers for service by using c11 atomic one-way barrier operations.

---
 lib/librte_eal/common/rte_service.c | 45

++++++++++++++++++++----------------

(quoted diff) +0 -1
-
 1 file changed, 25 insertions(+), 20 deletions(-)
lib/librte_eal/common/rte_service.c +0 -0
index c033224..d31663e 100644

lib/librte_eal/common/rte_service.c rte_service_set_stats_enable(uint32_t id, int32_t · line 183

Quoted 6 lines
lib/librte_eal/common/rte_service.c +1 -1
@@ -179,9 +179,11 @@ rte_service_set_stats_enable(uint32_t id, int32_t
 enabled)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, 0);
 
 	if (enabled)
-		s->internal_flags |= SERVICE_F_STATS_ENABLED;
+		__atomic_or_fetch(&s->internal_flags,

SERVICE_F_STATS_ENABLED,

Quoted 8 lines

lib/librte_eal/common/rte_service.c rte_service_set_stats_enable(uint32_t id, int32_t enabled) · line 186

Quoted 5 lines
lib/librte_eal/common/rte_service.c +3 -1
+			__ATOMIC_RELEASE);
 	else
-		s->internal_flags &= ~(SERVICE_F_STATS_ENABLED);
+		__atomic_and_fetch(&s->internal_flags,
+			~(SERVICE_F_STATS_ENABLED), __ATOMIC_RELEASE);

Not sure why these have to become stores with RELEASE memory ordering? (More occurances of same Q below, just answer here?)

Agree, 'internal_flags' is not acting as a guard variable, this should be RELAXED (similarly for the others below). Though I suggest keeping it atomic.

Quoted 11 lines

lib/librte_eal/common/rte_service.c rte_service_set_runstate_mapped_check(uint32_t · line 199

Quoted 10 lines
lib/librte_eal/common/rte_service.c +1 -1
 	return 0;
 }
@@ -193,9 +195,11 @@ rte_service_set_runstate_mapped_check(uint32_t
 id, int32_t enabled)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, 0);
 
 	if (enabled)
-		s->internal_flags |= SERVICE_F_START_CHECK;
+		__atomic_or_fetch(&s->internal_flags,

SERVICE_F_START_CHECK,

Quoted 8 lines

lib/librte_eal/common/rte_service.c rte_service_set_runstate_mapped_check(uint32_t id, int32_t enabled)

Quoted 4 lines
lib/librte_eal/common/rte_service.c +2 -1
+			__ATOMIC_RELEASE);
 	else
-		s->internal_flags &= ~(SERVICE_F_START_CHECK);
+		__atomic_and_fetch(&s->internal_flags,

~(SERVICE_F_START_CHECK),

lib/librte_eal/common/rte_service.c rte_service_set_runstate_mapped_check(uint32_t · line 203

Quoted 1 line
lib/librte_eal/common/rte_service.c +1 -0
+			__ATOMIC_RELEASE);

Same as above, why do these require RELEASE?

Agree

Quoted 13 lines

Remainder of patch below seems to make sense - there's a wmb() involved hence RELEASE m/o.

lib/librte_eal/common/rte_service.c rte_service_component_register(const struct · line 270

Quoted 6 lines
lib/librte_eal/common/rte_service.c +0 -0
 	return 0;
 }
@@ -264,8 +268,8 @@ rte_service_component_register(const struct
 rte_service_spec *spec,
 	s->spec = *spec;
 	s->internal_flags |= SERVICE_F_REGISTERED |

SERVICE_F_START_CHECK;

Quoted 1 line
Quoted 8 lines

lib/librte_eal/common/rte_service.c rte_service_component_register(const struct rte_service_spec *spec, · line 272

Quoted 4 lines
lib/librte_eal/common/rte_service.c +2 -2
-	rte_smp_wmb();
-	rte_service_count++;
+	/* make sure the counter update after the state change. */
+	__atomic_add_fetch(&rte_service_count, 1, __ATOMIC_RELEASE);

This makes sense to me - the RELEASE ensures that previous stores to the s->internal_flags are visible to other cores before rte_service_count increments atomically.

rte_service_count is not a guard variable, does not need RELEASE order. It is also not used by the reader. It looks like it is just a statistic being maintained.

Quoted 7 lines
Quoted 7 lines
lib/librte_eal/common/rte_service.c +0 -0
 	if (id_ptr)
 		*id_ptr = free_slot;
@@ -281,9 +285,10 @@ rte_service_component_unregister(uint32_t id)
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
Quoted 7 lines

lib/librte_eal/common/rte_service.c rte_service_component_unregister(uint32_t id) · line 291

Quoted 7 lines
lib/librte_eal/common/rte_service.c +3 -2
 	rte_service_count--;
-	rte_smp_wmb();
 
-	s->internal_flags &= ~(SERVICE_F_REGISTERED);
+	/* make sure the counter update before the state change. */
+	__atomic_and_fetch(&s->internal_flags, ~(SERVICE_F_REGISTERED),
+			   __ATOMIC_RELEASE);

RELAXED is enough.

Quoted 8 lines
Quoted 8 lines
lib/librte_eal/common/rte_service.c +0 -0
 	/* clear the run-bit in all cores */
 	for (i = 0; i < RTE_MAX_LCORE; i++)
@@ -301,11 +306,12 @@ rte_service_component_runstate_set(uint32_t id,

uint32_t
runstate)
SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);

Quoted 8 lines

lib/librte_eal/common/rte_service.c rte_service_component_runstate_set(uint32_t id, uint32_t runstate) · line 313

Quoted 8 lines
lib/librte_eal/common/rte_service.c +4 -2
 	if (runstate)
-		s->comp_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->comp_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->comp_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);

Here we need a thread_fence to prevent the memory operations from a subsequent call to 'rte_service_component_unregister' from getting hoisted above this. The user should be forced to call rte_service_component_unregister before calling rte_service_component_runstate_set. I suggest adding a check in rte_service_component_unregister to ensure that the state is set to RUNSTATE_STOPPED. In fact, the user needs to make sure that the service is stopped for sure before calling rte_service_component_unregister.

Quoted 23 lines

lib/librte_eal/common/rte_service.c rte_service_runstate_set(uint32_t id, uint32_t

Quoted 7 lines
lib/librte_eal/common/rte_service.c +0 -1
-	rte_smp_wmb();
 	return 0;
 }
@@ -316,11 +322,12 @@ rte_service_runstate_set(uint32_t id, uint32_t

runstate)

Quoted 15 lines
lib/librte_eal/common/rte_service.c +4 -3
 	SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL);
 
 	if (runstate)
-		s->app_runstate = RUNSTATE_RUNNING;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_RUNNING,
+				__ATOMIC_RELEASE);
 	else
-		s->app_runstate = RUNSTATE_STOPPED;
+		__atomic_store_n(&s->app_runstate, RUNSTATE_STOPPED,
+				__ATOMIC_RELEASE);
 
-	rte_smp_wmb();
 	return 0;
 }
 
Quoted 7 lines

lib/librte_eal/common/rte_service.c service_runner_func(void *arg) · line 453

Quoted 7 lines
lib/librte_eal/common/rte_service.c +2 -1
@@ -442,7 +449,8 @@ service_runner_func(void *arg)
 	const int lcore = rte_lcore_id();
 	struct core_state *cs = &lcore_states[lcore];
 
-	while (lcore_states[lcore].runstate == RUNSTATE_RUNNING) {
+	while (__atomic_load_n(&cs->runstate,
+		    __ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) {

This can be RELAXED, lcore's runstate is not acting as a guard variable.
However, note that the writer thread wants to communicate the 'runstate' (4B) to the reader thread. This ordering needs to be handled in 'rte_eal_remote_launch' and 'eal_thread_loop' functions. We have to note that in some other use case, the writer wants to communicate more than 4B to reader. Currently, the 'write' and 'read' system calls may have enough barriers to make things work fine. But, I suggest using the ' lcore_config[lcore].f' as the guard variable to make it explicit and not depend on 'write' and 'read'. We can take up the EAL things in a later patch as it does not cause any issues right now.

Quoted 12 lines

lib/librte_eal/common/rte_service.c service_runner_func(void *arg)

Quoted 3 lines
lib/librte_eal/common/rte_service.c +0 -0
 		const uint64_t service_mask = cs->service_mask;
 
 		for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) { @@ -453,8

+461,6 @@

Quoted 8 lines

service_runner_func(void *arg)

lib/librte_eal/common/rte_service.c +0 -2
 		}
 
 		cs->loops++;
-
-		rte_smp_rmb();
 	}
Quoted 8 lines

lib/librte_eal/common/rte_service.c rte_service_lcore_add(uint32_t lcore) · line 668

Quoted 8 lines
lib/librte_eal/common/rte_service.c +0 -3
 	lcore_config[lcore].state = WAIT;
@@ -663,9 +669,8 @@ rte_service_lcore_add(uint32_t lcore)
 
 	/* ensure that after adding a core the mask and state are defaults */
 	lcore_states[lcore].service_mask = 0;
-	lcore_states[lcore].runstate = RUNSTATE_STOPPED;
-
-	rte_smp_wmb();

Agree.

Quoted 2 lines

lib/librte_eal/common/rte_service.c rte_service_lcore_add(uint32_t lcore) · line 673

Quoted 2 lines
lib/librte_eal/common/rte_service.c +2 -0
+	__atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED,
+			__ATOMIC_RELEASE);

This can be relaxed.

Quoted 5 lines
Quoted 5 lines
lib/librte_eal/common/rte_service.c +0 -1
 	return rte_eal_wait_lcore(lcore);
 }
--

2.7.4

#128 lore raw

Re: [dpdk-dev] [PATCH v3 12/12] service: relax barriers with C11 atomic operations

Quoted 33 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Monday, April 6, 2020 6:06 PM
To: Van Haaren, Harry <[email protected]>; Phil Yang <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Gavin Hu <[email protected]>; Ruifeng Wang <[email protected]>; Joyce Kong <[email protected]>; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH v3 12/12] service: relax barriers with C11 atomic operations

<snip>
Just to get us on the same page on 'techniques to communicate data from writer to reader' (apologies if it is too trivial)

Let us say that the writer has 512B (key point is - cannot be written atomically) that needs to be communicated to the reader.

Since the data cannot be written atomically, we need a guard variable (which can be written atomically, can be a flag or pointer to data). So, the writer will store 512B in non-atomic way and write to guard variable with release memory order. So, if the guard variable is valid (set in the case of flag or not null in the case of pointer), it guarantees that 512B is written.

The reader will read the guard variable with acquire memory order and read the 512B data only if the guard variable is valid. So, the acquire memory order on the guard variable guarantees that the load of 512B does not happen before the guard variable is read. The validity check on the guard variable guarantees that 512B was written before it was read.

Quoted 4 lines

The store(guard_variable, RELEASE) on the writer and the load(guard_variable, ACQUIRE) can be said as synchronizing with each other.

(the guard variable technique applies even if we are not using C11 atomics)

Yep agreed on the above.

Quoted 17 lines

Let us say that the writer has 4B (key point is - can be written atomically) that needs to be communicated to the reader. The writer is free to write this atomically with no constraints on memory ordering as long as this data is not acting as a guard variable for any other data.

In my understanding, the sequence of APIs to call to start a service (writer) are as follows:
1) rte_service_init
2) rte_service_component_register
3) <possible configuration of the service>
4) rte_service_component_runstate_set (the reader is allowed at this point to read the information about the service - written by rte_service_component_register API. This API should not be called before rte_service_component_register)
5) <possible configuration of the service>
6) rte_service_runstate_set (the reader is allowed at this point to read the information about the service - written by rte_service_component_register API

Quoted 8 lines

and run the service. This API can be called anytime. But, the reader should not attempt to run the service before this API is called)
7) rte_lcore_service_add (multiple of these probably, can be called before this, can't be called later)
8) rte_service_map_lcore_set (this can be called anytime. Can be called even if the service is not registered)
9) rte_service_lcore_start (again, this can be called anytime, even before the service is registered)

I think this can be simplified, if we look at calling threads:
- one thread is the writer/config thread, and is allowed to call anything --- Any updates/changes must be atomically correct in how the other threads can read state.

- service lcores, which fundamentally spin in run(), and call services mapped to it --- here we need to ensure any service mapped to it is atomic, and the service is valid to run.

- other application threads using "run on app lcore" function
--- similar to service lcore, check for service in valid state, and allow to run.

Services are not allowed to be unregistered e.g. while running. I'd like to avoid the "explosion of combinations" by enforcing simple limitations (and documenting them more clearly if/where required).

Quoted 8 lines

So, there are 2 guard variables - 'comp_runstate' and 'app_runstate'. Only these 2 need to have RELEASE ordering in writer and ACQUIRE ordering in reader.

We can write test cases with different orders of these API calls to prove that the memory orders we use are sufficient.

Few comments are inline based on this assessment.

Sure. As per other email thread, splitting changes into bugfix/cleanup/C11 would likely help to try keep track of changes etc required per patch, its getting hard to follow the various topics being discussed in parallel.

Quoted 20 lines
Quoted 20 lines

Subject: RE: [PATCH v3 12/12] service: relax barriers with C11 atomic operations

Quoted 9 lines

From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected]

Quoted 7 lines

Subject: [PATCH v3 12/12] service: relax barriers with C11 atomic operations

To guarantee the inter-threads visibility of the shareable domain, it uses a lot of rte_smp_r/wmb in the service library. This patch relaxed these barriers for service by using c11 atomic one-way barrier operations.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
---
 lib/librte_eal/common/rte_service.c | 45

<snip patch review comments>

#129 lore raw
In reply to Phil Yang

Hi Phil & Honnappa,

Quoted 21 lines

-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Tuesday, March 17, 2020 1:18 AM
To: [email protected]; Van Haaren, Harry <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v3 00/12] generic rte atomic APIs deprecate proposal

DPDK provides generic rte_atomic APIs to do several atomic operations. These APIs are using the deprecated __sync built-ins and enforce full memory barriers on aarch64. However, full barriers are not necessary in many use cases. In order to address such use cases, C language offers C11 atomic APIs. The C11 atomic APIs provide finer memory barrier control by making use of the memory ordering parameter provided by the user. Various patches submitted in the past [2] and the patches in this series indicate significant performance gains on multiple aarch64 CPUs and no performance loss on x86.

Quoted 8 lines

But the existing rte_atomic API implementations cannot be changed as the APIs do not take the memory ordering parameter. The only choice available is replacing the usage of the rte_atomic APIs with C11 atomic APIs. In order to make this change, the following steps are proposed:

[1] deprecate rte_atomic APIs so that future patches do not use rte_atomic APIs (a script is added to flag the usages).
[2] refactor the code that uses rte_atomic APIs to use c11 atomic APIs.

On [1] above, I feel deprecating DPDKs atomic functions and failing checkpatch is a bit sudden. Perhaps noting that in a future release (20.11?) DPDK will move to a C11 based atomics model is a more gradual step to achieving the goal, and at that point add a checkpatch warning for additions of rte_atomic*?

More on [2] in context below.

The above is my point-of-view, of course I'd like more people from the DPDK community to provide their input too.

Quoted 8 lines

This patchset contains:
1) the checkpatch script changes to flag rte_atomic API usage in patches.
2) changes to programmer guide describing writing efficient code for aarch64.
3) changes to various libraries to make use of c11 atomic APIs.

We are planning to replicate this idea across all the other libraries, drivers, examples, test applications. In the next phase, we will add changes to the mbuf, the EAL interrupts and the event timer adapter libraries.

About ~6/12 patches of this C11 set are targeting the Service Cores area of DPDK. I have some concerns over increased complexity of C11 implementation vs the (already complex) rte_atomic implementation today. I see other patchsets enabling C11 across other DPDK components, so maybe we should also discuss C11 enabling in a wider context that just service cores?

I don't think it fair to expect all developers to be well versed in C11 atomic semantics like understanding the complex interactions between the various C11 RELEASE, AQUIRE barriers requires.

As maintainer of Service Cores I'm hesitant to accept the large-scale refactor of atomic-implementation, as it could lead to racey bugs that are likely extremely difficult to track down. (The recent race-on-exit has proven the difficulty in reproducing, and that's with an atomics model I'm quite familiar with).

Let me be very clear: I don't wish to block a C11 atomic implementation, but I'd like to discuss how we (DPDK community) can best port-to and maintain a complex multi-threaded service library with best-in-class performance for the workload.

To put some discussions/solutions on the table:
- Shared Maintainership of a component?
Split in functionality and C11 atomics implementation
Obviously there would be collaboration required in such a case.
- Maybe shared maintainership is too much?
A gentlemans/womans agreement of "helping out" with C11 atomics debug is enough?

Hope my concerns are understandable, and of course input/feedback welcomed! -Harry

PS: Apologies for the delay in reply - was OOO on Irish national holiday.

Quoted 41 lines

v3:
add libatomic dependency for 32-bit clang

v2:
1. fix Clang '-Wincompatible-pointer-types' WARNING.
2. fix typos.

Honnappa Nagarahalli (2):
service: avoid race condition for MT unsafe service
service: identify service running on another core correctly

Phil Yang (10):
doc: add generic atomic deprecation section
devtools: prevent use of rte atomic APIs in future patches eal/build: add libatomic dependency for 32-bit clang
build: remove redundant code
vhost: optimize broadcast rarp sync with c11 atomic
ipsec: optimize with c11 atomic for sa outbound sqn update
service: remove rte prefix from static functions
service: remove redundant code
service: optimize with c11 one-way barrier
service: relax barriers with C11 atomic operations

 devtools/checkpatches.sh                         |   9 ++
 doc/guides/prog_guide/writing_efficient_code.rst |  60 +++++++-
 drivers/event/octeontx/meson.build               |   5 -
 drivers/event/octeontx2/meson.build              |   5 -
 drivers/event/opdl/meson.build                   |   5 -
 lib/librte_eal/common/rte_service.c              | 175 ++++++++++++----------
-
 lib/librte_eal/meson.build                       |   6 +
 lib/librte_ipsec/ipsec_sqn.h                     |   3 +-
 lib/librte_ipsec/sa.h                            |   2 +-
 lib/librte_rcu/meson.build                       |   5 -
 lib/librte_vhost/vhost.h                         |   2 +-
 lib/librte_vhost/vhost_user.c                    |   7 +-
 lib/librte_vhost/virtio_net.c                    |  16 ++-
 13 files changed, 181 insertions(+), 119 deletions(-)

--
2.7.4

#130 lore raw

18/03/2020 15:01, Van Haaren, Harry:

Quoted 67 lines

Hi Phil & Honnappa,

From: Phil Yang <[email protected]>

Quoted 11 lines

DPDK provides generic rte_atomic APIs to do several atomic operations. These APIs are using the deprecated __sync built-ins and enforce full memory barriers on aarch64. However, full barriers are not necessary in many use cases. In order to address such use cases, C language offers C11 atomic APIs. The C11 atomic APIs provide finer memory barrier control by making use of the memory ordering parameter provided by the user. Various patches submitted in the past [2] and the patches in this series indicate significant performance gains on multiple aarch64 CPUs and no performance loss on x86.

Quoted 8 lines

But the existing rte_atomic API implementations cannot be changed as the APIs do not take the memory ordering parameter. The only choice available is replacing the usage of the rte_atomic APIs with C11 atomic APIs. In order to make this change, the following steps are proposed:

[1] deprecate rte_atomic APIs so that future patches do not use rte_atomic APIs (a script is added to flag the usages).
[2] refactor the code that uses rte_atomic APIs to use c11 atomic APIs.

On [1] above, I feel deprecating DPDKs atomic functions and failing checkpatch is a bit sudden. Perhaps noting that in a future release (20.11?) DPDK will move to a C11 based atomics model is a more gradual step to achieving the goal, and at that point add a checkpatch warning for additions of rte_atomic*?

More on [2] in context below.

The above is my point-of-view, of course I'd like more people from the DPDK community to provide their input too.

Quoted 8 lines

This patchset contains:
1) the checkpatch script changes to flag rte_atomic API usage in patches.
2) changes to programmer guide describing writing efficient code for aarch64.
3) changes to various libraries to make use of c11 atomic APIs.

We are planning to replicate this idea across all the other libraries, drivers, examples, test applications. In the next phase, we will add changes to the mbuf, the EAL interrupts and the event timer adapter libraries.

About ~6/12 patches of this C11 set are targeting the Service Cores area of DPDK. I have some concerns over increased complexity of C11 implementation vs the (already complex) rte_atomic implementation today. I see other patchsets enabling C11 across other DPDK components, so maybe we should also discuss C11 enabling in a wider context that just service cores?

I don't think it fair to expect all developers to be well versed in C11 atomic semantics like understanding the complex interactions between the various C11 RELEASE, AQUIRE barriers requires.

As maintainer of Service Cores I'm hesitant to accept the large-scale refactor of atomic-implementation, as it could lead to racey bugs that are likely extremely difficult to track down. (The recent race-on-exit has proven the difficulty in reproducing, and that's with an atomics model I'm quite familiar with).

Let me be very clear: I don't wish to block a C11 atomic implementation, but I'd like to discuss how we (DPDK community) can best port-to and maintain a complex multi-threaded service library with best-in-class performance for the workload.

To put some discussions/solutions on the table:
- Shared Maintainership of a component?
Split in functionality and C11 atomics implementation
Obviously there would be collaboration required in such a case.
- Maybe shared maintainership is too much?
A gentlemans/womans agreement of "helping out" with C11 atomics debug is enough?

Quoted 1 line

Hope my concerns are understandable, and of course input/feedback welcomed! -Harry

Thanks for raising the issue Harry.

I think we should have at least two official maintainers for C11 atomics in general. C11 conversion is a progressive effort being done, and should be merged step by step. If C11 maintainers fail to fix some issues on time, then we can hold the effort. Does it make sense?