Skip to contentSkip to conversation

Add non-blocking ring

PATCH Superseded Gage Eads 10 participants 102 messages Jan 10, 2019 – Apr 4, 2019 1 message not in the archive Arm/Amlogic Meson SoC support

lore
Share & tools
lore

Gage Eads

author
#0 lore raw

cover Add non-blocking ring

For some users, the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

This patchset introduces a non-blocking ring, on top of which a mempool can run. Crucially, the non-blocking algorithm relies on a 128-bit compare-and-swap, so it is limited to x86_64 machines.

The ring uses more compare-and-swap atomic operations than the regular rte ring:
With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a non-blocking ring based mempool (a likely use case for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. For ease-of-use, existing ring enqueue/dequeue functions work with both "regular" and non-blocking rings.

This patchset also adds non-blocking versions of ring_autotest and ring_perf_autotest, and a non-blocking ring based mempool.

This patchset makes ABI changes, and thus an ABI update announcement and deprecation cycle are required.

This patchset depends on the non-blocking stack patchset[1].

[1] http://mails.dpdk.org/archives/dev/2019-January/122923.html

Gage Eads (6):
ring: change head and tail to pointer-width size
ring: add a non-blocking implementation
test_ring: add non-blocking ring autotest
test_ring_perf: add non-blocking ring perf test mempool/ring: add non-blocking ring handlers
doc: add NB ring comment to EAL "known issues"

 doc/guides/prog_guide/env_abstraction_layer.rst |   2 +-
 drivers/mempool/ring/rte_mempool_ring.c         |  58 ++-
 lib/librte_eventdev/rte_event_ring.h            |   6 +-
 lib/librte_ring/rte_ring.c                      |  53 ++-
 lib/librte_ring/rte_ring.h                      | 555 ++++++++++++++++++++++--
 lib/librte_ring/rte_ring_generic.h              |  16 +-
 lib/librte_ring/rte_ring_version.map            |   7 +
 test/test/test_ring.c                           |  57 ++-
 test/test/test_ring_perf.c                      |  19 +-
 9 files changed, 689 insertions(+), 84 deletions(-)
Signature · 1 line
2.13.6
author
#1 lore raw

PATCH 1/6 ring: change head and tail to pointer-width size

For 64-bit architectures, doubling the head and tail index widths greatly increases the time it takes for them to wrap-around (with current CPU speeds, it won't happen within the author's lifetime). This is important in avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming non-blocking ring implementation. Using a 64-bit index makes the possibility of this occurring effectively zero.

I tested this commit's performance impact with an x86_64 build on a dual-socket Xeon E5-2699 v4 using ring_perf_autotest, and the change made no significant difference -- the few differences appear to be system noise. (The test ran on isolcpus cores using a tickless scheduler, but some variation was stll observed.) Each test was run three times and the results were averaged:

| 64b head/tail cycle cost minus
Test | 32b head/tail cycle cost
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | 0.00
SP/SC burst enq/dequeue (size 8)  | 0.00
MP/MC burst enq/dequeue (size 8)  | 1.00
SP/SC burst enq/dequeue (size 32) | 0.00
MP/MC burst enq/dequeue (size 32) | -1.00
SC empty dequeue                  | 0.01
MC empty dequeue                  | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | -0.36
MP/MC bulk enq/dequeue (size 8)   | 0.99
SP/SC bulk enq/dequeue (size 32)  | -0.40
MP/MC bulk enq/dequeue (size 32)  | -0.57

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | -0.49
MP/MC bulk enq/dequeue (size 8)   | 0.19
SP/SC bulk enq/dequeue (size 32)  | -0.28
MP/MC bulk enq/dequeue (size 32)  | -0.62

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | 3.25
MP/MC bulk enq/dequeue (size 8)   | 1.87
SP/SC bulk enq/dequeue (size 32)  | -0.44
MP/MC bulk enq/dequeue (size 32)  | -1.10

An earlier version of this patch changed the head and tail indexes to uint64_t, but that caused a performance drop on 32-bit builds. With uintptr_t, no performance difference is observed on an i686 build.

---
 lib/librte_eventdev/rte_event_ring.h |  6 +++---
 lib/librte_ring/rte_ring.c           | 10 +++++-----
 lib/librte_ring/rte_ring.h           | 20 ++++++++++----------
 lib/librte_ring/rte_ring_generic.h   | 16 +++++++++-------
 4 files changed, 27 insertions(+), 25 deletions(-)
lib/librte_eventdev/rte_event_ring.h +3 -3
index 827a3209e..eae70f904 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2016-2017 Intel Corporation
+ * Copyright(c) 2016-2019 Intel Corporation
  */
 
 /**
@@ -88,7 +88,7 @@ rte_event_ring_enqueue_burst(struct rte_event_ring *r,
 		const struct rte_event *events,
 		unsigned int n, uint16_t *free_space)
 {
-	uint32_t prod_head, prod_next;
+	uintptr_t prod_head, prod_next;
 	uint32_t free_entries;
 
 	n = __rte_ring_move_prod_head(&r->r, r->r.prod.single, n,
@@ -129,7 +129,7 @@ rte_event_ring_dequeue_burst(struct rte_event_ring *r,
 		struct rte_event *events,
 		unsigned int n, uint16_t *available)
 {
-	uint32_t cons_head, cons_next;
+	uintptr_t cons_head, cons_next;
 	uint32_t entries;
 
 	n = __rte_ring_move_cons_head(&r->r, r->r.cons.single, n,
lib/librte_ring/rte_ring.c +5 -5
index d215acecc..b15ee0eb3 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2015 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007,2008 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -227,10 +227,10 @@ rte_ring_dump(FILE *f, const struct rte_ring *r)
 	fprintf(f, "  flags=%x\n", r->flags);
 	fprintf(f, "  size=%"PRIu32"\n", r->size);
 	fprintf(f, "  capacity=%"PRIu32"\n", r->capacity);
-	fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
-	fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
-	fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
-	fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	fprintf(f, "  ct=%"PRIuPTR"\n", r->cons.tail);
+	fprintf(f, "  ch=%"PRIuPTR"\n", r->cons.head);
+	fprintf(f, "  pt=%"PRIuPTR"\n", r->prod.tail);
+	fprintf(f, "  ph=%"PRIuPTR"\n", r->prod.head);
 	fprintf(f, "  used=%u\n", rte_ring_count(r));
 	fprintf(f, "  avail=%u\n", rte_ring_free_count(r));
 }
lib/librte_ring/rte_ring.h +10 -10
index af5444a9f..12af64e13 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007-2009 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -65,8 +65,8 @@ struct rte_memzone; /* forward declaration, so as not to require memzone.h */
 
 /* structure to hold a pair of head/tail values and other metadata */
 struct rte_ring_headtail {
-	volatile uint32_t head;  /**< Prod/consumer head. */
-	volatile uint32_t tail;  /**< Prod/consumer tail. */
+	volatile uintptr_t head;  /**< Prod/consumer head. */
+	volatile uintptr_t tail;  /**< Prod/consumer tail. */
 	uint32_t single;         /**< True if single prod/cons */
 };
 
@@ -242,7 +242,7 @@ void rte_ring_dump(FILE *f, const struct rte_ring *r);
 #define ENQUEUE_PTRS(r, ring_start, prod_head, obj_table, n, obj_type) do { \
 	unsigned int i; \
 	const uint32_t size = (r)->size; \
-	uint32_t idx = prod_head & (r)->mask; \
+	uintptr_t idx = prod_head & (r)->mask; \
 	obj_type *ring = (obj_type *)ring_start; \
 	if (likely(idx + n < size)) { \
 		for (i = 0; i < (n & ((~(unsigned)0x3))); i+=4, idx+=4) { \
@@ -272,7 +272,7 @@ void rte_ring_dump(FILE *f, const struct rte_ring *r);
  * single and multi consumer dequeue functions */
 #define DEQUEUE_PTRS(r, ring_start, cons_head, obj_table, n, obj_type) do { \
 	unsigned int i; \
-	uint32_t idx = cons_head & (r)->mask; \
+	uintptr_t idx = cons_head & (r)->mask; \
 	const uint32_t size = (r)->size; \
 	obj_type *ring = (obj_type *)ring_start; \
 	if (likely(idx + n < size)) { \
@@ -338,7 +338,7 @@ __rte_ring_do_enqueue(struct rte_ring *r, void * const *obj_table,
 		 unsigned int n, enum rte_ring_queue_behavior behavior,
 		 unsigned int is_sp, unsigned int *free_space)
 {
-	uint32_t prod_head, prod_next;
+	uintptr_t prod_head, prod_next;
 	uint32_t free_entries;
 
 	n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
@@ -380,7 +380,7 @@ __rte_ring_do_dequeue(struct rte_ring *r, void **obj_table,
 		 unsigned int n, enum rte_ring_queue_behavior behavior,
 		 unsigned int is_sc, unsigned int *available)
 {
-	uint32_t cons_head, cons_next;
+	uintptr_t cons_head, cons_next;
 	uint32_t entries;
 
 	n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
@@ -681,9 +681,9 @@ rte_ring_dequeue(struct rte_ring *r, void **obj_p)
 static inline unsigned
 rte_ring_count(const struct rte_ring *r)
 {
-	uint32_t prod_tail = r->prod.tail;
-	uint32_t cons_tail = r->cons.tail;
-	uint32_t count = (prod_tail - cons_tail) & r->mask;
+	uintptr_t prod_tail = r->prod.tail;
+	uintptr_t cons_tail = r->cons.tail;
+	uintptr_t count = (prod_tail - cons_tail) & r->mask;
 	return (count > r->capacity) ? r->capacity : count;
 }
 
lib/librte_ring/rte_ring_generic.h +9 -7
index ea7dbe5b9..3fd1150f6 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007-2009 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -11,7 +11,7 @@
 #define _RTE_RING_GENERIC_H_
 
 static __rte_always_inline void
-update_tail(struct rte_ring_headtail *ht, uint32_t old_val, uint32_t new_val,
+update_tail(struct rte_ring_headtail *ht, uintptr_t old_val, uintptr_t new_val,
 		uint32_t single, uint32_t enqueue)
 {
 	if (enqueue)
@@ -55,7 +55,7 @@ update_tail(struct rte_ring_headtail *ht, uint32_t old_val, uint32_t new_val,
 static __rte_always_inline unsigned int
 __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,
 		unsigned int n, enum rte_ring_queue_behavior behavior,
-		uint32_t *old_head, uint32_t *new_head,
+		uintptr_t *old_head, uintptr_t *new_head,
 		uint32_t *free_entries)
 {
 	const uint32_t capacity = r->capacity;
@@ -93,7 +93,8 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,
 		if (is_sp)
 			r->prod.head = *new_head, success = 1;
 		else
-			success = rte_atomic32_cmpset(&r->prod.head,
+			/* Built-in used to handle variable-sized head index. */
+			success = __sync_bool_compare_and_swap(&r->prod.head,
 					*old_head, *new_head);
 	} while (unlikely(success == 0));
 	return n;
@@ -125,7 +126,7 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,
 static __rte_always_inline unsigned int
 __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
 		unsigned int n, enum rte_ring_queue_behavior behavior,
-		uint32_t *old_head, uint32_t *new_head,
+		uintptr_t *old_head, uintptr_t *new_head,
 		uint32_t *entries)
 {
 	unsigned int max = n;
@@ -161,8 +162,9 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
 		if (is_sc)
 			r->cons.head = *new_head, success = 1;
 		else
-			success = rte_atomic32_cmpset(&r->cons.head, *old_head,
-					*new_head);
+			/* Built-in used to handle variable-sized head index. */
+			success = __sync_bool_compare_and_swap(&r->cons.head,
+					*old_head, *new_head);
 	} while (unlikely(success == 0));
 	return n;
 }
Signature · 1 line
2.13.6
#2 lore raw

Re: [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 126 lines · On Thu, 10 Jan 2019 15:01:17 -0600 Gage Eads <[email protected]> wrote:

For 64-bit architectures, doubling the head and tail index widths greatly increases the time it takes for them to wrap-around (with current CPU speeds, it won't happen within the author's lifetime). This is important in avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming non-blocking ring implementation. Using a 64-bit index makes the possibility of this occurring effectively zero.

I tested this commit's performance impact with an x86_64 build on a dual-socket Xeon E5-2699 v4 using ring_perf_autotest, and the change made no significant difference -- the few differences appear to be system noise. (The test ran on isolcpus cores using a tickless scheduler, but some variation was stll observed.) Each test was run three times and the results were averaged:

| 64b head/tail cycle cost minus
Test | 32b head/tail cycle cost
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | 0.00
SP/SC burst enq/dequeue (size 8)  | 0.00
MP/MC burst enq/dequeue (size 8)  | 1.00
SP/SC burst enq/dequeue (size 32) | 0.00
MP/MC burst enq/dequeue (size 32) | -1.00
SC empty dequeue                  | 0.01
MC empty dequeue                  | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | -0.36
MP/MC bulk enq/dequeue (size 8)   | 0.99
SP/SC bulk enq/dequeue (size 32)  | -0.40
MP/MC bulk enq/dequeue (size 32)  | -0.57

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | -0.49
MP/MC bulk enq/dequeue (size 8)   | 0.19
SP/SC bulk enq/dequeue (size 32)  | -0.28
MP/MC bulk enq/dequeue (size 32)  | -0.62

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | 3.25
MP/MC bulk enq/dequeue (size 8)   | 1.87
SP/SC bulk enq/dequeue (size 32)  | -0.44
MP/MC bulk enq/dequeue (size 32)  | -1.10

An earlier version of this patch changed the head and tail indexes to uint64_t, but that caused a performance drop on 32-bit builds. With uintptr_t, no performance difference is observed on an i686 build.

---
 lib/librte_eventdev/rte_event_ring.h |  6 +++---
 lib/librte_ring/rte_ring.c           | 10 +++++-----
 lib/librte_ring/rte_ring.h           | 20 ++++++++++----------
 lib/librte_ring/rte_ring_generic.h   | 16 +++++++++-------
 4 files changed, 27 insertions(+), 25 deletions(-)
lib/librte_eventdev/rte_event_ring.h +3 -3
index 827a3209e..eae70f904 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2016-2017 Intel Corporation
+ * Copyright(c) 2016-2019 Intel Corporation
  */
 
 /**
@@ -88,7 +88,7 @@ rte_event_ring_enqueue_burst(struct rte_event_ring *r,
 		const struct rte_event *events,
 		unsigned int n, uint16_t *free_space)
 {
-	uint32_t prod_head, prod_next;
+	uintptr_t prod_head, prod_next;
 	uint32_t free_entries;
 
 	n = __rte_ring_move_prod_head(&r->r, r->r.prod.single, n,
@@ -129,7 +129,7 @@ rte_event_ring_dequeue_burst(struct rte_event_ring *r,
 		struct rte_event *events,
 		unsigned int n, uint16_t *available)
 {
-	uint32_t cons_head, cons_next;
+	uintptr_t cons_head, cons_next;
 	uint32_t entries;
 
 	n = __rte_ring_move_cons_head(&r->r, r->r.cons.single, n,
lib/librte_ring/rte_ring.c +5 -5
index d215acecc..b15ee0eb3 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2015 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007,2008 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -227,10 +227,10 @@ rte_ring_dump(FILE *f, const struct rte_ring *r)
 	fprintf(f, "  flags=%x\n", r->flags);
 	fprintf(f, "  size=%"PRIu32"\n", r->size);
 	fprintf(f, "  capacity=%"PRIu32"\n", r->capacity);
-	fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
-	fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
-	fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
-	fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	fprintf(f, "  ct=%"PRIuPTR"\n", r->cons.tail);
+	fprintf(f, "  ch=%"PRIuPTR"\n", r->cons.head);
+	fprintf(f, "  pt=%"PRIuPTR"\n", r->prod.tail);
+	fprintf(f, "  ph=%"PRIuPTR"\n", r->prod.head);
 	fprintf(f, "  used=%u\n", rte_ring_count(r));
 	fprintf(f, "  avail=%u\n", rte_ring_free_count(r));
 }
lib/librte_ring/rte_ring.h +1 -1
index af5444a9f..12af64e13 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007-2009 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h

lib/librte_ring/rte_ring.h struct rte_memzone; /* forward declaration, so as not to require memzone.h */ · line 71

Quoted 9 lines
lib/librte_ring/rte_ring.h +2 -2
@@ -65,8 +65,8 @@ struct rte_memzone; /* forward declaration, so as not to require memzone.h */
 
 /* structure to hold a pair of head/tail values and other metadata */
 struct rte_ring_headtail {
-	volatile uint32_t head;  /**< Prod/consumer head. */
-	volatile uint32_t tail;  /**< Prod/consumer tail. */
+	volatile uintptr_t head;  /**< Prod/consumer head. */
+	volatile uintptr_t tail;  /**< Prod/consumer tail. */
 	uint32_t single;         /**< True if single prod/cons */
 };

Isn't this a major ABI change which will break existing applications?

author
#3 lore raw

Re: [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 142 lines

-----Original Message-----
From: Stephen Hemminger [mailto:[email protected]]
Sent: Thursday, January 10, 2019 10:39 PM
To: Eads, Gage <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 7 lines · On Thu, 10 Jan 2019 15:01:17 -0600 Gage Eads <[email protected]> wrote:

For 64-bit architectures, doubling the head and tail index widths greatly increases the time it takes for them to wrap-around (with current CPU speeds, it won't happen within the author's lifetime). This is important in avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming non-blocking ring implementation. Using a 64-bit index makes the possibility of

this occurring effectively zero.

Quoted 4 lines

I tested this commit's performance impact with an x86_64 build on a dual-socket Xeon E5-2699 v4 using ring_perf_autotest, and the change made no significant difference -- the few differences appear to be system

noise.

Quoted 114 lines

(The test ran on isolcpus cores using a tickless scheduler, but some variation was stll observed.) Each test was run three times and the results were averaged:

| 64b head/tail cycle cost minus
Test | 32b head/tail cycle cost
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | 0.00
SP/SC burst enq/dequeue (size 8)  | 0.00 MP/MC burst enq/dequeue (size

8) | 1.00 SP/SC burst enq/dequeue (size 32) | 0.00 MP/MC burst enq/dequeue (size 32) | -1.00
SC empty dequeue | 0.01
MC empty dequeue | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | -0.36
MP/MC bulk enq/dequeue (size 8)   | 0.99
SP/SC bulk enq/dequeue (size 32)  | -0.40 MP/MC bulk enq/dequeue (size

32) | -0.57

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | -0.49
MP/MC bulk enq/dequeue (size 8)   | 0.19
SP/SC bulk enq/dequeue (size 32)  | -0.28 MP/MC bulk enq/dequeue (size

32) | -0.62

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | 3.25
MP/MC bulk enq/dequeue (size 8)   | 1.87
SP/SC bulk enq/dequeue (size 32)  | -0.44 MP/MC bulk enq/dequeue (size

32) | -1.10

An earlier version of this patch changed the head and tail indexes to uint64_t, but that caused a performance drop on 32-bit builds. With uintptr_t, no performance difference is observed on an i686 build.

---
 lib/librte_eventdev/rte_event_ring.h |  6 +++---
 lib/librte_ring/rte_ring.c           | 10 +++++-----
 lib/librte_ring/rte_ring.h           | 20 ++++++++++----------
 lib/librte_ring/rte_ring_generic.h   | 16 +++++++++-------
 4 files changed, 27 insertions(+), 25 deletions(-)
lib/librte_eventdev/rte_event_ring.h +4 -3
index 827a3209e..eae70f904 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2016-2017 Intel Corporation
+ * Copyright(c) 2016-2019 Intel Corporation
  */
 
 /**
@@ -88,7 +88,7 @@ rte_event_ring_enqueue_burst(struct rte_event_ring *r,
 		const struct rte_event *events,
 		unsigned int n, uint16_t *free_space)  {
-	uint32_t prod_head, prod_next;
+	uintptr_t prod_head, prod_next;
 	uint32_t free_entries;
 
 	n = __rte_ring_move_prod_head(&r->r, r->r.prod.single, n, @@ -129,7
+129,7 @@ rte_event_ring_dequeue_burst(struct rte_event_ring *r,
 		struct rte_event *events,
 		unsigned int n, uint16_t *available)  {
-	uint32_t cons_head, cons_next;
+	uintptr_t cons_head, cons_next;
 	uint32_t entries;
 
 	n = __rte_ring_move_cons_head(&r->r, r->r.cons.single, n, diff --git

a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c index d215acecc..b15ee0eb3 100644

lib/librte_ring/rte_ring.c +5 -5
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2015 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007,2008 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -227,10 +227,10 @@ rte_ring_dump(FILE *f, const struct rte_ring *r)
 	fprintf(f, "  flags=%x\n", r->flags);
 	fprintf(f, "  size=%"PRIu32"\n", r->size);
 	fprintf(f, "  capacity=%"PRIu32"\n", r->capacity);
-	fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
-	fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
-	fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
-	fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	fprintf(f, "  ct=%"PRIuPTR"\n", r->cons.tail);
+	fprintf(f, "  ch=%"PRIuPTR"\n", r->cons.head);
+	fprintf(f, "  pt=%"PRIuPTR"\n", r->prod.tail);
+	fprintf(f, "  ph=%"PRIuPTR"\n", r->prod.head);
 	fprintf(f, "  used=%u\n", rte_ring_count(r));
 	fprintf(f, "  avail=%u\n", rte_ring_free_count(r));  } diff --git

a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h index af5444a9f..12af64e13 100644

lib/librte_ring/rte_ring.h +1 -1
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007-2009 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -65,8 +65,8 @@ struct rte_memzone; /* forward declaration, so as

not to require memzone.h */

Quoted 10 lines

lib/librte_ring/rte_ring.h struct rte_memzone; /* forward declaration, so as not to require memzone.h */ · line 71

Quoted 8 lines

/* structure to hold a pair of head/tail values and other metadata */
struct rte_ring_headtail {

lib/librte_ring/rte_ring.h +2 -2
-	volatile uint32_t head;  /**< Prod/consumer head. */
-	volatile uint32_t tail;  /**< Prod/consumer tail. */
+	volatile uintptr_t head;  /**< Prod/consumer head. */
+	volatile uintptr_t tail;  /**< Prod/consumer tail. */
 	uint32_t single;         /**< True if single prod/cons */
 };

Isn't this a major ABI change which will break existing applications?

Correct, and this patch needs to be reworked with the RTE_NEXT_ABI ifdef, as described in the versioning guidelines. I had misunderstood the ABI change procedure, but I'll fix this in v2.

#4 lore raw

Re: [PATCH 1/6] ring: change head and tail to pointer-width size

In reply to Gage Eads
Quoted 45 lines · Gage Eads

For 64-bit architectures, doubling the head and tail index widths greatly increases the time it takes for them to wrap-around (with current CPU speeds, it won't happen within the author's lifetime). This is important in avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming non-blocking ring implementation. Using a 64-bit index makes the possibility of this occurring effectively zero.

I tested this commit's performance impact with an x86_64 build on a dual-socket Xeon E5-2699 v4 using ring_perf_autotest, and the change made no significant difference -- the few differences appear to be system noise. (The test ran on isolcpus cores using a tickless scheduler, but some variation was stll observed.) Each test was run three times and the results were averaged:

| 64b head/tail cycle cost minus
Test | 32b head/tail cycle cost
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | 0.00
SP/SC burst enq/dequeue (size 8)  | 0.00
MP/MC burst enq/dequeue (size 8)  | 1.00
SP/SC burst enq/dequeue (size 32) | 0.00
MP/MC burst enq/dequeue (size 32) | -1.00
SC empty dequeue                  | 0.01
MC empty dequeue                  | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | -0.36
MP/MC bulk enq/dequeue (size 8)   | 0.99
SP/SC bulk enq/dequeue (size 32)  | -0.40
MP/MC bulk enq/dequeue (size 32)  | -0.57

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | -0.49
MP/MC bulk enq/dequeue (size 8)   | 0.19
SP/SC bulk enq/dequeue (size 32)  | -0.28
MP/MC bulk enq/dequeue (size 32)  | -0.62

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | 3.25
MP/MC bulk enq/dequeue (size 8)   | 1.87
SP/SC bulk enq/dequeue (size 32)  | -0.44
MP/MC bulk enq/dequeue (size 32)  | -1.10
Quoted 6 lines

An earlier version of this patch changed the head and tail indexes to uint64_t, but that caused a performance drop on 32-bit builds. With uintptr_t, no performance difference is observed on an i686 build.

---

You're breaking the ABI - version bump for affected libraries is needed.

Signature · 2 lines
Thanks,
Anatoly
author
#5 lore raw

Re: [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 63 lines

-----Original Message-----
From: Burakov, Anatoly
Sent: Friday, January 11, 2019 4:25 AM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 7 lines · Gage Eads

For 64-bit architectures, doubling the head and tail index widths greatly increases the time it takes for them to wrap-around (with current CPU speeds, it won't happen within the author's lifetime). This is important in avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming non-blocking ring implementation. Using a 64-bit index makes the possibility of

this occurring effectively zero.

Quoted 4 lines

I tested this commit's performance impact with an x86_64 build on a dual-socket Xeon E5-2699 v4 using ring_perf_autotest, and the change made no significant difference -- the few differences appear to be system

noise.

Quoted 37 lines

(The test ran on isolcpus cores using a tickless scheduler, but some variation was stll observed.) Each test was run three times and the results were averaged:

| 64b head/tail cycle cost minus
Test | 32b head/tail cycle cost
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | 0.00
SP/SC burst enq/dequeue (size 8)  | 0.00 MP/MC burst enq/dequeue (size

8) | 1.00 SP/SC burst enq/dequeue (size 32) | 0.00 MP/MC burst enq/dequeue (size 32) | -1.00
SC empty dequeue | 0.01
MC empty dequeue | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | -0.36
MP/MC bulk enq/dequeue (size 8)   | 0.99
SP/SC bulk enq/dequeue (size 32)  | -0.40 MP/MC bulk enq/dequeue (size

32) | -0.57

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | -0.49
MP/MC bulk enq/dequeue (size 8)   | 0.19
SP/SC bulk enq/dequeue (size 32)  | -0.28 MP/MC bulk enq/dequeue (size

32) | -0.62

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | 3.25
MP/MC bulk enq/dequeue (size 8)   | 1.87
SP/SC bulk enq/dequeue (size 32)  | -0.44 MP/MC bulk enq/dequeue (size

32) | -1.10

An earlier version of this patch changed the head and tail indexes to uint64_t, but that caused a performance drop on 32-bit builds. With uintptr_t, no performance difference is observed on an i686 build.

Quoted 8 lines
Quoted 2 lines

---

You're breaking the ABI - version bump for affected libraries is needed.

--
Thanks,
Anatoly

If I'm reading the versioning guidelines correctly, I'll need to gate the changes with the RTE_NEXT_ABI macro and provide a deprecation notice, then after a full deprecation cycle we can revert that and bump the library version. Not to mention the 3 ML ACKs.

I'll address this in v2.

Thanks,
Gage

#6 lore raw

Re: [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 66 lines · Gage
Quoted 66 lines

-----Original Message-----
From: Burakov, Anatoly
Sent: Friday, January 11, 2019 4:25 AM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 7 lines · Gage Eads

For 64-bit architectures, doubling the head and tail index widths greatly increases the time it takes for them to wrap-around (with current CPU speeds, it won't happen within the author's lifetime). This is important in avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming non-blocking ring implementation. Using a 64-bit index makes the possibility of

this occurring effectively zero.

Quoted 4 lines

I tested this commit's performance impact with an x86_64 build on a dual-socket Xeon E5-2699 v4 using ring_perf_autotest, and the change made no significant difference -- the few differences appear to be system

noise.

Quoted 31 lines

(The test ran on isolcpus cores using a tickless scheduler, but some variation was stll observed.) Each test was run three times and the results were averaged:

| 64b head/tail cycle cost minus
Test | 32b head/tail cycle cost
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | 0.00
SP/SC burst enq/dequeue (size 8)  | 0.00 MP/MC burst enq/dequeue (size

8) | 1.00 SP/SC burst enq/dequeue (size 32) | 0.00 MP/MC burst enq/dequeue (size 32) | -1.00
SC empty dequeue | 0.01
MC empty dequeue | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | -0.36
MP/MC bulk enq/dequeue (size 8)   | 0.99
SP/SC bulk enq/dequeue (size 32)  | -0.40 MP/MC bulk enq/dequeue (size

32) | -0.57

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | -0.49
MP/MC bulk enq/dequeue (size 8)   | 0.19
SP/SC bulk enq/dequeue (size 32)  | -0.28 MP/MC bulk enq/dequeue (size

32) | -0.62

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | 3.25
MP/MC bulk enq/dequeue (size 8)   | 1.87
SP/SC bulk enq/dequeue (size 32)  | -0.44 MP/MC bulk enq/dequeue (size
Quoted 8 lines

32) | -1.10

An earlier version of this patch changed the head and tail indexes to uint64_t, but that caused a performance drop on 32-bit builds. With uintptr_t, no performance difference is observed on an i686 build.

---

You're breaking the ABI - version bump for affected libraries is needed.

Quoted 7 lines
Quoted 3 lines

--
Thanks,
Anatoly

If I'm reading the versioning guidelines correctly, I'll need to gate the changes with the RTE_NEXT_ABI macro and provide a deprecation notice, then after a full deprecation cycle we can revert that and bump the library version. Not to mention the 3 ML ACKs.

I'll address this in v2.

My understanding is that RTE_NEXT_API method is not used any more. Replaced by rte_experimental. But this kind of change is more of a flag day event. Which means it needs to be pushed off to a release that is planned as an ABI break (usually once a year) which would mean 19.11.

author
#7 lore raw

Re: [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 97 lines

-----Original Message-----
From: Stephen Hemminger [mailto:[email protected]]
Sent: Friday, January 11, 2019 1:55 PM
To: Eads, Gage <[email protected]>
Cc: Burakov, Anatoly <[email protected]>; [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 70 lines · Gage
Quoted 70 lines

-----Original Message-----
From: Burakov, Anatoly
Sent: Friday, January 11, 2019 4:25 AM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 8 lines · Gage Eads

For 64-bit architectures, doubling the head and tail index widths greatly increases the time it takes for them to wrap-around (with current CPU speeds, it won't happen within the author's lifetime). This is important in avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming non-blocking ring implementation. Using a 64-bit index makes the possibility of

this occurring effectively zero.

Quoted 5 lines

I tested this commit's performance impact with an x86_64 build on a dual-socket Xeon E5-2699 v4 using ring_perf_autotest, and the change made no significant difference -- the few differences appear to be system

noise.

Quoted 43 lines

(The test ran on isolcpus cores using a tickless scheduler, but some variation was stll observed.) Each test was run three times and the results were averaged:

| 64b head/tail cycle cost minus
Test | 32b head/tail cycle cost
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | 0.00
SP/SC burst enq/dequeue (size 8)  | 0.00 MP/MC burst enq/dequeue

(size
8) | 1.00 SP/SC burst enq/dequeue (size 32) | 0.00 MP/MC burst enq/dequeue (size 32) | -1.00
SC empty dequeue | 0.01
MC empty dequeue | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | -0.36
MP/MC bulk enq/dequeue (size 8)   | 0.99
SP/SC bulk enq/dequeue (size 32)  | -0.40 MP/MC bulk enq/dequeue

(size
32) | -0.57

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | -0.49
MP/MC bulk enq/dequeue (size 8)   | 0.19
SP/SC bulk enq/dequeue (size 32)  | -0.28 MP/MC bulk enq/dequeue

(size
32) | -0.62

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | 3.25
MP/MC bulk enq/dequeue (size 8)   | 1.87
SP/SC bulk enq/dequeue (size 32)  | -0.44 MP/MC bulk enq/dequeue

(size
32) | -1.10

An earlier version of this patch changed the head and tail indexes to uint64_t, but that caused a performance drop on 32-bit builds. With uintptr_t, no performance difference is observed on an i686 build.

---

Quoted 7 lines
Quoted 5 lines

You're breaking the ABI - version bump for affected libraries is needed.

--
Thanks,
Anatoly

If I'm reading the versioning guidelines correctly, I'll need to gate the changes

with the RTE_NEXT_ABI macro and provide a deprecation notice, then after a full deprecation cycle we can revert that and bump the library version. Not to mention the 3 ML ACKs.

Quoted 1 line
Quoted 7 lines
Quoted 1 line

I'll address this in v2.

My understanding is that RTE_NEXT_API method is not used any more. Replaced by rte_experimental.
But this kind of change is more of a flag day event. Which means it needs to be pushed off to a release that is planned as an ABI break (usually once a year) which would mean 19.11.

In recent release notes, I see ABI changes can happen more frequently than once per year; 18.11, 18.05, 17.11, and 17.08 have ABI changes -- and soon 19.02 will too.

At any rate, I'll create a separate deprecation notice patch and update this patchset accordingly.

Thanks,
Gage

#8 lore raw

Re: [PATCH 1/6] ring: change head and tail to pointer-width size

In reply to Gage Eads

<...>

lib/librte_eventdev/rte_event_ring.h rte_event_ring_enqueue_burst(struct rte_event_ring *r, · line 91

Quoted 10 lines

+ * Copyright(c) 2016-2019 Intel Corporation
*/

/**

lib/librte_eventdev/rte_event_ring.h +1 -1
@@ -88,7 +88,7 @@ rte_event_ring_enqueue_burst(struct rte_event_ring *r,
  		const struct rte_event *events,
  		unsigned int n, uint16_t *free_space)
  {
-	uint32_t prod_head, prod_next;
+	uintptr_t prod_head, prod_next;

I would also question the use of uinptr_t. I think semnatically, size_t is more appropriate.

Signature · 2 lines
Thanks,
Anatoly
#9 lore raw

Re: [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 15 lines · Anatoly

<...>

lib/librte_eventdev/rte_event_ring.h rte_event_ring_enqueue_burst(struct rte_event_ring *r, · line 91

Quoted 9 lines

+ * Copyright(c) 2016-2019 Intel Corporation
*/
/**

lib/librte_eventdev/rte_event_ring.h +1 -1
@@ -88,7 +88,7 @@ rte_event_ring_enqueue_burst(struct rte_event_ring *r,
  		const struct rte_event *events,
  		unsigned int n, uint16_t *free_space)
  {
-	uint32_t prod_head, prod_next;
+	uintptr_t prod_head, prod_next;

I would also question the use of uinptr_t. I think semnatically, size_t is more appropriate.

Yes, it would, but I believe in this case they want to use the largest size of (unsigned)int where there exists an atomic for manipulating 2 of them simultaneously. [The largest size is to minimize any chance of an ABA issue occuring]. Therefore we need 32-bit values on 32-bit and 64-bit on 64, and I suspect the best way to guarantee this is to use pointer-sized values. If size_t is guaranteed across all OS's to have the same size as uintptr_t it could also be used, though.

/Bruce

#10 lore raw

Re: [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 24 lines · Bruce Richardson
Quoted 15 lines · Anatoly

<...>

lib/librte_eventdev/rte_event_ring.h rte_event_ring_enqueue_burst(struct rte_event_ring *r, · line 91

Quoted 9 lines

+ * Copyright(c) 2016-2019 Intel Corporation

    */

/**

lib/librte_eventdev/rte_event_ring.h +1 -1
@@ -88,7 +88,7 @@ rte_event_ring_enqueue_burst(struct rte_event_ring *r,
   		const struct rte_event *events,
   		unsigned int n, uint16_t *free_space)
   {
-	uint32_t prod_head, prod_next;
+	uintptr_t prod_head, prod_next;

I would also question the use of uinptr_t. I think semnatically, size_t is more appropriate.

Yes, it would, but I believe in this case they want to use the largest size of (unsigned)int where there exists an atomic for manipulating 2 of them simultaneously. [The largest size is to minimize any chance of an ABA issue occuring]. Therefore we need 32-bit values on 32-bit and 64-bit on 64, and I suspect the best way to guarantee this is to use pointer-sized values. If size_t is guaranteed across all OS's to have the same size as uintptr_t it could also be used, though.

Quoted 2 lines

/Bruce

Technically, size_t and uintptr_t are not guaranteed to match. In practice, they won't match only on architectures that DPDK doesn't intend to run on (such as 16-bit segmented archs, where size_t would be 16-bit but uinptr_t would be 32-bit).

In all the rest of DPDK code, we use size_t for this kind of thing.

Signature · 2 lines
Thanks,
Anatoly

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#12 lore raw

Re: [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 48 lines

-----Original Message-----
From: Richardson, Bruce
Sent: Friday, January 11, 2019 5:59 AM
To: Burakov, Anatoly <[email protected]>
Cc: Eads, Gage <[email protected]>; [email protected]; [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 8 lines · Anatoly
Quoted 7 lines · Bruce Richardson
Quoted 6 lines · Anatoly

<...>

Review rte_event_ring_enqueue_burst(struct

Quoted 4 lines

+ * Copyright(c) 2016-2019 Intel Corporation

    */

/**

(quoted diff) +0 -0
@@ -88,7 +88,7 @@ rte_event_ring_enqueue_burst(struct

rte_event_ring *r,

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/librte_eventdev/rte_event_ring.h rte_event_ring_enqueue_burst(struct rte_event_ring *r, · line 91

Quoted 5 lines
lib/librte_eventdev/rte_event_ring.h +1 -1
   		const struct rte_event *events,
   		unsigned int n, uint16_t *free_space)
   {
-	uint32_t prod_head, prod_next;
+	uintptr_t prod_head, prod_next;

I would also question the use of uinptr_t. I think semnatically, size_t is more appropriate.

Quoted 7 lines
Quoted 7 lines

Yes, it would, but I believe in this case they want to use the largest size of (unsigned)int where there exists an atomic for manipulating 2 of them simultaneously. [The largest size is to minimize any chance of an ABA issue occuring]. Therefore we need 32-bit values on 32-bit and 64-bit on 64, and I suspect the best way to guarantee this is to use pointer-sized values. If size_t is guaranteed across all OS's to have the same size as uintptr_t it could also be

used, though.

Quoted 9 lines
Quoted 3 lines

/Bruce

Technically, size_t and uintptr_t are not guaranteed to match. In practice, they won't match only on architectures that DPDK doesn't intend to run on (such as 16-bit segmented archs, where size_t would be 16-bit but uinptr_t would be 32-bit).

Quoted 6 lines
Quoted 2 lines

In all the rest of DPDK code, we use size_t for this kind of thing.

Ok.
If we do use size_t, I think we also need to add a compile-time check into the build too, to error out if sizeof(size_t) != sizeof(uintptr_t).

Ok, I wasn't aware of the precedent of using size_t for this purpose. I'll change it and look into adding a static assert.

Thanks,
Gage

#13 lore raw

Re: [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 48 lines · Gage
Quoted 48 lines

-----Original Message-----
From: Richardson, Bruce
Sent: Friday, January 11, 2019 5:59 AM
To: Burakov, Anatoly <[email protected]>
Cc: Eads, Gage <[email protected]>; [email protected]; [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 8 lines · Anatoly
Quoted 7 lines · Bruce Richardson
Quoted 6 lines · Anatoly

<...>

Review rte_event_ring_enqueue_burst(struct

Quoted 4 lines

+ * Copyright(c) 2016-2019 Intel Corporation

     */
    /**
(quoted diff) +0 -0
@@ -88,7 +88,7 @@ rte_event_ring_enqueue_burst(struct

rte_event_ring *r,

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/librte_eventdev/rte_event_ring.h rte_event_ring_enqueue_burst(struct rte_event_ring *r, · line 91

Quoted 5 lines
lib/librte_eventdev/rte_event_ring.h +1 -1
    		const struct rte_event *events,
    		unsigned int n, uint16_t *free_space)
    {
-	uint32_t prod_head, prod_next;
+	uintptr_t prod_head, prod_next;

I would also question the use of uinptr_t. I think semnatically, size_t is more appropriate.

Quoted 7 lines
Quoted 7 lines

Yes, it would, but I believe in this case they want to use the largest size of (unsigned)int where there exists an atomic for manipulating 2 of them simultaneously. [The largest size is to minimize any chance of an ABA issue occuring]. Therefore we need 32-bit values on 32-bit and 64-bit on 64, and I suspect the best way to guarantee this is to use pointer-sized values. If size_t is guaranteed across all OS's to have the same size as uintptr_t it could also be

used, though.

Quoted 9 lines
Quoted 3 lines

/Bruce

Technically, size_t and uintptr_t are not guaranteed to match. In practice, they won't match only on architectures that DPDK doesn't intend to run on (such as 16-bit segmented archs, where size_t would be 16-bit but uinptr_t would be 32-bit).

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

In all the rest of DPDK code, we use size_t for this kind of thing.

Ok.
If we do use size_t, I think we also need to add a compile-time check into the build too, to error out if sizeof(size_t) != sizeof(uintptr_t).

Ok, I wasn't aware of the precedent of using size_t for this purpose. I'll change it and look into adding a static assert.

RTE_BUILD_BUG_ON?

Quoted 4 lines

Thanks,
Gage

Signature · 2 lines
Thanks,
Anatoly
author
#14 lore raw

Re: [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 65 lines

-----Original Message-----
From: Burakov, Anatoly
Sent: Monday, January 21, 2019 8:15 AM
To: Eads, Gage <[email protected]>; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 52 lines · Gage
Quoted 52 lines

-----Original Message-----
From: Richardson, Bruce
Sent: Friday, January 11, 2019 5:59 AM
To: Burakov, Anatoly <[email protected]>
Cc: Eads, Gage <[email protected]>; [email protected]; [email protected]; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH 1/6] ring: change head and tail to pointer-width size

Quoted 8 lines · Anatoly
Quoted 7 lines · Bruce Richardson
Quoted 6 lines · Anatoly

<...>

Review rte_event_ring_enqueue_burst(struct

Quoted 4 lines

+ * Copyright(c) 2016-2019 Intel Corporation

     */
    /**
(quoted diff) +0 -0
@@ -88,7 +88,7 @@ rte_event_ring_enqueue_burst(struct

rte_event_ring *r,

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Review rte_event_ring_enqueue_burst(struct · line 91

Quoted 5 lines
(quoted diff) +1 -1
    		const struct rte_event *events,
    		unsigned int n, uint16_t *free_space)
    {
-	uint32_t prod_head, prod_next;
+	uintptr_t prod_head, prod_next;

I would also question the use of uinptr_t. I think semnatically, size_t is more appropriate.

Quoted 8 lines
Quoted 8 lines

Yes, it would, but I believe in this case they want to use the largest size of (unsigned)int where there exists an atomic for manipulating 2 of them simultaneously. [The largest size is to minimize any chance of an ABA issue occuring]. Therefore we need 32-bit values on 32-bit and 64-bit on 64, and I suspect the best way to guarantee this is to use pointer-sized values. If size_t is guaranteed across all OS's to have the same size as uintptr_t it could also be

used, though.

Quoted 11 lines
Quoted 3 lines

/Bruce

Technically, size_t and uintptr_t are not guaranteed to match. In practice, they won't match only on architectures that DPDK doesn't intend to run on (such as 16-bit segmented archs, where size_t would be 16-bit but uinptr_t would be 32-bit).

In all the rest of DPDK code, we use size_t for this kind of thing.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Ok.
If we do use size_t, I think we also need to add a compile-time check into the build too, to error out if sizeof(size_t) != sizeof(uintptr_t).

Ok, I wasn't aware of the precedent of using size_t for this purpose. I'll change

it and look into adding a static assert.

RTE_BUILD_BUG_ON?

Appreciate the pointer, but with the changes needed to preserve ABI compatibility* this is no longer necessary.

*http://mails.dpdk.org/archives/dev/2019-January/123775.html

Quoted 10 lines
Quoted 5 lines

Thanks,
Gage

--
Thanks,
Anatoly

author
#15 lore raw

PATCH 2/6 ring: add a non-blocking implementation

In reply to Gage Eads

This commit adds support for non-blocking circular ring enqueue and dequeue functions. The ring uses a 128-bit compare-and-swap instruction, and thus is limited to x86_64.

The algorithm is based on the original rte ring (derived from FreeBSD's bufring.h) and inspired by Michael and Scott's non-blocking concurrent queue. Importantly, it adds a modification counter to each ring entry to ensure only one thread can write to an unused entry.

-----
Algorithm:

Multi-producer non-blocking enqueue:
1. Move the producer head index 'n' locations forward, effectively reserving 'n' locations.
2. For each pointer:
a. Read the producer tail index, then ring[tail]. If ring[tail]'s
modification counter isn't 'tail', retry.
b. Construct the new entry: {pointer, tail + ring size}
c. Compare-and-swap the old entry with the new. If unsuccessful, the
next loop iteration will try to enqueue this pointer again.
d. Compare-and-swap the tail index with 'tail + 1', whether or not step 2c
succeeded. This guarantees threads can make forward progress.

Multi-consumer non-blocking dequeue:
1. Move the consumer head index 'n' locations forward, effectively reserving 'n' pointers to be dequeued.
2. Copy 'n' pointers into the caller's object table (ignoring the modification counter), starting from ring[tail], then compare-and-swap the tail index with 'tail + n'. If unsuccessful, repeat step 2.

-----
Discussion:

There are two cases where the ABA problem is mitigated:
1. Enqueueing a pointer to the ring: without a modification counter tied to the tail index, the index could become stale by the time the enqueue happens, causing it to overwrite valid data. Tying the counter to the tail index gives us an expected value (as opposed to, say, a monotonically incrementing counter).

Since the counter will eventually wrap, there is potential for the ABA problem. However, using a 64-bit counter makes this likelihood effectively zero.

2. Updating a tail index: the ABA problem can occur if the thread is preempted and the tail index wraps around. However, using 64-bit indexes makes this likelihood effectively zero.

With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a non-blocking ring based mempool (a likely use case for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. Because the ring's memsize is now a function of its flags (the non-blocking ring requires 128b for each entry), this commit adds a new argument ('flags') to rte_ring_get_memsize().

For ease-of-use, existing ring enqueue and dequeue functions work on both regular and non-blocking rings. This introduces an additional branch in the datapath, but this should be a highly predictable branch. ring_perf_autotest shows a negligible performance impact; it's hard to distinguish a real difference versus system noise.

| ring_perf_autotest cycles with branch -
Test | ring_perf_autotest cycles without
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | -4.00
SP/SC burst enq/dequeue (size 8)  | 0.00
MP/MC burst enq/dequeue (size 8)  | 0.00
SP/SC burst enq/dequeue (size 32) | 0.00
MP/MC burst enq/dequeue (size 32) | 0.00
SC empty dequeue                  | 1.00
MC empty dequeue                  | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | 0.49
MP/MC bulk enq/dequeue (size 8)   | 0.08
SP/SC bulk enq/dequeue (size 32)  | 0.07
MP/MC bulk enq/dequeue (size 32)  | 0.09

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | 0.19
MP/MC bulk enq/dequeue (size 8)   | -0.37
SP/SC bulk enq/dequeue (size 32)  | 0.09
MP/MC bulk enq/dequeue (size 32)  | -0.05

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | -1.96
MP/MC bulk enq/dequeue (size 8)   | 0.88
SP/SC bulk enq/dequeue (size 32)  | 0.10
MP/MC bulk enq/dequeue (size 32)  | 0.46

Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. Each test run three times and the results averaged.

---
 lib/librte_ring/rte_ring.c           |  43 ++-
 lib/librte_ring/rte_ring.h           | 535 +++++++++++++++++++++++++++++++++--
 lib/librte_ring/rte_ring_version.map |   7 +
 3 files changed, 554 insertions(+), 31 deletions(-)
lib/librte_ring/rte_ring.c +39 -4
index b15ee0eb3..bd1282eac 100644
@@ -45,9 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
 
 /* return the size of memory occupied by a ring */
 ssize_t
-rte_ring_get_memsize(unsigned count)
+rte_ring_get_memsize_v1902(unsigned int count, unsigned int flags)
 {
-	ssize_t sz;
+	ssize_t sz, elt_sz;
 
 	/* count must be a power of 2 */
 	if ((!POWEROF2(count)) || (count > RTE_RING_SZ_MASK )) {
@@ -57,10 +57,23 @@ rte_ring_get_memsize(unsigned count)
 		return -EINVAL;
 	}
 
-	sz = sizeof(struct rte_ring) + count * sizeof(void *);
+	elt_sz = (flags & RING_F_NB) ? 2 * sizeof(void *) : sizeof(void *);
+
+	sz = sizeof(struct rte_ring) + count * elt_sz;
 	sz = RTE_ALIGN(sz, RTE_CACHE_LINE_SIZE);
 	return sz;
 }
+BIND_DEFAULT_SYMBOL(rte_ring_get_memsize, _v1902, 19.02);
+MAP_STATIC_SYMBOL(ssize_t rte_ring_get_memsize(unsigned int count,
+					       unsigned int flags),
+		  rte_ring_get_memsize_v1902);
+
+ssize_t
+rte_ring_get_memsize_v20(unsigned int count)
+{
+	return rte_ring_get_memsize_v1902(count, 0);
+}
+VERSION_SYMBOL(rte_ring_get_memsize, _v20, 2.0);
 
 int
 rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
@@ -103,6 +116,20 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 	r->prod.head = r->cons.head = 0;
 	r->prod.tail = r->cons.tail = 0;
 
+	if (flags & RING_F_NB) {
+		uint64_t i;
+
+		for (i = 0; i < r->size; i++) {
+			struct nb_ring_entry *ring_ptr, *base;
+
+			base = ((struct nb_ring_entry *) &r[1]);
+
+			ring_ptr = &base[i & r->mask];
+
+			ring_ptr->cnt = i;
+		}
+	}
+
 	return 0;
 }
 
@@ -123,11 +150,19 @@ rte_ring_create(const char *name, unsigned count, int socket_id,
 
 	ring_list = RTE_TAILQ_CAST(rte_ring_tailq.head, rte_ring_list);
 
+#if !defined(RTE_ARCH_X86_64)
+	if (flags & RING_F_NB) {
+		printf("RING_F_NB is only supported on x86-64 platforms\n");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+#endif
+
 	/* for an exact size ring, round up from count to a power of two */
 	if (flags & RING_F_EXACT_SZ)
 		count = rte_align32pow2(count + 1);
 
-	ring_size = rte_ring_get_memsize(count);
+	ring_size = rte_ring_get_memsize(count, flags);
 	if (ring_size < 0) {
 		rte_errno = ring_size;
 		return NULL;
lib/librte_ring/rte_ring.h +508 -27

658 lines not on this page · raw message

lib/librte_ring/rte_ring_version.map +7 -0
index d935efd0d..8969467af 100644
@@ -17,3 +17,10 @@ DPDK_2.2 {
 	rte_ring_free;
 
 } DPDK_2.0;
+
+DPDK_19.05 {
+	global:
+
+	rte_ring_get_memsize;
+
+} DPDK_2.2;
Signature · 1 line
2.13.6
author
#16 lore raw

PATCH 3/6 test_ring: add non-blocking ring autotest

In reply to Gage Eads

ring_nb_autotest re-uses the ring_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 test/test/test_ring.c | 57 ++++++++++++++++++++++++++++++++-------------------
 1 file changed, 36 insertions(+), 21 deletions(-)
test/test/test_ring.c +36 -21
index aaf1e70ad..ff410d978 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 #include <string.h>
@@ -601,18 +601,20 @@ test_ring_burst_basic(struct rte_ring *r)
  * it will always fail to create ring with a wrong ring size number in this function
  */
 static int
-test_ring_creation_with_wrong_size(void)
+test_ring_creation_with_wrong_size(unsigned int flags)
 {
 	struct rte_ring * rp = NULL;
 
 	/* Test if ring size is not power of 2 */
-	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1,
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
 
 	/* Test if ring size is exceeding the limit */
-	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1), SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1),
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
@@ -623,11 +625,11 @@ test_ring_creation_with_wrong_size(void)
  * it tests if it would always fail to create ring with an used ring name
  */
 static int
-test_ring_creation_with_an_used_name(void)
+test_ring_creation_with_an_used_name(unsigned int flags)
 {
 	struct rte_ring * rp;
 
-	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (NULL != rp)
 		return -1;
 
@@ -639,10 +641,10 @@ test_ring_creation_with_an_used_name(void)
  * function to fail correctly
  */
 static int
-test_create_count_odd(void)
+test_create_count_odd(unsigned int flags)
 {
 	struct rte_ring *r = rte_ring_create("test_ring_count",
-			4097, SOCKET_ID_ANY, 0 );
+			4097, SOCKET_ID_ANY, flags);
 	if(r != NULL){
 		return -1;
 	}
@@ -665,7 +667,7 @@ test_lookup_null(void)
  * it tests some more basic ring operations
  */
 static int
-test_ring_basic_ex(void)
+test_ring_basic_ex(unsigned int flags)
 {
 	int ret = -1;
 	unsigned i;
@@ -679,7 +681,7 @@ test_ring_basic_ex(void)
 	}
 
 	rp = rte_ring_create("test_ring_basic_ex", RING_SIZE, SOCKET_ID_ANY,
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (rp == NULL) {
 		printf("test_ring_basic_ex fail to create ring\n");
 		goto fail_test;
@@ -737,7 +739,7 @@ test_ring_basic_ex(void)
 }
 
 static int
-test_ring_with_exact_size(void)
+test_ring_with_exact_size(unsigned int flags)
 {
 	struct rte_ring *std_ring = NULL, *exact_sz_ring = NULL;
 	void *ptr_array[16];
@@ -746,13 +748,13 @@ test_ring_with_exact_size(void)
 	int ret = -1;
 
 	std_ring = rte_ring_create("std", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (std_ring == NULL) {
 		printf("%s: error, can't create std ring\n", __func__);
 		goto end;
 	}
 	exact_sz_ring = rte_ring_create("exact sz", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ);
+		RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ | flags);
 	if (exact_sz_ring == NULL) {
 		printf("%s: error, can't create exact size ring\n", __func__);
 		goto end;
@@ -808,17 +810,17 @@ test_ring_with_exact_size(void)
 }
 
 static int
-test_ring(void)
+__test_ring(unsigned int flags)
 {
 	struct rte_ring *r = NULL;
 
 	/* some more basic operations */
-	if (test_ring_basic_ex() < 0)
+	if (test_ring_basic_ex(flags) < 0)
 		goto test_fail;
 
 	rte_atomic32_init(&synchro);
 
-	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (r == NULL)
 		goto test_fail;
 
@@ -837,27 +839,27 @@ test_ring(void)
 		goto test_fail;
 
 	/* basic operations */
-	if ( test_create_count_odd() < 0){
+	if (test_create_count_odd(flags) < 0) {
 		printf("Test failed to detect odd count\n");
 		goto test_fail;
 	} else
 		printf("Test detected odd count\n");
 
-	if ( test_lookup_null() < 0){
+	if (test_lookup_null() < 0) {
 		printf("Test failed to detect NULL ring lookup\n");
 		goto test_fail;
 	} else
 		printf("Test detected NULL ring lookup\n");
 
 	/* test of creating ring with wrong size */
-	if (test_ring_creation_with_wrong_size() < 0)
+	if (test_ring_creation_with_wrong_size(flags) < 0)
 		goto test_fail;
 
 	/* test of creation ring with an used name */
-	if (test_ring_creation_with_an_used_name() < 0)
+	if (test_ring_creation_with_an_used_name(flags) < 0)
 		goto test_fail;
 
-	if (test_ring_with_exact_size() < 0)
+	if (test_ring_with_exact_size(flags) < 0)
 		goto test_fail;
 
 	/* dump the ring status */
@@ -873,4 +875,17 @@ test_ring(void)
 	return -1;
 }
 
+static int
+test_ring(void)
+{
+	return __test_ring(0);
+}
+
+static int
+test_nb_ring(void)
+{
+	return __test_ring(RING_F_NB);
+}
+
 REGISTER_TEST_COMMAND(ring_autotest, test_ring);
+REGISTER_TEST_COMMAND(ring_nb_autotest, test_nb_ring);
Signature · 1 line
2.13.6
author
#17 lore raw

PATCH 4/6 test_ring_perf: add non-blocking ring perf test

In reply to Gage Eads

nb_ring_perf_autotest re-uses the ring_perf_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 test/test/test_ring_perf.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
test/test/test_ring_perf.c +16 -3
index ebb3939f5..380c4b4a1 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 
@@ -363,12 +363,12 @@ test_bulk_enqueue_dequeue(struct rte_ring *r)
 }
 
 static int
-test_ring_perf(void)
+__test_ring_perf(unsigned int flags)
 {
 	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
 
-	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), 0);
+	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), flags);
 	if (r == NULL)
 		return -1;
 
@@ -398,4 +398,17 @@ test_ring_perf(void)
 	return 0;
 }
 
+static int
+test_ring_perf(void)
+{
+	return __test_ring_perf(0);
+}
+
+static int
+test_nb_ring_perf(void)
+{
+	return __test_ring_perf(RING_F_NB);
+}
+
 REGISTER_TEST_COMMAND(ring_perf_autotest, test_ring_perf);
+REGISTER_TEST_COMMAND(ring_nb_perf_autotest, test_nb_ring_perf);
Signature · 1 line
2.13.6
author
#18 lore raw

PATCH 5/6 mempool/ring: add non-blocking ring handlers

In reply to Gage Eads

These handlers allow an application to create a mempool based on the non-blocking ring, with any combination of single/multi producer/consumer.

---
 drivers/mempool/ring/rte_mempool_ring.c | 58 +++++++++++++++++++++++++++++++--
 1 file changed, 55 insertions(+), 3 deletions(-)
drivers/mempool/ring/rte_mempool_ring.c +55 -3
index bc123fc52..013dac3bc 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2016 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 #include <stdio.h>
@@ -47,11 +47,11 @@ common_ring_get_count(const struct rte_mempool *mp)
 
 
 static int
-common_ring_alloc(struct rte_mempool *mp)
+__common_ring_alloc(struct rte_mempool *mp, int rg_flags)
 {
-	int rg_flags = 0, ret;
 	char rg_name[RTE_RING_NAMESIZE];
 	struct rte_ring *r;
+	int ret;
 
 	ret = snprintf(rg_name, sizeof(rg_name),
 		RTE_MEMPOOL_MZ_FORMAT, mp->name);
@@ -82,6 +82,18 @@ common_ring_alloc(struct rte_mempool *mp)
 	return 0;
 }
 
+static int
+common_ring_alloc(struct rte_mempool *mp)
+{
+	return __common_ring_alloc(mp, 0);
+}
+
+static int
+common_ring_alloc_nb(struct rte_mempool *mp)
+{
+	return __common_ring_alloc(mp, RING_F_NB);
+}
+
 static void
 common_ring_free(struct rte_mempool *mp)
 {
@@ -130,7 +142,47 @@ static const struct rte_mempool_ops ops_sp_mc = {
 	.get_count = common_ring_get_count,
 };
 
+static const struct rte_mempool_ops ops_mp_mc_nb = {
+	.name = "ring_mp_mc_nb",
+	.alloc = common_ring_alloc_nb,
+	.free = common_ring_free,
+	.enqueue = common_ring_mp_enqueue,
+	.dequeue = common_ring_mc_dequeue,
+	.get_count = common_ring_get_count,
+};
+
+static const struct rte_mempool_ops ops_sp_sc_nb = {
+	.name = "ring_sp_sc_nb",
+	.alloc = common_ring_alloc_nb,
+	.free = common_ring_free,
+	.enqueue = common_ring_sp_enqueue,
+	.dequeue = common_ring_sc_dequeue,
+	.get_count = common_ring_get_count,
+};
+
+static const struct rte_mempool_ops ops_mp_sc_nb = {
+	.name = "ring_mp_sc_nb",
+	.alloc = common_ring_alloc_nb,
+	.free = common_ring_free,
+	.enqueue = common_ring_mp_enqueue,
+	.dequeue = common_ring_sc_dequeue,
+	.get_count = common_ring_get_count,
+};
+
+static const struct rte_mempool_ops ops_sp_mc_nb = {
+	.name = "ring_sp_mc_nb",
+	.alloc = common_ring_alloc_nb,
+	.free = common_ring_free,
+	.enqueue = common_ring_sp_enqueue,
+	.dequeue = common_ring_mc_dequeue,
+	.get_count = common_ring_get_count,
+};
+
 MEMPOOL_REGISTER_OPS(ops_mp_mc);
 MEMPOOL_REGISTER_OPS(ops_sp_sc);
 MEMPOOL_REGISTER_OPS(ops_mp_sc);
 MEMPOOL_REGISTER_OPS(ops_sp_mc);
+MEMPOOL_REGISTER_OPS(ops_mp_mc_nb);
+MEMPOOL_REGISTER_OPS(ops_sp_sc_nb);
+MEMPOOL_REGISTER_OPS(ops_mp_sc_nb);
+MEMPOOL_REGISTER_OPS(ops_sp_mc_nb);
Signature · 1 line
2.13.6
author
#20 lore raw

PATCH 6/6 doc: add NB ring comment to EAL "known issues"

In reply to Gage Eads

This comment makes users aware of the non-blocking ring option and its caveats.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
doc/guides/prog_guide/env_abstraction_layer.rst +1 -1
index 9497b879c..b6ac236d6 100644
@@ -541,7 +541,7 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
-  Alternatively, x86_64 applications can use the non-blocking stack mempool handler. When considering this handler, note that:
+  Alternatively, x86_64 applications can use the non-blocking ring or stack mempool handlers. When considering one of them, note that:
 
   - it is limited to the x86_64 platform, because it uses an instruction (16-byte compare-and-swap) that is not available on other platforms.
   - it has worse average-case performance than the non-preemptive rte_ring, but software caching (e.g. the mempool cache) can mitigate this by reducing the number of handler operations.
Signature · 1 line
2.13.6
#21 lore raw

Re: [PATCH 6/6] doc: add NB ring comment to EAL "known issues"

Hi Gage,

Humble suggestion from my end, as per DPDK 19.02-rc1 the documentation and code change have to be in same patch. Can you please take a look into it.

Quoted 40 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Gage Eads
Sent: Friday, January 11, 2019 2:31 AM
To: [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>
Subject: [dpdk-dev] [PATCH 6/6] doc: add NB ring comment to EAL "known issues"

This comment makes users aware of the non-blocking ring option and its caveats.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
doc/guides/prog_guide/env_abstraction_layer.rst +0 -0
index 9497b879c..b6ac236d6 100644
@@ -541,7 +541,7 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose

scheduling policies are SCHED_FIFO or SCHED_RR.

- Alternatively, x86_64 applications can use the non-blocking stack mempool handler. When considering this handler, note that:
+ Alternatively, x86_64 applications can use the non-blocking ring or stack mempool handlers. When considering one of them, note that:

- it is limited to the x86_64 platform, because it uses an instruction (16-byte
compare-and-swap) that is not available on other platforms.
- it has worse average-case performance than the non-preemptive rte_ring,
but software caching (e.g. the mempool cache) can mitigate this by reducing the number of handler operations.
--
2.13.6

author
#22 lore raw

Re: [PATCH 6/6] doc: add NB ring comment to EAL "known issues"

Quoted 5 lines

Hi Gage,

Humble suggestion from my end, as per DPDK 19.02-rc1 the documentation and code change have to be in same patch. Can you please take a look into it.

Certainly, and I'll fix this in my other docs change patch (http://mails.dpdk.org/archives/dev/2019-January/122926.html). Thanks for the heads up.

#23 lore raw

Re: [PATCH 6/6] doc: add NB ring comment to EAL "known issues"

Thanks for understanding

Quoted 20 lines

-----Original Message-----
From: Eads, Gage
Sent: Saturday, January 12, 2019 1:01 AM
To: Varghese, Vipin <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>
Subject: RE: [dpdk-dev] [PATCH 6/6] doc: add NB ring comment to EAL "known issues"

Quoted 4 lines

Hi Gage,

Humble suggestion from my end, as per DPDK 19.02-rc1 the documentation and code change have to be in same patch. Can you please take a look into

it.

Quoted 1 line

Certainly, and I'll fix this in my other docs change patch
(http://mails.dpdk.org/archives/dev/2019-January/122926.html). Thanks for the heads up.

author
#24 lore raw

v2 cover Add non-blocking ring

In reply to Gage Eads

For some users, the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

This patchset introduces a non-blocking ring, on top of which a mempool can run. Crucially, the non-blocking algorithm relies on a 128-bit compare-and-swap, so it is limited to x86_64 machines.

The ring uses more compare-and-swap atomic operations than the regular rte ring:
With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a non-blocking ring based mempool (a likely use case for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. For ease-of-use, existing ring enqueue/dequeue functions work with both "regular" and non-blocking rings.

This patchset also adds non-blocking versions of ring_autotest and ring_perf_autotest, and a non-blocking ring based mempool.

This patchset makes ABI and API changes; a deprecation notice will be posted in a separate commit.

This patchset depends on the non-blocking stack patchset[1].

[1] http://mails.dpdk.org/archives/dev/2019-January/123470.html

v2:
- Merge separate docs commit into patch #5
- Convert uintptr_t to size_t
- Add a compile-time check for the size of size_t
- Fix a space-after-typecast issue
- Fix an unnecessary-parentheses checkpatch warning
- Bump librte_ring's library version

Gage Eads (5):
ring: change head and tail to pointer-width size
ring: add a non-blocking implementation
test_ring: add non-blocking ring autotest
test_ring_perf: add non-blocking ring perf test mempool/ring: add non-blocking ring handlers

 doc/guides/prog_guide/env_abstraction_layer.rst |   2 +-
 drivers/mempool/ring/rte_mempool_ring.c         |  58 ++-
 lib/librte_eventdev/rte_event_ring.h            |   6 +-
 lib/librte_ring/Makefile                        |   2 +-
 lib/librte_ring/meson.build                     |   2 +-
 lib/librte_ring/rte_ring.c                      |  53 ++-
 lib/librte_ring/rte_ring.h                      | 564 ++++++++++++++++++++++--
 lib/librte_ring/rte_ring_generic.h              |  16 +-
 lib/librte_ring/rte_ring_version.map            |   7 +
 test/test/test_ring.c                           |  57 ++-
 test/test/test_ring_perf.c                      |  19 +-
 11 files changed, 699 insertions(+), 87 deletions(-)
Signature · 1 line
2.13.6
author
#25 lore raw

PATCH v2 1/5 ring: change head and tail to pointer-width size

For 64-bit architectures, doubling the head and tail index widths greatly increases the time it takes for them to wrap-around (with current CPU speeds, it won't happen within the author's lifetime). This is important in avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming non-blocking ring implementation. Using a 64-bit index makes the possibility of this occurring effectively zero.

I tested this commit's performance impact with an x86_64 build on a dual-socket Xeon E5-2699 v4 using ring_perf_autotest, and the change made no significant difference -- the few differences appear to be system noise. (The test ran on isolcpus cores using a tickless scheduler, but some variation was stll observed.) Each test was run three times and the results were averaged:

| 64b head/tail cycle cost minus
Test | 32b head/tail cycle cost
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | 0.00
SP/SC burst enq/dequeue (size 8)  | 0.00
MP/MC burst enq/dequeue (size 8)  | 1.00
SP/SC burst enq/dequeue (size 32) | 0.00
MP/MC burst enq/dequeue (size 32) | -1.00
SC empty dequeue                  | 0.01
MC empty dequeue                  | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | -0.36
MP/MC bulk enq/dequeue (size 8)   | 0.99
SP/SC bulk enq/dequeue (size 32)  | -0.40
MP/MC bulk enq/dequeue (size 32)  | -0.57

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | -0.49
MP/MC bulk enq/dequeue (size 8)   | 0.19
SP/SC bulk enq/dequeue (size 32)  | -0.28
MP/MC bulk enq/dequeue (size 32)  | -0.62

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | 3.25
MP/MC bulk enq/dequeue (size 8)   | 1.87
SP/SC bulk enq/dequeue (size 32)  | -0.44
MP/MC bulk enq/dequeue (size 32)  | -1.10

An earlier version of this patch changed the head and tail indexes to uint64_t, but that caused a performance drop on 32-bit builds. With size_t, no performance difference is observed on an i686 build.

---
 lib/librte_eventdev/rte_event_ring.h |  6 +++---
 lib/librte_ring/Makefile             |  2 +-
 lib/librte_ring/meson.build          |  2 +-
 lib/librte_ring/rte_ring.c           | 10 +++++-----
 lib/librte_ring/rte_ring.h           | 29 ++++++++++++++++++-----------
 lib/librte_ring/rte_ring_generic.h   | 16 +++++++++-------
 6 files changed, 37 insertions(+), 28 deletions(-)
lib/librte_eventdev/rte_event_ring.h +3 -3
index 827a3209e..da8886c08 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2016-2017 Intel Corporation
+ * Copyright(c) 2016-2019 Intel Corporation
  */
 
 /**
@@ -88,7 +88,7 @@ rte_event_ring_enqueue_burst(struct rte_event_ring *r,
 		const struct rte_event *events,
 		unsigned int n, uint16_t *free_space)
 {
-	uint32_t prod_head, prod_next;
+	size_t prod_head, prod_next;
 	uint32_t free_entries;
 
 	n = __rte_ring_move_prod_head(&r->r, r->r.prod.single, n,
@@ -129,7 +129,7 @@ rte_event_ring_dequeue_burst(struct rte_event_ring *r,
 		struct rte_event *events,
 		unsigned int n, uint16_t *available)
 {
-	uint32_t cons_head, cons_next;
+	size_t cons_head, cons_next;
 	uint32_t entries;
 
 	n = __rte_ring_move_cons_head(&r->r, r->r.cons.single, n,
lib/librte_ring/Makefile +1 -1
index 21a36770d..c106f9908 100644
@@ -11,7 +11,7 @@ LDLIBS += -lrte_eal
 
 EXPORT_MAP := rte_ring_version.map
 
-LIBABIVER := 2
+LIBABIVER := 3
 
 # all source are stored in SRCS-y
 SRCS-$(CONFIG_RTE_LIBRTE_RING) := rte_ring.c
lib/librte_ring/meson.build +1 -1
index ab8b0b469..3603fdb7a 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
-version = 2
+version = 3
 sources = files('rte_ring.c')
 headers = files('rte_ring.h',
 		'rte_ring_c11_mem.h',
lib/librte_ring/rte_ring.c +5 -5
index d215acecc..b15ee0eb3 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2015 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007,2008 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -227,10 +227,10 @@ rte_ring_dump(FILE *f, const struct rte_ring *r)
 	fprintf(f, "  flags=%x\n", r->flags);
 	fprintf(f, "  size=%"PRIu32"\n", r->size);
 	fprintf(f, "  capacity=%"PRIu32"\n", r->capacity);
-	fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
-	fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
-	fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
-	fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	fprintf(f, "  ct=%"PRIuPTR"\n", r->cons.tail);
+	fprintf(f, "  ch=%"PRIuPTR"\n", r->cons.head);
+	fprintf(f, "  pt=%"PRIuPTR"\n", r->prod.tail);
+	fprintf(f, "  ph=%"PRIuPTR"\n", r->prod.head);
 	fprintf(f, "  used=%u\n", rte_ring_count(r));
 	fprintf(f, "  avail=%u\n", rte_ring_free_count(r));
 }
lib/librte_ring/rte_ring.h +18 -11
index af5444a9f..213c50708 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007-2009 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -63,11 +63,18 @@ enum rte_ring_queue_behavior {
 
 struct rte_memzone; /* forward declaration, so as not to require memzone.h */
 
+/* Verify that size_t is the same as uintptr_t, which rte_ring (among other
+ * components) assumes.
+ */
+#if UINTPTR_MAX != SIZE_MAX
+#error "DPDK requires sizeof(size_t) == sizeof(uintptr_t)"
+#endif
+
 /* structure to hold a pair of head/tail values and other metadata */
 struct rte_ring_headtail {
-	volatile uint32_t head;  /**< Prod/consumer head. */
-	volatile uint32_t tail;  /**< Prod/consumer tail. */
-	uint32_t single;         /**< True if single prod/cons */
+	volatile size_t head;  /**< Prod/consumer head. */
+	volatile size_t tail;  /**< Prod/consumer tail. */
+	uint32_t single;       /**< True if single prod/cons */
 };
 
 /**
@@ -242,7 +249,7 @@ void rte_ring_dump(FILE *f, const struct rte_ring *r);
 #define ENQUEUE_PTRS(r, ring_start, prod_head, obj_table, n, obj_type) do { \
 	unsigned int i; \
 	const uint32_t size = (r)->size; \
-	uint32_t idx = prod_head & (r)->mask; \
+	size_t idx = prod_head & (r)->mask; \
 	obj_type *ring = (obj_type *)ring_start; \
 	if (likely(idx + n < size)) { \
 		for (i = 0; i < (n & ((~(unsigned)0x3))); i+=4, idx+=4) { \
@@ -272,7 +279,7 @@ void rte_ring_dump(FILE *f, const struct rte_ring *r);
  * single and multi consumer dequeue functions */
 #define DEQUEUE_PTRS(r, ring_start, cons_head, obj_table, n, obj_type) do { \
 	unsigned int i; \
-	uint32_t idx = cons_head & (r)->mask; \
+	size_t idx = cons_head & (r)->mask; \
 	const uint32_t size = (r)->size; \
 	obj_type *ring = (obj_type *)ring_start; \
 	if (likely(idx + n < size)) { \
@@ -338,7 +345,7 @@ __rte_ring_do_enqueue(struct rte_ring *r, void * const *obj_table,
 		 unsigned int n, enum rte_ring_queue_behavior behavior,
 		 unsigned int is_sp, unsigned int *free_space)
 {
-	uint32_t prod_head, prod_next;
+	size_t prod_head, prod_next;
 	uint32_t free_entries;
 
 	n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
@@ -380,7 +387,7 @@ __rte_ring_do_dequeue(struct rte_ring *r, void **obj_table,
 		 unsigned int n, enum rte_ring_queue_behavior behavior,
 		 unsigned int is_sc, unsigned int *available)
 {
-	uint32_t cons_head, cons_next;
+	size_t cons_head, cons_next;
 	uint32_t entries;
 
 	n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
@@ -681,9 +688,9 @@ rte_ring_dequeue(struct rte_ring *r, void **obj_p)
 static inline unsigned
 rte_ring_count(const struct rte_ring *r)
 {
-	uint32_t prod_tail = r->prod.tail;
-	uint32_t cons_tail = r->cons.tail;
-	uint32_t count = (prod_tail - cons_tail) & r->mask;
+	size_t prod_tail = r->prod.tail;
+	size_t cons_tail = r->cons.tail;
+	size_t count = (prod_tail - cons_tail) & r->mask;
 	return (count > r->capacity) ? r->capacity : count;
 }
 
lib/librte_ring/rte_ring_generic.h +9 -7
index ea7dbe5b9..31e38adcc 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007-2009 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -11,7 +11,7 @@
 #define _RTE_RING_GENERIC_H_
 
 static __rte_always_inline void
-update_tail(struct rte_ring_headtail *ht, uint32_t old_val, uint32_t new_val,
+update_tail(struct rte_ring_headtail *ht, size_t old_val, size_t new_val,
 		uint32_t single, uint32_t enqueue)
 {
 	if (enqueue)
@@ -55,7 +55,7 @@ update_tail(struct rte_ring_headtail *ht, uint32_t old_val, uint32_t new_val,
 static __rte_always_inline unsigned int
 __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,
 		unsigned int n, enum rte_ring_queue_behavior behavior,
-		uint32_t *old_head, uint32_t *new_head,
+		size_t *old_head, size_t *new_head,
 		uint32_t *free_entries)
 {
 	const uint32_t capacity = r->capacity;
@@ -93,7 +93,8 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,
 		if (is_sp)
 			r->prod.head = *new_head, success = 1;
 		else
-			success = rte_atomic32_cmpset(&r->prod.head,
+			/* Built-in used to handle variable-sized head index. */
+			success = __sync_bool_compare_and_swap(&r->prod.head,
 					*old_head, *new_head);
 	} while (unlikely(success == 0));
 	return n;
@@ -125,7 +126,7 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,
 static __rte_always_inline unsigned int
 __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
 		unsigned int n, enum rte_ring_queue_behavior behavior,
-		uint32_t *old_head, uint32_t *new_head,
+		size_t *old_head, size_t *new_head,
 		uint32_t *entries)
 {
 	unsigned int max = n;
@@ -161,8 +162,9 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
 		if (is_sc)
 			r->cons.head = *new_head, success = 1;
 		else
-			success = rte_atomic32_cmpset(&r->cons.head, *old_head,
-					*new_head);
+			/* Built-in used to handle variable-sized head index. */
+			success = __sync_bool_compare_and_swap(&r->cons.head,
+					*old_head, *new_head);
 	} while (unlikely(success == 0));
 	return n;
 }
Signature · 1 line
2.13.6
author
#26 lore raw

PATCH v2 2/5 ring: add a non-blocking implementation

In reply to Gage Eads

This commit adds support for non-blocking circular ring enqueue and dequeue functions. The ring uses a 128-bit compare-and-swap instruction, and thus is limited to x86_64.

The algorithm is based on the original rte ring (derived from FreeBSD's bufring.h) and inspired by Michael and Scott's non-blocking concurrent queue. Importantly, it adds a modification counter to each ring entry to ensure only one thread can write to an unused entry.

-----
Algorithm:

Multi-producer non-blocking enqueue:
1. Move the producer head index 'n' locations forward, effectively reserving 'n' locations.
2. For each pointer:
a. Read the producer tail index, then ring[tail]. If ring[tail]'s
modification counter isn't 'tail', retry.
b. Construct the new entry: {pointer, tail + ring size}
c. Compare-and-swap the old entry with the new. If unsuccessful, the
next loop iteration will try to enqueue this pointer again.
d. Compare-and-swap the tail index with 'tail + 1', whether or not step 2c
succeeded. This guarantees threads can make forward progress.

Multi-consumer non-blocking dequeue:
1. Move the consumer head index 'n' locations forward, effectively reserving 'n' pointers to be dequeued.
2. Copy 'n' pointers into the caller's object table (ignoring the modification counter), starting from ring[tail], then compare-and-swap the tail index with 'tail + n'. If unsuccessful, repeat step 2.

-----
Discussion:

There are two cases where the ABA problem is mitigated:
1. Enqueueing a pointer to the ring: without a modification counter tied to the tail index, the index could become stale by the time the enqueue happens, causing it to overwrite valid data. Tying the counter to the tail index gives us an expected value (as opposed to, say, a monotonically incrementing counter).

Since the counter will eventually wrap, there is potential for the ABA problem. However, using a 64-bit counter makes this likelihood effectively zero.

2. Updating a tail index: the ABA problem can occur if the thread is preempted and the tail index wraps around. However, using 64-bit indexes makes this likelihood effectively zero.

With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a non-blocking ring based mempool (a likely use case for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. Because the ring's memsize is now a function of its flags (the non-blocking ring requires 128b for each entry), this commit adds a new argument ('flags') to rte_ring_get_memsize().

For ease-of-use, existing ring enqueue and dequeue functions work on both regular and non-blocking rings. This introduces an additional branch in the datapath, but this should be a highly predictable branch. ring_perf_autotest shows a negligible performance impact; it's hard to distinguish a real difference versus system noise.

| ring_perf_autotest cycles with branch -
Test | ring_perf_autotest cycles without
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | -4.00
SP/SC burst enq/dequeue (size 8)  | 0.00
MP/MC burst enq/dequeue (size 8)  | 0.00
SP/SC burst enq/dequeue (size 32) | 0.00
MP/MC burst enq/dequeue (size 32) | 0.00
SC empty dequeue                  | 1.00
MC empty dequeue                  | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | 0.49
MP/MC bulk enq/dequeue (size 8)   | 0.08
SP/SC bulk enq/dequeue (size 32)  | 0.07
MP/MC bulk enq/dequeue (size 32)  | 0.09

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | 0.19
MP/MC bulk enq/dequeue (size 8)   | -0.37
SP/SC bulk enq/dequeue (size 32)  | 0.09
MP/MC bulk enq/dequeue (size 32)  | -0.05

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | -1.96
MP/MC bulk enq/dequeue (size 8)   | 0.88
SP/SC bulk enq/dequeue (size 32)  | 0.10
MP/MC bulk enq/dequeue (size 32)  | 0.46

Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. Each test run three times and the results averaged.

---
 lib/librte_ring/rte_ring.c           |  43 ++-
 lib/librte_ring/rte_ring.h           | 535 +++++++++++++++++++++++++++++++++--
 lib/librte_ring/rte_ring_version.map |   7 +
 3 files changed, 554 insertions(+), 31 deletions(-)
lib/librte_ring/rte_ring.c +39 -4
index b15ee0eb3..783c96568 100644
@@ -45,9 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
 
 /* return the size of memory occupied by a ring */
 ssize_t
-rte_ring_get_memsize(unsigned count)
+rte_ring_get_memsize_v1905(unsigned int count, unsigned int flags)
 {
-	ssize_t sz;
+	ssize_t sz, elt_sz;
 
 	/* count must be a power of 2 */
 	if ((!POWEROF2(count)) || (count > RTE_RING_SZ_MASK )) {
@@ -57,10 +57,23 @@ rte_ring_get_memsize(unsigned count)
 		return -EINVAL;
 	}
 
-	sz = sizeof(struct rte_ring) + count * sizeof(void *);
+	elt_sz = (flags & RING_F_NB) ? 2 * sizeof(void *) : sizeof(void *);
+
+	sz = sizeof(struct rte_ring) + count * elt_sz;
 	sz = RTE_ALIGN(sz, RTE_CACHE_LINE_SIZE);
 	return sz;
 }
+BIND_DEFAULT_SYMBOL(rte_ring_get_memsize, _v1905, 19.05);
+MAP_STATIC_SYMBOL(ssize_t rte_ring_get_memsize(unsigned int count,
+					       unsigned int flags),
+		  rte_ring_get_memsize_v1905);
+
+ssize_t
+rte_ring_get_memsize_v20(unsigned int count)
+{
+	return rte_ring_get_memsize_v1905(count, 0);
+}
+VERSION_SYMBOL(rte_ring_get_memsize, _v20, 2.0);
 
 int
 rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
@@ -103,6 +116,20 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 	r->prod.head = r->cons.head = 0;
 	r->prod.tail = r->cons.tail = 0;
 
+	if (flags & RING_F_NB) {
+		uint64_t i;
+
+		for (i = 0; i < r->size; i++) {
+			struct nb_ring_entry *ring_ptr, *base;
+
+			base = ((struct nb_ring_entry *)&r[1]);
+
+			ring_ptr = &base[i & r->mask];
+
+			ring_ptr->cnt = i;
+		}
+	}
+
 	return 0;
 }
 
@@ -123,11 +150,19 @@ rte_ring_create(const char *name, unsigned count, int socket_id,
 
 	ring_list = RTE_TAILQ_CAST(rte_ring_tailq.head, rte_ring_list);
 
+#if !defined(RTE_ARCH_X86_64)
+	if (flags & RING_F_NB) {
+		printf("RING_F_NB is only supported on x86-64 platforms\n");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+#endif
+
 	/* for an exact size ring, round up from count to a power of two */
 	if (flags & RING_F_EXACT_SZ)
 		count = rte_align32pow2(count + 1);
 
-	ring_size = rte_ring_get_memsize(count);
+	ring_size = rte_ring_get_memsize(count, flags);
 	if (ring_size < 0) {
 		rte_errno = ring_size;
 		return NULL;
lib/librte_ring/rte_ring.h +508 -27

658 lines not on this page · raw message

lib/librte_ring/rte_ring_version.map +7 -0
index d935efd0d..8969467af 100644
@@ -17,3 +17,10 @@ DPDK_2.2 {
 	rte_ring_free;
 
 } DPDK_2.0;
+
+DPDK_19.05 {
+	global:
+
+	rte_ring_get_memsize;
+
+} DPDK_2.2;
Signature · 1 line
2.13.6
author
#27 lore raw

PATCH v2 3/5 test_ring: add non-blocking ring autotest

In reply to Gage Eads

ring_nb_autotest re-uses the ring_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 test/test/test_ring.c | 57 ++++++++++++++++++++++++++++++++-------------------
 1 file changed, 36 insertions(+), 21 deletions(-)
test/test/test_ring.c +36 -21
index aaf1e70ad..ff410d978 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 #include <string.h>
@@ -601,18 +601,20 @@ test_ring_burst_basic(struct rte_ring *r)
  * it will always fail to create ring with a wrong ring size number in this function
  */
 static int
-test_ring_creation_with_wrong_size(void)
+test_ring_creation_with_wrong_size(unsigned int flags)
 {
 	struct rte_ring * rp = NULL;
 
 	/* Test if ring size is not power of 2 */
-	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1,
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
 
 	/* Test if ring size is exceeding the limit */
-	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1), SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1),
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
@@ -623,11 +625,11 @@ test_ring_creation_with_wrong_size(void)
  * it tests if it would always fail to create ring with an used ring name
  */
 static int
-test_ring_creation_with_an_used_name(void)
+test_ring_creation_with_an_used_name(unsigned int flags)
 {
 	struct rte_ring * rp;
 
-	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (NULL != rp)
 		return -1;
 
@@ -639,10 +641,10 @@ test_ring_creation_with_an_used_name(void)
  * function to fail correctly
  */
 static int
-test_create_count_odd(void)
+test_create_count_odd(unsigned int flags)
 {
 	struct rte_ring *r = rte_ring_create("test_ring_count",
-			4097, SOCKET_ID_ANY, 0 );
+			4097, SOCKET_ID_ANY, flags);
 	if(r != NULL){
 		return -1;
 	}
@@ -665,7 +667,7 @@ test_lookup_null(void)
  * it tests some more basic ring operations
  */
 static int
-test_ring_basic_ex(void)
+test_ring_basic_ex(unsigned int flags)
 {
 	int ret = -1;
 	unsigned i;
@@ -679,7 +681,7 @@ test_ring_basic_ex(void)
 	}
 
 	rp = rte_ring_create("test_ring_basic_ex", RING_SIZE, SOCKET_ID_ANY,
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (rp == NULL) {
 		printf("test_ring_basic_ex fail to create ring\n");
 		goto fail_test;
@@ -737,7 +739,7 @@ test_ring_basic_ex(void)
 }
 
 static int
-test_ring_with_exact_size(void)
+test_ring_with_exact_size(unsigned int flags)
 {
 	struct rte_ring *std_ring = NULL, *exact_sz_ring = NULL;
 	void *ptr_array[16];
@@ -746,13 +748,13 @@ test_ring_with_exact_size(void)
 	int ret = -1;
 
 	std_ring = rte_ring_create("std", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (std_ring == NULL) {
 		printf("%s: error, can't create std ring\n", __func__);
 		goto end;
 	}
 	exact_sz_ring = rte_ring_create("exact sz", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ);
+		RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ | flags);
 	if (exact_sz_ring == NULL) {
 		printf("%s: error, can't create exact size ring\n", __func__);
 		goto end;
@@ -808,17 +810,17 @@ test_ring_with_exact_size(void)
 }
 
 static int
-test_ring(void)
+__test_ring(unsigned int flags)
 {
 	struct rte_ring *r = NULL;
 
 	/* some more basic operations */
-	if (test_ring_basic_ex() < 0)
+	if (test_ring_basic_ex(flags) < 0)
 		goto test_fail;
 
 	rte_atomic32_init(&synchro);
 
-	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (r == NULL)
 		goto test_fail;
 
@@ -837,27 +839,27 @@ test_ring(void)
 		goto test_fail;
 
 	/* basic operations */
-	if ( test_create_count_odd() < 0){
+	if (test_create_count_odd(flags) < 0) {
 		printf("Test failed to detect odd count\n");
 		goto test_fail;
 	} else
 		printf("Test detected odd count\n");
 
-	if ( test_lookup_null() < 0){
+	if (test_lookup_null() < 0) {
 		printf("Test failed to detect NULL ring lookup\n");
 		goto test_fail;
 	} else
 		printf("Test detected NULL ring lookup\n");
 
 	/* test of creating ring with wrong size */
-	if (test_ring_creation_with_wrong_size() < 0)
+	if (test_ring_creation_with_wrong_size(flags) < 0)
 		goto test_fail;
 
 	/* test of creation ring with an used name */
-	if (test_ring_creation_with_an_used_name() < 0)
+	if (test_ring_creation_with_an_used_name(flags) < 0)
 		goto test_fail;
 
-	if (test_ring_with_exact_size() < 0)
+	if (test_ring_with_exact_size(flags) < 0)
 		goto test_fail;
 
 	/* dump the ring status */
@@ -873,4 +875,17 @@ test_ring(void)
 	return -1;
 }
 
+static int
+test_ring(void)
+{
+	return __test_ring(0);
+}
+
+static int
+test_nb_ring(void)
+{
+	return __test_ring(RING_F_NB);
+}
+
 REGISTER_TEST_COMMAND(ring_autotest, test_ring);
+REGISTER_TEST_COMMAND(ring_nb_autotest, test_nb_ring);
Signature · 1 line
2.13.6
author
#28 lore raw

PATCH v2 4/5 test_ring_perf: add non-blocking ring perf test

In reply to Gage Eads

nb_ring_perf_autotest re-uses the ring_perf_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 test/test/test_ring_perf.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
test/test/test_ring_perf.c +16 -3
index ebb3939f5..380c4b4a1 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 
@@ -363,12 +363,12 @@ test_bulk_enqueue_dequeue(struct rte_ring *r)
 }
 
 static int
-test_ring_perf(void)
+__test_ring_perf(unsigned int flags)
 {
 	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
 
-	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), 0);
+	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), flags);
 	if (r == NULL)
 		return -1;
 
@@ -398,4 +398,17 @@ test_ring_perf(void)
 	return 0;
 }
 
+static int
+test_ring_perf(void)
+{
+	return __test_ring_perf(0);
+}
+
+static int
+test_nb_ring_perf(void)
+{
+	return __test_ring_perf(RING_F_NB);
+}
+
 REGISTER_TEST_COMMAND(ring_perf_autotest, test_ring_perf);
+REGISTER_TEST_COMMAND(ring_nb_perf_autotest, test_nb_ring_perf);
Signature · 1 line
2.13.6
author
#29 lore raw

PATCH v2 5/5 mempool/ring: add non-blocking ring handlers

In reply to Gage Eads

These handlers allow an application to create a mempool based on the non-blocking ring, with any combination of single/multi producer/consumer.

Also, add a note to the programmer's guide's "known issues" section.

---
 doc/guides/prog_guide/env_abstraction_layer.rst |  2 +-
 drivers/mempool/ring/rte_mempool_ring.c         | 58 +++++++++++++++++++++++--
 2 files changed, 56 insertions(+), 4 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +1 -1
index 9497b879c..b6ac236d6 100644
@@ -541,7 +541,7 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
-  Alternatively, x86_64 applications can use the non-blocking stack mempool handler. When considering this handler, note that:
+  Alternatively, x86_64 applications can use the non-blocking ring or stack mempool handlers. When considering one of them, note that:
 
   - it is limited to the x86_64 platform, because it uses an instruction (16-byte compare-and-swap) that is not available on other platforms.
   - it has worse average-case performance than the non-preemptive rte_ring, but software caching (e.g. the mempool cache) can mitigate this by reducing the number of handler operations.
drivers/mempool/ring/rte_mempool_ring.c +55 -3

84 lines not on this page · raw message

Signature · 1 line
2.13.6
#30 lore raw
In reply to Gage Eads
Quoted 49 lines · On Tue, 15 Jan 2019 17:52:22 -0600 Gage Eads <[email protected]> wrote:

For some users, the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

This patchset introduces a non-blocking ring, on top of which a mempool can run. Crucially, the non-blocking algorithm relies on a 128-bit compare-and-swap, so it is limited to x86_64 machines.

The ring uses more compare-and-swap atomic operations than the regular rte ring:
With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a non-blocking ring based mempool (a likely use case for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. For ease-of-use, existing ring enqueue/dequeue functions work with both "regular" and non-blocking rings.

This patchset also adds non-blocking versions of ring_autotest and ring_perf_autotest, and a non-blocking ring based mempool.

This patchset makes ABI and API changes; a deprecation notice will be posted in a separate commit.

This patchset depends on the non-blocking stack patchset[1].

[1] http://mails.dpdk.org/archives/dev/2019-January/123470.html

v2:
- Merge separate docs commit into patch #5
- Convert uintptr_t to size_t
- Add a compile-time check for the size of size_t
- Fix a space-after-typecast issue
- Fix an unnecessary-parentheses checkpatch warning
- Bump librte_ring's library version

Gage Eads (5):
ring: change head and tail to pointer-width size
ring: add a non-blocking implementation
test_ring: add non-blocking ring autotest
test_ring_perf: add non-blocking ring perf test mempool/ring: add non-blocking ring handlers

Quoted 13 lines
 doc/guides/prog_guide/env_abstraction_layer.rst |   2 +-
 drivers/mempool/ring/rte_mempool_ring.c         |  58 ++-
 lib/librte_eventdev/rte_event_ring.h            |   6 +-
 lib/librte_ring/Makefile                        |   2 +-
 lib/librte_ring/meson.build                     |   2 +-
 lib/librte_ring/rte_ring.c                      |  53 ++-
 lib/librte_ring/rte_ring.h                      | 564 ++++++++++++++++++++++--
 lib/librte_ring/rte_ring_generic.h              |  16 +-
 lib/librte_ring/rte_ring_version.map            |   7 +
 test/test/test_ring.c                           |  57 ++-
 test/test/test_ring_perf.c                      |  19 +-
 11 files changed, 699 insertions(+), 87 deletions(-)

Just bumping the version number is not enough. This looks like an ABI breakage for existing users.

author
#31 lore raw

v3 cover Add non-blocking ring

In reply to Gage Eads

For some users, the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

This patchset introduces a non-blocking ring, on top of which a mempool can run. Crucially, the non-blocking algorithm relies on a 128-bit compare-and-swap, so it is currently limited to x86_64 machines. This is also an experimental API, so RING_F_NB users must build with the ALLOW_EXPERIMENTAL_API flag.

The ring uses more compare-and-swap atomic operations than the regular rte ring:
With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a non-blocking ring based mempool (a likely use case for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. For ease-of-use, existing ring enqueue/dequeue functions work with both "regular" and non-blocking rings.

This patchset also adds non-blocking versions of ring_autotest and ring_perf_autotest, and a non-blocking ring based mempool.

This patchset makes one API change; a deprecation notice will be posted in a separate commit.

This patchset depends on the non-blocking stack patchset[1].

[1] http://mails.dpdk.org/archives/dev/2019-January/123653.html

v3:
- Avoid the ABI break by putting 64-bit head and tail values in the same cacheline as struct rte_ring's prod and cons members.
- Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of libraries.
- Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case someone tries to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
- Update the ring mempool to use experimental APIs
- Clarify that RINB_F_NB is only limited to x86_64 currently; ARMv8.1-A builds can eventually support it with the CASP instruction.

v2:
- Merge separate docs commit into patch #5
- Convert uintptr_t to size_t
- Add a compile-time check for the size of size_t
- Fix a space-after-typecast issue
- Fix an unnecessary-parentheses checkpatch warning
- Bump librte_ring's library version

Gage Eads (5):
ring: add 64-bit headtail structure
ring: add a non-blocking implementation
test_ring: add non-blocking ring autotest
test_ring_perf: add non-blocking ring perf test
mempool/ring: add non-blocking ring handlers

 doc/guides/prog_guide/env_abstraction_layer.rst |   2 +-
 drivers/mempool/ring/Makefile                   |   1 +
 drivers/mempool/ring/meson.build                |   2 +
 drivers/mempool/ring/rte_mempool_ring.c         |  58 ++-
 lib/librte_eventdev/rte_event_ring.h            |   2 +-
 lib/librte_ring/Makefile                        |   3 +-
 lib/librte_ring/rte_ring.c                      |  72 ++-
 lib/librte_ring/rte_ring.h                      | 574 ++++++++++++++++++++++--
 lib/librte_ring/rte_ring_generic_64.h           | 152 +++++++
 lib/librte_ring/rte_ring_version.map            |   7 +
 test/test/test_ring.c                           |  57 ++-
 test/test/test_ring_perf.c                      |  19 +-
 12 files changed, 874 insertions(+), 75 deletions(-)
 create mode 100644 lib/librte_ring/rte_ring_generic_64.h
Signature · 1 line
2.13.6
author
#32 lore raw

PATCH v3 1/5 ring: add 64-bit headtail structure

64-bit head and tail index widths greatly increases the time it takes for them to wrap-around (with current CPU speeds, it won't happen within the author's lifetime). This is important in avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming non-blocking ring implementation. Using a 64-bit index makes the possibility of this occurring effectively zero.

This commit places the new producer and consumer structures in the same location in struct rte_ring as their 32-bit counterparts. Since the 32-bit versions are padded out to a cache line, there is space for the new structure without affecting the layout of struct rte_ring. Thus, the ABI is preserved.

---
 lib/librte_eventdev/rte_event_ring.h  |   2 +-
 lib/librte_ring/Makefile              |   3 +-
 lib/librte_ring/rte_ring.h            |  24 +++++-
 lib/librte_ring/rte_ring_generic_64.h | 152 ++++++++++++++++++++++++++++++++++
 4 files changed, 176 insertions(+), 5 deletions(-)
 create mode 100644 lib/librte_ring/rte_ring_generic_64.h
lib/librte_eventdev/rte_event_ring.h +1 -1
index 827a3209e..5fcb2d5f7 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2016-2017 Intel Corporation
+ * Copyright(c) 2016-2019 Intel Corporation
  */
 
 /**
lib/librte_ring/Makefile +2 -1
index 21a36770d..18c48fbc8 100644
@@ -19,6 +19,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_RING) := rte_ring.c
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
 					rte_ring_generic.h \
-					rte_ring_c11_mem.h
+					rte_ring_c11_mem.h \
+					rte_ring_generic_64.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_ring/rte_ring.h +21 -3
index af5444a9f..b270a4746 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007-2009 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -70,6 +70,15 @@ struct rte_ring_headtail {
 	uint32_t single;         /**< True if single prod/cons */
 };
 
+/* 64-bit version of rte_ring_headtail, for use by rings that need to avoid
+ * head/tail wrap-around.
+ */
+struct rte_ring_headtail_64 {
+	volatile uint64_t head;  /**< Prod/consumer head. */
+	volatile uint64_t tail;  /**< Prod/consumer tail. */
+	uint32_t single;       /**< True if single prod/cons */
+};
+
 /**
  * An RTE ring structure.
  *
@@ -97,11 +106,19 @@ struct rte_ring {
 	char pad0 __rte_cache_aligned; /**< empty cache line */
 
 	/** Ring producer status. */
-	struct rte_ring_headtail prod __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail prod __rte_cache_aligned;
+		struct rte_ring_headtail_64 prod_64 __rte_cache_aligned;
+	};
 	char pad1 __rte_cache_aligned; /**< empty cache line */
 
 	/** Ring consumer status. */
-	struct rte_ring_headtail cons __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail cons __rte_cache_aligned;
+		struct rte_ring_headtail_64 cons_64 __rte_cache_aligned;
+	};
 	char pad2 __rte_cache_aligned; /**< empty cache line */
 };
 
@@ -312,6 +329,7 @@ void rte_ring_dump(FILE *f, const struct rte_ring *r);
 #else
 #include "rte_ring_generic.h"
 #endif
+#include "rte_ring_generic_64.h"
 
 /**
  * @internal Enqueue several objects on the ring
lib/librte_ring/rte_ring_generic_64.h new file +152 -0

152 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#33 lore raw

PATCH v3 2/5 ring: add a non-blocking implementation

In reply to Gage Eads

This commit adds support for non-blocking circular ring enqueue and dequeue functions. The ring uses a 128-bit compare-and-swap instruction, and thus is currently limited to x86_64.

The algorithm is based on the original rte ring (derived from FreeBSD's bufring.h) and inspired by Michael and Scott's non-blocking concurrent queue. Importantly, it adds a modification counter to each ring entry to ensure only one thread can write to an unused entry.

-----
Algorithm:

Multi-producer non-blocking enqueue:
1. Move the producer head index 'n' locations forward, effectively reserving 'n' locations.
2. For each pointer:
a. Read the producer tail index, then ring[tail]. If ring[tail]'s
modification counter isn't 'tail', retry.
b. Construct the new entry: {pointer, tail + ring size}
c. Compare-and-swap the old entry with the new. If unsuccessful, the
next loop iteration will try to enqueue this pointer again.
d. Compare-and-swap the tail index with 'tail + 1', whether or not step 2c
succeeded. This guarantees threads can make forward progress.

Multi-consumer non-blocking dequeue:
1. Move the consumer head index 'n' locations forward, effectively reserving 'n' pointers to be dequeued.
2. Copy 'n' pointers into the caller's object table (ignoring the modification counter), starting from ring[tail], then compare-and-swap the tail index with 'tail + n'. If unsuccessful, repeat step 2.

-----
Discussion:

There are two cases where the ABA problem is mitigated:
1. Enqueueing a pointer to the ring: without a modification counter tied to the tail index, the index could become stale by the time the enqueue happens, causing it to overwrite valid data. Tying the counter to the tail index gives us an expected value (as opposed to, say, a monotonically incrementing counter).

Since the counter will eventually wrap, there is potential for the ABA problem. However, using a 64-bit counter makes this likelihood effectively zero.

2. Updating a tail index: the ABA problem can occur if the thread is preempted and the tail index wraps around. However, using 64-bit indexes makes this likelihood effectively zero.

With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a non-blocking ring based mempool (a likely use case for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. Because the ring's memsize is now a function of its flags (the non-blocking ring requires 128b for each entry), this commit adds a new argument ('flags') to rte_ring_get_memsize(). An API deprecation notice will be sent in a separate commit.

For ease-of-use, existing ring enqueue and dequeue functions work on both regular and non-blocking rings. This introduces an additional branch in the datapath, but this should be a highly predictable branch. ring_perf_autotest shows a negligible performance impact; it's hard to distinguish a real difference versus system noise.

| ring_perf_autotest cycles with branch -
Test | ring_perf_autotest cycles without
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | -4.00
SP/SC burst enq/dequeue (size 8)  | 0.00
MP/MC burst enq/dequeue (size 8)  | 0.00
SP/SC burst enq/dequeue (size 32) | 0.00
MP/MC burst enq/dequeue (size 32) | 0.00
SC empty dequeue                  | 1.00
MC empty dequeue                  | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | 0.49
MP/MC bulk enq/dequeue (size 8)   | 0.08
SP/SC bulk enq/dequeue (size 32)  | 0.07
MP/MC bulk enq/dequeue (size 32)  | 0.09

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | 0.19
MP/MC bulk enq/dequeue (size 8)   | -0.37
SP/SC bulk enq/dequeue (size 32)  | 0.09
MP/MC bulk enq/dequeue (size 32)  | -0.05

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | -1.96
MP/MC bulk enq/dequeue (size 8)   | 0.88
SP/SC bulk enq/dequeue (size 32)  | 0.10
MP/MC bulk enq/dequeue (size 32)  | 0.46

Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. Each test run three times and the results averaged.

---
 lib/librte_ring/rte_ring.c           |  72 ++++-
 lib/librte_ring/rte_ring.h           | 550 +++++++++++++++++++++++++++++++++--
 lib/librte_ring/rte_ring_version.map |   7 +
 3 files changed, 587 insertions(+), 42 deletions(-)
lib/librte_ring/rte_ring.c +60 -12
index d215acecc..f3378dccd 100644
@@ -45,9 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
 
 /* return the size of memory occupied by a ring */
 ssize_t
-rte_ring_get_memsize(unsigned count)
+rte_ring_get_memsize_v1905(unsigned int count, unsigned int flags)
 {
-	ssize_t sz;
+	ssize_t sz, elt_sz;
 
 	/* count must be a power of 2 */
 	if ((!POWEROF2(count)) || (count > RTE_RING_SZ_MASK )) {
@@ -57,10 +57,23 @@ rte_ring_get_memsize(unsigned count)
 		return -EINVAL;
 	}
 
-	sz = sizeof(struct rte_ring) + count * sizeof(void *);
+	elt_sz = (flags & RING_F_NB) ? 2 * sizeof(void *) : sizeof(void *);
+
+	sz = sizeof(struct rte_ring) + count * elt_sz;
 	sz = RTE_ALIGN(sz, RTE_CACHE_LINE_SIZE);
 	return sz;
 }
+BIND_DEFAULT_SYMBOL(rte_ring_get_memsize, _v1905, 19.05);
+MAP_STATIC_SYMBOL(ssize_t rte_ring_get_memsize(unsigned int count,
+					       unsigned int flags),
+		  rte_ring_get_memsize_v1905);
+
+ssize_t
+rte_ring_get_memsize_v20(unsigned int count)
+{
+	return rte_ring_get_memsize_v1905(count, 0);
+}
+VERSION_SYMBOL(rte_ring_get_memsize, _v20, 2.0);
 
 int
 rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
@@ -82,8 +95,6 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 	if (ret < 0 || ret >= (int)sizeof(r->name))
 		return -ENAMETOOLONG;
 	r->flags = flags;
-	r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
-	r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
 
 	if (flags & RING_F_EXACT_SZ) {
 		r->size = rte_align32pow2(count + 1);
@@ -100,8 +111,30 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 		r->mask = count - 1;
 		r->capacity = r->mask;
 	}
-	r->prod.head = r->cons.head = 0;
-	r->prod.tail = r->cons.tail = 0;
+
+	if (flags & RING_F_NB) {
+		uint64_t i;
+
+		r->prod_64.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
+		r->cons_64.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+		r->prod_64.head = r->cons_64.head = 0;
+		r->prod_64.tail = r->cons_64.tail = 0;
+
+		for (i = 0; i < r->size; i++) {
+			struct nb_ring_entry *ring_ptr, *base;
+
+			base = ((struct nb_ring_entry *)&r[1]);
+
+			ring_ptr = &base[i & r->mask];
+
+			ring_ptr->cnt = i;
+		}
+	} else {
+		r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
+		r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+		r->prod.head = r->cons.head = 0;
+		r->prod.tail = r->cons.tail = 0;
+	}
 
 	return 0;
 }
@@ -123,11 +156,19 @@ rte_ring_create(const char *name, unsigned count, int socket_id,
 
 	ring_list = RTE_TAILQ_CAST(rte_ring_tailq.head, rte_ring_list);
 
+#if !defined(RTE_ARCH_X86_64)
+	if (flags & RING_F_NB) {
+		printf("RING_F_NB is only supported on x86-64 platforms\n");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+#endif
+
 	/* for an exact size ring, round up from count to a power of two */
 	if (flags & RING_F_EXACT_SZ)
 		count = rte_align32pow2(count + 1);
 
-	ring_size = rte_ring_get_memsize(count);
+	ring_size = rte_ring_get_memsize(count, flags);
 	if (ring_size < 0) {
 		rte_errno = ring_size;
 		return NULL;
@@ -227,10 +268,17 @@ rte_ring_dump(FILE *f, const struct rte_ring *r)
 	fprintf(f, "  flags=%x\n", r->flags);
 	fprintf(f, "  size=%"PRIu32"\n", r->size);
 	fprintf(f, "  capacity=%"PRIu32"\n", r->capacity);
-	fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
-	fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
-	fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
-	fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	if (r->flags & RING_F_NB) {
+		fprintf(f, "  ct=%"PRIu64"\n", r->cons_64.tail);
+		fprintf(f, "  ch=%"PRIu64"\n", r->cons_64.head);
+		fprintf(f, "  pt=%"PRIu64"\n", r->prod_64.tail);
+		fprintf(f, "  ph=%"PRIu64"\n", r->prod_64.head);
+	} else {
+		fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
+		fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
+		fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
+		fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	}
 	fprintf(f, "  used=%u\n", rte_ring_count(r));
 	fprintf(f, "  avail=%u\n", rte_ring_free_count(r));
 }
lib/librte_ring/rte_ring.h +520 -30

679 lines not on this page · raw message

lib/librte_ring/rte_ring_version.map +7 -0
index d935efd0d..8969467af 100644
@@ -17,3 +17,10 @@ DPDK_2.2 {
 	rte_ring_free;
 
 } DPDK_2.0;
+
+DPDK_19.05 {
+	global:
+
+	rte_ring_get_memsize;
+
+} DPDK_2.2;
Signature · 1 line
2.13.6
#34 lore raw

Re: [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 474 lines · Gage Eads

This commit adds support for non-blocking circular ring enqueue and dequeue
functions. The ring uses a 128-bit compare-and-swap instruction, and thus
is currently limited to x86_64.

The algorithm is based on the original rte ring (derived from FreeBSD's
bufring.h) and inspired by Michael and Scott's non-blocking concurrent
queue. Importantly, it adds a modification counter to each ring entry to
ensure only one thread can write to an unused entry.
-----
Algorithm:

Multi-producer non-blocking enqueue:
1. Move the producer head index 'n' locations forward, effectively reserving 'n' locations.
2. For each pointer:
a. Read the producer tail index, then ring[tail]. If ring[tail]'s
modification counter isn't 'tail', retry.
b. Construct the new entry: {pointer, tail + ring size} c. Compare-and-swap the old entry with the new. If unsuccessful, the
next loop iteration will try to enqueue this pointer again.
d. Compare-and-swap the tail index with 'tail + 1', whether or not
step 2c
succeeded. This guarantees threads can make forward progress.

Multi-consumer non-blocking dequeue:
1. Move the consumer head index 'n' locations forward, effectively reserving 'n' pointers to be dequeued.
2. Copy 'n' pointers into the caller's object table (ignoring the modification counter), starting from ring[tail], then compare-and-
swap
the tail index with 'tail + n'. If unsuccessful, repeat step 2.

-----
Discussion:

There are two cases where the ABA problem is mitigated:
1. Enqueueing a pointer to the ring: without a modification counter tied to the tail index, the index could become stale by the time
the
enqueue happens, causing it to overwrite valid data. Tying the counter to the tail index gives us an expected value (as opposed
to,
say, a monotonically incrementing counter).

Since the counter will eventually wrap, there is potential for the
ABA
problem. However, using a 64-bit counter makes this likelihood effectively zero.

2. Updating a tail index: the ABA problem can occur if the thread is preempted and the tail index wraps around. However, using 64-bit
indexes
makes this likelihood effectively zero.

With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations
and a dequeue of n pointers uses 2. This algorithm has worse average- case
performance than the regular rte ring (particularly a highly- contended ring
with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based
algorithms. For example, a non-blocking ring based mempool (a
likely use
case for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. Because the
ring's memsize is now a function of its flags (the non-blocking ring requires 128b for each entry), this commit adds a new argument ('flags') to
rte_ring_get_memsize(). An API deprecation notice will be sent in a separate commit.

For ease-of-use, existing ring enqueue and dequeue functions work on both
regular and non-blocking rings. This introduces an additional branch in
the datapath, but this should be a highly predictable branch. ring_perf_autotest shows a negligible performance impact; it's hard to
distinguish a real difference versus system noise.

| ring_perf_autotest cycles with
branch -
Test | ring_perf_autotest cycles
without
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | -4.00
SP/SC burst enq/dequeue (size 8)  | 0.00
MP/MC burst enq/dequeue (size 8)  | 0.00
SP/SC burst enq/dequeue (size 32) | 0.00
MP/MC burst enq/dequeue (size 32) | 0.00
SC empty dequeue                  | 1.00
MC empty dequeue                  | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | 0.49
MP/MC bulk enq/dequeue (size 8)   | 0.08
SP/SC bulk enq/dequeue (size 32)  | 0.07
MP/MC bulk enq/dequeue (size 32)  | 0.09

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | 0.19
MP/MC bulk enq/dequeue (size 8)   | -0.37
SP/SC bulk enq/dequeue (size 32)  | 0.09
MP/MC bulk enq/dequeue (size 32)  | -0.05

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | -1.96
MP/MC bulk enq/dequeue (size 8)   | 0.88
SP/SC bulk enq/dequeue (size 32)  | 0.10
MP/MC bulk enq/dequeue (size 32)  | 0.46

Test setup: x86_64 build with default config, dual-socket Xeon E5- 2699 v4,
running on isolcpus cores with a tickless scheduler. Each test run three
times and the results averaged.

---
 lib/librte_ring/rte_ring.c           |  72 ++++-
 lib/librte_ring/rte_ring.h           | 550
+++++++++++++++++++++++++++++++++--
 lib/librte_ring/rte_ring_version.map |   7 +
 3 files changed, 587 insertions(+), 42 deletions(-)
lib/librte_ring/rte_ring.c +16 -3
index d215acecc..f3378dccd 100644
@@ -45,9 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
 
 /* return the size of memory occupied by a ring */
 ssize_t
-rte_ring_get_memsize(unsigned count)
+rte_ring_get_memsize_v1905(unsigned int count, unsigned int flags)
 {
-ssize_t sz;
+ssize_t sz, elt_sz;
 
 /* count must be a power of 2 */
 if ((!POWEROF2(count)) || (count > RTE_RING_SZ_MASK )) {
@@ -57,10 +57,23 @@ rte_ring_get_memsize(unsigned count)
 return -EINVAL;
 }
 
-sz = sizeof(struct rte_ring) + count * sizeof(void *);
+elt_sz = (flags & RING_F_NB) ? 2 * sizeof(void *) :
 sizeof(void *);
+
+sz = sizeof(struct rte_ring) + count * elt_sz;
 sz = RTE_ALIGN(sz, RTE_CACHE_LINE_SIZE);
 return sz;
 }
+BIND_DEFAULT_SYMBOL(rte_ring_get_memsize, _v1905, 19.05);
+MAP_STATIC_SYMBOL(ssize_t rte_ring_get_memsize(unsigned int count,
+       unsigned int flags),
+  rte_ring_get_memsize_v1905);
+
+ssize_t
+rte_ring_get_memsize_v20(unsigned int count)
+{
+return rte_ring_get_memsize_v1905(count, 0);
+}
+VERSION_SYMBOL(rte_ring_get_memsize, _v20, 2.0);
 
 int

rte_ring_init(struct rte_ring *r, const char *name, unsigned count,

lib/librte_ring/rte_ring.c +0 -2
@@ -82,8 +95,6 @@ rte_ring_init(struct rte_ring *r, const char *name,
 unsigned count,
 if (ret < 0 || ret >= (int)sizeof(r->name))
 return -ENAMETOOLONG;
 r->flags = flags;
-r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP :
 __IS_MP;
-r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC :
 __IS_MC;

if (flags & RING_F_EXACT_SZ) {
r->size = rte_align32pow2(count + 1);

lib/librte_ring/rte_ring.c +20 -2
@@ -100,8 +111,30 @@ rte_ring_init(struct rte_ring *r, const char
 *name, unsigned count,
 r->mask = count - 1;
 r->capacity = r->mask;
 }
-r->prod.head = r->cons.head = 0;
-r->prod.tail = r->cons.tail = 0;
+
+if (flags & RING_F_NB) {
+uint64_t i;
+
+r->prod_64.single = (flags & RING_F_SP_ENQ) ?
 __IS_SP : __IS_MP;
+r->cons_64.single = (flags & RING_F_SC_DEQ) ?
 __IS_SC : __IS_MC;
+r->prod_64.head = r->cons_64.head = 0;
+r->prod_64.tail = r->cons_64.tail = 0;
+
+for (i = 0; i < r->size; i++) {
+struct nb_ring_entry *ring_ptr, *base;
+
+base = ((struct nb_ring_entry *)&r[1]);
+
+ring_ptr = &base[i & r->mask];
+
+ring_ptr->cnt = i;
+}
+} else {
+r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP :

__IS_MP;
+r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC :
__IS_MC;

lib/librte_ring/rte_ring.c +11 -0
+r->prod.head = r->cons.head = 0;
+r->prod.tail = r->cons.tail = 0;
+}
 
 return 0;
 }
@@ -123,11 +156,19 @@ rte_ring_create(const char *name, unsigned
 count, int socket_id,
 
 ring_list = RTE_TAILQ_CAST(rte_ring_tailq.head,
 rte_ring_list);
 
+#if !defined(RTE_ARCH_X86_64)
+if (flags & RING_F_NB) {
+printf("RING_F_NB is only supported on x86-64
 platforms\n");
+rte_errno = EINVAL;
+return NULL;
+}
+#endif
+
 /* for an exact size ring, round up from count to a power of

two */
if (flags & RING_F_EXACT_SZ)
count = rte_align32pow2(count + 1);

lib/librte_ring/rte_ring.c +12 -5
-ring_size = rte_ring_get_memsize(count);
+ring_size = rte_ring_get_memsize(count, flags);
 if (ring_size < 0) {
 rte_errno = ring_size;
 return NULL;
@@ -227,10 +268,17 @@ rte_ring_dump(FILE *f, const struct rte_ring
 *r)
 fprintf(f, "  flags=%x\n", r->flags);
 fprintf(f, "  size=%"PRIu32"\n", r->size);
 fprintf(f, "  capacity=%"PRIu32"\n", r->capacity);
-fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
-fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
-fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
-fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+if (r->flags & RING_F_NB) {
+fprintf(f, "  ct=%"PRIu64"\n", r->cons_64.tail);
+fprintf(f, "  ch=%"PRIu64"\n", r->cons_64.head);
+fprintf(f, "  pt=%"PRIu64"\n", r->prod_64.tail);
+fprintf(f, "  ph=%"PRIu64"\n", r->prod_64.head);
+} else {
+fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
+fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
+fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
+fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+}
 fprintf(f, "  used=%u\n", rte_ring_count(r));
 fprintf(f, "  avail=%u\n", rte_ring_free_count(r));

}

lib/librte_ring/rte_ring.h +2 -0
index b270a4746..08c9de6a6 100644
@@ -134,6 +134,18 @@ struct rte_ring {
  */
 #define RING_F_EXACT_SZ 0x0004
 #define RTE_RING_SZ_MASK  (0x7fffffffU) /**< Ring size mask */
+/**
+ * The ring uses non-blocking enqueue and dequeue functions. These

functions
+ * do not have the "non-preemptive" constraint of a regular rte ring, and thus
+ * are suited for applications using preemptible pthreads. However, the
+ * non-blocking functions have worse average-case performance than their
+ * regular rte ring counterparts. When used as the handler for a mempool,
+ * per-thread caching can mitigate the performance difference by reducing the
+ * number (and contention) of ring accesses.
+ *
+ * This flag is only supported on x86_64 platforms.
+ */
+#define RING_F_NB 0x0008

/* @internal defines for passing to the enqueue dequeue worker
functions */
#define __IS_SP 1

lib/librte_ring/rte_ring.h +5 -1
@@ -151,11 +163,15 @@ struct rte_ring {
  *
  * @param count
  *   The number of elements in the ring (must be a power of 2).
+ * @param flags
+ *   The flags the ring will be created with.
  * @return
  *   - The memory size needed for the ring on success.
  *   - -EINVAL if count is not a power of 2.
  */
-ssize_t rte_ring_get_memsize(unsigned count);
+ssize_t rte_ring_get_memsize(unsigned int count, unsigned int
 flags);
+ssize_t rte_ring_get_memsize_v20(unsigned int count);
+ssize_t rte_ring_get_memsize_v1905(unsigned int count, unsigned int
 flags);
 

/**
* Initialize a ring structure.

lib/librte_ring/rte_ring.h +0 -0
@@ -188,6 +204,10 @@ ssize_t rte_ring_get_memsize(unsigned count);
  *    - RING_F_SC_DEQ: If this flag is set, the default behavior

when
* using ``rte_ring_dequeue()`` or ``rte_ring_dequeue_bulk()``
* is "single-consumer". Otherwise, it is "multi-consumers".
+ * - RING_F_EXACT_SZ: If this flag is set, count can be a non-
power-of-2
+ * number, but up to half the ring space may be wasted.
+ * - RING_F_NB: (x86_64 only) If this flag is set, the ring uses
+ * non-blocking variants of the dequeue and enqueue functions.
* @return
* 0 on success, or a negative value on error.
*/

lib/librte_ring/rte_ring.h +0 -0
@@ -223,12 +243,17 @@ int rte_ring_init(struct rte_ring *r, const
 char *name, unsigned count,
  *    - RING_F_SC_DEQ: If this flag is set, the default behavior

when
* using ``rte_ring_dequeue()`` or ``rte_ring_dequeue_bulk()``
* is "single-consumer". Otherwise, it is "multi-consumers".
+ * - RING_F_EXACT_SZ: If this flag is set, count can be a non-
power-of-2
+ * number, but up to half the ring space may be wasted.
+ * - RING_F_NB: (x86_64 only) If this flag is set, the ring uses
+ * non-blocking variants of the dequeue and enqueue functions.
* @return
* On success, the pointer to the new allocated ring. NULL on
error with
* rte_errno set appropriately. Possible errno values include:
* - E_RTE_NO_CONFIG - function could not get pointer to
rte_config structure
* - E_RTE_SECONDARY - function was called from a secondary
process instance
- * - EINVAL - count provided is not a power of 2
+ * - EINVAL - count provided is not a power of 2, or RING_F_NB is
used on an
+ * unsupported platform
* - ENOSPC - the maximum number of memzones has already been
allocated
* - EEXIST - a memzone with the same name already exists
* - ENOMEM - no appropriate memory area found in which to create
memzone

lib/librte_ring/rte_ring.h +2 -0
@@ -284,6 +309,50 @@ void rte_ring_dump(FILE *f, const struct
 rte_ring *r);
 } \
 } while (0)
 
+/* The actual enqueue of pointers on the ring.
+ * Used only by the single-producer non-blocking enqueue function,

but
+ * out-lined here for code readability.
+ */
+#define ENQUEUE_PTRS_NB(r, ring_start, prod_head, obj_table, n) do { \

lib/librte_ring/rte_ring.h +5 -0
+unsigned int i; \
+const uint32_t size = (r)->size; \
+size_t idx = prod_head & (r)->mask; \
+size_t new_cnt = prod_head + size; \
+struct nb_ring_entry *ring = (struct nb_ring_entry

*)ring_start; \

lib/librte_ring/rte_ring.h +2 -0
+if (likely(idx + n < size)) { \
+for (i = 0; i < (n & ((~(unsigned)0x3))); i += 4,

idx += 4) { \

lib/librte_ring/rte_ring.h +13 -0
+ring[idx].ptr = obj_table[i]; \
+ring[idx].cnt = new_cnt + i;  \
+ring[idx + 1].ptr = obj_table[i + 1]; \
+ring[idx + 1].cnt = new_cnt + i + 1;  \
+ring[idx + 2].ptr = obj_table[i + 2]; \
+ring[idx + 2].cnt = new_cnt + i + 2;  \
+ring[idx + 3].ptr = obj_table[i + 3]; \
+ring[idx + 3].cnt = new_cnt + i + 3;  \
+} \
+switch (n & 0x3) { \
+case 3: \
+ring[idx].cnt = new_cnt + i; \
+ring[idx++].ptr = obj_table[i++]; /*

fallthrough */ \

lib/librte_ring/rte_ring.h +3 -0
+case 2: \
+ring[idx].cnt = new_cnt + i; \
+ring[idx++].ptr = obj_table[i++]; /*

fallthrough */ \

lib/librte_ring/rte_ring.h +36 -0
+case 1: \
+ring[idx].cnt = new_cnt + i; \
+ring[idx++].ptr = obj_table[i++]; \
+} \
+} else { \
+for (i = 0; idx < size; i++, idx++) { \
+ring[idx].cnt = new_cnt + i;  \
+ring[idx].ptr = obj_table[i]; \
+} \
+for (idx = 0; i < n; i++, idx++) {    \
+ring[idx].cnt = new_cnt + i;  \
+ring[idx].ptr = obj_table[i]; \
+} \
+} \
+} while (0)
+
 /* the actual copy of pointers on the ring to obj_table.
  * Placed here since identical code needed in both
  * single and multi consumer dequeue functions */
@@ -315,6 +384,39 @@ void rte_ring_dump(FILE *f, const struct
 rte_ring *r);
 } \
 } while (0)
 
+/* The actual copy of pointers on the ring to obj_table.
+ * Placed here since identical code needed in both
+ * single and multi consumer non-blocking dequeue functions.
+ */
+#define DEQUEUE_PTRS_NB(r, ring_start, cons_head, obj_table, n) do {
 \
+unsigned int i; \
+size_t idx = cons_head & (r)->mask; \
+const uint32_t size = (r)->size; \
+struct nb_ring_entry *ring = (struct nb_ring_entry
 *)ring_start; \
+if (likely(idx + n < size)) { \
+for (i = 0; i < (n & (~(unsigned)0x3)); i += 4, idx
+= 4) {\
+obj_table[i] = ring[idx].ptr; \
+obj_table[i + 1] = ring[idx + 1].ptr; \
+obj_table[i + 2] = ring[idx + 2].ptr; \
+obj_table[i + 3] = ring[idx + 3].ptr; \
+} \
+switch (n & 0x3) { \
+case 3: \
+obj_table[i++] = ring[idx++].ptr; /*

fallthrough */ \

lib/librte_ring/rte_ring.h +2 -0
+case 2: \
+obj_table[i++] = ring[idx++].ptr; /*

fallthrough */ \

lib/librte_ring/rte_ring.h +12 -0
+case 1: \
+obj_table[i++] = ring[idx++].ptr; \
+} \
+} else { \
+for (i = 0; idx < size; i++, idx++) \
+obj_table[i] = ring[idx].ptr; \
+for (idx = 0; i < n; i++, idx++) \
+obj_table[i] = ring[idx].ptr; \
+} \
+} while (0)
+
+
 /* Between load and load. there might be cpu reorder in weak model
  * (powerpc/arm).
  * There are 2 choices for the users

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct · line 440

Quoted 8 lines
lib/librte_ring/rte_ring.h +4 -0
@@ -331,6 +433,319 @@ void rte_ring_dump(FILE *f, const struct
 rte_ring *r);
 #endif
 #include "rte_ring_generic_64.h"
 
+/* @internal 128-bit structure used by the non-blocking ring */
+struct nb_ring_entry {
+void *ptr; /**< Data pointer */
+uint64_t cnt; /**< Modification counter */

Why not make 'cnt' uintptr_t? This way 32-bit architectures will also be supported.

Quoted 45 lines
lib/librte_ring/rte_ring.h +3 -0
+};
+
+/* The non-blocking ring algorithm is based on the original rte ring

(derived
+ * from FreeBSD's bufring.h) and inspired by Michael and Scott's non-blocking
+ * concurrent queue.
+ */

lib/librte_ring/rte_ring.h +4 -0
+
+/**
+ * @internal
+ *   Enqueue several objects on the non-blocking ring (single-

producer only)
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items to the
ring
+ * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible to
the ring
+ * @param free_space
+ * returns the amount of space after the enqueue operation has
finished
+ * @return
+ * Actual number of objects enqueued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */

lib/librte_ring/rte_ring.h +2 -0
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue_sp(struct rte_ring *r, void * const

*obj_table,

lib/librte_ring/rte_ring.h +9 -0
+    unsigned int n,
+    enum rte_ring_queue_behavior behavior,
+    unsigned int *free_space)
+{
+uint32_t free_entries;
+size_t head, next;
+
+n = __rte_ring_move_prod_head_64(r, 1, n, behavior,
+ &head, &next,

&free_entries);

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 482

Quoted 6 lines
lib/librte_ring/rte_ring.h +6 -0
+if (n == 0)
+goto end;
+
+ENQUEUE_PTRS_NB(r, &r[1], head, obj_table, n);
+
+r->prod_64.tail += n;

Don't we need release order (or smp_wmb) between writing of the ring pointers and the update of tail? By updating the tail pointer, we are synchronising with a consumer.

Quoted 73 lines
lib/librte_ring/rte_ring.h +10 -0
+
+end:
+if (free_space != NULL)
+*free_space = free_entries - n;
+return n;
+}
+
+/**
+ * @internal
+ *   Enqueue several objects on the non-blocking ring (multi-

producer safe)
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items to the
ring
+ * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible to
the ring
+ * @param free_space
+ * returns the amount of space after the enqueue operation has
finished
+ * @return
+ * Actual number of objects enqueued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */

lib/librte_ring/rte_ring.h +2 -0
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue_mp(struct rte_ring *r, void * const

*obj_table,

lib/librte_ring/rte_ring.h +24 -0
+    unsigned int n,
+    enum rte_ring_queue_behavior behavior,
+    unsigned int *free_space)
+{
+#if !defined(RTE_ARCH_X86_64) || !defined(ALLOW_EXPERIMENTAL_API)
+RTE_SET_USED(r);
+RTE_SET_USED(obj_table);
+RTE_SET_USED(n);
+RTE_SET_USED(behavior);
+RTE_SET_USED(free_space);
+#ifndef ALLOW_EXPERIMENTAL_API
+printf("[%s()] RING_F_NB requires an experimental API."
+       " Recompile with ALLOW_EXPERIMENTAL_API to use it.\n"
+       , __func__);
+#endif
+return 0;
+#endif
+#if defined(RTE_ARCH_X86_64) && defined(ALLOW_EXPERIMENTAL_API)
+size_t head, next, tail;
+uint32_t free_entries;
+unsigned int i;
+
+n = __rte_ring_move_prod_head_64(r, 0, n, behavior,
+ &head, &next,

&free_entries);

lib/librte_ring/rte_ring.h +4 -0
+if (n == 0)
+goto end;
+
+for (i = 0; i < n; /* i incremented if enqueue succeeds */)

{

lib/librte_ring/rte_ring.h +4 -0
+struct nb_ring_entry old_value, new_value;
+struct nb_ring_entry *ring_ptr;
+
+/* Enqueue to the tail entry. If another thread wins

the race,
+ * retry with the new tail.
+ */

lib/librte_ring/rte_ring.h +3 -0
+tail = r->prod_64.tail;
+
+ring_ptr = &((struct nb_ring_entry *)&r[1])[tail &
Quoted 1 line

r->mask];

This is a very ugly cast. Also I think it is unnecessary. What's preventing this from being written without a cast? Perhaps the ring array needs to be a union of "void *" and struct nb_ring_entry?

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 555

Quoted 9 lines
lib/librte_ring/rte_ring.h +4 -0
+
+old_value = *ring_ptr;
+
+/* If the tail entry's modification counter doesn't

match the
+ * producer tail index, it's already been updated.
+ */

lib/librte_ring/rte_ring.h +2 -0
+if (old_value.cnt != tail)
+continue;

Continue restarts the loop at the condition test in the for statement, 'i' and 'n' are unchanged. Then we re-read 'prod_64.tail' and 'ring[tail]'. If some other thread never updates 'prod_64.tail', the test here (ring[tail].cnt != tail) will still be false and we will spin forever.
Waiting for other threads <=> blocking behaviour so this is not a non- blocking design.

Quoted 17 lines
lib/librte_ring/rte_ring.h +2 -0
+
+/* Prepare the new entry. The cnt field mitigates

the ABA
+ * problem on the ring write.
+ */

lib/librte_ring/rte_ring.h +4 -0
+new_value.ptr = obj_table[i];
+new_value.cnt = tail + r->size;
+
+if (rte_atomic128_cmpset((volatile rte_int128_t

*)ring_ptr,
+ (rte_int128_t *)&old_value,
+ (rte_int128_t
*)&new_value))

lib/librte_ring/rte_ring.h +3 -0
+i++;
+
+/* Every thread attempts the cmpset, so they don't

have to wait

Quoted 8 lines

+ * for the thread that successfully enqueued to the ring.
+ * Using a 64-bit tail mitigates the ABA problem here.
+ *
+ * Built-in used to handle variable-sized tail index.
+ */

But prod_64.tail is 64 bits so not really variable size?

Quoted 2 lines

+__sync_bool_compare_and_swap(&r->prod_64.tail, tail, tail + 1);

What memory order is required here? Why not use __atomic_compare_exchange() with explicit memory order parameters?

Quoted 82 lines
lib/librte_ring/rte_ring.h +20 -0
+}
+
+end:
+if (free_space != NULL)
+*free_space = free_entries - n;
+return n;
+#endif
+}
+
+/**
+ * @internal Enqueue several objects on the non-blocking ring
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items to the

ring
+ * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible to
the ring
+ * @param is_sp
+ * Indicates whether to use single producer or multi-producer head
update
+ * @param free_space
+ * returns the amount of space after the enqueue operation has
finished
+ * @return
+ * Actual number of objects enqueued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */

lib/librte_ring/rte_ring.h +2 -0
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue(struct rte_ring *r, void * const

*obj_table,
+ unsigned int n, enum
rte_ring_queue_behavior behavior,
+ unsigned int is_sp, unsigned int
*free_space)

lib/librte_ring/rte_ring.h +4 -0
+{
+if (is_sp)
+return __rte_ring_do_nb_enqueue_sp(r, obj_table, n,
+   behavior,

free_space);

lib/librte_ring/rte_ring.h +3 -0
+else
+return __rte_ring_do_nb_enqueue_mp(r, obj_table, n,
+   behavior,

free_space);

lib/librte_ring/rte_ring.h +5 -0
+}
+
+/**
+ * @internal
+ *   Dequeue several objects from the non-blocking ring (single-

consumer only)
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to pull from the ring.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from
the ring
+ * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
the ring
+ * @param available
+ * returns the number of remaining ring entries after the dequeue
has finished
+ * @return
+ * - Actual number of objects dequeued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n
only.
+ */

lib/librte_ring/rte_ring.h +7 -0
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_dequeue_sc(struct rte_ring *r, void **obj_table,
+    unsigned int n,
+    enum rte_ring_queue_behavior behavior,
+    unsigned int *available)
+{
+size_t head, next;

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 652

Quoted 10 lines
lib/librte_ring/rte_ring.h +10 -0
+uint32_t entries;
+
+n = __rte_ring_move_cons_head_64(r, 1, n, behavior,
+ &head, &next, &entries);
+if (n == 0)
+goto end;
+
+DEQUEUE_PTRS_NB(r, &r[1], head, obj_table, n);
+
+r->cons_64.tail += n;

Memory ordering? Consumer synchronises with producer.

Quoted 395 lines
lib/librte_ring/rte_ring.h +10 -0
+
+end:
+if (available != NULL)
+*available = entries - n;
+return n;
+}
+
+/**
+ * @internal
+ *   Dequeue several objects from the non-blocking ring (multi-

consumer safe)
+ *
+ * @param r
+ * A pointer to the ring structure.
+ * @param obj_table
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to pull from the ring.
+ * @param behavior
+ * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items from
the ring
+ * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
the ring
+ * @param available
+ * returns the number of remaining ring entries after the dequeue
has finished
+ * @return
+ * - Actual number of objects dequeued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n
only.
+ */

lib/librte_ring/rte_ring.h +18 -0
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_dequeue_mc(struct rte_ring *r, void **obj_table,
+    unsigned int n,
+    enum rte_ring_queue_behavior behavior,
+    unsigned int *available)
+{
+size_t head, next;
+uint32_t entries;
+
+n = __rte_ring_move_cons_head_64(r, 0, n, behavior,
+ &head, &next, &entries);
+if (n == 0)
+goto end;
+
+while (1) {
+size_t tail = r->cons_64.tail;
+
+/* Dequeue from the cons tail onwards. If multiple

threads read
+ * the same pointers, the thread that successfully performs the
+ * CAS will keep them and the other(s) will retry.
+ */

lib/librte_ring/rte_ring.h +5 -0
+DEQUEUE_PTRS_NB(r, &r[1], tail, obj_table, n);
+
+next = tail + n;
+
+/* Built-in used to handle variable-sized tail

index. */
+if (__sync_bool_compare_and_swap(&r->cons_64.tail, tail, next))
+/* There is potential for the ABA problem here, but
+ * that is mitigated by the large (64-bit) tail.
+ */

lib/librte_ring/rte_ring.h +20 -0
+break;
+}
+
+end:
+if (available != NULL)
+*available = entries - n;
+return n;
+}
+
+/**
+ * @internal Dequeue several objects from the non-blocking ring
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from

the ring
+ * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
the ring
+ * @param available
+ * returns the number of remaining ring entries after the dequeue
has finished
+ * @return
+ * - Actual number of objects dequeued.
+ * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n
only.
+ */

lib/librte_ring/rte_ring.h +3 -0
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_dequeue(struct rte_ring *r, void **obj_table,
+ unsigned int n, enum rte_ring_queue_behavior

behavior,
+ unsigned int is_sc, unsigned int *available)

lib/librte_ring/rte_ring.h +4 -0
+{
+if (is_sc)
+return __rte_ring_do_nb_dequeue_sc(r, obj_table, n,
+   behavior,

available);

lib/librte_ring/rte_ring.h +3 -0
+else
+return __rte_ring_do_nb_dequeue_mc(r, obj_table, n,
+   behavior,

available);

lib/librte_ring/rte_ring.h +9 -2
+}
+
 /**
  * @internal Enqueue several objects on the ring
  *
@@ -438,8 +853,14 @@ static __rte_always_inline unsigned int
 rte_ring_mp_enqueue_bulk(struct rte_ring *r, void * const
 *obj_table,
  unsigned int n, unsigned int *free_space)
 {
-return __rte_ring_do_enqueue(r, obj_table, n,
 RTE_RING_QUEUE_FIXED,
-__IS_MP, free_space);
+if (r->flags & RING_F_NB)
+return __rte_ring_do_nb_enqueue(r, obj_table, n,
+RTE_RING_QUEUE_FIXED
 , __IS_MP,
+free_space);
+else
+return __rte_ring_do_enqueue(r, obj_table, n,
+     RTE_RING_QUEUE_FIXED,

__IS_MP,

lib/librte_ring/rte_ring.h +8 -2
+     free_space);
 }
 
 /**
@@ -461,8 +882,14 @@ static __rte_always_inline unsigned int
 rte_ring_sp_enqueue_bulk(struct rte_ring *r, void * const
 *obj_table,
  unsigned int n, unsigned int *free_space)
 {
-return __rte_ring_do_enqueue(r, obj_table, n,
 RTE_RING_QUEUE_FIXED,
-__IS_SP, free_space);
+if (r->flags & RING_F_NB)
+return __rte_ring_do_nb_enqueue(r, obj_table, n,
+RTE_RING_QUEUE_FIXED
 , __IS_SP,
+free_space);
+else
+return __rte_ring_do_enqueue(r, obj_table, n,
+     RTE_RING_QUEUE_FIXED,

__IS_SP,

lib/librte_ring/rte_ring.h +9 -2
+     free_space);
 }
 
 /**
@@ -488,8 +915,14 @@ static __rte_always_inline unsigned int
 rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
       unsigned int n, unsigned int *free_space)
 {
-return __rte_ring_do_enqueue(r, obj_table, n,
 RTE_RING_QUEUE_FIXED,
-r->prod.single, free_space);
+if (r->flags & RING_F_NB)
+return __rte_ring_do_nb_enqueue(r, obj_table, n,
+RTE_RING_QUEUE_FIXED
 ,
+r->prod_64.single,
 free_space);
+else
+return __rte_ring_do_enqueue(r, obj_table, n,
+     RTE_RING_QUEUE_FIXED,
+     r->prod.single,

free_space);
}

/**

lib/librte_ring/rte_ring.h +8 -2
@@ -572,8 +1005,14 @@ static __rte_always_inline unsigned int
 rte_ring_mc_dequeue_bulk(struct rte_ring *r, void **obj_table,
 unsigned int n, unsigned int *available)
 {
-return __rte_ring_do_dequeue(r, obj_table, n,
 RTE_RING_QUEUE_FIXED,
-__IS_MC, available);
+if (r->flags & RING_F_NB)
+return __rte_ring_do_nb_dequeue(r, obj_table, n,
+RTE_RING_QUEUE_FIXED
 , __IS_MC,
+available);
+else
+return __rte_ring_do_dequeue(r, obj_table, n,
+     RTE_RING_QUEUE_FIXED,
 __IS_MC,
+     available);

}

/**

lib/librte_ring/rte_ring.h +8 -2
@@ -596,8 +1035,14 @@ static __rte_always_inline unsigned int
 rte_ring_sc_dequeue_bulk(struct rte_ring *r, void **obj_table,
 unsigned int n, unsigned int *available)
 {
-return __rte_ring_do_dequeue(r, obj_table, n,
 RTE_RING_QUEUE_FIXED,
-__IS_SC, available);
+if (r->flags & RING_F_NB)
+return __rte_ring_do_nb_dequeue(r, obj_table, n,
+RTE_RING_QUEUE_FIXED
 , __IS_SC,
+available);
+else
+return __rte_ring_do_dequeue(r, obj_table, n,
+     RTE_RING_QUEUE_FIXED,
 __IS_SC,
+     available);

}

/**

lib/librte_ring/rte_ring.h +4 -2
@@ -623,8 +1068,14 @@ static __rte_always_inline unsigned int
 rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned
 int n,
 unsigned int *available)
 {
-return __rte_ring_do_dequeue(r, obj_table, n,
 RTE_RING_QUEUE_FIXED,
-r->cons.single, available);
+if (r->flags & RING_F_NB)
+return __rte_ring_do_nb_dequeue(r, obj_table, n,
+RTE_RING_QUEUE_FIXED
 ,
+r->cons_64.single,

available);

lib/librte_ring/rte_ring.h +4 -0
+else
+return __rte_ring_do_dequeue(r, obj_table, n,
+     RTE_RING_QUEUE_FIXED,
+     r->cons.single,

available);
}

/**

lib/librte_ring/rte_ring.h +7 -3
@@ -699,9 +1150,13 @@ rte_ring_dequeue(struct rte_ring *r, void
 **obj_p)
 static inline unsigned
 rte_ring_count(const struct rte_ring *r)
 {
-uint32_t prod_tail = r->prod.tail;
-uint32_t cons_tail = r->cons.tail;
-uint32_t count = (prod_tail - cons_tail) & r->mask;
+uint32_t count;
+
+if (r->flags & RING_F_NB)
+count = (r->prod_64.tail - r->cons_64.tail) & r-
 >mask;
+else
+count = (r->prod.tail - r->cons.tail) & r->mask;
+
 return (count > r->capacity) ? r->capacity : count;

}

lib/librte_ring/rte_ring.h +4 -2
@@ -821,8 +1276,14 @@ static __rte_always_inline unsigned
 rte_ring_mp_enqueue_burst(struct rte_ring *r, void * const
 *obj_table,
  unsigned int n, unsigned int *free_space)
 {
-return __rte_ring_do_enqueue(r, obj_table, n,
-RTE_RING_QUEUE_VARIABLE, __IS_MP,
 free_space);
+if (r->flags & RING_F_NB)
+return __rte_ring_do_nb_enqueue(r, obj_table, n,
+RTE_RING_QUEUE_VARIA
 BLE,
+__IS_MP,

free_space);

lib/librte_ring/rte_ring.h +3 -0
+else
+return __rte_ring_do_enqueue(r, obj_table, n,
+     RTE_RING_QUEUE_VARIABLE

,

lib/librte_ring/rte_ring.h +5 -2
+     __IS_MP, free_space);
 }
 
 /**
@@ -844,8 +1305,14 @@ static __rte_always_inline unsigned
 rte_ring_sp_enqueue_burst(struct rte_ring *r, void * const
 *obj_table,
  unsigned int n, unsigned int *free_space)
 {
-return __rte_ring_do_enqueue(r, obj_table, n,
-RTE_RING_QUEUE_VARIABLE, __IS_SP,
 free_space);
+if (r->flags & RING_F_NB)
+return __rte_ring_do_nb_enqueue(r, obj_table, n,
+RTE_RING_QUEUE_VARIA
 BLE,
+__IS_SP,

free_space);

lib/librte_ring/rte_ring.h +3 -0
+else
+return __rte_ring_do_enqueue(r, obj_table, n,
+     RTE_RING_QUEUE_VARIABLE

,

lib/librte_ring/rte_ring.h +8 -2
+     __IS_SP, free_space);
 }
 
 /**
@@ -871,8 +1338,14 @@ static __rte_always_inline unsigned
 rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
       unsigned int n, unsigned int *free_space)
 {
-return __rte_ring_do_enqueue(r, obj_table, n,
 RTE_RING_QUEUE_VARIABLE,
-r->prod.single, free_space);
+if (r->flags & RING_F_NB)
+return __rte_ring_do_nb_enqueue(r, obj_table, n,
+RTE_RING_QUEUE_VARIA
 BLE,
+r->prod_64.single,
 free_space);
+else
+return __rte_ring_do_enqueue(r, obj_table, n,
+     RTE_RING_QUEUE_VARIABLE

,
+ r->prod.single,
free_space);
}

/**

lib/librte_ring/rte_ring.h +8 -2
@@ -899,8 +1372,14 @@ static __rte_always_inline unsigned
 rte_ring_mc_dequeue_burst(struct rte_ring *r, void **obj_table,
 unsigned int n, unsigned int *available)
 {
-return __rte_ring_do_dequeue(r, obj_table, n,
-RTE_RING_QUEUE_VARIABLE, __IS_MC,
 available);
+if (r->flags & RING_F_NB)
+return __rte_ring_do_nb_dequeue(r, obj_table, n,
+RTE_RING_QUEUE_VARIA
 BLE,
+__IS_MC, available);
+else
+return __rte_ring_do_dequeue(r, obj_table, n,
+     RTE_RING_QUEUE_VARIABLE
 ,
+     __IS_MC, available);

}

/**

lib/librte_ring/rte_ring.h +8 -2
@@ -924,8 +1403,14 @@ static __rte_always_inline unsigned
 rte_ring_sc_dequeue_burst(struct rte_ring *r, void **obj_table,
 unsigned int n, unsigned int *available)
 {
-return __rte_ring_do_dequeue(r, obj_table, n,
-RTE_RING_QUEUE_VARIABLE, __IS_SC,
 available);
+if (r->flags & RING_F_NB)
+return __rte_ring_do_nb_dequeue(r, obj_table, n,
+RTE_RING_QUEUE_VARIA
 BLE,
+__IS_SC, available);
+else
+return __rte_ring_do_dequeue(r, obj_table, n,
+     RTE_RING_QUEUE_VARIABLE
 ,
+     __IS_SC, available);

}

/**

lib/librte_ring/rte_ring.h +8 -3
@@ -951,9 +1436,14 @@ static __rte_always_inline unsigned
 rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
 unsigned int n, unsigned int *available)
 {
-return __rte_ring_do_dequeue(r, obj_table, n,
-RTE_RING_QUEUE_VARIABLE,
-r->cons.single, available);
+if (r->flags & RING_F_NB)
+return __rte_ring_do_nb_dequeue(r, obj_table, n,
+RTE_RING_QUEUE_VARIA
 BLE,
+r->cons_64.single,
 available);
+else
+return __rte_ring_do_dequeue(r, obj_table, n,
+     RTE_RING_QUEUE_VARIABLE
 ,
+     r->cons.single,

available);
}

#ifdef __cplusplus

lib/librte_ring/rte_ring_version.map +0 -0
index d935efd0d..8969467af 100644

lib/librte_ring/rte_ring_version.map DPDK_2.2 { · line 26

Quoted 10 lines
lib/librte_ring/rte_ring_version.map +7 -0
@@ -17,3 +17,10 @@ DPDK_2.2 {
 rte_ring_free;
 
 } DPDK_2.0;
+
+DPDK_19.05 {
+global:
+
+rte_ring_get_memsize;
+
+} DPDK_2.2;

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#35 lore raw

Re: [PATCH v3 2/5] ring: add a non-blocking implementation

In reply to Gage Eads

(resending without the confidential footer, think I figured it out, ignore the previous email from me in this thread)

-- Ola

Quoted 417 lines · Gage Eads

This commit adds support for non-blocking circular ring enqueue and dequeue functions. The ring uses a 128-bit compare-and-swap instruction, and thus is currently limited to x86_64.

The algorithm is based on the original rte ring (derived from FreeBSD's bufring.h) and inspired by Michael and Scott's non-blocking concurrent queue. Importantly, it adds a modification counter to each ring entry to ensure only one thread can write to an unused entry.

-----
Algorithm:

Multi-producer non-blocking enqueue:
1. Move the producer head index 'n' locations forward, effectively reserving 'n' locations.
2. For each pointer:
 a. Read the producer tail index, then ring[tail]. If ring[tail]'s modification counter isn't 'tail', retry.
 b. Construct the new entry: {pointer, tail + ring size}
 c. Compare-and-swap the old entry with the new. If unsuccessful, the next loop iteration will try to enqueue this pointer again. d. Compare-and-swap the tail index with 'tail + 1', whether or not step 2c succeeded. This guarantees threads can make forward progress.

Multi-consumer non-blocking dequeue:
1. Move the consumer head index 'n' locations forward, effectively reserving 'n' pointers to be dequeued.
2. Copy 'n' pointers into the caller's object table (ignoring the modification counter), starting from ring[tail], then compare-and-swap the tail index with 'tail + n'.  If unsuccessful, repeat step 2.

-----
Discussion:

There are two cases where the ABA problem is mitigated:
1. Enqueueing a pointer to the ring: without a modification counter tied to the tail index, the index could become stale by the time the enqueue happens, causing it to overwrite valid data. Tying the counter to the tail index gives us an expected value (as opposed to, say, a monotonically incrementing counter).

   Since the counter will eventually wrap, there is potential for the ABA problem. However, using a 64-bit counter makes this likelihood effectively zero.

2. Updating a tail index: the ABA problem can occur if the thread is preempted and the tail index wraps around. However, using 64-bit indexes makes this likelihood effectively zero.

With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a non-blocking ring based mempool (a likely use case for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. Because the ring's memsize is now a function of its flags (the non-blocking ring requires 128b for each entry), this commit adds a new argument ('flags') to rte_ring_get_memsize(). An API deprecation notice will be sent in a separate commit.

For ease-of-use, existing ring enqueue and dequeue functions work on both regular and non-blocking rings. This introduces an additional branch in the datapath, but this should be a highly predictable branch. ring_perf_autotest shows a negligible performance impact; it's hard to distinguish a real difference versus system noise.

                                  | ring_perf_autotest cycles with branch - Test                 |   ring_perf_autotest cycles without
------------------------------------------------------------------
SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | -4.00
SP/SC burst enq/dequeue (size 8)  | 0.00
MP/MC burst enq/dequeue (size 8)  | 0.00
SP/SC burst enq/dequeue (size 32) | 0.00
MP/MC burst enq/dequeue (size 32) | 0.00
SC empty dequeue                  | 1.00
MC empty dequeue                  | 0.00

Single lcore:
SP/SC bulk enq/dequeue (size 8)   | 0.49
MP/MC bulk enq/dequeue (size 8)   | 0.08
SP/SC bulk enq/dequeue (size 32)  | 0.07
MP/MC bulk enq/dequeue (size 32)  | 0.09

Two physical cores:
SP/SC bulk enq/dequeue (size 8)   | 0.19
MP/MC bulk enq/dequeue (size 8)   | -0.37
SP/SC bulk enq/dequeue (size 32)  | 0.09
MP/MC bulk enq/dequeue (size 32)  | -0.05

Two NUMA nodes:
SP/SC bulk enq/dequeue (size 8)   | -1.96
MP/MC bulk enq/dequeue (size 8)   | 0.88
SP/SC bulk enq/dequeue (size 32)  | 0.10
MP/MC bulk enq/dequeue (size 32)  | 0.46

Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. Each test run three times and the results averaged.

---
 lib/librte_ring/rte_ring.c           |  72 ++++- lib/librte_ring/rte_ring.h           | 550 +++++++++++++++++++++++++++++++++- -
 lib/librte_ring/rte_ring_version.map |   7 +

 3 files changed, 587 insertions(+), 42 deletions(-)
lib/librte_ring/rte_ring.c +0 -0
index d215acecc..f3378dccd 100644
@@ -45,9 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)


 /* return the size of memory occupied by a ring */ ssize_t

lib/librte_ring/rte_ring.c +1 -1
-rte_ring_get_memsize(unsigned count)
+rte_ring_get_memsize_v1905(unsigned int count, unsigned int flags)

 {

lib/librte_ring/rte_ring.c +1 -1
-	ssize_t sz;
+	ssize_t sz, elt_sz;


  /* count must be a power of 2 */
  if ((!POWEROF2(count)) || (count > RTE_RING_SZ_MASK )) {

lib/librte_ring/rte_ring.c +0 -0
@@ -57,10 +57,23 @@ rte_ring_get_memsize(unsigned count)
  		return -EINVAL;
  	}

lib/librte_ring/rte_ring.c +3 -1
-	sz = sizeof(struct rte_ring) + count * sizeof(void *);
+	elt_sz = (flags & RING_F_NB) ? 2 * sizeof(void *) : sizeof(void *);
+
+	sz = sizeof(struct rte_ring) + count * elt_sz;

  sz = RTE_ALIGN(sz, RTE_CACHE_LINE_SIZE);
  return sz;
 }

lib/librte_ring/rte_ring.c +11 -0
+BIND_DEFAULT_SYMBOL(rte_ring_get_memsize, _v1905, 19.05);
+MAP_STATIC_SYMBOL(ssize_t rte_ring_get_memsize(unsigned int count,
+					       unsigned int flags),
+		  rte_ring_get_memsize_v1905);
+
+ssize_t
+rte_ring_get_memsize_v20(unsigned int count)
+{
+	return rte_ring_get_memsize_v1905(count, 0);
+}
+VERSION_SYMBOL(rte_ring_get_memsize, _v20, 2.0);


 int
 rte_ring_init(struct rte_ring *r, const char *name, unsigned count,

lib/librte_ring/rte_ring.c +0 -2
@@ -82,8 +95,6 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned
 count,
  	if (ret < 0 || ret >= (int)sizeof(r->name))
  		return -ENAMETOOLONG;
  	r->flags = flags;
-	r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
-	r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;


  if (flags & RING_F_EXACT_SZ) { r->size = rte_align32pow2(count + 1);

lib/librte_ring/rte_ring.c +24 -2
@@ -100,8 +111,30 @@ rte_ring_init(struct rte_ring *r, const char *name,
 unsigned count,
  		r->mask = count - 1;
  		r->capacity = r->mask;
  	}
-	r->prod.head = r->cons.head = 0;
-	r->prod.tail = r->cons.tail = 0;
+
+	if (flags & RING_F_NB) {
+		uint64_t i;
+
+		r->prod_64.single = (flags & RING_F_SP_ENQ) ? __IS_SP :
 __IS_MP;
+		r->cons_64.single = (flags & RING_F_SC_DEQ) ? __IS_SC :
 __IS_MC;
+		r->prod_64.head = r->cons_64.head = 0;
+		r->prod_64.tail = r->cons_64.tail = 0;
+
+		for (i = 0; i < r->size; i++) {
+			struct nb_ring_entry *ring_ptr, *base;
+
+			base = ((struct nb_ring_entry *)&r[1]);
+
+			ring_ptr = &base[i & r->mask];
+
+			ring_ptr->cnt = i;
+		}
+	} else {
+		r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
+		r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+		r->prod.head = r->cons.head = 0;
+		r->prod.tail = r->cons.tail = 0;
+	}


  return 0;
 }

lib/librte_ring/rte_ring.c +0 -0
@@ -123,11 +156,19 @@ rte_ring_create(const char *name, unsigned count, int
 socket_id,


  ring_list = RTE_TAILQ_CAST(rte_ring_tailq.head, rte_ring_list);

lib/librte_ring/rte_ring.c +8 -0
+#if !defined(RTE_ARCH_X86_64)
+	if (flags & RING_F_NB) {
+		printf("RING_F_NB is only supported on x86-64 platforms\n");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+#endif
+

  /* for an exact size ring, round up from count to a power of two */ if (flags & RING_F_EXACT_SZ)
  count = rte_align32pow2(count + 1);

lib/librte_ring/rte_ring.c +1 -1
-	ring_size = rte_ring_get_memsize(count);
+	ring_size = rte_ring_get_memsize(count, flags);

  if (ring_size < 0) {
  rte_errno = ring_size;
  return NULL;

lib/librte_ring/rte_ring.c +11 -4
@@ -227,10 +268,17 @@ rte_ring_dump(FILE *f, const struct rte_ring *r)
  	fprintf(f, "  flags=%x\n", r->flags);
  	fprintf(f, "  size=%"PRIu32"\n", r->size);
  	fprintf(f, "  capacity=%"PRIu32"\n", r->capacity);
-	fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
-	fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
-	fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
-	fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	if (r->flags & RING_F_NB) {
+		fprintf(f, "  ct=%"PRIu64"\n", r->cons_64.tail);
+		fprintf(f, "  ch=%"PRIu64"\n", r->cons_64.head);
+		fprintf(f, "  pt=%"PRIu64"\n", r->prod_64.tail);
+		fprintf(f, "  ph=%"PRIu64"\n", r->prod_64.head);
+	} else {
+		fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
+		fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
+		fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
+		fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	}
  	fprintf(f, "  used=%u\n", rte_ring_count(r));
  	fprintf(f, "  avail=%u\n", rte_ring_free_count(r));
  }
lib/librte_ring/rte_ring.h +3 -0
index b270a4746..08c9de6a6 100644
@@ -134,6 +134,18 @@ struct rte_ring {
   */
  #define RING_F_EXACT_SZ 0x0004
  #define RTE_RING_SZ_MASK  (0x7fffffffU) /**< Ring size mask */
+/**
+ * The ring uses non-blocking enqueue and dequeue functions. These functions
+ * do not have the "non-preemptive" constraint of a regular rte ring, and

thus
+ * are suited for applications using preemptible pthreads. However, the
+ * non-blocking functions have worse average-case performance than their
+ * regular rte ring counterparts. When used as the handler for a mempool,
+ * per-thread caching can mitigate the performance difference by reducing the
+ * number (and contention) of ring accesses.
+ *
+ * This flag is only supported on x86_64 platforms.
+ */
+#define RING_F_NB 0x0008

 /* @internal defines for passing to the enqueue dequeue worker functions */ #define __IS_SP 1

lib/librte_ring/rte_ring.h +5 -1
@@ -151,11 +163,15 @@ struct rte_ring {
   *
   * @param count
   *   The number of elements in the ring (must be a power of 2).
+ * @param flags
+ *   The flags the ring will be created with.
   * @return
   *   - The memory size needed for the ring on success.
   *   - -EINVAL if count is not a power of 2.
   */
-ssize_t rte_ring_get_memsize(unsigned count);
+ssize_t rte_ring_get_memsize(unsigned int count, unsigned int flags);
+ssize_t rte_ring_get_memsize_v20(unsigned int count);
+ssize_t rte_ring_get_memsize_v1905(unsigned int count, unsigned int flags);


 /**
  * Initialize a ring structure.

lib/librte_ring/rte_ring.h +8 -0
@@ -188,6 +204,10 @@ ssize_t rte_ring_get_memsize(unsigned count);
   *    - RING_F_SC_DEQ: If this flag is set, the default behavior when
   *      using ``rte_ring_dequeue()`` or ``rte_ring_dequeue_bulk()``
   *      is "single-consumer". Otherwise, it is "multi-consumers".
+ *    - RING_F_EXACT_SZ: If this flag is set, count can be a non-power-of-2
+ *      number, but up to half the ring space may be wasted.
+ *    - RING_F_NB: (x86_64 only) If this flag is set, the ring uses
+ *      non-blocking variants of the dequeue and enqueue functions.
   * @return
   *   0 on success, or a negative value on error.
   */
@@ -223,12 +243,17 @@ int rte_ring_init(struct rte_ring *r, const char *name,
 unsigned count,
   *    - RING_F_SC_DEQ: If this flag is set, the default behavior when
   *      using ``rte_ring_dequeue()`` or ``rte_ring_dequeue_bulk()``
   *      is "single-consumer". Otherwise, it is "multi-consumers".
+ *    - RING_F_EXACT_SZ: If this flag is set, count can be a non-power-of-2
+ *      number, but up to half the ring space may be wasted.
+ *    - RING_F_NB: (x86_64 only) If this flag is set, the ring uses
+ *      non-blocking variants of the dequeue and enqueue functions.
   * @return
   *   On success, the pointer to the new allocated ring. NULL on error with
   *    rte_errno set appropriately. Possible errno values include:
   *    - E_RTE_NO_CONFIG - function could not get pointer to rte_config

structure
  *    - E_RTE_SECONDARY - function was called from a secondary process instance
- *    - EINVAL - count provided is not a power of 2
+ *    - EINVAL - count provided is not a power of 2, or RING_F_NB is used on an
+ *      unsupported platform
  *    - ENOSPC - the maximum number of memzones has already been allocated *    - EEXIST - a memzone with the same name already exists
  *    - ENOMEM - no appropriate memory area found in which to create memzone

lib/librte_ring/rte_ring.h +0 -0
@@ -284,6 +309,50 @@ void rte_ring_dump(FILE *f, const struct rte_ring *r);
  	} \
  } while (0)

lib/librte_ring/rte_ring.h +12 -0
+/* The actual enqueue of pointers on the ring.
+ * Used only by the single-producer non-blocking enqueue function, but
+ * out-lined here for code readability.
+ */
+#define ENQUEUE_PTRS_NB(r, ring_start, prod_head, obj_table, n) do { \
+	unsigned int i; \
+	const uint32_t size = (r)->size; \
+	size_t idx = prod_head & (r)->mask; \
+	size_t new_cnt = prod_head + size; \
+	struct nb_ring_entry *ring = (struct nb_ring_entry *)ring_start; \
+	if (likely(idx + n < size)) { \
+		for (i = 0; i < (n & ((~(unsigned)0x3))); i += 4, idx += 4) {

\

lib/librte_ring/rte_ring.h +32 -0
+			ring[idx].ptr = obj_table[i]; \
+			ring[idx].cnt = new_cnt + i;  \
+			ring[idx + 1].ptr = obj_table[i + 1]; \
+			ring[idx + 1].cnt = new_cnt + i + 1;  \
+			ring[idx + 2].ptr = obj_table[i + 2]; \
+			ring[idx + 2].cnt = new_cnt + i + 2;  \
+			ring[idx + 3].ptr = obj_table[i + 3]; \
+			ring[idx + 3].cnt = new_cnt + i + 3;  \
+		} \
+		switch (n & 0x3) { \
+		case 3: \
+			ring[idx].cnt = new_cnt + i; \
+			ring[idx++].ptr = obj_table[i++]; /* fallthrough */ \
+		case 2: \
+			ring[idx].cnt = new_cnt + i; \
+			ring[idx++].ptr = obj_table[i++]; /* fallthrough */ \
+		case 1: \
+			ring[idx].cnt = new_cnt + i; \
+			ring[idx++].ptr = obj_table[i++]; \
+		} \
+	} else { \
+		for (i = 0; idx < size; i++, idx++) { \
+			ring[idx].cnt = new_cnt + i;  \
+			ring[idx].ptr = obj_table[i]; \
+		} \
+		for (idx = 0; i < n; i++, idx++) {    \
+			ring[idx].cnt = new_cnt + i;  \
+			ring[idx].ptr = obj_table[i]; \
+		} \
+	} \
+} while (0)
+

 /* the actual copy of pointers on the ring to obj_table.
  * Placed here since identical code needed in both
  * single and multi consumer dequeue functions */

lib/librte_ring/rte_ring.h +0 -0
@@ -315,6 +384,39 @@ void rte_ring_dump(FILE *f, const struct rte_ring *r);
  	} \
  } while (0)

lib/librte_ring/rte_ring.h +33 -0
+/* The actual copy of pointers on the ring to obj_table.
+ * Placed here since identical code needed in both
+ * single and multi consumer non-blocking dequeue functions.
+ */
+#define DEQUEUE_PTRS_NB(r, ring_start, cons_head, obj_table, n) do { \
+	unsigned int i; \
+	size_t idx = cons_head & (r)->mask; \
+	const uint32_t size = (r)->size; \
+	struct nb_ring_entry *ring = (struct nb_ring_entry *)ring_start; \
+	if (likely(idx + n < size)) { \
+		for (i = 0; i < (n & (~(unsigned)0x3)); i += 4, idx += 4) {\
+			obj_table[i] = ring[idx].ptr; \
+			obj_table[i + 1] = ring[idx + 1].ptr; \
+			obj_table[i + 2] = ring[idx + 2].ptr; \
+			obj_table[i + 3] = ring[idx + 3].ptr; \
+		} \
+		switch (n & 0x3) { \
+		case 3: \
+			obj_table[i++] = ring[idx++].ptr; /* fallthrough */ \
+		case 2: \
+			obj_table[i++] = ring[idx++].ptr; /* fallthrough */ \
+		case 1: \
+			obj_table[i++] = ring[idx++].ptr; \
+		} \
+	} else { \
+		for (i = 0; idx < size; i++, idx++) \
+			obj_table[i] = ring[idx].ptr; \
+		for (idx = 0; i < n; i++, idx++) \
+			obj_table[i] = ring[idx].ptr; \
+	} \
+} while (0)
+
+

 /* Between load and load. there might be cpu reorder in weak model
  * (powerpc/arm).
  * There are 2 choices for the users

lib/librte_ring/rte_ring.h +0 -0
@@ -331,6 +433,319 @@ void rte_ring_dump(FILE *f, const struct rte_ring *r);
  #endif
  #include "rte_ring_generic_64.h"

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 439

Quoted 4 lines
lib/librte_ring/rte_ring.h +4 -0
+/* @internal 128-bit structure used by the non-blocking ring */
+struct nb_ring_entry {
+	void *ptr; /**< Data pointer */
+	uint64_t cnt; /**< Modification counter */

Why not make 'cnt' uintptr_t? This way 32-bit architectures will also be supported. I think there are some claims that DPDK still supports e.g. ARMv7a and possibly also 32-bit x86?

Quoted 33 lines
lib/librte_ring/rte_ring.h +33 -0
+};
+
+/* The non-blocking ring algorithm is based on the original rte ring (derived
+ * from FreeBSD's bufring.h) and inspired by Michael and Scott's non-blocking
+ * concurrent queue.
+ */
+
+/**
+ * @internal
+ *   Enqueue several objects on the non-blocking ring (single-producer only)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items to the ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible to the ring
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue_sp(struct rte_ring *r, void * const *obj_table,
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *free_space)
+{
+	uint32_t free_entries;

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 482

Quoted 10 lines
lib/librte_ring/rte_ring.h +10 -0
+	size_t head, next;
+
+	n = __rte_ring_move_prod_head_64(r, 1, n, behavior,
+					 &head, &next, &free_entries);
+	if (n == 0)
+		goto end;
+
+	ENQUEUE_PTRS_NB(r, &r[1], head, obj_table, n);
+
+	r->prod_64.tail += n;

Don't we need release order when (or smp_wmb between) writing of the ring pointers and the update of tail? By updating the tail pointer, we are synchronising with a consumer.

I prefer using __atomic operations even for load and store. You can see which parts of the code that synchronise with each other, e.g. store-release to some location synchronises with load-acquire from the same location. If you don't know how different threads synchronise with each other, you are very likely to make mistakes.

Quoted 61 lines
lib/librte_ring/rte_ring.h +60 -0
+
+end:
+	if (free_space != NULL)
+		*free_space = free_entries - n;
+	return n;
+}
+
+/**
+ * @internal
+ *   Enqueue several objects on the non-blocking ring (multi-producer safe)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items to the ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible to the ring
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue_mp(struct rte_ring *r, void * const *obj_table,
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *free_space)
+{
+#if !defined(RTE_ARCH_X86_64) || !defined(ALLOW_EXPERIMENTAL_API)
+	RTE_SET_USED(r);
+	RTE_SET_USED(obj_table);
+	RTE_SET_USED(n);
+	RTE_SET_USED(behavior);
+	RTE_SET_USED(free_space);
+#ifndef ALLOW_EXPERIMENTAL_API
+	printf("[%s()] RING_F_NB requires an experimental API."
+	       " Recompile with ALLOW_EXPERIMENTAL_API to use it.\n"
+	       , __func__);
+#endif
+	return 0;
+#endif
+#if defined(RTE_ARCH_X86_64) && defined(ALLOW_EXPERIMENTAL_API)
+	size_t head, next, tail;
+	uint32_t free_entries;
+	unsigned int i;
+
+	n = __rte_ring_move_prod_head_64(r, 0, n, behavior,
+					 &head, &next, &free_entries);
+	if (n == 0)
+		goto end;
+
+	for (i = 0; i < n; /* i incremented if enqueue succeeds */) {
+		struct nb_ring_entry old_value, new_value;
+		struct nb_ring_entry *ring_ptr;
+
+		/* Enqueue to the tail entry. If another thread wins the

race,

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 547

Quoted 5 lines
lib/librte_ring/rte_ring.h +5 -0
+		 * retry with the new tail.
+		 */
+		tail = r->prod_64.tail;
+
+		ring_ptr = &((struct nb_ring_entry *)&r[1])[tail & r->mask];

This is an ugly expression and cast. Also I think it is unnecessary. What's preventing this from being written without a cast? Perhaps the ring array needs to be a union of "void *" and struct nb_ring_entry?

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 555

Quoted 8 lines
lib/librte_ring/rte_ring.h +8 -0
+
+		old_value = *ring_ptr;
+
+		/* If the tail entry's modification counter doesn't match the
+		 * producer tail index, it's already been updated.
+		 */
+		if (old_value.cnt != tail)
+			continue;

Continue restarts the loop at the condition test in the for statement, 'i' and 'n' are unchanged. Then we re-read 'prod_64.tail' and 'ring[tail & mask]'. If some other thread never updates 'prod_64.tail', the test here (ring[tail].cnt != tail) will still be true and we will spin forever.

Waiting for other threads <=> blocking behaviour so this is not a non- blocking design.

Quoted 14 lines
lib/librte_ring/rte_ring.h +13 -0
+
+		/* Prepare the new entry. The cnt field mitigates the ABA
+		 * problem on the ring write.
+		 */
+		new_value.ptr = obj_table[i];
+		new_value.cnt = tail + r->size;
+
+		if (rte_atomic128_cmpset((volatile rte_int128_t *)ring_ptr,
+					 (rte_int128_t *)&old_value,
+					 (rte_int128_t *)&new_value))
+			i++;
+
+		/* Every thread attempts the cmpset, so they don't have to

wait

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 573

Quoted 5 lines
lib/librte_ring/rte_ring.h +5 -0
+		 * for the thread that successfully enqueued to the ring.
+		 * Using a 64-bit tail mitigates the ABA problem here.
+		 *
+		 * Built-in used to handle variable-sized tail index.
+		 */

But prod_64.tail is 64 bits so not really variable size?

Quoted 2 lines
lib/librte_ring/rte_ring.h +1 -0
+		__sync_bool_compare_and_swap(&r->prod_64.tail, tail, tail +

1);

What memory order is required here? Why not use __atomic_compare_exchange() with explicit memory order parameters?

Quoted 71 lines
lib/librte_ring/rte_ring.h +32 -0
+	}
+
+end:
+	if (free_space != NULL)
+		*free_space = free_entries - n;
+	return n;
+#endif
+}
+
+/**
+ * @internal Enqueue several objects on the non-blocking ring
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items to the ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible to the ring
+ * @param is_sp
+ *   Indicates whether to use single producer or multi-producer head update
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue(struct rte_ring *r, void * const *obj_table,
+			 unsigned int n, enum rte_ring_queue_behavior

behavior,

lib/librte_ring/rte_ring.h +13 -0
+			 unsigned int is_sp, unsigned int *free_space)
+{
+	if (is_sp)
+		return __rte_ring_do_nb_enqueue_sp(r, obj_table, n,
+						   behavior, free_space);
+	else
+		return __rte_ring_do_nb_enqueue_mp(r, obj_table, n,
+						   behavior, free_space);
+}
+
+/**
+ * @internal
+ *   Dequeue several objects from the non-blocking ring (single-consumer

only)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from the ring
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from the ring
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has finished
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */

lib/librte_ring/rte_ring.h +7 -0
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_dequeue_sc(struct rte_ring *r, void **obj_table,
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *available)
+{
+	size_t head, next;

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 652

Quoted 10 lines
lib/librte_ring/rte_ring.h +10 -0
+	uint32_t entries;
+
+	n = __rte_ring_move_cons_head_64(r, 1, n, behavior,
+					 &head, &next, &entries);
+	if (n == 0)
+		goto end;
+
+	DEQUEUE_PTRS_NB(r, &r[1], head, obj_table, n);
+
+	r->cons_64.tail += n;

Memory ordering? Consumer synchronises with producer.

Quoted 351 lines
lib/librte_ring/rte_ring.h +22 -0
+
+end:
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+/**
+ * @internal
+ *   Dequeue several objects from the non-blocking ring (multi-consumer safe)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from the ring
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from the ring
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has

finished
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */

lib/librte_ring/rte_ring.h +18 -0
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_dequeue_mc(struct rte_ring *r, void **obj_table,
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *available)
+{
+	size_t head, next;
+	uint32_t entries;
+
+	n = __rte_ring_move_cons_head_64(r, 0, n, behavior,
+					 &head, &next, &entries);
+	if (n == 0)
+		goto end;
+
+	while (1) {
+		size_t tail = r->cons_64.tail;
+
+		/* Dequeue from the cons tail onwards. If multiple threads

read

lib/librte_ring/rte_ring.h +1 -0
+		 * the same pointers, the thread that successfully performs

the

lib/librte_ring/rte_ring.h +8 -0
+		 * CAS will keep them and the other(s) will retry.
+		 */
+		DEQUEUE_PTRS_NB(r, &r[1], tail, obj_table, n);
+
+		next = tail + n;
+
+		/* Built-in used to handle variable-sized tail index. */
+		if (__sync_bool_compare_and_swap(&r->cons_64.tail, tail,

next))

lib/librte_ring/rte_ring.h +26 -0
+			/* There is potential for the ABA problem here, but
+			 * that is mitigated by the large (64-bit) tail.
+			 */
+			break;
+	}
+
+end:
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the non-blocking ring
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from the ring
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from the ring
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has

finished
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */

lib/librte_ring/rte_ring.h +13 -0
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_dequeue(struct rte_ring *r, void **obj_table,
+		 unsigned int n, enum rte_ring_queue_behavior behavior,
+		 unsigned int is_sc, unsigned int *available)
+{
+	if (is_sc)
+		return __rte_ring_do_nb_dequeue_sc(r, obj_table, n,
+						   behavior, available);
+	else
+		return __rte_ring_do_nb_dequeue_mc(r, obj_table, n,
+						   behavior, available);
+}
+

 /**
  * @internal Enqueue several objects on the ring
  *

lib/librte_ring/rte_ring.h +8 -2
@@ -438,8 +853,14 @@ static __rte_always_inline unsigned int
  rte_ring_mp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
  			 unsigned int n, unsigned int *free_space)
  {
-	return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
-			__IS_MP, free_space);
+	if (r->flags & RING_F_NB)
+		return __rte_ring_do_nb_enqueue(r, obj_table, n,
+						RTE_RING_QUEUE_FIXED,
 __IS_MP,
+						free_space);
+	else
+		return __rte_ring_do_enqueue(r, obj_table, n,
+					     RTE_RING_QUEUE_FIXED, __IS_MP,
+					     free_space);
  }


 /**

lib/librte_ring/rte_ring.h +8 -2
@@ -461,8 +882,14 @@ static __rte_always_inline unsigned int
  rte_ring_sp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
  			 unsigned int n, unsigned int *free_space)
  {
-	return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
-			__IS_SP, free_space);
+	if (r->flags & RING_F_NB)
+		return __rte_ring_do_nb_enqueue(r, obj_table, n,
+						RTE_RING_QUEUE_FIXED,
 __IS_SP,
+						free_space);
+	else
+		return __rte_ring_do_enqueue(r, obj_table, n,
+					     RTE_RING_QUEUE_FIXED, __IS_SP,
+					     free_space);
  }


 /**

lib/librte_ring/rte_ring.h +8 -2
@@ -488,8 +915,14 @@ static __rte_always_inline unsigned int
  rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
  		      unsigned int n, unsigned int *free_space)
  {
-	return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
-			r->prod.single, free_space);
+	if (r->flags & RING_F_NB)
+		return __rte_ring_do_nb_enqueue(r, obj_table, n,
+						RTE_RING_QUEUE_FIXED,
+						r->prod_64.single,
 free_space);
+	else
+		return __rte_ring_do_enqueue(r, obj_table, n,
+					     RTE_RING_QUEUE_FIXED,
+					     r->prod.single, free_space);
  }


 /**

lib/librte_ring/rte_ring.h +8 -2
@@ -572,8 +1005,14 @@ static __rte_always_inline unsigned int
  rte_ring_mc_dequeue_bulk(struct rte_ring *r, void **obj_table,
  		unsigned int n, unsigned int *available)
  {
-	return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
-			__IS_MC, available);
+	if (r->flags & RING_F_NB)
+		return __rte_ring_do_nb_dequeue(r, obj_table, n,
+						RTE_RING_QUEUE_FIXED,
 __IS_MC,
+						available);
+	else
+		return __rte_ring_do_dequeue(r, obj_table, n,
+					     RTE_RING_QUEUE_FIXED, __IS_MC,
+					     available);
  }


 /**

lib/librte_ring/rte_ring.h +8 -2
@@ -596,8 +1035,14 @@ static __rte_always_inline unsigned int
  rte_ring_sc_dequeue_bulk(struct rte_ring *r, void **obj_table,
  		unsigned int n, unsigned int *available)
  {
-	return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
-			__IS_SC, available);
+	if (r->flags & RING_F_NB)
+		return __rte_ring_do_nb_dequeue(r, obj_table, n,
+						RTE_RING_QUEUE_FIXED,
 __IS_SC,
+						available);
+	else
+		return __rte_ring_do_dequeue(r, obj_table, n,
+					     RTE_RING_QUEUE_FIXED, __IS_SC,
+					     available);
  }


 /**

lib/librte_ring/rte_ring.h +8 -2
@@ -623,8 +1068,14 @@ static __rte_always_inline unsigned int
  rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
  		unsigned int *available)
  {
-	return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
-				r->cons.single, available);
+	if (r->flags & RING_F_NB)
+		return __rte_ring_do_nb_dequeue(r, obj_table, n,
+						RTE_RING_QUEUE_FIXED,
+						r->cons_64.single,
 available);
+	else
+		return __rte_ring_do_dequeue(r, obj_table, n,
+					     RTE_RING_QUEUE_FIXED,
+					     r->cons.single, available);
  }


 /**

lib/librte_ring/rte_ring.h +0 -0
@@ -699,9 +1150,13 @@ rte_ring_dequeue(struct rte_ring *r, void **obj_p)

 static inline unsigned
 rte_ring_count(const struct rte_ring *r)
 {

lib/librte_ring/rte_ring.h +7 -3
-	uint32_t prod_tail = r->prod.tail;
-	uint32_t cons_tail = r->cons.tail;
-	uint32_t count = (prod_tail - cons_tail) & r->mask;
+	uint32_t count;
+
+	if (r->flags & RING_F_NB)
+		count = (r->prod_64.tail - r->cons_64.tail) & r->mask;
+	else
+		count = (r->prod.tail - r->cons.tail) & r->mask;
+

  return (count > r->capacity) ? r->capacity : count; }

lib/librte_ring/rte_ring.h +8 -2
@@ -821,8 +1276,14 @@ static __rte_always_inline unsigned
  rte_ring_mp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
  			 unsigned int n, unsigned int *free_space)
  {
-	return __rte_ring_do_enqueue(r, obj_table, n,
-			RTE_RING_QUEUE_VARIABLE, __IS_MP, free_space);
+	if (r->flags & RING_F_NB)
+		return __rte_ring_do_nb_enqueue(r, obj_table, n,
+						RTE_RING_QUEUE_VARIABLE,
+						__IS_MP, free_space);
+	else
+		return __rte_ring_do_enqueue(r, obj_table, n,
+					     RTE_RING_QUEUE_VARIABLE,
+					     __IS_MP, free_space);
  }


 /**

lib/librte_ring/rte_ring.h +8 -2
@@ -844,8 +1305,14 @@ static __rte_always_inline unsigned
  rte_ring_sp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
  			 unsigned int n, unsigned int *free_space)
  {
-	return __rte_ring_do_enqueue(r, obj_table, n,
-			RTE_RING_QUEUE_VARIABLE, __IS_SP, free_space);
+	if (r->flags & RING_F_NB)
+		return __rte_ring_do_nb_enqueue(r, obj_table, n,
+						RTE_RING_QUEUE_VARIABLE,
+						__IS_SP, free_space);
+	else
+		return __rte_ring_do_enqueue(r, obj_table, n,
+					     RTE_RING_QUEUE_VARIABLE,
+					     __IS_SP, free_space);
  }


 /**

lib/librte_ring/rte_ring.h +8 -2
@@ -871,8 +1338,14 @@ static __rte_always_inline unsigned
  rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
  		      unsigned int n, unsigned int *free_space)
  {
-	return __rte_ring_do_enqueue(r, obj_table, n,
 RTE_RING_QUEUE_VARIABLE,
-			r->prod.single, free_space);
+	if (r->flags & RING_F_NB)
+		return __rte_ring_do_nb_enqueue(r, obj_table, n,
+						RTE_RING_QUEUE_VARIABLE,
+						r->prod_64.single,
 free_space);
+	else
+		return __rte_ring_do_enqueue(r, obj_table, n,
+					     RTE_RING_QUEUE_VARIABLE,
+					     r->prod.single, free_space);
  }


 /**

lib/librte_ring/rte_ring.h +8 -2
@@ -899,8 +1372,14 @@ static __rte_always_inline unsigned
  rte_ring_mc_dequeue_burst(struct rte_ring *r, void **obj_table,
  		unsigned int n, unsigned int *available)
  {
-	return __rte_ring_do_dequeue(r, obj_table, n,
-			RTE_RING_QUEUE_VARIABLE, __IS_MC, available);
+	if (r->flags & RING_F_NB)
+		return __rte_ring_do_nb_dequeue(r, obj_table, n,
+						RTE_RING_QUEUE_VARIABLE,
+						__IS_MC, available);
+	else
+		return __rte_ring_do_dequeue(r, obj_table, n,
+					     RTE_RING_QUEUE_VARIABLE,
+					     __IS_MC, available);
  }


 /**

lib/librte_ring/rte_ring.h +8 -2
@@ -924,8 +1403,14 @@ static __rte_always_inline unsigned
  rte_ring_sc_dequeue_burst(struct rte_ring *r, void **obj_table,
  		unsigned int n, unsigned int *available)
  {
-	return __rte_ring_do_dequeue(r, obj_table, n,
-			RTE_RING_QUEUE_VARIABLE, __IS_SC, available);
+	if (r->flags & RING_F_NB)
+		return __rte_ring_do_nb_dequeue(r, obj_table, n,
+						RTE_RING_QUEUE_VARIABLE,
+						__IS_SC, available);
+	else
+		return __rte_ring_do_dequeue(r, obj_table, n,
+					     RTE_RING_QUEUE_VARIABLE,
+					     __IS_SC, available);
  }


 /**

lib/librte_ring/rte_ring.h +8 -3
@@ -951,9 +1436,14 @@ static __rte_always_inline unsigned
  rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
  		unsigned int n, unsigned int *available)
  {
-	return __rte_ring_do_dequeue(r, obj_table, n,
-				RTE_RING_QUEUE_VARIABLE,
-				r->cons.single, available);
+	if (r->flags & RING_F_NB)
+		return __rte_ring_do_nb_dequeue(r, obj_table, n,
+						RTE_RING_QUEUE_VARIABLE,
+						r->cons_64.single,
 available);
+	else
+		return __rte_ring_do_dequeue(r, obj_table, n,
+					     RTE_RING_QUEUE_VARIABLE,
+					     r->cons.single, available);
  }


 #ifdef __cplusplus

lib/librte_ring/rte_ring_version.map +0 -0
index d935efd0d..8969467af 100644
@@ -17,3 +17,10 @@ DPDK_2.2 {
  	rte_ring_free;


 } DPDK_2.0;

lib/librte_ring/rte_ring_version.map +7 -0
+
+DPDK_19.05 {
+	global:
+
+	rte_ring_get_memsize;
+
+} DPDK_2.2;
Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
author
#36 lore raw

Re: [PATCH v3 2/5] ring: add a non-blocking implementation

Hi Ola,

<snip>

Quoted 5 lines
Quoted 5 lines
lib/librte_ring/rte_ring.h +0 -0
@@ -331,6 +433,319 @@ void rte_ring_dump(FILE *f, const struct
 rte_ring *r);
  #endif
  #include "rte_ring_generic_64.h"
 
Quoted 7 lines

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 439

Quoted 4 lines
lib/librte_ring/rte_ring.h +4 -0
+/* @internal 128-bit structure used by the non-blocking ring */
+struct nb_ring_entry {
+	void *ptr; /**< Data pointer */
+	uint64_t cnt; /**< Modification counter */

Why not make 'cnt' uintptr_t? This way 32-bit architectures will also be supported. I think there are some claims that DPDK still supports e.g. ARMv7a and possibly also 32-bit x86?

I chose a 64-bit modification counter because (practically speaking) the ABA problem will not occur with such a large counter -- definitely not within my lifetime. See the "Discussion" section of the commit message for more information.

With a 32-bit counter, there is a very (very) low likelihood of it, but it is possible. Personally, I don't feel comfortable providing such code, because a) I doubt all users would understand the implementation well enough to do the risk/reward analysis, and b) such a bug would be near impossible to reproduce and root-cause if it did occur.

Quoted 54 lines
Quoted 40 lines
lib/librte_ring/rte_ring.h +39 -0
+};
+
+/* The non-blocking ring algorithm is based on the original rte ring
+(derived
+ * from FreeBSD's bufring.h) and inspired by Michael and Scott's
+non-blocking
+ * concurrent queue.
+ */
+
+/**
+ * @internal
+ *   Enqueue several objects on the non-blocking ring
+(single-producer only)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items to the
+ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible to
+the ring
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has
+finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue_sp(struct rte_ring *r, void * const *obj_table,
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *free_space)
+{
+	uint32_t free_entries;

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 482

Quoted 10 lines
lib/librte_ring/rte_ring.h +10 -0
+	size_t head, next;
+
+	n = __rte_ring_move_prod_head_64(r, 1, n, behavior,
+					 &head, &next, &free_entries);
+	if (n == 0)
+		goto end;
+
+	ENQUEUE_PTRS_NB(r, &r[1], head, obj_table, n);
+
+	r->prod_64.tail += n;

Don't we need release order when (or smp_wmb between) writing of the ring pointers and the update of tail? By updating the tail pointer, we are synchronising with a consumer.

Quoted 6 lines

I prefer using __atomic operations even for load and store. You can see which parts of the code that synchronise with each other, e.g. store-release to some location synchronises with load-acquire from the same location. If you don't know how different threads synchronise with each other, you are very likely to make mistakes.

You can tell this code was written when I thought x86-64 was the only viable target :). Yes, you are correct.

With regards to using __atomic intrinsics, I'm planning on taking a similar approach to the functions duplicated in rte_ring_generic.h and rte_ring_c11_mem.h: one version that uses rte_atomic functions (and thus stricter memory ordering) and one that uses __atomic intrinsics (and thus can benefit from more relaxed memory ordering).

Quoted 65 lines
Quoted 65 lines
lib/librte_ring/rte_ring.h +64 -0
+
+end:
+	if (free_space != NULL)
+		*free_space = free_entries - n;
+	return n;
+}
+
+/**
+ * @internal
+ *   Enqueue several objects on the non-blocking ring (multi-producer
+safe)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items to the
+ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible to
+the ring
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has
+finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue_mp(struct rte_ring *r, void * const *obj_table,
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *free_space)
+{
+#if !defined(RTE_ARCH_X86_64) || !defined(ALLOW_EXPERIMENTAL_API)
+	RTE_SET_USED(r);
+	RTE_SET_USED(obj_table);
+	RTE_SET_USED(n);
+	RTE_SET_USED(behavior);
+	RTE_SET_USED(free_space);
+#ifndef ALLOW_EXPERIMENTAL_API
+	printf("[%s()] RING_F_NB requires an experimental API."
+	       " Recompile with ALLOW_EXPERIMENTAL_API to use it.\n"
+	       , __func__);
+#endif
+	return 0;
+#endif
+#if defined(RTE_ARCH_X86_64) && defined(ALLOW_EXPERIMENTAL_API)
+	size_t head, next, tail;
+	uint32_t free_entries;
+	unsigned int i;
+
+	n = __rte_ring_move_prod_head_64(r, 0, n, behavior,
+					 &head, &next, &free_entries);
+	if (n == 0)
+		goto end;
+
+	for (i = 0; i < n; /* i incremented if enqueue succeeds */) {
+		struct nb_ring_entry old_value, new_value;
+		struct nb_ring_entry *ring_ptr;
+
+		/* Enqueue to the tail entry. If another thread wins the
 race,
Quoted 8 lines

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 547

Quoted 5 lines
lib/librte_ring/rte_ring.h +5 -0
+		 * retry with the new tail.
+		 */
+		tail = r->prod_64.tail;
+
+		ring_ptr = &((struct nb_ring_entry *)&r[1])[tail & r->mask];

This is an ugly expression and cast. Also I think it is unnecessary. What's preventing this from being written without a cast? Perhaps the ring array needs to be a union of "void *" and struct nb_ring_entry?

The cast is necessary for the correct pointer arithmetic (let "uintptr_t base == &r[1]"):
- With cast: ring_ptr = base + sizeof(struct nb_ring_entry) * (tail & r->mask);
- W/o cast: ring_ptr = base + sizeof(struct rte_ring) * (tail & r->mask);

FWIW, this is essentially the same as is done with the second argument (&r[1]) to ENQUEUE_PTRS and DEQUEUE_PTRS, but there it's split across multiple lines of code. The equivalent here would be:

struct nb_ring_entry *ring_base = (struct nb_ring_entry*)&r[1]; ring_ptr = ring_base[tail & r->mask];

Which is more legible, I think.

There is no ring array structure in which to add a union; the ring array is a contiguous chunk of memory that immediately follows after the end of a struct rte_ring. We interpret the memory there according to the ring entry data type (void * for regular rings and struct nb_ring_entry for non-blocking rings).

Quoted 14 lines

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 555

Quoted 9 lines
lib/librte_ring/rte_ring.h +8 -0
+
+		old_value = *ring_ptr;
+
+		/* If the tail entry's modification counter doesn't match the
+		 * producer tail index, it's already been updated.
+		 */
+		if (old_value.cnt != tail)
+			continue;

Continue restarts the loop at the condition test in the for statement, 'i' and 'n' are unchanged. Then we re-read 'prod_64.tail' and 'ring[tail & mask]'. If some other thread never updates 'prod_64.tail', the test here (ring[tail].cnt != tail) will still be true and we will spin forever.

Quoted 3 lines

Waiting for other threads <=> blocking behaviour so this is not a non- blocking design.

You're absolutely right. The if-statement was added as optimization to avoid 128-bit cmpset operations that are known to fail, but in this form it violates the non-blocking design.

I see two solutions: 1) drop the if-statement altogether, or 2) attempt to update prod_64.tail before continuing. Both require every thread to attempt to update prod_64.tail on every iteration, but #2 will result in fewer failed 128-bit cmpsets.

Quoted 12 lines
Quoted 12 lines
lib/librte_ring/rte_ring.h +12 -0
+
+		/* Prepare the new entry. The cnt field mitigates the ABA
+		 * problem on the ring write.
+		 */
+		new_value.ptr = obj_table[i];
+		new_value.cnt = tail + r->size;
+
+		if (rte_atomic128_cmpset((volatile rte_int128_t *)ring_ptr,
+					 (rte_int128_t *)&old_value,
+					 (rte_int128_t *)&new_value))
+			i++;
+
Quoted 9 lines

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 573

Quoted 7 lines
(quoted diff) +1 -0
+		/* Every thread attempts the cmpset, so they don't have to

wait

lib/librte_ring/rte_ring.h +5 -0
+		 * for the thread that successfully enqueued to the ring.
+		 * Using a 64-bit tail mitigates the ABA problem here.
+		 *
+		 * Built-in used to handle variable-sized tail index.
+		 */

But prod_64.tail is 64 bits so not really variable size?

(See next comment)

Quoted 5 lines
Quoted 2 lines
(quoted diff) +1 -0
+		__sync_bool_compare_and_swap(&r->prod_64.tail, tail, tail +

1);

What memory order is required here? Why not use __atomic_compare_exchange() with explicit memory order parameters?

This is an artifact from an older patchset that used uintptr_t, and before I learned that other platforms could support this non-blocking algorithm (hence the __sync type intrinsic was sufficient).

At any rate, as described earlier in this response, I plan on writing these functions using the __atomic builtins in the next patchset.

Quoted 81 lines
Quoted 81 lines
lib/librte_ring/rte_ring.h +36 -0
+	}
+
+end:
+	if (free_space != NULL)
+		*free_space = free_entries - n;
+	return n;
+#endif
+}
+
+/**
+ * @internal Enqueue several objects on the non-blocking ring
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items to the
+ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible to
+the ring
+ * @param is_sp
+ *   Indicates whether to use single producer or multi-producer head
+update
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has
+finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue(struct rte_ring *r, void * const *obj_table,
+			 unsigned int n, enum rte_ring_queue_behavior

behavior,

lib/librte_ring/rte_ring.h +13 -0
+			 unsigned int is_sp, unsigned int *free_space) {
+	if (is_sp)
+		return __rte_ring_do_nb_enqueue_sp(r, obj_table, n,
+						   behavior, free_space);
+	else
+		return __rte_ring_do_nb_enqueue_mp(r, obj_table, n,
+						   behavior, free_space);
+}
+
+/**
+ * @internal
+ *   Dequeue several objects from the non-blocking ring
+(single-consumer

only)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from
+ the ring
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
+ the ring
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue
+ has
finished
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */

lib/librte_ring/rte_ring.h +10 -0
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_dequeue_sc(struct rte_ring *r, void **obj_table,
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *available)
+{
+	size_t head, next;
+	uint32_t entries;
+
+	n = __rte_ring_move_cons_head_64(r, 1, n, behavior,
Quoted 9 lines

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 652

Quoted 7 lines
lib/librte_ring/rte_ring.h +7 -0
+					 &head, &next, &entries);
+	if (n == 0)
+		goto end;
+
+	DEQUEUE_PTRS_NB(r, &r[1], head, obj_table, n);
+
+	r->cons_64.tail += n;

Memory ordering? Consumer synchronises with producer.

Agreed, that is missing here. Will fix.

Thanks,
Gage

Quoted 3 lines

--
Ola Liljedahl, Networking System Architect, Arm Phone +46706866373, Skype ola.liljedahl

#37 lore raw

Re: [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 24 lines · Gage

Hi Ola,

<snip>

Quoted 7 lines
Quoted 7 lines
lib/librte_ring/rte_ring.h +0 -0
@@ -331,6 +433,319 @@ void rte_ring_dump(FILE *f, const struct
 rte_ring *r);
  #endif
  #include "rte_ring_generic_64.h"
 
Quoted 8 lines

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 439

Quoted 4 lines
lib/librte_ring/rte_ring.h +4 -0
+/* @internal 128-bit structure used by the non-blocking ring */
+struct nb_ring_entry {
+	void *ptr; /**< Data pointer */
+	uint64_t cnt; /**< Modification counter */

Why not make 'cnt' uintptr_t? This way 32-bit architectures will also be supported. I think there are some claims that DPDK still supports e.g. ARMv7a
and possibly also 32-bit x86?

I chose a 64-bit modification counter because (practically speaking) the ABA problem will not occur with such a large counter -- definitely not within my lifetime. See the "Discussion" section of the commit message for more information.

Quoted 5 lines

With a 32-bit counter, there is a very (very) low likelihood of it, but it is possible. Personally, I don't feel comfortable providing such code, because a) I doubt all users would understand the implementation well enough to do the risk/reward analysis, and b) such a bug would be near impossible to reproduce and root-cause if it did occur.

With a 64-bit counter (and 32-bit pointer), 32-bit architectures (e.g. ARMv7a and probably x86 as well) won't be able to support this as they at best support 64-bit CAS (ARMv7a has LDREXD/STREXD). So you are essentially putting a 64-bit (and 128-bit CAS) requirement on the implementation.

Quoted 66 lines
Quoted 57 lines
Quoted 43 lines
lib/librte_ring/rte_ring.h +39 -0
+};
+
+/* The non-blocking ring algorithm is based on the original rte ring
+(derived
+ * from FreeBSD's bufring.h) and inspired by Michael and Scott's
+non-blocking
+ * concurrent queue.
+ */
+
+/**
+ * @internal
+ *   Enqueue several objects on the non-blocking ring
+(single-producer only)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items to the
+ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible to
+the ring
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has
+finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue_sp(struct rte_ring *r, void * const *obj_table,
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *free_space)
+{
+	uint32_t free_entries;

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 482

Quoted 10 lines
lib/librte_ring/rte_ring.h +10 -0
+	size_t head, next;
+
+	n = __rte_ring_move_prod_head_64(r, 1, n, behavior,
+					 &head, &next, &free_entries);
+	if (n == 0)
+		goto end;
+
+	ENQUEUE_PTRS_NB(r, &r[1], head, obj_table, n);
+
+	r->prod_64.tail += n;

Don't we need release order when (or smp_wmb between) writing of the ring pointers and the update of tail? By updating the tail pointer, we are synchronising with a consumer.

Quoted 9 lines

I prefer using __atomic operations even for load and store. You can see which
parts of the code that synchronise with each other, e.g. store-release to some
location synchronises with load-acquire from the same location. If you don't know how different threads synchronise with each other, you are very likely to
make mistakes.

Quoted 8 lines

You can tell this code was written when I thought x86-64 was the only viable target :). Yes, you are correct.

With regards to using __atomic intrinsics, I'm planning on taking a similar approach to the functions duplicated in rte_ring_generic.h and rte_ring_c11_mem.h: one version that uses rte_atomic functions (and thus stricter memory ordering) and one that uses __atomic intrinsics (and thus can benefit from more relaxed memory ordering).

What's the advantage of having two different implementations? What is the disadvantage?

The existing ring buffer code originally had only the "legacy" implementation which was kept when the __atomic implementation was added. The reason claimed was that some older compilers for x86 do not support GCC __atomic builtins. But I thought there was consensus that new functionality could have only __atomic implementations.

Does the non-blocking ring buffer implementation have to support these older compilers? Will the applications that require these older compiler be updated to utilise the non-blocking ring buffer?

Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
lib/librte_ring/rte_ring.h +69 -0
+
+end:
+	if (free_space != NULL)
+		*free_space = free_entries - n;
+	return n;
+}
+
+/**
+ * @internal
+ *   Enqueue several objects on the non-blocking ring (multi-producer
+safe)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items to the
+ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible to
+the ring
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has
+finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue_mp(struct rte_ring *r, void * const *obj_table,
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *free_space)
+{
+#if !defined(RTE_ARCH_X86_64) || !defined(ALLOW_EXPERIMENTAL_API)
+	RTE_SET_USED(r);
+	RTE_SET_USED(obj_table);
+	RTE_SET_USED(n);
+	RTE_SET_USED(behavior);
+	RTE_SET_USED(free_space);
+#ifndef ALLOW_EXPERIMENTAL_API
+	printf("[%s()] RING_F_NB requires an experimental API."
+	       " Recompile with ALLOW_EXPERIMENTAL_API to use it.\n"
+	       , __func__);
+#endif
+	return 0;
+#endif
+#if defined(RTE_ARCH_X86_64) && defined(ALLOW_EXPERIMENTAL_API)
+	size_t head, next, tail;
+	uint32_t free_entries;
+	unsigned int i;
+
+	n = __rte_ring_move_prod_head_64(r, 0, n, behavior,
+					 &head, &next, &free_entries);
+	if (n == 0)
+		goto end;
+
+	for (i = 0; i < n; /* i incremented if enqueue succeeds */) {
+		struct nb_ring_entry old_value, new_value;
+		struct nb_ring_entry *ring_ptr;
+
+		/* Enqueue to the tail entry. If another thread wins the
 race,
+		 * retry with the new tail.
+		 */
+		tail = r->prod_64.tail;
+
+		ring_ptr = &((struct nb_ring_entry *)&r[1])[tail & r-
Quoted 7 lines
Quoted 5 lines

Review void rte_ring_dump(FILE *f, const struct · line 560

Quoted 1 line
(quoted diff) +0 -0
 >mask];

This is an ugly expression and cast. Also I think it is unnecessary. What's preventing this from being written without a cast? Perhaps the ring array needs
to be a union of "void *" and struct nb_ring_entry?

The cast is necessary for the correct pointer arithmetic (let "uintptr_t base == &r[1]"):

Yes I know the C language.

Quoted 4 lines

- With cast: ring_ptr = base + sizeof(struct nb_ring_entry) * (tail & r-

Quoted 1 line

mask);

- W/o cast: ring_ptr = base + sizeof(struct rte_ring) * (tail & r->mask);

Quoted 8 lines

FWIW, this is essentially the same as is done with the second argument (&r[1]) to ENQUEUE_PTRS and DEQUEUE_PTRS, but there it's split across multiple lines of code. The equivalent here would be:

struct nb_ring_entry *ring_base = (struct nb_ring_entry*)&r[1]; ring_ptr = ring_base[tail & r->mask];

Which is more legible, I think.

The RTE ring buffer code is not very legible to start with.

Quoted 5 lines

There is no ring array structure in which to add a union; the ring array is a contiguous chunk of memory that immediately follows after the end of a struct rte_ring. We interpret the memory there according to the ring entry data type (void * for regular rings and struct nb_ring_entry for non-blocking rings).

My worry is that we are abusing the C language and creating a monster of fragile C code that will be more and more difficult to understand and to maintain. At some point you have to think the question "Are we doing the right thing?".

Quoted 69 lines
Quoted 21 lines

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 555

Quoted 13 lines
lib/librte_ring/rte_ring.h +4 -0
+
+		old_value = *ring_ptr;
+
+		/* If the tail entry's modification counter doesn't match

the

lib/librte_ring/rte_ring.h +4 -0
+		 * producer tail index, it's already been updated.
+		 */
+		if (old_value.cnt != tail)
+			continue;

Continue restarts the loop at the condition test in the for statement, 'i' and 'n'
are unchanged. Then we re-read 'prod_64.tail' and 'ring[tail & mask]'. If some
other thread never updates 'prod_64.tail', the test here (ring[tail].cnt != tail) will
still be true and we will spin forever.

Quoted 4 lines

Waiting for other threads <=> blocking behaviour so this is not a non- blocking
design.

You're absolutely right. The if-statement was added as optimization to avoid 128-bit cmpset operations that are known to fail, but in this form it violates the non-blocking design.

I see two solutions: 1) drop the if-statement altogether, or 2) attempt to update prod_64.tail before continuing. Both require every thread to attempt to update prod_64.tail on every iteration, but #2 will result in fewer failed 128-bit cmpsets.

Quoted 16 lines
Quoted 16 lines
lib/librte_ring/rte_ring.h +8 -0
+
+		/* Prepare the new entry. The cnt field mitigates the ABA
+		 * problem on the ring write.
+		 */
+		new_value.ptr = obj_table[i];
+		new_value.cnt = tail + r->size;
+
+		if (rte_atomic128_cmpset((volatile rte_int128_t

*)ring_ptr,

lib/librte_ring/rte_ring.h +5 -0
+					 (rte_int128_t *)&old_value,
+					 (rte_int128_t *)&new_value))
+			i++;
+
+		/* Every thread attempts the cmpset, so they don't have
Quoted 9 lines

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 573

Quoted 7 lines

to
wait

lib/librte_ring/rte_ring.h +5 -0
+		 * for the thread that successfully enqueued to the ring.
+		 * Using a 64-bit tail mitigates the ABA problem here.
+		 *
+		 * Built-in used to handle variable-sized tail index.
+		 */

But prod_64.tail is 64 bits so not really variable size?

(See next comment)

Quoted 8 lines
Quoted 5 lines
(quoted diff) +2 -0
+		__sync_bool_compare_and_swap(&r->prod_64.tail, tail, tail
+

1);

What memory order is required here? Why not use __atomic_compare_exchange() with explicit memory order parameters?

Quoted 6 lines

This is an artifact from an older patchset that used uintptr_t, and before I learned that other platforms could support this non-blocking algorithm (hence the __sync type intrinsic was sufficient).

At any rate, as described earlier in this response, I plan on writing these functions using the __atomic builtins in the next patchset.

Great.

Quoted 102 lines
Quoted 84 lines
Quoted 84 lines
lib/librte_ring/rte_ring.h +36 -0
+	}
+
+end:
+	if (free_space != NULL)
+		*free_space = free_entries - n;
+	return n;
+#endif
+}
+
+/**
+ * @internal Enqueue several objects on the non-blocking ring
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items to the
+ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible to
+the ring
+ * @param is_sp
+ *   Indicates whether to use single producer or multi-producer head
+update
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has
+finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue(struct rte_ring *r, void * const *obj_table,
+			 unsigned int n, enum rte_ring_queue_behavior

behavior,

lib/librte_ring/rte_ring.h +13 -0
+			 unsigned int is_sp, unsigned int *free_space) {
+	if (is_sp)
+		return __rte_ring_do_nb_enqueue_sp(r, obj_table, n,
+						   behavior, free_space);
+	else
+		return __rte_ring_do_nb_enqueue_mp(r, obj_table, n,
+						   behavior, free_space);
+}
+
+/**
+ * @internal
+ *   Dequeue several objects from the non-blocking ring
+(single-consumer

only)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from
+ the ring
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
+ the ring
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue
+ has
finished
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */

lib/librte_ring/rte_ring.h +10 -0
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_dequeue_sc(struct rte_ring *r, void **obj_table,
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *available)
+{
+	size_t head, next;
+	uint32_t entries;
+
+	n = __rte_ring_move_cons_head_64(r, 1, n, behavior,
Quoted 9 lines

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 652

Quoted 7 lines
lib/librte_ring/rte_ring.h +7 -0
+					 &head, &next, &entries);
+	if (n == 0)
+		goto end;
+
+	DEQUEUE_PTRS_NB(r, &r[1], head, obj_table, n);
+
+	r->cons_64.tail += n;

Memory ordering? Consumer synchronises with producer.

Agreed, that is missing here. Will fix.

Thanks,
Gage

Quoted 4 lines

--
Ola Liljedahl, Networking System Architect, Arm Phone +46706866373, Skype ola.liljedahl

Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
author
#38 lore raw

Re: [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 42 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Wednesday, January 23, 2019 4:16 AM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 24 lines · Gage

Hi Ola,

<snip>

Quoted 7 lines
Quoted 7 lines
lib/librte_ring/rte_ring.h +0 -0
@@ -331,6 +433,319 @@ void rte_ring_dump(FILE *f, const struct
 rte_ring *r);
  #endif
  #include "rte_ring_generic_64.h"
 
Quoted 8 lines

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 439

Quoted 4 lines
lib/librte_ring/rte_ring.h +4 -0
+/* @internal 128-bit structure used by the non-blocking ring */
+struct nb_ring_entry {
+	void *ptr; /**< Data pointer */
+	uint64_t cnt; /**< Modification counter */

Why not make 'cnt' uintptr_t? This way 32-bit architectures will also be supported. I think there are some claims that DPDK still supports e.g. ARMv7a
and possibly also 32-bit x86?

I chose a 64-bit modification counter because (practically speaking) the ABA problem will not occur with such a large counter -- definitely not within my lifetime. See the "Discussion" section of the commit message for more information.

Quoted 5 lines

With a 32-bit counter, there is a very (very) low likelihood of it, but it is possible. Personally, I don't feel comfortable providing such code, because a) I doubt all users would understand the implementation well enough to do the risk/reward analysis, and b) such a bug would be near impossible to reproduce and root-cause if it did occur.

Quoted 5 lines

With a 64-bit counter (and 32-bit pointer), 32-bit architectures (e.g. ARMv7a and probably x86 as well) won't be able to support this as they at best support 64-bit CAS (ARMv7a has LDREXD/STREXD). So you are essentially putting a 64-bit (and 128-bit CAS) requirement on the implementation.

Yes, I am. I tried to make that clear in the cover letter.

Quoted 71 lines
Quoted 63 lines
Quoted 57 lines
Quoted 43 lines
lib/librte_ring/rte_ring.h +39 -0
+};
+
+/* The non-blocking ring algorithm is based on the original rte
+ring (derived
+ * from FreeBSD's bufring.h) and inspired by Michael and Scott's
+non-blocking
+ * concurrent queue.
+ */
+
+/**
+ * @internal
+ *   Enqueue several objects on the non-blocking ring
+(single-producer only)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items to
+the ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible
+to the ring
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has
+finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue_sp(struct rte_ring *r, void * const *obj_table,
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *free_space)
+{
+	uint32_t free_entries;

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 482

Quoted 10 lines
lib/librte_ring/rte_ring.h +10 -0
+	size_t head, next;
+
+	n = __rte_ring_move_prod_head_64(r, 1, n, behavior,
+					 &head, &next, &free_entries);
+	if (n == 0)
+		goto end;
+
+	ENQUEUE_PTRS_NB(r, &r[1], head, obj_table, n);
+
+	r->prod_64.tail += n;

Don't we need release order when (or smp_wmb between) writing of the ring pointers and the update of tail? By updating the tail pointer, we are synchronising with a consumer.

Quoted 6 lines

I prefer using __atomic operations even for load and store. You can see which parts of the code that synchronise with each other, e.g. store-release to some location synchronises with load-acquire from the same location. If you don't know how different threads synchronise with each other, you are very likely to make mistakes.

Quoted 8 lines

You can tell this code was written when I thought x86-64 was the only viable target :). Yes, you are correct.

With regards to using __atomic intrinsics, I'm planning on taking a similar approach to the functions duplicated in rte_ring_generic.h and rte_ring_c11_mem.h: one version that uses rte_atomic functions (and thus stricter memory ordering) and one that uses __atomic intrinsics (and thus can benefit from more relaxed memory ordering).

Quoted 9 lines

What's the advantage of having two different implementations? What is the disadvantage?

The existing ring buffer code originally had only the "legacy" implementation which was kept when the __atomic implementation was added. The reason claimed was that some older compilers for x86 do not support GCC __atomic builtins. But I thought there was consensus that new functionality could have only __atomic implementations.

When CONFIG_RTE_RING_USE_C11_MEM_MODEL was introduced, it was left disabled for thunderx[1] for performance reasons. Assuming that hasn't changed, the advantage to having two versions is to best support all of DPDK's platforms. The disadvantage is of course duplicated code and the additional maintenance burden.

That said, if the thunderx maintainers are ok with it, I'm certainly open to only doing the __atomic version. Note that even in the __atomic version, based on Honnapa's findings[2], using a DPDK-defined rte_atomic128_cmpset() (with additional arguments to support machines with weak consistency) appears to be a better option than __atomic_compare_exchange_16.

I couldn't find the discussion about new functionality using __atomic going forward -- can you send a link?

[1] https://mails.dpdk.org/archives/dev/2017-December/082853.html
[2] http://mails.dpdk.org/archives/dev/2019-January/124002.html

Quoted 4 lines

Does the non-blocking ring buffer implementation have to support these older compilers? Will the applications that require these older compiler be updated to utilise the non-blocking ring buffer?

(See above -- compiler versions wasn't a consideration here.)

Quoted 105 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
lib/librte_ring/rte_ring.h +25 -0
+
+end:
+	if (free_space != NULL)
+		*free_space = free_entries - n;
+	return n;
+}
+
+/**
+ * @internal
+ *   Enqueue several objects on the non-blocking ring
+(multi-producer
+safe)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items to
+the ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible
+to the ring
+ * @param free_space
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r);

Quoted 8 lines
lib/librte_ring/rte_ring.h +8 -0
+ *   returns the amount of space after the enqueue operation has
+finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue_mp(struct rte_ring *r, void * const

*obj_table,

Quoted 38 lines
Quoted 38 lines
Quoted 38 lines
lib/librte_ring/rte_ring.h +37 -0
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *free_space)
+{
+#if !defined(RTE_ARCH_X86_64) || !defined(ALLOW_EXPERIMENTAL_API)
+	RTE_SET_USED(r);
+	RTE_SET_USED(obj_table);
+	RTE_SET_USED(n);
+	RTE_SET_USED(behavior);
+	RTE_SET_USED(free_space);
+#ifndef ALLOW_EXPERIMENTAL_API
+	printf("[%s()] RING_F_NB requires an experimental API."
+	       " Recompile with ALLOW_EXPERIMENTAL_API to use it.\n"
+	       , __func__);
+#endif
+	return 0;
+#endif
+#if defined(RTE_ARCH_X86_64) && defined(ALLOW_EXPERIMENTAL_API)
+	size_t head, next, tail;
+	uint32_t free_entries;
+	unsigned int i;
+
+	n = __rte_ring_move_prod_head_64(r, 0, n, behavior,
+					 &head, &next, &free_entries);
+	if (n == 0)
+		goto end;
+
+	for (i = 0; i < n; /* i incremented if enqueue succeeds */) {
+		struct nb_ring_entry old_value, new_value;
+		struct nb_ring_entry *ring_ptr;
+
+		/* Enqueue to the tail entry. If another thread wins the
 race,
+		 * retry with the new tail.
+		 */
+		tail = r->prod_64.tail;
+
+		ring_ptr = &((struct nb_ring_entry *)&r[1])[tail & r-
Quoted 7 lines
Quoted 5 lines

Review void rte_ring_dump(FILE *f, const struct · line 561

Quoted 1 line
(quoted diff) +0 -0
 >mask];

This is an ugly expression and cast. Also I think it is unnecessary. What's preventing this from being written without a cast? Perhaps the ring array needs to be a union of "void *" and struct nb_ring_entry?

The cast is necessary for the correct pointer arithmetic (let "uintptr_t base == &r[1]"):

Yes I know the C language.

Quoted 6 lines

- With cast: ring_ptr = base + sizeof(struct nb_ring_entry) * (tail & r-

Quoted 1 line

mask);

- W/o cast: ring_ptr = base + sizeof(struct rte_ring) * (tail & r->mask);

Quoted 8 lines

FWIW, this is essentially the same as is done with the second argument (&r[1]) to ENQUEUE_PTRS and DEQUEUE_PTRS, but there it's split across multiple lines of code. The equivalent here would be:

struct nb_ring_entry *ring_base = (struct nb_ring_entry*)&r[1]; ring_ptr = ring_base[tail & r->mask];

Which is more legible, I think.

The RTE ring buffer code is not very legible to start with.

Quoted 5 lines

There is no ring array structure in which to add a union; the ring array is a contiguous chunk of memory that immediately follows after the end of a struct rte_ring. We interpret the memory there according to the ring entry data type (void * for regular rings and struct nb_ring_entry for

Quoted 6 lines

non-blocking rings).
My worry is that we are abusing the C language and creating a monster of fragile C code that will be more and more difficult to understand and to maintain. At some point you have to think the question "Are we doing the right thing?".

I'm not aware of any fragility/maintainability issues in the ring code (though perhaps the maintainers have a different view!), and personally I find the code fairly legible. If you have a specific suggestion, I'll look into incorporating it.

Thanks,
Gage

</snip>

#39 lore raw

Re: [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 48 lines · Gage
Quoted 48 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Wednesday, January 23, 2019 4:16 AM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 34 lines · Gage

Hi Ola,

<snip>

Quoted 9 lines
Quoted 9 lines
lib/librte_ring/rte_ring.h +0 -0
@@ -331,6 +433,319 @@ void rte_ring_dump(FILE *f, const struct
 rte_ring *r);
  #endif
  #include "rte_ring_generic_64.h"
 
Quoted 9 lines

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 439

Quoted 4 lines
lib/librte_ring/rte_ring.h +4 -0
+/* @internal 128-bit structure used by the non-blocking ring */
+struct nb_ring_entry {
+	void *ptr; /**< Data pointer */
+	uint64_t cnt; /**< Modification counter */

Why not make 'cnt' uintptr_t? This way 32-bit architectures will also be supported. I think there are some claims that DPDK still supports e.g.
ARMv7a
and possibly also 32-bit x86?

I chose a 64-bit modification counter because (practically speaking) the ABA problem will not occur with such a large counter -- definitely not within my lifetime. See the "Discussion" section of the commit message for more information.

With a 32-bit counter, there is a very (very) low likelihood of it, but it is possible. Personally, I don't feel comfortable providing such code, because a) I doubt all users would understand the implementation well enough to do the risk/reward analysis, and b) such a bug would be near impossible to reproduce and root-cause if it did occur.

Quoted 86 lines
Quoted 7 lines

With a 64-bit counter (and 32-bit pointer), 32-bit architectures (e.g. ARMv7a and
probably x86 as well) won't be able to support this as they at best support 64-bit
CAS (ARMv7a has LDREXD/STREXD). So you are essentially putting a 64-bit (and 128-bit CAS) requirement on the implementation.

Yes, I am. I tried to make that clear in the cover letter.

Quoted 77 lines
Quoted 77 lines
Quoted 63 lines
Quoted 49 lines
lib/librte_ring/rte_ring.h +40 -0
+};
+
+/* The non-blocking ring algorithm is based on the original rte
+ring (derived
+ * from FreeBSD's bufring.h) and inspired by Michael and Scott's
+non-blocking
+ * concurrent queue.
+ */
+
+/**
+ * @internal
+ *   Enqueue several objects on the non-blocking ring
+(single-producer only)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items to
+the ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible
+to the ring
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has
+finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue_sp(struct rte_ring *r, void * const
 *obj_table,
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *free_space)
+{
+	uint32_t free_entries;
+	size_t head, next;

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 482

Quoted 10 lines
lib/librte_ring/rte_ring.h +9 -0
+
+	n = __rte_ring_move_prod_head_64(r, 1, n, behavior,
+					 &head, &next,
 &free_entries);
+	if (n == 0)
+		goto end;
+
+	ENQUEUE_PTRS_NB(r, &r[1], head, obj_table, n);
+
+	r->prod_64.tail += n;

Don't we need release order when (or smp_wmb between) writing of the ring pointers and the update of tail? By updating the tail pointer, we are synchronising with a consumer.

Quoted 6 lines

I prefer using __atomic operations even for load and store. You can see which parts of the code that synchronise with each other, e.g. store-release to some location synchronises with load-acquire from the same location. If you don't know how different threads synchronise with each other, you are very likely to make mistakes.

You can tell this code was written when I thought x86-64 was the only viable target :). Yes, you are correct.

With regards to using __atomic intrinsics, I'm planning on taking a similar approach to the functions duplicated in rte_ring_generic.h and rte_ring_c11_mem.h: one version that uses rte_atomic functions (and thus stricter memory ordering) and one that uses __atomic intrinsics (and thus can benefit from more relaxed memory ordering).

From a code point of view, I strongly prefer the atomic operations to be visible in the top level code, not hidden in subroutines. For correctness, it is vital that memory accesses are performed with the required ordering and that acquire and release matches up. Hiding e.g. load-acquire and store-release in subroutines (in a different file!) make this difficult. There have already been such bugs found in rte_ring.

Quoted 11 lines
Quoted 11 lines

What's the advantage of having two different implementations? What is the disadvantage?

The existing ring buffer code originally had only the "legacy" implementation
which was kept when the __atomic implementation was added. The reason claimed was that some older compilers for x86 do not support GCC __atomic builtins. But I thought there was consensus that new functionality could have
only __atomic implementations.

Quoted 5 lines

When CONFIG_RTE_RING_USE_C11_MEM_MODEL was introduced, it was left disabled for thunderx[1] for performance reasons. Assuming that hasn't changed, the advantage to having two versions is to best support all of DPDK's platforms. The disadvantage is of course duplicated code and the additional maintenance burden.

The only way I see that a C11 memory model implementation can be slower than using smp_wmb/rmb is if you need to order loads before a synchronizing store and there are also outstanding stores which do not require ordering. smp_rmb() handles this while store-release will also (unnecessarily) order those outstanding stores. This situation occurs e.g. in ring buffer dequeue operations where ring slots are read (and possibly written to thread-private memory) before the ring slots are release (e.g. using CAS-release or store-release).

I imagine that the LSU/cache subsystem on ThunderX/OCTEON-TX also have something to do with this problem. If there are a large amounts of stores pending in the load/store unit, store-release might have to wait for a long time before the synchronizing store can complete.

Quoted 6 lines

That said, if the thunderx maintainers are ok with it, I'm certainly open to only doing the __atomic version. Note that even in the __atomic version, based on Honnapa's findings[2], using a DPDK-defined rte_atomic128_cmpset() (with additional arguments to support machines with weak consistency) appears to be a better option than __atomic_compare_exchange_16.

__atomic_compare_exchange_16() is not guaranteed to be lock-free. It is not lock-free on ARM/AArch64 and the support in GCC is formally broken (can't use cmpexchg16b to implement __atomic_load_16).

So yes, I think DPDK will have to define and implement the 128-bit atomic compare and exchange operation (whatever it will be called). For compatibility with ARMv8.0, we can't require the "old" value returned by a failed compare- exchange operation to be read atomically (LDXP does not guaranteed atomicity by itself). But this is seldom a problem, many designs read the memory location using two separate 64-bit loads (so not atomic) anyway, it is a successful atomic compare exchange operation which provides atomicity.

Quoted 150 lines

I couldn't find the discussion about new functionality using __atomic going forward -- can you send a link?

[1] https://mails.dpdk.org/archives/dev/2017-December/082853.html
[2] http://mails.dpdk.org/archives/dev/2019-January/124002.html

Quoted 6 lines

Does the non-blocking ring buffer implementation have to support these older compilers? Will the applications that require these older compiler be updated to
utilise the non-blocking ring buffer?

(See above -- compiler versions wasn't a consideration here.)

Quoted 119 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
lib/librte_ring/rte_ring.h +25 -0
+
+end:
+	if (free_space != NULL)
+		*free_space = free_entries - n;
+	return n;
+}
+
+/**
+ * @internal
+ *   Enqueue several objects on the non-blocking ring
+(multi-producer
+safe)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items to
+the ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible
+to the ring
+ * @param free_space
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r);

Quoted 8 lines
lib/librte_ring/rte_ring.h +8 -0
+ *   returns the amount of space after the enqueue operation has
+finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue_mp(struct rte_ring *r, void * const

*obj_table,

Quoted 44 lines
Quoted 44 lines
Quoted 44 lines
lib/librte_ring/rte_ring.h +24 -0
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *free_space)
+{
+#if !defined(RTE_ARCH_X86_64) || !defined(ALLOW_EXPERIMENTAL_API)
+	RTE_SET_USED(r);
+	RTE_SET_USED(obj_table);
+	RTE_SET_USED(n);
+	RTE_SET_USED(behavior);
+	RTE_SET_USED(free_space);
+#ifndef ALLOW_EXPERIMENTAL_API
+	printf("[%s()] RING_F_NB requires an experimental API."
+	       " Recompile with ALLOW_EXPERIMENTAL_API to use it.\n"
+	       , __func__);
+#endif
+	return 0;
+#endif
+#if defined(RTE_ARCH_X86_64) && defined(ALLOW_EXPERIMENTAL_API)
+	size_t head, next, tail;
+	uint32_t free_entries;
+	unsigned int i;
+
+	n = __rte_ring_move_prod_head_64(r, 0, n, behavior,
+					 &head, &next,

&free_entries);

lib/librte_ring/rte_ring.h +8 -0
+	if (n == 0)
+		goto end;
+
+	for (i = 0; i < n; /* i incremented if enqueue succeeds */) {
+		struct nb_ring_entry old_value, new_value;
+		struct nb_ring_entry *ring_ptr;
+
+		/* Enqueue to the tail entry. If another thread wins

the
race,

lib/librte_ring/rte_ring.h +5 -0
+		 * retry with the new tail.
+		 */
+		tail = r->prod_64.tail;
+
+		ring_ptr = &((struct nb_ring_entry *)&r[1])[tail & r-
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line

mask];

This is an ugly expression and cast. Also I think it is unnecessary. What's preventing this from being written without a cast? Perhaps the ring array needs to be a union of "void *" and struct nb_ring_entry?

The cast is necessary for the correct pointer arithmetic (let "uintptr_t base == &r[1]"):

Yes I know the C language.

Quoted 8 lines

- With cast: ring_ptr = base + sizeof(struct nb_ring_entry) * (tail & r-

Quoted 2 lines

mask);

- W/o cast: ring_ptr = base + sizeof(struct rte_ring) * (tail & r->mask);

Quoted 8 lines

FWIW, this is essentially the same as is done with the second argument (&r[1]) to ENQUEUE_PTRS and DEQUEUE_PTRS, but there it's split across multiple lines of code. The equivalent here would be:

struct nb_ring_entry *ring_base = (struct nb_ring_entry*)&r[1]; ring_ptr = ring_base[tail & r->mask];

Which is more legible, I think.

The RTE ring buffer code is not very legible to start with.

Quoted 7 lines

There is no ring array structure in which to add a union; the ring array is a contiguous chunk of memory that immediately follows after the end of a struct rte_ring. We interpret the memory there according to the ring entry data type (void * for regular rings and struct nb_ring_entry for

Quoted 7 lines

non-blocking rings).
My worry is that we are abusing the C language and creating a monster of fragile C code that will be more and more difficult to understand and to maintain. At some point you have to think the question "Are we doing the right
thing?".

I'm not aware of any fragility/maintainability issues in the ring code (though perhaps the maintainers have a different view!), and personally I find the code fairly legible. If you have a specific suggestion, I'll look into incorporating it.

Thanks,
Gage

</snip>

Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
author
#40 lore raw

Re: [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 148 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Monday, January 28, 2019 4:36 AM
To: [email protected]; [email protected]; Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 48 lines · Gage
Quoted 48 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Wednesday, January 23, 2019 4:16 AM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 34 lines · Gage

Hi Ola,

<snip>

Quoted 9 lines
Quoted 9 lines
lib/librte_ring/rte_ring.h +0 -0
@@ -331,6 +433,319 @@ void rte_ring_dump(FILE *f, const struct
 rte_ring *r);
  #endif
  #include "rte_ring_generic_64.h"
 
Quoted 9 lines

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 439

Quoted 4 lines
lib/librte_ring/rte_ring.h +4 -0
+/* @internal 128-bit structure used by the non-blocking ring
+*/ struct nb_ring_entry {
+	void *ptr; /**< Data pointer */
+	uint64_t cnt; /**< Modification counter */

Why not make 'cnt' uintptr_t? This way 32-bit architectures will also be supported. I think there are some claims that DPDK still supports e.g.
ARMv7a
and possibly also 32-bit x86?

I chose a 64-bit modification counter because (practically speaking) the ABA problem will not occur with such a large counter -- definitely not within my lifetime. See the "Discussion" section of the commit message for more information.

With a 32-bit counter, there is a very (very) low likelihood of it, but it is possible. Personally, I don't feel comfortable providing such code, because a) I doubt all users would understand the implementation well enough to do the risk/reward analysis, and
b) such a bug would be near impossible to reproduce and root-cause if it did occur.

Quoted 86 lines
Quoted 7 lines

With a 64-bit counter (and 32-bit pointer), 32-bit architectures (e.g. ARMv7a and
probably x86 as well) won't be able to support this as they at best support 64-bit CAS (ARMv7a has LDREXD/STREXD). So you are essentially putting a 64-bit (and 128-bit CAS) requirement on the implementation.

Yes, I am. I tried to make that clear in the cover letter.

Quoted 77 lines
Quoted 77 lines
Quoted 62 lines
Quoted 48 lines
lib/librte_ring/rte_ring.h +39 -0
+};
+
+/* The non-blocking ring algorithm is based on the original
+rte ring (derived
+ * from FreeBSD's bufring.h) and inspired by Michael and
+Scott's non-blocking
+ * concurrent queue.
+ */
+
+/**
+ * @internal
+ *   Enqueue several objects on the non-blocking ring
+(single-producer only)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items
+to the ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as
+possible to the ring
+ * @param free_space
+ *   returns the amount of space after the enqueue operation
+has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue_sp(struct rte_ring *r, void * const
 *obj_table,
+			    unsigned int n,
+			    enum rte_ring_queue_behavior behavior,
+			    unsigned int *free_space) {
+	uint32_t free_entries;
+	size_t head, next;

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 482

Quoted 10 lines
lib/librte_ring/rte_ring.h +9 -0
+
+	n = __rte_ring_move_prod_head_64(r, 1, n, behavior,
+					 &head, &next,
 &free_entries);
+	if (n == 0)
+		goto end;
+
+	ENQUEUE_PTRS_NB(r, &r[1], head, obj_table, n);
+
+	r->prod_64.tail += n;

Don't we need release order when (or smp_wmb between) writing of the ring pointers and the update of tail? By updating the tail pointer, we are synchronising with a consumer.

Quoted 6 lines

I prefer using __atomic operations even for load and store. You can see which parts of the code that synchronise with each other, e.g. store-release to some location synchronises with load-acquire from the same location. If you don't know how different threads synchronise with each other, you are very likely to make mistakes.

You can tell this code was written when I thought x86-64 was the only viable target :). Yes, you are correct.

With regards to using __atomic intrinsics, I'm planning on taking a similar approach to the functions duplicated in rte_ring_generic.h and
rte_ring_c11_mem.h: one version that uses rte_atomic functions (and thus stricter memory ordering) and one that uses __atomic intrinsics (and thus can benefit from more relaxed memory ordering).

Quoted 7 lines

From a code point of view, I strongly prefer the atomic operations to be visible in the top level code, not hidden in subroutines. For correctness, it is vital that memory accesses are performed with the required ordering and that acquire and release matches up. Hiding e.g. load-acquire and store-release in subroutines (in a different file!) make this difficult. There have already been such bugs found in rte_ring.

After working on the acq/rel ordering this weekend, I agree. This'll be easier/cleaner if we end up only using the C11 version.

Quoted 40 lines
Quoted 10 lines
Quoted 10 lines

What's the advantage of having two different implementations? What is the disadvantage?

The existing ring buffer code originally had only the "legacy" implementation
which was kept when the __atomic implementation was added. The reason claimed was that some older compilers for x86 do not support GCC __atomic builtins. But I thought there was consensus that new functionality could have only __atomic implementations.

Quoted 3 lines

When CONFIG_RTE_RING_USE_C11_MEM_MODEL was introduced, it was left disabled for thunderx[1] for performance reasons. Assuming that hasn't changed, the advantage to having two versions is to best support all of DPDK's

platforms.

Quoted 2 lines

The disadvantage is of course duplicated code and the additional maintenance burden.

The only way I see that a C11 memory model implementation can be slower than using smp_wmb/rmb is if you need to order loads before a synchronizing store and there are also outstanding stores which do not require ordering. smp_rmb() handles this while store-release will also (unnecessarily) order those outstanding stores. This situation occurs e.g. in ring buffer dequeue operations where ring slots are read (and possibly written to thread-private memory) before the ring slots are release (e.g. using CAS-release or store-release).

I imagine that the LSU/cache subsystem on ThunderX/OCTEON-TX also have something to do with this problem. If there are a large amounts of stores pending in the load/store unit, store-release might have to wait for a long time before the synchronizing store can complete.

Quoted 6 lines

That said, if the thunderx maintainers are ok with it, I'm certainly open to only doing the __atomic version. Note that even in the __atomic version, based on Honnapa's findings[2], using a DPDK-defined rte_atomic128_cmpset() (with additional arguments to support machines with weak consistency) appears to be a better option than

__atomic_compare_exchange_16.
__atomic_compare_exchange_16() is not guaranteed to be lock-free. It is not lock-free on ARM/AArch64 and the support in GCC is formally broken (can't use cmpexchg16b to implement __atomic_load_16).

Quoted 8 lines

So yes, I think DPDK will have to define and implement the 128-bit atomic compare and exchange operation (whatever it will be called). For compatibility with ARMv8.0, we can't require the "old" value returned by a failed compare- exchange operation to be read atomically (LDXP does not guaranteed atomicity by itself). But this is seldom a problem, many designs read the memory location using two separate 64-bit loads (so not atomic) anyway, it is a successful atomic compare exchange operation which provides atomicity.

Ok. I agree, I don't expect that to be a problem. The 128-bit CAS patch I just submitted[1] (which was developed before reading this) will have to be changed.

[1] http://mails.dpdk.org/archives/dev/2019-January/124159.html

Thanks,
Gage

</snip>

#41 lore raw

Re: [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 173 lines · Gage
Quoted 161 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Monday, January 28, 2019 4:36 AM
To: [email protected]; [email protected]; Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 53 lines · Gage
Quoted 53 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Wednesday, January 23, 2019 4:16 AM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 37 lines · Gage

Hi Ola,

<snip>

Quoted 11 lines
Quoted 11 lines
(quoted diff) +0 -0
@@ -331,6 +433,319 @@ void rte_ring_dump(FILE *f, const struct
 rte_ring *r);
  #endif
  #include "rte_ring_generic_64.h"
 
Quoted 9 lines

Review void rte_ring_dump(FILE *f, const struct · line 440

Quoted 4 lines
(quoted diff) +4 -0
+/* @internal 128-bit structure used by the non-blocking ring
+*/ struct nb_ring_entry {
+	void *ptr; /**< Data pointer */
+	uint64_t cnt; /**< Modification counter */

Why not make 'cnt' uintptr_t? This way 32-bit architectures will also be supported. I think there are some claims that DPDK still supports e.g.
ARMv7a
and possibly also 32-bit x86?

I chose a 64-bit modification counter because (practically speaking) the ABA problem will not occur with such a large counter -- definitely not within my lifetime. See the "Discussion" section of the commit message for more information.

With a 32-bit counter, there is a very (very) low likelihood of it, but it is possible. Personally, I don't feel comfortable providing such code, because a) I doubt all users would understand the implementation well enough to do the risk/reward analysis, and
b) such a bug would be near impossible to reproduce and root-cause if it did occur.

Quoted 93 lines
Quoted 7 lines

With a 64-bit counter (and 32-bit pointer), 32-bit architectures (e.g. ARMv7a and
probably x86 as well) won't be able to support this as they at best support 64-bit CAS (ARMv7a has LDREXD/STREXD). So you are essentially putting a 64-bit (and 128-bit CAS) requirement on the implementation.

Yes, I am. I tried to make that clear in the cover letter.

Quoted 84 lines
Quoted 84 lines
Quoted 68 lines
Quoted 58 lines
(quoted diff) +31 -0
+};
+
+/* The non-blocking ring algorithm is based on the original
+rte ring (derived
+ * from FreeBSD's bufring.h) and inspired by Michael and
+Scott's non-blocking
+ * concurrent queue.
+ */
+
+/**
+ * @internal
+ *   Enqueue several objects on the non-blocking ring
+(single-producer only)
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items
+to the ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as
+possible to the ring
+ * @param free_space
+ *   returns the amount of space after the enqueue operation
+has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n

only.
+ */

(quoted diff) +2 -0
+static __rte_always_inline unsigned int
+__rte_ring_do_nb_enqueue_sp(struct rte_ring *r, void * const

*obj_table,

(quoted diff) +2 -0
+			    unsigned int n,
+			    enum rte_ring_queue_behavior

behavior,

(quoted diff) +6 -0
+			    unsigned int *free_space) {
+	uint32_t free_entries;
+	size_t head, next;
+
+	n = __rte_ring_move_prod_head_64(r, 1, n, behavior,
+					 &head, &next,

&free_entries);

Review

Quoted 6 lines
(quoted diff) +6 -0
+	if (n == 0)
+		goto end;
+
+	ENQUEUE_PTRS_NB(r, &r[1], head, obj_table, n);
+
+	r->prod_64.tail += n;

Don't we need release order when (or smp_wmb between) writing of the ring pointers and the update of tail? By updating the tail pointer, we are synchronising with a consumer.

Quoted 7 lines

I prefer using __atomic operations even for load and store. You can see which parts of the code that synchronise with each other, e.g.
store-release to some location synchronises with load-acquire from the same location. If you don't know how different threads synchronise with each other, you are very likely to make mistakes.

You can tell this code was written when I thought x86-64 was the only viable target :). Yes, you are correct.

With regards to using __atomic intrinsics, I'm planning on taking a similar approach to the functions duplicated in rte_ring_generic.h and
rte_ring_c11_mem.h: one version that uses rte_atomic functions (and thus stricter memory ordering) and one that uses __atomic intrinsics (and thus can benefit from more relaxed memory ordering).

Quoted 12 lines

From a code point of view, I strongly prefer the atomic operations to be visible
in the top level code, not hidden in subroutines. For correctness, it is vital that
memory accesses are performed with the required ordering and that acquire and
release matches up. Hiding e.g. load-acquire and store-release in subroutines (in
a different file!) make this difficult. There have already been such bugs found in
rte_ring.

Quoted 2 lines

After working on the acq/rel ordering this weekend, I agree. This'll be easier/cleaner if we end up only using the C11 version.

Fabulous!

As I wrote in a response to Jerin, with a small cheat (LoadStore fence+store- relaxed instead of store-release in the dequeue function where we only read shared data in the critical section), C11 should provide the same ordering and thus the same performance as the explicit barrier version. Benchmarking will show.

Quoted 63 lines
Quoted 52 lines
Quoted 14 lines
Quoted 7 lines

What's the advantage of having two different implementations? What is the disadvantage?

Quoted 7 lines

The existing ring buffer code originally had only the "legacy" implementation
which was kept when the __atomic implementation was added. The reason claimed was that some older compilers for x86 do not support GCC __atomic builtins. But I thought there was consensus that new functionality could have only __atomic implementations.

Quoted 4 lines

When CONFIG_RTE_RING_USE_C11_MEM_MODEL was introduced, it was left disabled for thunderx[1] for performance reasons. Assuming that hasn't changed, the advantage to having two versions is to best support all of DPDK's

platforms.

Quoted 3 lines

The disadvantage is of course duplicated code and the additional maintenance burden.

The only way I see that a C11 memory model implementation can be slower than using smp_wmb/rmb is if you need to order loads before a synchronizing store and there are also outstanding stores which do not require ordering. smp_rmb() handles this while store-release will also (unnecessarily) order those
outstanding stores. This situation occurs e.g. in ring buffer dequeue operations
where ring slots are read (and possibly written to thread-private memory) before
the ring slots are release (e.g. using CAS-release or store-release).

I imagine that the LSU/cache subsystem on ThunderX/OCTEON-TX also have something to do with this problem. If there are a large amounts of stores pending in the load/store unit, store-release might have to wait for a long time
before the synchronizing store can complete.

Quoted 7 lines

That said, if the thunderx maintainers are ok with it, I'm certainly open to only doing the __atomic version. Note that even in the __atomic version, based on Honnapa's findings[2], using a DPDK-defined rte_atomic128_cmpset() (with additional arguments to support machines with weak consistency) appears to be a better option than

__atomic_compare_exchange_16.
__atomic_compare_exchange_16() is not guaranteed to be lock-free. It is not lock-free on ARM/AArch64 and the support in GCC is formally broken (can't use
cmpexchg16b to implement __atomic_load_16).

Quoted 11 lines

So yes, I think DPDK will have to define and implement the 128-bit atomic compare and exchange operation (whatever it will be called). For compatibility
with ARMv8.0, we can't require the "old" value returned by a failed compare- exchange operation to be read atomically (LDXP does not guaranteed atomicity by itself). But this is seldom a problem, many designs read the memory location
using two separate 64-bit loads (so not atomic) anyway, it is a successful atomic
compare exchange operation which provides atomicity.

Quoted 5 lines

Ok. I agree, I don't expect that to be a problem. The 128-bit CAS patch I just submitted[1] (which was developed before reading this) will have to be changed.

[1] http://mails.dpdk.org/archives/dev/2019-January/124159.html

I will take a look and comnment on this.

Quoted 5 lines

Thanks,
Gage

</snip>

Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
#42 lore raw

Re: [PATCH v3 2/5] ring: add a non-blocking implementation

In reply to Eads, Gage
Quoted 41 lines · Gage
Quoted 30 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Wednesday, January 23, 2019 4:16 AM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 2/5] ring: add a non-blocking implementation

s.

Quoted 5 lines
Quoted 1 line

You can tell this code was written when I thought x86-64 was the only
viable target :). Yes, you are correct.

Quoted 9 lines

With regards to using __atomic intrinsics, I'm planning on taking a
similar approach to the functions duplicated in rte_ring_generic.h and
rte_ring_c11_mem.h: one version that uses rte_atomic functions (and
thus stricter memory ordering) and one that uses __atomic intrinsics
(and thus can benefit from more relaxed memory ordering).

What's the advantage of having two different implementations? What is the
disadvantage?

Quoted 10 lines

The existing ring buffer code originally had only the "legacy" implementation
which was kept when the __atomic implementation was added. The reason
claimed was that some older compilers for x86 do not support GCC __atomic
builtins. But I thought there was consensus that new functionality could have
only __atomic implementations.

Quoted 7 lines

When CONFIG_RTE_RING_USE_C11_MEM_MODEL was introduced, it was left disabled for thunderx[1] for performance reasons. Assuming that hasn't changed, the advantage to having two versions is to best support all of DPDK's platforms. The disadvantage is of course duplicated code and the additional maintenance burden.

That said, if the thunderx maintainers are ok with it, I'm certainly

The ring code was so fundamental building block for DPDK, there was difference in performance and there was already legacy code so introducing C11_MEM_MODEL was justified IMO.

For the nonblocking implementation, I am happy to test with three ARM64 microarchitectures and share the result with C11_MEM_MODEL vs non C11_MEM_MODLE performance. We may need to consider PPC also here. So IMO, based on the overall performance result may be can decide the new code direction.

#43 lore raw

Re: [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 51 lines · Jerin Jacob Kollanukkaran
Quoted 50 lines · Gage
Quoted 34 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Wednesday, January 23, 2019 4:16 AM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 2/5] ring: add a non-blocking implementation

s.

Quoted 7 lines
Quoted 3 lines

You can tell this code was written when I thought x86-64 was the only
viable target :). Yes, you are correct.

Quoted 9 lines

With regards to using __atomic intrinsics, I'm planning on taking a
similar approach to the functions duplicated in rte_ring_generic.h and
rte_ring_c11_mem.h: one version that uses rte_atomic functions (and
thus stricter memory ordering) and one that uses __atomic intrinsics
(and thus can benefit from more relaxed memory ordering).

What's the advantage of having two different implementations? What is the
disadvantage?

Quoted 10 lines

The existing ring buffer code originally had only the "legacy" implementation
which was kept when the __atomic implementation was added. The reason
claimed was that some older compilers for x86 do not support GCC __atomic
builtins. But I thought there was consensus that new functionality could have
only __atomic implementations.

When CONFIG_RTE_RING_USE_C11_MEM_MODEL was introduced, it was left disabled for thunderx[1] for performance reasons. Assuming that hasn't changed, the advantage to having two versions is to best support all of DPDK's platforms. The disadvantage is of course duplicated code and the additional maintenance burden.

Quoted 8 lines
Quoted 1 line

That said, if the thunderx maintainers are ok with it, I'm certainly

The ring code was so fundamental building block for DPDK, there was difference in performance and there was already legacy code so introducing C11_MEM_MODEL was justified IMO.

For the nonblocking implementation, I am happy to test with three ARM64 microarchitectures and share the result with C11_MEM_MODEL vs non C11_MEM_MODLE performance.

We should ensure the C11 memory model version enforces minimal ordering requirements:
1) when computing number of available slots, allow for underflow (head and tail observed in unexpected order) instead of imposing read order with an additional read barrier.
2) We could cheat a little and use an explicit LoadStore barrier instead of store-release/cas-release in dequeue (which only reads the ring). At least see if this improves performance. See such a patch here:
https://github.com/ARM-software/progress64/commit/84c48e9c84100eb5b2d15e54f0dbf7
8dfa468805

Ideally, C/C++ would have an __ATOMIC_RELEASE_READSONLY memory model to use in situations where the shared data was only read before being released.

Quoted 3 lines

We may need to consider PPC also
here. So IMO, based on the overall performance result may be can decide the new code direction.

Does PPC (64-bit POWER?) have support for double-word (128-bit) CAS?

Quoted 1 line
Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
#44 lore raw

Re: [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 62 lines · Ola Liljedahl
Quoted 61 lines · Jerin Jacob Kollanukkaran
Quoted 53 lines · Gage
Quoted 34 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Wednesday, January 23, 2019 4:16 AM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 2/5] ring: add a non-blocking implementation

s.

Quoted 6 lines
Quoted 1 line

You can tell this code was written when I thought x86-64 was the
only
viable target :). Yes, you are correct.

Quoted 11 lines

With regards to using __atomic intrinsics, I'm planning on taking
a
similar approach to the functions duplicated in rte_ring_generic.h and
rte_ring_c11_mem.h: one version that uses rte_atomic functions
(and
thus stricter memory ordering) and one that uses __atomic intrinsics
(and thus can benefit from more relaxed memory ordering).

What's the advantage of having two different implementations? What
is the
disadvantage?

Quoted 12 lines

The existing ring buffer code originally had only the "legacy" implementation
which was kept when the __atomic implementation was added. The reason
claimed was that some older compilers for x86 do not support GCC
__atomic
builtins. But I thought there was consensus that new functionality
could have
only __atomic implementations.

When CONFIG_RTE_RING_USE_C11_MEM_MODEL was introduced, it was left
disabled for thunderx[1] for performance reasons. Assuming that hasn't changed, the advantage to having two versions is to best support all of DPDK's platforms. The disadvantage is of course duplicated code and the additional maintenance burden.

Quoted 2 lines

That said, if the thunderx maintainers are ok with it, I'm certainly

The ring code was so fundamental building block for DPDK, there was
difference in performance and there was already legacy code so introducing C11_MEM_MODEL was justified IMO.

Quoted 7 lines
Quoted 4 lines

For the nonblocking implementation, I am happy to test with three ARM64 microarchitectures and share the result with C11_MEM_MODEL
vs non C11_MEM_MODLE performance.

We should ensure the C11 memory model version enforces minimal ordering
requirements:

I agree.

I think, We should have enough test case for performance measurement in order to choose algorithms and quantify the other variables like C11 vs non C11, LDXP/STXP vs CASP etc.

Quoted 17 lines

1) when computing number of available slots, allow for underflow (head and tail
observed in unexpected order) instead of imposing read order with an additional
read barrier.
2) We could cheat a little and use an explicit LoadStore barrier instead of
store-release/cas-release in dequeue (which only reads the ring). At
least see
if this improves performance. See such a patch here:
https://github.com/ARM-software/progress64/commit/84c48e9c84100eb5b2d15e54f0dbf7
8dfa468805

Ideally, C/C++ would have an __ATOMIC_RELEASE_READSONLY memory model to use in
situations where the shared data was only read before being released.

Quoted 5 lines
Quoted 4 lines

We may need to consider PPC also
here. So IMO, based on the overall performance result may be can decide
the new code direction.

Does PPC (64-bit POWER?) have support for double-word (128-bit) CAS?

I dont know, I was telling wrt in general C11 mem model for PPC.

Quoted 5 lines
Signature · 3 lines
-- 
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
#45 lore raw

Re: [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 3 lines · Jerin Jacob Kollanukkaran
Quoted 1 line

Does PPC (64-bit POWER?) have support for double-word (128-bit) CAS?

I dont know, I was telling wrt in general C11 mem model for PPC.

Sorry, I misunderstood.

Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
author
#46 lore raw

Re: [PATCH v3 2/5] ring: add a non-blocking implementation

In reply to Jerin Jacob Kollanukkaran
Quoted 59 lines

-----Original Message-----
From: Jerin Jacob Kollanukkaran [mailto:[email protected]]
Sent: Monday, January 28, 2019 7:34 AM
To: [email protected]; Maciej Czekaj <[email protected]>; Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 2/5] ring: add a non-blocking implementation

Quoted 34 lines · Gage
Quoted 26 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Wednesday, January 23, 2019 4:16 AM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 2/5] ring: add a non-blocking implementation

s.

Quoted 11 lines
Quoted 1 line

You can tell this code was written when I thought x86-64 was the only viable target :). Yes, you are correct.

With regards to using __atomic intrinsics, I'm planning on taking a similar approach to the functions duplicated in rte_ring_generic.h and
rte_ring_c11_mem.h: one version that uses rte_atomic functions (and thus stricter memory ordering) and one that uses __atomic intrinsics (and thus can benefit from more relaxed memory ordering).

What's the advantage of having two different implementations? What is the disadvantage?

Quoted 7 lines

The existing ring buffer code originally had only the "legacy" implementation
which was kept when the __atomic implementation was added. The reason claimed was that some older compilers for x86 do not support GCC __atomic builtins. But I thought there was consensus that new functionality could have only __atomic implementations.

Quoted 7 lines

When CONFIG_RTE_RING_USE_C11_MEM_MODEL was introduced, it was left disabled for thunderx[1] for performance reasons. Assuming that hasn't changed, the advantage to having two versions is to best support all of DPDK's platforms. The disadvantage is of course duplicated code and the additional maintenance burden.

That said, if the thunderx maintainers are ok with it, I'm certainly

The ring code was so fundamental building block for DPDK, there was difference in performance and there was already legacy code so introducing C11_MEM_MODEL was justified IMO.

Quoted 5 lines

For the nonblocking implementation, I am happy to test with three ARM64 microarchitectures and share the result with C11_MEM_MODEL vs non C11_MEM_MODLE performance. We may need to consider PPC also here. So IMO, based on the overall performance result may be can decide the new code direction.

Appreciate the help. Please hold off any testing until we've had a chance to incorporate ideas from lfring, which will definitely affect performance.

author
#47 lore raw

PATCH v3 3/5 test_ring: add non-blocking ring autotest

In reply to Gage Eads

ring_nb_autotest re-uses the ring_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 test/test/test_ring.c | 57 ++++++++++++++++++++++++++++++++-------------------
 1 file changed, 36 insertions(+), 21 deletions(-)
test/test/test_ring.c +36 -21
index aaf1e70ad..ff410d978 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 #include <string.h>
@@ -601,18 +601,20 @@ test_ring_burst_basic(struct rte_ring *r)
  * it will always fail to create ring with a wrong ring size number in this function
  */
 static int
-test_ring_creation_with_wrong_size(void)
+test_ring_creation_with_wrong_size(unsigned int flags)
 {
 	struct rte_ring * rp = NULL;
 
 	/* Test if ring size is not power of 2 */
-	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1,
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
 
 	/* Test if ring size is exceeding the limit */
-	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1), SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1),
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
@@ -623,11 +625,11 @@ test_ring_creation_with_wrong_size(void)
  * it tests if it would always fail to create ring with an used ring name
  */
 static int
-test_ring_creation_with_an_used_name(void)
+test_ring_creation_with_an_used_name(unsigned int flags)
 {
 	struct rte_ring * rp;
 
-	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (NULL != rp)
 		return -1;
 
@@ -639,10 +641,10 @@ test_ring_creation_with_an_used_name(void)
  * function to fail correctly
  */
 static int
-test_create_count_odd(void)
+test_create_count_odd(unsigned int flags)
 {
 	struct rte_ring *r = rte_ring_create("test_ring_count",
-			4097, SOCKET_ID_ANY, 0 );
+			4097, SOCKET_ID_ANY, flags);
 	if(r != NULL){
 		return -1;
 	}
@@ -665,7 +667,7 @@ test_lookup_null(void)
  * it tests some more basic ring operations
  */
 static int
-test_ring_basic_ex(void)
+test_ring_basic_ex(unsigned int flags)
 {
 	int ret = -1;
 	unsigned i;
@@ -679,7 +681,7 @@ test_ring_basic_ex(void)
 	}
 
 	rp = rte_ring_create("test_ring_basic_ex", RING_SIZE, SOCKET_ID_ANY,
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (rp == NULL) {
 		printf("test_ring_basic_ex fail to create ring\n");
 		goto fail_test;
@@ -737,7 +739,7 @@ test_ring_basic_ex(void)
 }
 
 static int
-test_ring_with_exact_size(void)
+test_ring_with_exact_size(unsigned int flags)
 {
 	struct rte_ring *std_ring = NULL, *exact_sz_ring = NULL;
 	void *ptr_array[16];
@@ -746,13 +748,13 @@ test_ring_with_exact_size(void)
 	int ret = -1;
 
 	std_ring = rte_ring_create("std", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (std_ring == NULL) {
 		printf("%s: error, can't create std ring\n", __func__);
 		goto end;
 	}
 	exact_sz_ring = rte_ring_create("exact sz", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ);
+		RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ | flags);
 	if (exact_sz_ring == NULL) {
 		printf("%s: error, can't create exact size ring\n", __func__);
 		goto end;
@@ -808,17 +810,17 @@ test_ring_with_exact_size(void)
 }
 
 static int
-test_ring(void)
+__test_ring(unsigned int flags)
 {
 	struct rte_ring *r = NULL;
 
 	/* some more basic operations */
-	if (test_ring_basic_ex() < 0)
+	if (test_ring_basic_ex(flags) < 0)
 		goto test_fail;
 
 	rte_atomic32_init(&synchro);
 
-	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (r == NULL)
 		goto test_fail;
 
@@ -837,27 +839,27 @@ test_ring(void)
 		goto test_fail;
 
 	/* basic operations */
-	if ( test_create_count_odd() < 0){
+	if (test_create_count_odd(flags) < 0) {
 		printf("Test failed to detect odd count\n");
 		goto test_fail;
 	} else
 		printf("Test detected odd count\n");
 
-	if ( test_lookup_null() < 0){
+	if (test_lookup_null() < 0) {
 		printf("Test failed to detect NULL ring lookup\n");
 		goto test_fail;
 	} else
 		printf("Test detected NULL ring lookup\n");
 
 	/* test of creating ring with wrong size */
-	if (test_ring_creation_with_wrong_size() < 0)
+	if (test_ring_creation_with_wrong_size(flags) < 0)
 		goto test_fail;
 
 	/* test of creation ring with an used name */
-	if (test_ring_creation_with_an_used_name() < 0)
+	if (test_ring_creation_with_an_used_name(flags) < 0)
 		goto test_fail;
 
-	if (test_ring_with_exact_size() < 0)
+	if (test_ring_with_exact_size(flags) < 0)
 		goto test_fail;
 
 	/* dump the ring status */
@@ -873,4 +875,17 @@ test_ring(void)
 	return -1;
 }
 
+static int
+test_ring(void)
+{
+	return __test_ring(0);
+}
+
+static int
+test_nb_ring(void)
+{
+	return __test_ring(RING_F_NB);
+}
+
 REGISTER_TEST_COMMAND(ring_autotest, test_ring);
+REGISTER_TEST_COMMAND(ring_nb_autotest, test_nb_ring);
Signature · 1 line
2.13.6
author
#48 lore raw

PATCH v3 4/5 test_ring_perf: add non-blocking ring perf test

In reply to Gage Eads

nb_ring_perf_autotest re-uses the ring_perf_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 test/test/test_ring_perf.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
test/test/test_ring_perf.c +16 -3
index ebb3939f5..380c4b4a1 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 
@@ -363,12 +363,12 @@ test_bulk_enqueue_dequeue(struct rte_ring *r)
 }
 
 static int
-test_ring_perf(void)
+__test_ring_perf(unsigned int flags)
 {
 	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
 
-	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), 0);
+	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), flags);
 	if (r == NULL)
 		return -1;
 
@@ -398,4 +398,17 @@ test_ring_perf(void)
 	return 0;
 }
 
+static int
+test_ring_perf(void)
+{
+	return __test_ring_perf(0);
+}
+
+static int
+test_nb_ring_perf(void)
+{
+	return __test_ring_perf(RING_F_NB);
+}
+
 REGISTER_TEST_COMMAND(ring_perf_autotest, test_ring_perf);
+REGISTER_TEST_COMMAND(ring_nb_perf_autotest, test_nb_ring_perf);
Signature · 1 line
2.13.6
author
#49 lore raw

PATCH v3 5/5 mempool/ring: add non-blocking ring handlers

In reply to Gage Eads

These handlers allow an application to create a mempool based on the non-blocking ring, with any combination of single/multi producer/consumer.

Also, add a note to the programmer's guide's "known issues" section.

---
 doc/guides/prog_guide/env_abstraction_layer.rst |  2 +-
 drivers/mempool/ring/Makefile                   |  1 +
 drivers/mempool/ring/meson.build                |  2 +
 drivers/mempool/ring/rte_mempool_ring.c         | 58 +++++++++++++++++++++++--
 4 files changed, 59 insertions(+), 4 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +1 -1
index 9497b879c..b6ac236d6 100644
@@ -541,7 +541,7 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
-  Alternatively, x86_64 applications can use the non-blocking stack mempool handler. When considering this handler, note that:
+  Alternatively, x86_64 applications can use the non-blocking ring or stack mempool handlers. When considering one of them, note that:
 
   - it is limited to the x86_64 platform, because it uses an instruction (16-byte compare-and-swap) that is not available on other platforms.
   - it has worse average-case performance than the non-preemptive rte_ring, but software caching (e.g. the mempool cache) can mitigate this by reducing the number of handler operations.
drivers/mempool/ring/Makefile +1 -0
index ddab522fe..012ba6966 100644
@@ -10,6 +10,7 @@ LIB = librte_mempool_ring.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 LDLIBS += -lrte_eal -lrte_mempool -lrte_ring
 
 EXPORT_MAP := rte_mempool_ring_version.map
drivers/mempool/ring/meson.build +2 -0
index a021e908c..b1cb673cc 100644
@@ -1,4 +1,6 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
+allow_experimental_apis = true
+
 sources = files('rte_mempool_ring.c')
drivers/mempool/ring/rte_mempool_ring.c +55 -3

84 lines not on this page · raw message

Signature · 1 line
2.13.6
#50 lore raw
In reply to Gage Eads
Quoted 50 lines · Gage Eads

For some users, the rte ring's "non-preemptive" constraint is not acceptable;
for example, if the application uses a mixture of pinned high- priority threads
and multiplexed low-priority threads that share a mempool.

This patchset introduces a non-blocking ring, on top of which a mempool can run.
Crucially, the non-blocking algorithm relies on a 128-bit compare- and-swap, so
it is currently limited to x86_64 machines. This is also an experimental API,
so RING_F_NB users must build with the ALLOW_EXPERIMENTAL_API flag.

The ring uses more compare-and-swap atomic operations than the regular rte ring:
With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a
dequeue of n pointers uses 2. This algorithm has worse average-case performance
than the regular rte ring (particularly a highly-contended ring with large bulk
accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case
performance (i.e. one thread being preempted in the update_tail()
critical
section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based
algorithms. For example, a non-blocking ring based mempool (a
likely use case
for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. For ease- of-use,
existing ring enqueue/dequeue functions work with both "regular" and non-blocking rings.

This patchset also adds non-blocking versions of ring_autotest and ring_perf_autotest, and a non-blocking ring based mempool.

This patchset makes one API change; a deprecation notice will be posted in a
separate commit.

This patchset depends on the non-blocking stack patchset[1].

[1] http://mails.dpdk.org/archives/dev/2019-January/123653.html

Quoted 14 lines

v3:
- Avoid the ABI break by putting 64-bit head and tail values in the
same
cacheline as struct rte_ring's prod and cons members.
- Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of
libraries.
- Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case
someone tries
to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
- Update the ring mempool to use experimental APIs
- Clarify that RINB_F_NB is only limited to x86_64 currently;
ARMv8.1-A builds
can eventually support it with the CASP instruction.

ARMv8.0 should be able to implement a 128-bit atomic compare exchange operation using LDXP/STXP.

From an ARM perspective, I want all atomic operations to take memory ordering arguments (e.g. acquire, release). Not all usages of e.g. atomic compare exchange require sequential consistency (which I think what x86 cmpxchg instruction provides). DPDK functions should not be modelled after x86 behaviour.

Lock-free 128-bit atomics implementations for ARM/AArch64 and x86-64 are available here:
https://github.com/ARM-software/progress64/blob/master/src/lockfree.h

Quoted 16 lines

v2:
- Merge separate docs commit into patch #5
- Convert uintptr_t to size_t
- Add a compile-time check for the size of size_t
- Fix a space-after-typecast issue
- Fix an unnecessary-parentheses checkpatch warning
- Bump librte_ring's library version

Gage Eads (5):
ring: add 64-bit headtail structure
ring: add a non-blocking implementation
test_ring: add non-blocking ring autotest
test_ring_perf: add non-blocking ring perf test
mempool/ring: add non-blocking ring handlers

Quoted 16 lines
 doc/guides/prog_guide/env_abstraction_layer.rst |   2 +-
 drivers/mempool/ring/Makefile                   |   1 +
 drivers/mempool/ring/meson.build                |   2 +
 drivers/mempool/ring/rte_mempool_ring.c         |  58 ++-
 lib/librte_eventdev/rte_event_ring.h            |   2 +-
 lib/librte_ring/Makefile                        |   3 +-
 lib/librte_ring/rte_ring.c                      |  72 ++-
 lib/librte_ring/rte_ring.h                      | 574
++++++++++++++++++++++--
 lib/librte_ring/rte_ring_generic_64.h           | 152 +++++++
 lib/librte_ring/rte_ring_version.map            |   7 +
 test/test/test_ring.c                           |  57 ++-
 test/test/test_ring_perf.c                      |  19 +-
 12 files changed, 874 insertions(+), 75 deletions(-)
 create mode 100644 lib/librte_ring/rte_ring_generic_64.h

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#51 lore raw

Sorry about the confidental footer. I tried to remove it using some Exhange magic but it seems not to work with Evolution. I'll try some other way.

-- Ola

Quoted 8 lines · Ola Liljedahl
Quoted 7 lines · Gage Eads

For some users, the rte ring's "non-preemptive" constraint is not acceptable;
for example, if the application uses a mixture of pinned high- priority threads
and multiplexed low-priority threads that share a mempool.

I. This patchset introduces a non-blocking ring, on top of which a

Quoted 103 lines
Quoted 43 lines

mempool can run.
Crucially, the non-blocking algorithm relies on a 128-bit compare- and-swap, so
it is currently limited to x86_64 machines. This is also an experimental API,
so RING_F_NB users must build with the ALLOW_EXPERIMENTAL_API flag.

The ring uses more compare-and-swap atomic operations than the regular rte ring:
With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a
dequeue of n pointers uses 2. This algorithm has worse average-case performance
than the regular rte ring (particularly a highly-contended ring with large bulk
accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case
performance (i.e. one thread being preempted in the update_tail()
critical
section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based
algorithms. For example, a non-blocking ring based mempool (a
likely use case
for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. For ease- of-use,
existing ring enqueue/dequeue functions work with both "regular" and non-blocking rings.

This patchset also adds non-blocking versions of ring_autotest and ring_perf_autotest, and a non-blocking ring based mempool.

This patchset makes one API change; a deprecation notice will be posted in a
separate commit.

This patchset depends on the non-blocking stack patchset[1].

[1] http://mails.dpdk.org/archives/dev/2019-January/123653.html

Quoted 14 lines

v3:
- Avoid the ABI break by putting 64-bit head and tail values in the
same
cacheline as struct rte_ring's prod and cons members.
- Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of
libraries.
- Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case
someone tries
to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
- Update the ring mempool to use experimental APIs
- Clarify that RINB_F_NB is only limited to x86_64 currently;
ARMv8.1-A builds
can eventually support it with the CASP instruction.

ARMv8.0 should be able to implement a 128-bit atomic compare exchange operation using LDXP/STXP.

From an ARM perspective, I want all atomic operations to take memory ordering arguments (e.g. acquire, release). Not all usages of e.g. atomic compare exchange require sequential consistency (which I think what x86 cmpxchg instruction provides). DPDK functions should not be modelled after x86 behaviour.

Lock-free 128-bit atomics implementations for ARM/AArch64 and x86-64 are available here:
https://github.com/ARM-software/progress64/blob/master/src/lockfree.h

Quoted 17 lines

v2:
- Merge separate docs commit into patch #5
- Convert uintptr_t to size_t
- Add a compile-time check for the size of size_t
- Fix a space-after-typecast issue
- Fix an unnecessary-parentheses checkpatch warning
- Bump librte_ring's library version

Gage Eads (5):
ring: add 64-bit headtail structure
ring: add a non-blocking implementation
test_ring: add non-blocking ring autotest
test_ring_perf: add non-blocking ring perf test
mempool/ring: add non-blocking ring handlers

Quoted 16 lines
 doc/guides/prog_guide/env_abstraction_layer.rst |   2 +-
 drivers/mempool/ring/Makefile                   |   1 +
 drivers/mempool/ring/meson.build                |   2 +
 drivers/mempool/ring/rte_mempool_ring.c         |  58 ++-
 lib/librte_eventdev/rte_event_ring.h            |   2 +-
 lib/librte_ring/Makefile                        |   3 +-
 lib/librte_ring/rte_ring.c                      |  72 ++-
 lib/librte_ring/rte_ring.h                      | 574
++++++++++++++++++++++--
 lib/librte_ring/rte_ring_generic_64.h           | 152 +++++++
 lib/librte_ring/rte_ring_version.map            |   7 +
 test/test/test_ring.c                           |  57 ++-
 test/test/test_ring_perf.c                      |  19 +-
 12 files changed, 874 insertions(+), 75 deletions(-)
 create mode 100644 lib/librte_ring/rte_ring_generic_64.h
Quoted 5 lines

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

author
#52 lore raw
In reply to Ola Liljedahl
Quoted 69 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Tuesday, January 22, 2019 3:28 AM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Ananyev, Konstantin <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 0/5] Add non-blocking ring

Quoted 43 lines · Gage Eads

For some users, the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high- priority threads and multiplexed low-priority threads that share a mempool.

This patchset introduces a non-blocking ring, on top of which a mempool can run.
Crucially, the non-blocking algorithm relies on a 128-bit compare- and-swap, so it is currently limited to x86_64 machines. This is also an experimental API, so RING_F_NB users must build with the ALLOW_EXPERIMENTAL_API flag.

The ring uses more compare-and-swap atomic operations than the regular rte ring:
With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case
performance (i.e. one thread being preempted in the update_tail()
critical
section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based
algorithms. For example, a non-blocking ring based mempool (a likely
use case
for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. For ease- of-use, existing ring enqueue/dequeue functions work with both "regular" and non-blocking rings.

This patchset also adds non-blocking versions of ring_autotest and ring_perf_autotest, and a non-blocking ring based mempool.

This patchset makes one API change; a deprecation notice will be posted in a separate commit.

This patchset depends on the non-blocking stack patchset[1].

[1] http://mails.dpdk.org/archives/dev/2019-January/123653.html

Quoted 14 lines

v3:
- Avoid the ABI break by putting 64-bit head and tail values in the
same
cacheline as struct rte_ring's prod and cons members.
- Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of
libraries.
- Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case
someone tries
to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
- Update the ring mempool to use experimental APIs
- Clarify that RINB_F_NB is only limited to x86_64 currently;
ARMv8.1-A builds
can eventually support it with the CASP instruction.

Quoted 2 lines

ARMv8.0 should be able to implement a 128-bit atomic compare exchange operation using LDXP/STXP.

I see, I wasn't aware these instructions were available.

Quoted 11 lines

From an ARM perspective, I want all atomic operations to take memory ordering arguments (e.g. acquire, release). Not all usages of e.g.
atomic compare exchange require sequential consistency (which I think what x86 cmpxchg instruction provides). DPDK functions should not be modelled after x86 behaviour.

Lock-free 128-bit atomics implementations for ARM/AArch64 and x86-64 are available here:
https://github.com/ARM-software/progress64/blob/master/src/lockfree.h

Sure, I'll address this in the next patchset.

Thanks,
Gage

#53 lore raw
In reply to Ola Liljedahl
Quoted 16 lines · Ola Liljedahl
Quoted 7 lines · Gage Eads

v3:
- Avoid the ABI break by putting 64-bit head and tail values in
the
same
cacheline as struct rte_ring's prod and cons members.
- Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of

Quoted 8 lines

libraries.
- Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case
someone tries
to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
- Update the ring mempool to use experimental APIs
- Clarify that RINB_F_NB is only limited to x86_64 currently;
ARMv8.1-A builds
can eventually support it with the CASP instruction.

Quoted 2 lines

ARMv8.0 should be able to implement a 128-bit atomic compare exchange operation using LDXP/STXP.

Just wondering what would the performance difference between CASP vs LDXP/STXP on LSE supported machine?

I think, We can not detect the presese of LSE support in compile time. Right?

The dynamic one will be costly like,

if (hwcaps & HWCAP_ATOMICS) {
casp
} else {
ldxp
stxp
}

Quoted 2 lines

From an ARM perspective, I want all atomic operations to take memory ordering arguments (e.g. acquire, release). Not all usages of e.g.

+1

Quoted 8 lines

atomic compare exchange require sequential consistency (which I think what x86 cmpxchg instruction provides). DPDK functions should not be modelled after x86 behaviour.

Lock-free 128-bit atomics implementations for ARM/AArch64 and x86-64 are available here:
https://github.com/ARM-software/progress64/blob/master/src/lockfree.h

#54 lore raw
Quoted 19 lines · Jerin Jacob Kollanukkaran
Quoted 18 lines · Ola Liljedahl
Quoted 16 lines · Gage Eads

v3:
 - Avoid the ABI break by putting 64-bit head and tail values in the
same
   cacheline as struct rte_ring's prod and cons members.
 - Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of libraries.
 - Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case someone tries
   to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
 - Update the ring mempool to use experimental APIs
 - Clarify that RINB_F_NB is only limited to x86_64 currently; ARMv8.1-A builds
   can eventually support it with the CASP instruction.

Quoted 4 lines
Quoted 2 lines

ARMv8.0 should be able to implement a 128-bit atomic compare exchange operation using LDXP/STXP.

Just wondering what would the performance difference between CASP vs LDXP/STXP on LSE supported machine?

I think that is up to the microarchitecture. But one the ideas behind introducing the LSE atomics was that they should be "better" than the equivalent code using exclusives. I think non-conditional LDxxx and STxxx atomics could be better than using exclusives while conditional atomics (CAS, CASP) might not be so different (the reason has to do with cache coherency, a core can speculatively snoop-unique the cache line which is targetted by an atomic instruction but to what extent that provides a benefit could be depend on whether the atomic actually performs a store or not).

Quoted 3 lines

I think, We can not detect the presese of LSE support in compile time. Right?

Unfortunately, AFAIK GCC doesn't notify the source code that it is targetting v8.1+ with LSE support. If there were intrinsics for (certain) LSE instructions (e.g. those not generated by the compiler, e.g. STxxx and CASP), we could use some corresponding preprocessor define to detect the presence of such intrinsics (they exist for other intrinsics, e.g. __ARM_FEATURE_QRDMX for SQRDMLAH/SQRDMLSH instructions and corresponding intrinsics).

I have tried to interest the Arm GCC developers in this but have not yet succeeded. Perhaps if we have more use cases were atomics intrinsics would be useful, we could convince them to add such intrinsics to the ACLE (ARM C Language Extensions). But we will never get intrinsics for exclusives, they are deemed unsafe for explicit use from C. Instead need to provide inline assembler that contains the complete exclusives sequence. But in practice it seems to work with using inline assembler for LDXR and STXR as I do in the lockfree code linked below.

Quoted 2 lines

The dynamic one will be costly like,

Do you think so? Shouldn't this branch be perfectly predictable? Once in a while it will fall out of the branch history table but doesn't that mean the application hasn't been executing this code for some time so not really performance critical?

Quoted 22 lines

if (hwcaps & HWCAP_ATOMICS) {
casp
} else {
ldxp
stxp
}

Quoted 3 lines

From an ARM perspective, I want all atomic operations to take memory ordering arguments (e.g. acquire, release). Not all usages of e.g.

+1

Quoted 9 lines

atomic compare exchange require sequential consistency (which I think what x86 cmpxchg instruction provides). DPDK functions should not be modelled after x86 behaviour.

Lock-free 128-bit atomics implementations for ARM/AArch64 and x86-64 are available here:
https://github.com/ARM-software/progress64/blob/master/src/lockfree.h

Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
#55 lore raw
Quoted 25 lines · Ola Liljedahl

External Email

-------------------------------------------------------------------
---

Quoted 20 lines · Jerin Jacob Kollanukkaran
Quoted 19 lines · Ola Liljedahl
Quoted 18 lines · Gage Eads

v3:
- Avoid the ABI break by putting 64-bit head and tail values
in
the
same
cacheline as struct rte_ring's prod and cons members.
- Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number
of
libraries.
- Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in
case
someone tries
to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
- Update the ring mempool to use experimental APIs
- Clarify that RINB_F_NB is only limited to x86_64 currently;
ARMv8.1-A builds
can eventually support it with the CASP instruction.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

ARMv8.0 should be able to implement a 128-bit atomic compare exchange
operation using LDXP/STXP.

Just wondering what would the performance difference between CASP vs
LDXP/STXP on LSE supported machine?

I think that is up to the microarchitecture. But one the ideas behind

Yes. This is where things are getting little messy to have generic code where a lot of stuff is defined based on micro architecture/IMPLEMENTATION DEFINED as arm spec. Al least, I am dealing with three different micro archirectures now with a lot of difference. Including the arm cores and qualcomm cores there could around >6ish different micro archtectures.

Quoted 44 lines

introducing the LSE atomics was that they should be "better" than the equivalent
code using exclusives. I think non-conditional LDxxx and STxxx atomics could be
better than using exclusives while conditional atomics (CAS, CASP) might not be
so different (the reason has to do with cache coherency, a core can speculatively snoop-unique the cache line which is targetted by an atomic
instruction but to what extent that provides a benefit could be depend on
whether the atomic actually performs a store or not).

Quoted 3 lines

I think, We can not detect the presese of LSE support in compile time.
Right?

Unfortunately, AFAIK GCC doesn't notify the source code that it is targetting
v8.1+ with LSE support. If there were intrinsics for (certain) LSE instructions
(e.g. those not generated by the compiler, e.g. STxxx and CASP), we could use
some corresponding preprocessor define to detect the presence of such intrinsics
(they exist for other intrinsics, e.g. __ARM_FEATURE_QRDMX for SQRDMLAH/SQRDMLSH
instructions and corresponding intrinsics).

I have tried to interest the Arm GCC developers in this but have not yet
succeeded. Perhaps if we have more use cases were atomics intrinsics would be
useful, we could convince them to add such intrinsics to the ACLE (ARM C
Language Extensions). But we will never get intrinsics for exclusives, they are
deemed unsafe for explicit use from C. Instead need to provide inline assembler
that contains the complete exclusives sequence. But in practice it seems to work
with using inline assembler for LDXR and STXR as I do in the lockfree code
linked below.

Quoted 2 lines
Quoted 1 line

The dynamic one will be costly like,

Do you think so? Shouldn't this branch be perfectly predictable? Once

Not just branch predication. Right? Corresponding Load and need for more I cache etc.

I think, for the generic build we can have either run time detection or stick with LDXR/STXR.

We can give a compile time option for CASP based code so that for given micro architecture if it optimized it can make use of it.(Something we can easily expressed on meson build with MIDR value)

Quoted 35 lines

in a while
it will fall out of the branch history table but doesn't that mean the
application hasn't been executing this code for some time so not really
performance critical?

Quoted 24 lines

if (hwcaps & HWCAP_ATOMICS) {
casp
} else {
ldxp
stxp
}

Quoted 4 lines

From an ARM perspective, I want all atomic operations to take memory
ordering arguments (e.g. acquire, release). Not all usages of e.g.

+1

Quoted 11 lines

atomic compare exchange require sequential consistency (which I think
what x86 cmpxchg instruction provides). DPDK functions should not be
modelled after x86 behaviour.

Lock-free 128-bit atomics implementations for ARM/AArch64 and x86-64
are available here:
https://github.com/ARM-software/progress64/blob/master/src/lockfree.h

Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
#56 lore raw
In reply to Gage Eads

Hi Gage,
Thank you for this patch. Arm (Ola Liljedahl) had worked on a non-blocking ring algorithm. We were planning to add it to DPDK at some point this year. I am wondering if you would be open to take a look at the algorithm and collaborate?

I am yet to fully understand both the algorithms. But, Ola has reviewed your patch and can provide a quick overview of the differences here.

If you agree, we can send a RFC patch. You can review that and do performance benchmarking on your platforms. I can also benchmark your patch (may be once you fix the issue identified in __rte_ring_do_nb_enqueue_mp function?) on Arm platforms. May be we can end up with a better combined algorithm.

Hi Thomas/Bruce,
Please let me know if this is ok and if there is a better way to do this.

Thank you,
Honnappa

Quoted 92 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Gage Eads
Sent: Friday, January 18, 2019 9:23 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [dpdk-dev] [PATCH v3 0/5] Add non-blocking ring

For some users, the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

This patchset introduces a non-blocking ring, on top of which a mempool can run.
Crucially, the non-blocking algorithm relies on a 128-bit compare-and-swap, so it is currently limited to x86_64 machines. This is also an experimental API, so RING_F_NB users must build with the ALLOW_EXPERIMENTAL_API flag.

The ring uses more compare-and-swap atomic operations than the regular rte ring:
With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a non-blocking ring based mempool (a likely use
case
for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. For ease-of-use, existing ring enqueue/dequeue functions work with both "regular" and non- blocking rings.

This patchset also adds non-blocking versions of ring_autotest and ring_perf_autotest, and a non-blocking ring based mempool.

This patchset makes one API change; a deprecation notice will be posted in a separate commit.

This patchset depends on the non-blocking stack patchset[1].

[1] http://mails.dpdk.org/archives/dev/2019-January/123653.html

v3:
- Avoid the ABI break by putting 64-bit head and tail values in the same cacheline as struct rte_ring's prod and cons members.
- Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of libraries.
- Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case someone
tries
to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
- Update the ring mempool to use experimental APIs
- Clarify that RINB_F_NB is only limited to x86_64 currently; ARMv8.1-A
builds
can eventually support it with the CASP instruction.

v2:
- Merge separate docs commit into patch #5
- Convert uintptr_t to size_t
- Add a compile-time check for the size of size_t
- Fix a space-after-typecast issue
- Fix an unnecessary-parentheses checkpatch warning
- Bump librte_ring's library version

Gage Eads (5):
ring: add 64-bit headtail structure
ring: add a non-blocking implementation
test_ring: add non-blocking ring autotest
test_ring_perf: add non-blocking ring perf test
mempool/ring: add non-blocking ring handlers

 doc/guides/prog_guide/env_abstraction_layer.rst |   2 +-
 drivers/mempool/ring/Makefile                   |   1 +
 drivers/mempool/ring/meson.build                |   2 +
 drivers/mempool/ring/rte_mempool_ring.c         |  58 ++-
 lib/librte_eventdev/rte_event_ring.h            |   2 +-
 lib/librte_ring/Makefile                        |   3 +-
 lib/librte_ring/rte_ring.c                      |  72 ++-
 lib/librte_ring/rte_ring.h                      | 574 ++++++++++++++++++++++--
 lib/librte_ring/rte_ring_generic_64.h           | 152 +++++++
 lib/librte_ring/rte_ring_version.map            |   7 +
 test/test/test_ring.c                           |  57 ++-
 test/test/test_ring_perf.c                      |  19 +-
 12 files changed, 874 insertions(+), 75 deletions(-)  create mode 100644

lib/librte_ring/rte_ring_generic_64.h

--
2.13.6

author
#57 lore raw

Hi Honnappa,

Works for me -- I'm in favor of the best performing implementation, whoever provides it.

To allow an apples-to-apples comparison, I suggest Ola's/ARM's implementation be made to fit into the rte_ring API with an associated mempool handler. That'll allow us to use the existing ring and mempool performance tests as well. Feel free to use code from this patchset for the rte_ring integration, if that helps, of course.

I expect to have v4 available within the next week.

Thanks,
Gage

Quoted 128 lines

-----Original Message-----
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Thursday, January 24, 2019 11:21 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; nd <[email protected]>; [email protected]; Ola Liljedahl <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; Song Zhu (Arm Technology China) <[email protected]>; nd <[email protected]>
Subject: RE: [dpdk-dev] [PATCH v3 0/5] Add non-blocking ring

Hi Gage,
Thank you for this patch. Arm (Ola Liljedahl) had worked on a non-
blocking ring algorithm. We were planning to add it to DPDK at some point this year. I am wondering if you would be open to take a look at the algorithm and collaborate?

I am yet to fully understand both the algorithms. But, Ola has reviewed your patch and can provide a quick overview of the differences here.

If you agree, we can send a RFC patch. You can review that and do performance benchmarking on your platforms. I can also benchmark your patch (may be once you fix the issue identified in __rte_ring_do_nb_enqueue_mp function?) on Arm platforms. May be we can end up with a better combined algorithm.

Hi Thomas/Bruce,
Please let me know if this is ok and if there is a better way to do this.

Thank you,
Honnappa

Quoted 9 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Gage Eads
Sent: Friday, January 18, 2019 9:23 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [dpdk-dev] [PATCH v3 0/5] Add non-blocking ring

Quoted 3 lines

For some users, the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a

mempool.

Quoted 6 lines

This patchset introduces a non-blocking ring, on top of which a mempool can run.
Crucially, the non-blocking algorithm relies on a 128-bit compare-and-swap, so it is currently limited to x86_64 machines. This is also an experimental API, so RING_F_NB users must build with the

ALLOW_EXPERIMENTAL_API flag.

Quoted 75 lines

The ring uses more compare-and-swap atomic operations than the regular rte
ring:
With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a non-blocking ring based mempool (a likely
use case
for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. For ease-of-use, existing ring enqueue/dequeue functions work with both "regular" and non- blocking rings.

This patchset also adds non-blocking versions of ring_autotest and ring_perf_autotest, and a non-blocking ring based mempool.

This patchset makes one API change; a deprecation notice will be posted in a separate commit.

This patchset depends on the non-blocking stack patchset[1].

[1] http://mails.dpdk.org/archives/dev/2019-January/123653.html

v3:
- Avoid the ABI break by putting 64-bit head and tail values in the same cacheline as struct rte_ring's prod and cons members.
- Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of libraries.
- Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case
someone tries
to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
- Update the ring mempool to use experimental APIs
- Clarify that RINB_F_NB is only limited to x86_64 currently;
ARMv8.1-A builds
can eventually support it with the CASP instruction.

v2:
- Merge separate docs commit into patch #5
- Convert uintptr_t to size_t
- Add a compile-time check for the size of size_t
- Fix a space-after-typecast issue
- Fix an unnecessary-parentheses checkpatch warning
- Bump librte_ring's library version

Gage Eads (5):
ring: add 64-bit headtail structure
ring: add a non-blocking implementation
test_ring: add non-blocking ring autotest
test_ring_perf: add non-blocking ring perf test
mempool/ring: add non-blocking ring handlers

 doc/guides/prog_guide/env_abstraction_layer.rst |   2 +-
 drivers/mempool/ring/Makefile                   |   1 +
 drivers/mempool/ring/meson.build                |   2 +
 drivers/mempool/ring/rte_mempool_ring.c         |  58 ++-
 lib/librte_eventdev/rte_event_ring.h            |   2 +-
 lib/librte_ring/Makefile                        |   3 +-
 lib/librte_ring/rte_ring.c                      |  72 ++-
 lib/librte_ring/rte_ring.h                      | 574 ++++++++++++++++++++++--
 lib/librte_ring/rte_ring_generic_64.h           | 152 +++++++
 lib/librte_ring/rte_ring_version.map            |   7 +
 test/test/test_ring.c                           |  57 ++-
 test/test/test_ring_perf.c                      |  19 +-
 12 files changed, 874 insertions(+), 75 deletions(-)  create mode

100644 lib/librte_ring/rte_ring_generic_64.h

--
2.13.6

author
#58 lore raw
In reply to Honnappa Nagarahalli
Quoted 17 lines

-----Original Message-----
From: Eads, Gage
Sent: Friday, January 25, 2019 11:43 AM
To: 'Honnappa Nagarahalli' <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; nd <[email protected]>; [email protected]; Ola Liljedahl <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; Song Zhu (Arm Technology China) <[email protected]>; nd <[email protected]>
Subject: RE: [dpdk-dev] [PATCH v3 0/5] Add non-blocking ring

Hi Honnappa,

Quoted 9 lines

Works for me -- I'm in favor of the best performing implementation, whoever provides it.

To allow an apples-to-apples comparison, I suggest Ola's/ARM's implementation be made to fit into the rte_ring API with an associated mempool handler. That'll allow us to use the existing ring and mempool performance tests as well. Feel free to use code from this patchset for the rte_ring integration, if that helps, of course.

But also, if Ola/ARM's algorithm is sufficiently similar to this one, it's probably better to tweak this patchset's enqueue and dequeue functions with any improvements you can identify rather than creating an entirely separate implementation.

Quoted 139 lines

I expect to have v4 available within the next week.

Thanks,
Gage

Quoted 19 lines

-----Original Message-----
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Thursday, January 24, 2019 11:21 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; nd <[email protected]>; [email protected]; Ola Liljedahl <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; Song Zhu (Arm Technology China) <[email protected]>; nd <[email protected]>
Subject: RE: [dpdk-dev] [PATCH v3 0/5] Add non-blocking ring

Hi Gage,
Thank you for this patch. Arm (Ola Liljedahl) had worked on a non-
blocking ring algorithm. We were planning to add it to DPDK at some point this year. I am wondering if you would be open to take a look at the algorithm and collaborate?

Quoted 7 lines

I am yet to fully understand both the algorithms. But, Ola has reviewed your patch and can provide a quick overview of the differences here.

If you agree, we can send a RFC patch. You can review that and do performance benchmarking on your platforms. I can also benchmark your patch (may be once you fix the issue identified in __rte_ring_do_nb_enqueue_mp function?) on Arm platforms. May be we can

end up with a better combined algorithm.

Quoted 22 lines

Hi Thomas/Bruce,
Please let me know if this is ok and if there is a better way to do this.

Thank you,
Honnappa

Quoted 9 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Gage Eads
Sent: Friday, January 18, 2019 9:23 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [dpdk-dev] [PATCH v3 0/5] Add non-blocking ring

Quoted 4 lines

For some users, the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a

mempool.

Quoted 1 line
Quoted 16 lines
Quoted 6 lines

This patchset introduces a non-blocking ring, on top of which a mempool can run.
Crucially, the non-blocking algorithm relies on a 128-bit compare-and-swap, so it is currently limited to x86_64 machines. This is also an experimental API, so RING_F_NB users must build with the

ALLOW_EXPERIMENTAL_API flag.

Quoted 9 lines

The ring uses more compare-and-swap atomic operations than the regular rte
ring:
With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-

case

Quoted 3 lines
Quoted 3 lines

performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-

based

Quoted 16 lines
Quoted 16 lines

algorithms. For example, a non-blocking ring based mempool (a
likely use case
for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. For ease-of-use, existing ring enqueue/dequeue functions work with both "regular" and non- blocking rings.

This patchset also adds non-blocking versions of ring_autotest and ring_perf_autotest, and a non-blocking ring based mempool.

This patchset makes one API change; a deprecation notice will be posted in a separate commit.

This patchset depends on the non-blocking stack patchset[1].

Quoted 7 lines
Quoted 7 lines

[1] http://mails.dpdk.org/archives/dev/2019-January/123653.html

v3:
- Avoid the ABI break by putting 64-bit head and tail values in the same cacheline as struct rte_ring's prod and cons members.
- Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of

libraries.

Quoted 40 lines
Quoted 40 lines

- Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case
someone tries
to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
- Update the ring mempool to use experimental APIs
- Clarify that RINB_F_NB is only limited to x86_64 currently;
ARMv8.1-A builds
can eventually support it with the CASP instruction.

v2:
- Merge separate docs commit into patch #5
- Convert uintptr_t to size_t
- Add a compile-time check for the size of size_t
- Fix a space-after-typecast issue
- Fix an unnecessary-parentheses checkpatch warning
- Bump librte_ring's library version

Gage Eads (5):
ring: add 64-bit headtail structure
ring: add a non-blocking implementation
test_ring: add non-blocking ring autotest
test_ring_perf: add non-blocking ring perf test
mempool/ring: add non-blocking ring handlers

 doc/guides/prog_guide/env_abstraction_layer.rst |   2 +-
 drivers/mempool/ring/Makefile                   |   1 +
 drivers/mempool/ring/meson.build                |   2 +
 drivers/mempool/ring/rte_mempool_ring.c         |  58 ++-
 lib/librte_eventdev/rte_event_ring.h            |   2 +-
 lib/librte_ring/Makefile                        |   3 +-
 lib/librte_ring/rte_ring.c                      |  72 ++-
 lib/librte_ring/rte_ring.h                      | 574 ++++++++++++++++++++++--
 lib/librte_ring/rte_ring_generic_64.h           | 152 +++++++
 lib/librte_ring/rte_ring_version.map            |   7 +
 test/test/test_ring.c                           |  57 ++-
 test/test/test_ring_perf.c                      |  19 +-
 12 files changed, 874 insertions(+), 75 deletions(-)  create mode

100644 lib/librte_ring/rte_ring_generic_64.h

--
2.13.6

#59 lore raw
Quoted 30 lines · Gage
Quoted 20 lines

-----Original Message-----
From: Eads, Gage
Sent: Friday, January 25, 2019 11:43 AM
To: 'Honnappa Nagarahalli' <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; nd <[email protected]>; [email protected]; Ola Liljedahl <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; Song Zhu (Arm Technology China) <[email protected]>; nd <[email protected]>
Subject: RE: [dpdk-dev] [PATCH v3 0/5] Add non-blocking ring

Hi Honnappa,

Works for me -- I'm in favor of the best performing implementation, whoever provides it.

Quoted 10 lines

To allow an apples-to-apples comparison, I suggest Ola's/ARM's implementation
be made to fit into the rte_ring API with an associated mempool handler. That'll
allow us to use the existing ring and mempool performance tests as well. Feel
free to use code from this patchset for the rte_ring integration, if that helps, of
course.

Quoted 4 lines

But also, if Ola/ARM's algorithm is sufficiently similar to this one, it's probably better to tweak this patchset's enqueue and dequeue functions with any improvements you can identify rather than creating an entirely separate implementation.

There are strong similarities. But my implementation is separate from rte_ring (whose code is a mess) which also freed me from any interoperatibility with the rte_ring code and data structure (with two pairs of head+tail which is unnecessary for the lock-free ring buffer).

My design and implementation is here:
https://github.com/ARM-software/progress64/blob/master/src/p64_lfring.c
I have a DPDK version in flight. Merging the relevant changes into your patch makes sense. There are some differences we will have to agree on.

Quoted 159 lines
Quoted 159 lines

I expect to have v4 available within the next week.

Thanks,
Gage

Quoted 20 lines

-----Original Message-----
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Thursday, January 24, 2019 11:21 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; nd <[email protected]>; [email protected]; Ola Liljedahl <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; Song Zhu (Arm Technology China) <[email protected]>; nd <[email protected]>
Subject: RE: [dpdk-dev] [PATCH v3 0/5] Add non-blocking ring

Hi Gage,
Thank you for this patch. Arm (Ola Liljedahl) had worked on a non-
blocking ring algorithm. We were planning to add it to DPDK at some point this year. I am wondering if you would be open to take a look at the algorithm and collaborate?

Quoted 8 lines

I am yet to fully understand both the algorithms. But, Ola has reviewed your patch and can provide a quick overview of the differences here.

If you agree, we can send a RFC patch. You can review that and do performance benchmarking on your platforms. I can also benchmark your patch (may be once you fix the issue identified in __rte_ring_do_nb_enqueue_mp  function?) on Arm platforms. May be we can

end up with a better combined algorithm.

Quoted 26 lines

Hi Thomas/Bruce,
Please let me know if this is ok and if there is a better way to do
this.

Thank you,
Honnappa

Quoted 10 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Gage Eads
Sent: Friday, January 18, 2019 9:23 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [dpdk-dev] [PATCH v3 0/5] Add non-blocking ring

Quoted 4 lines

For some users, the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a

mempool.

Quoted 2 lines
Quoted 19 lines
Quoted 6 lines

This patchset introduces a non-blocking ring, on top of which a mempool can run.
Crucially, the non-blocking algorithm relies on a 128-bit compare-and-swap, so it is currently limited to x86_64 machines. This is also an experimental API, so RING_F_NB users must build with the

ALLOW_EXPERIMENTAL_API flag.

Quoted 12 lines

The ring uses more compare-and-swap atomic operations than the regular rte
ring:
With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-

case

Quoted 6 lines
Quoted 6 lines

  performance (i.e. one thread being preempted in the update_tail() critical
  section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-

based

Quoted 18 lines
Quoted 18 lines

  algorithms. For example, a non-blocking ring based mempool (a likely use case
  for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. For ease-of-use, existing ring enqueue/dequeue functions work with both "regular" and non- blocking rings.

This patchset also adds non-blocking versions of ring_autotest and ring_perf_autotest, and a non-blocking ring based mempool.

This patchset makes one API change; a deprecation notice will be posted in a separate commit.

This patchset depends on the non-blocking stack patchset[1].

Quoted 8 lines
Quoted 8 lines

[1] http://mails.dpdk.org/archives/dev/2019-January/123653.html

v3:
 - Avoid the ABI break by putting 64-bit head and tail values in the same
   cacheline as struct rte_ring's prod and cons members.
 - Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of

libraries.

Quoted 43 lines
Quoted 43 lines

 - Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case someone tries
   to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
 - Update the ring mempool to use experimental APIs
 - Clarify that RINB_F_NB is only limited to x86_64 currently; ARMv8.1-A builds
   can eventually support it with the CASP instruction.

v2:
 - Merge separate docs commit into patch #5
 - Convert uintptr_t to size_t
 - Add a compile-time check for the size of size_t
 - Fix a space-after-typecast issue
 - Fix an unnecessary-parentheses checkpatch warning
 - Bump librte_ring's library version

Gage Eads (5):
  ring: add 64-bit headtail structure
  ring: add a non-blocking implementation
  test_ring: add non-blocking ring autotest
  test_ring_perf: add non-blocking ring perf test
  mempool/ring: add non-blocking ring handlers

 doc/guides/prog_guide/env_abstraction_layer.rst |   2 +- drivers/mempool/ring/Makefile                   |   1 + drivers/mempool/ring/meson.build                |   2 + drivers/mempool/ring/rte_mempool_ring.c         |  58 ++- lib/librte_eventdev/rte_event_ring.h            |   2 +- lib/librte_ring/Makefile                        |   3 +- lib/librte_ring/rte_ring.c                      |  72 ++- lib/librte_ring/rte_ring.h                      | 574
++++++++++++++++++++++--
 lib/librte_ring/rte_ring_generic_64.h           | 152 +++++++ lib/librte_ring/rte_ring_version.map            |   7 + test/test/test_ring.c                           |  57 ++- test/test/test_ring_perf.c                      |  19 +-

 12 files changed, 874 insertions(+), 75 deletions(-)  create mode

100644 lib/librte_ring/rte_ring_generic_64.h

--
2.13.6

Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
author
#60 lore raw

v4 cover Add non-blocking ring

In reply to Gage Eads

For some users, the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

This patchset introduces a non-blocking ring, on top of which a mempool can run. Crucially, the non-blocking algorithm relies on a 128-bit compare-and-swap, so it is currently limited to x86_64 machines. This is also an experimental API, so RING_F_NB users must build with the ALLOW_EXPERIMENTAL_API flag.

The ring uses more compare-and-swap atomic operations than the regular rte ring:
With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a non-blocking ring based mempool (a likely use case for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. For ease-of-use, existing ring enqueue/dequeue functions work with both "regular" and non-blocking rings.

This patchset also adds non-blocking versions of ring_autotest and ring_perf_autotest, and a non-blocking ring based mempool.

This patchset makes one API change; a deprecation notice will be posted in a separate commit.

This patchset depends on the 128-bit compare-and-set patch[1].

[1] http://mails.dpdk.org/archives/dev/2019-January/124159.html

v4:
- Split out nb_enqueue and nb_dequeue functions in generic and C11 versions, with the necessary memory ordering behavior for weakly consistent machines.
- Convert size_t variables (from v2) to uint64_t and no-longer-applicable comment about variably-sized ring indexes.
- Fix bug in nb_enqueue_mp that the breaks the non-blocking guarantee.
- Split the ring_ptr cast into two lines.
- Change the dependent patchset from the non-blocking stack patch series to one only containing the 128b CAS commit

v3:
- Avoid the ABI break by putting 64-bit head and tail values in the same cacheline as struct rte_ring's prod and cons members.
- Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of libraries.
- Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case someone tries to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
- Update the ring mempool to use experimental APIs
- Clarify that RINB_F_NB is only limited to x86_64 currently; e.g ARMv8 has the ISA support for 128-bit CAS to eventually support it.

v2:
- Merge separate docs commit into patch #5
- Convert uintptr_t to size_t
- Add a compile-time check for the size of size_t
- Fix a space-after-typecast issue
- Fix an unnecessary-parentheses checkpatch warning
- Bump librte_ring's library version

Gage Eads (5):
ring: add 64-bit headtail structure
ring: add a non-blocking implementation
test_ring: add non-blocking ring autotest
test_ring_perf: add non-blocking ring perf test
mempool/ring: add non-blocking ring handlers

 doc/guides/prog_guide/env_abstraction_layer.rst |   5 +
 drivers/mempool/ring/Makefile                   |   1 +
 drivers/mempool/ring/meson.build                |   2 +
 drivers/mempool/ring/rte_mempool_ring.c         |  58 +++-
 lib/librte_ring/rte_ring.c                      |  72 +++-
 lib/librte_ring/rte_ring.h                      | 336 +++++++++++++++++--
 lib/librte_ring/rte_ring_c11_mem.h              | 427 ++++++++++++++++++++++++
 lib/librte_ring/rte_ring_generic.h              | 408 ++++++++++++++++++++++
 lib/librte_ring/rte_ring_version.map            |   7 +
 test/test/test_ring.c                           |  57 ++--
 test/test/test_ring_perf.c                      |  19 +-
 11 files changed, 1319 insertions(+), 73 deletions(-)
Signature · 1 line
2.13.6
author
#61 lore raw

PATCH v4 1/5 ring: add 64-bit headtail structure

64-bit head and tail index widths greatly increases the time it takes for them to wrap-around (with current CPU speeds, it won't happen within the author's lifetime). This is fundamental to avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming non-blocking ring implementation. Using a 64-bit index makes the possibility of this occurring effectively zero.

This commit places the new producer and consumer structures in the same location in struct rte_ring as their 32-bit counterparts. Since the 32-bit versions are padded out to a cache line, there is space for the new structure without affecting the layout of struct rte_ring. Thus, the ABI is preserved.

---
 lib/librte_ring/rte_ring.h         |  23 +++++-
 lib/librte_ring/rte_ring_c11_mem.h | 153 +++++++++++++++++++++++++++++++++++++
 lib/librte_ring/rte_ring_generic.h | 139 +++++++++++++++++++++++++++++++++
 3 files changed, 312 insertions(+), 3 deletions(-)
lib/librte_ring/rte_ring.h +20 -3
index af5444a9f..00dfb5b85 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007-2009 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -70,6 +70,15 @@ struct rte_ring_headtail {
 	uint32_t single;         /**< True if single prod/cons */
 };
 
+/* 64-bit version of rte_ring_headtail, for use by rings that need to avoid
+ * head/tail wrap-around.
+ */
+struct rte_ring_headtail_64 {
+	volatile uint64_t head;  /**< Prod/consumer head. */
+	volatile uint64_t tail;  /**< Prod/consumer tail. */
+	uint32_t single;       /**< True if single prod/cons */
+};
+
 /**
  * An RTE ring structure.
  *
@@ -97,11 +106,19 @@ struct rte_ring {
 	char pad0 __rte_cache_aligned; /**< empty cache line */
 
 	/** Ring producer status. */
-	struct rte_ring_headtail prod __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail prod __rte_cache_aligned;
+		struct rte_ring_headtail_64 prod_64 __rte_cache_aligned;
+	};
 	char pad1 __rte_cache_aligned; /**< empty cache line */
 
 	/** Ring consumer status. */
-	struct rte_ring_headtail cons __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail cons __rte_cache_aligned;
+		struct rte_ring_headtail_64 cons_64 __rte_cache_aligned;
+	};
 	char pad2 __rte_cache_aligned; /**< empty cache line */
 };
 
lib/librte_ring/rte_ring_c11_mem.h +153 -0

157 lines not on this page · raw message

lib/librte_ring/rte_ring_generic.h +139 -0

143 lines not on this page · raw message

Signature · 1 line
2.13.6
#62 lore raw

Re: [PATCH v4 1/5] ring: add 64-bit headtail structure

Quoted 7 lines · Gage Eads

64-bit head and tail index widths greatly increases the time it takes for them to wrap-around (with current CPU speeds, it won't happen within the author's lifetime). This is fundamental to avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming non-blocking ring implementation. Using a 64-bit index makes the possibility of this occurring effectively zero.

Just an observation.
The following invariant holds (using ring_size instead of mask):
∀ index: ring[index % ring_size].index % ring_size == index % ring_size i.e. the N (N=log2 ring size) lsb of ring[].index will always be the same (for a specific slot) so serve no purpose.

This means we don't have to store the whole index in each slot, it is enough to store "index / ring_size" (which I call the lap counter). This could be useful for an implementation for 32-bit platforms which support 64-bit CAS (to write the slot ptr & index (lap counter) atomically) and uses 64-bit head & tail indexes (to avoid the quick wrap around you would have with 32-bit ring indexes).

So
ring[index % ring_size].lap = index / ring_size;

An implementation could of course use bitwise-and instead of modulo and bitwise- right shift instead of division. The 2-logaritm of ring_size should also be pre- calcucated and stored in the ring buffer metadata.

-- Ola

Quoted 385 lines

This commit places the new producer and consumer structures in the same location in struct rte_ring as their 32-bit counterparts. Since the 32-bit versions are padded out to a cache line, there is space for the new structure without affecting the layout of struct rte_ring. Thus, the ABI is preserved.

---
 lib/librte_ring/rte_ring.h         |  23 +++++- lib/librte_ring/rte_ring_c11_mem.h | 153
+++++++++++++++++++++++++++++++++++++
 lib/librte_ring/rte_ring_generic.h | 139 +++++++++++++++++++++++++++++++++

 3 files changed, 312 insertions(+), 3 deletions(-)
lib/librte_ring/rte_ring.h +1 -1
index af5444a9f..00dfb5b85 100644
@@ -1,6 +1,6 @@
  /* SPDX-License-Identifier: BSD-3-Clause
   *
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
   * Copyright (c) 2007-2009 Kip Macy [email protected]
   * All rights reserved.
   * Derived from FreeBSD's bufring.h
@@ -70,6 +70,15 @@ struct rte_ring_headtail {

  uint32_t single;         /**< True if single prod/cons */ };

lib/librte_ring/rte_ring.h +9 -0
+/* 64-bit version of rte_ring_headtail, for use by rings that need to avoid
+ * head/tail wrap-around.
+ */
+struct rte_ring_headtail_64 {
+	volatile uint64_t head;  /**< Prod/consumer head. */
+	volatile uint64_t tail;  /**< Prod/consumer tail. */
+	uint32_t single;       /**< True if single prod/cons */
+};
+

 /**
  * An RTE ring structure.
  *

lib/librte_ring/rte_ring.h +0 -0
@@ -97,11 +106,19 @@ struct rte_ring {

  char pad0 __rte_cache_aligned; /**< empty cache line */
  /** Ring producer status. */

lib/librte_ring/rte_ring.h +5 -1
-	struct rte_ring_headtail prod __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail prod __rte_cache_aligned;
+		struct rte_ring_headtail_64 prod_64 __rte_cache_aligned;
+	};

  char pad1 __rte_cache_aligned; /**< empty cache line */
  /** Ring consumer status. */

lib/librte_ring/rte_ring.h +5 -1
-	struct rte_ring_headtail cons __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail cons __rte_cache_aligned;
+		struct rte_ring_headtail_64 cons_64 __rte_cache_aligned;
+	};

  char pad2 __rte_cache_aligned; /**< empty cache line */ };

lib/librte_ring/rte_ring_c11_mem.h +0 -0
index 0fb73a337..47acd4c7c 100644
@@ -178,4 +178,157 @@ __rte_ring_move_cons_head(struct rte_ring *r, int is_sc,
  	return n;
  }

lib/librte_ring/rte_ring_c11_mem.h +52 -0
+/**
+ * @internal This function updates the producer head for enqueue using
+ *	     64-bit head/tail values.
+ *
+ * @param r
+ *   A pointer to the ring structure
+ * @param is_sp
+ *   Indicates whether multi-producer path is needed or not
+ * @param n
+ *   The number of elements we will want to enqueue, i.e. how far should the
+ *   head be moved
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param old_head
+ *   Returns head value as it was before the move, i.e. where enqueue starts
+ * @param new_head
+ *   Returns the current/new head value i.e. where enqueue finishes
+ * @param free_entries
+ *   Returns the amount of free space in the ring BEFORE head was moved
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_move_prod_head_64(struct rte_ring *r, unsigned int is_sp,
+		unsigned int n, enum rte_ring_queue_behavior behavior,
+		uint64_t *old_head, uint64_t *new_head,
+		uint32_t *free_entries)
+{
+	const uint32_t capacity = r->capacity;
+	uint64_t cons_tail;
+	unsigned int max = n;
+	int success;
+
+	*old_head = __atomic_load_n(&r->prod_64.head, __ATOMIC_RELAXED);
+	do {
+		/* Reset n to the initial burst count */
+		n = max;
+
+		/* Ensure the head is read before tail */
+		__atomic_thread_fence(__ATOMIC_ACQUIRE);
+
+		/* load-acquire synchronize with store-release of ht->tail
+		 * in update_tail.
+		 */
+		cons_tail = __atomic_load_n(&r->cons_64.tail,
+					__ATOMIC_ACQUIRE);
+
+		/* The subtraction is done between two unsigned 32bits value
+		 * (the result is always modulo 32 bits even if we have
+		 * *old_head > cons_tail). So 'free_entries' is always

between 0

lib/librte_ring/rte_ring_c11_mem.h +18 -0
+		 * and capacity (which is < size).
+		 */
+		*free_entries = (capacity + cons_tail - *old_head);
+
+		/* check that we have enough room in ring */
+		if (unlikely(n > *free_entries))
+			n = (behavior == RTE_RING_QUEUE_FIXED) ?
+					0 : *free_entries;
+
+		if (n == 0)
+			return 0;
+
+		*new_head = *old_head + n;
+		if (is_sp)
+			r->prod_64.head = *new_head, success = 1;
+		else
+			/* on failure, *old_head is updated */
+			success = __atomic_compare_exchange_n(&r-
Quoted 1 line

prod_64.head,

lib/librte_ring/rte_ring_c11_mem.h +51 -0
+					old_head, *new_head,
+					0, __ATOMIC_RELAXED,
+					__ATOMIC_RELAXED);
+	} while (unlikely(success == 0));
+	return n;
+}
+
+/**
+ * @internal This function updates the consumer head for dequeue using
+ *	     64-bit head/tail values.
+ *
+ * @param r
+ *   A pointer to the ring structure
+ * @param is_sc
+ *   Indicates whether multi-consumer path is needed or not
+ * @param n
+ *   The number of elements we will want to enqueue, i.e. how far should the
+ *   head be moved
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param old_head
+ *   Returns head value as it was before the move, i.e. where dequeue starts
+ * @param new_head
+ *   Returns the current/new head value i.e. where dequeue finishes
+ * @param entries
+ *   Returns the number of entries in the ring BEFORE head was moved
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_move_cons_head_64(struct rte_ring *r, unsigned int is_sc,
+		unsigned int n, enum rte_ring_queue_behavior behavior,
+		uint64_t *old_head, uint64_t *new_head,
+		uint32_t *entries)
+{
+	unsigned int max = n;
+	uint64_t prod_tail;
+	int success;
+
+	/* move cons.head atomically */
+	*old_head = __atomic_load_n(&r->cons_64.head, __ATOMIC_RELAXED);
+	do {
+		/* Restore n as it may change every loop */
+		n = max;
+
+		/* Ensure the head is read before tail */
+		__atomic_thread_fence(__ATOMIC_ACQUIRE);
+
+		/* this load-acquire synchronize with store-release of ht-
Quoted 1 line

tail

lib/librte_ring/rte_ring_c11_mem.h +15 -0
+		 * in update_tail.
+		 */
+		prod_tail = __atomic_load_n(&r->prod_64.tail,
+					__ATOMIC_ACQUIRE);
+
+		/* The subtraction is done between two unsigned 32bits value
+		 * (the result is always modulo 32 bits even if we have
+		 * cons_head > prod_tail). So 'entries' is always between 0
+		 * and size(ring)-1.
+		 */
+		*entries = (prod_tail - *old_head);
+
+		/* Set the actual entries for dequeue */
+		if (n > *entries)
+			n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 :

*entries;

lib/librte_ring/rte_ring_c11_mem.h +10 -0
+
+		if (unlikely(n == 0))
+			return 0;
+
+		*new_head = *old_head + n;
+		if (is_sc)
+			r->cons_64.head = *new_head, success = 1;
+		else
+			/* on failure, *old_head will be updated */
+			success = __atomic_compare_exchange_n(&r-
Quoted 1 line

cons_64.head,

lib/librte_ring/rte_ring_c11_mem.h +7 -0
+							old_head, *new_head,
+							0, __ATOMIC_RELAXED,
+							__ATOMIC_RELAXED);
+	} while (unlikely(success == 0));
+	return n;
+}
+

 #endif /* _RTE_RING_C11_MEM_H_ */

lib/librte_ring/rte_ring_generic.h +0 -0
index ea7dbe5b9..2158e092a 100644
@@ -167,4 +167,143 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned
 int is_sc,
  	return n;
  }

lib/librte_ring/rte_ring_generic.h +49 -0
+/**
+ * @internal This function updates the producer head for enqueue using
+ *	     64-bit head/tail values.
+ *
+ * @param r
+ *   A pointer to the ring structure
+ * @param is_sp
+ *   Indicates whether multi-producer path is needed or not
+ * @param n
+ *   The number of elements we will want to enqueue, i.e. how far should the
+ *   head be moved
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param old_head
+ *   Returns head value as it was before the move, i.e. where enqueue starts
+ * @param new_head
+ *   Returns the current/new head value i.e. where enqueue finishes
+ * @param free_entries
+ *   Returns the amount of free space in the ring BEFORE head was moved
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_move_prod_head_64(struct rte_ring *r, unsigned int is_sp,
+		unsigned int n, enum rte_ring_queue_behavior behavior,
+		uint64_t *old_head, uint64_t *new_head,
+		uint32_t *free_entries)
+{
+	const uint32_t capacity = r->capacity;
+	unsigned int max = n;
+	int success;
+
+	do {
+		/* Reset n to the initial burst count */
+		n = max;
+
+		*old_head = r->prod_64.head;
+
+		/* add rmb barrier to avoid load/load reorder in weak
+		 * memory model. It is noop on x86
+		 */
+		rte_smp_rmb();
+
+		/*
+		 *  The subtraction is done between two unsigned 64bits value
+		 * (the result is always modulo 64 bits even if we have
+		 * *old_head > cons_tail). So 'free_entries' is always

between 0

lib/librte_ring/rte_ring_generic.h +76 -0
+		 * and capacity (which is < size).
+		 */
+		*free_entries = (capacity + r->cons_64.tail - *old_head);
+
+		/* check that we have enough room in ring */
+		if (unlikely(n > *free_entries))
+			n = (behavior == RTE_RING_QUEUE_FIXED) ?
+					0 : *free_entries;
+
+		if (n == 0)
+			return 0;
+
+		*new_head = *old_head + n;
+		if (is_sp)
+			r->prod_64.head = *new_head, success = 1;
+		else
+			success = rte_atomic64_cmpset(&r->prod_64.head,
+					*old_head, *new_head);
+	} while (unlikely(success == 0));
+	return n;
+}
+
+/**
+ * @internal This function updates the consumer head for dequeue using
+ *	     64-bit head/tail values.
+ *
+ * @param r
+ *   A pointer to the ring structure
+ * @param is_sc
+ *   Indicates whether multi-consumer path is needed or not
+ * @param n
+ *   The number of elements we will want to enqueue, i.e. how far should the
+ *   head be moved
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param old_head
+ *   Returns head value as it was before the move, i.e. where dequeue starts
+ * @param new_head
+ *   Returns the current/new head value i.e. where dequeue finishes
+ * @param entries
+ *   Returns the number of entries in the ring BEFORE head was moved
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_move_cons_head_64(struct rte_ring *r, unsigned int is_sc,
+		unsigned int n, enum rte_ring_queue_behavior behavior,
+		uint64_t *old_head, uint64_t *new_head,
+		uint32_t *entries)
+{
+	unsigned int max = n;
+	int success;
+
+	do {
+		/* Restore n as it may change every loop */
+		n = max;
+
+		*old_head = r->cons_64.head;
+
+		/* add rmb barrier to avoid load/load reorder in weak
+		 * memory model. It is noop on x86
+		 */
+		rte_smp_rmb();
+
+		/* The subtraction is done between two unsigned 64bits value
+		 * (the result is always modulo 64 bits even if we have
+		 * cons_head > prod_tail). So 'entries' is always between 0
+		 * and size(ring)-1.
+		 */
+		*entries = (r->prod_64.tail - *old_head);
+
+		/* Set the actual entries for dequeue */
+		if (n > *entries)
+			n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 :

*entries;

lib/librte_ring/rte_ring_generic.h +14 -0
+
+		if (unlikely(n == 0))
+			return 0;
+
+		*new_head = *old_head + n;
+		if (is_sc)
+			r->cons_64.head = *new_head, success = 1;
+		else
+			success = rte_atomic64_cmpset(&r->cons_64.head,
+					*old_head, *new_head);
+	} while (unlikely(success == 0));
+	return n;
+}
+

 #endif /* _RTE_RING_GENERIC_H_ */

Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
author
#63 lore raw

Re: [PATCH v4 1/5] ring: add 64-bit headtail structure

Quoted 32 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Tuesday, January 29, 2019 6:57 AM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v4 1/5] ring: add 64-bit headtail structure

Quoted 7 lines · Gage Eads

64-bit head and tail index widths greatly increases the time it takes for them to wrap-around (with current CPU speeds, it won't happen within the author's lifetime). This is fundamental to avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming non-blocking ring implementation. Using a 64-bit index makes the possibility of this occurring effectively zero.

Just an observation.
The following invariant holds (using ring_size instead of mask):
∀ index: ring[index % ring_size].index % ring_size == index % ring_size i.e. the N (N=log2 ring size) lsb of ring[].index will always be the same (for a specific slot) so serve no purpose.

This means we don't have to store the whole index in each slot, it is enough to store "index / ring_size" (which I call the lap counter). This could be useful for an implementation for 32-bit platforms which support 64-bit CAS (to write the slot ptr & index (lap counter) atomically) and uses 64-bit head & tail indexes (to avoid the quick wrap around you would have with 32-bit ring indexes).

Quoted 7 lines

So
ring[index % ring_size].lap = index / ring_size;

An implementation could of course use bitwise-and instead of modulo and bitwise- right shift instead of division. The 2-logaritm of ring_size should also be pre- calcucated and stored in the ring buffer metadata.

That's a pretty interesting idea. The question is, with such a design, what should DPDK's minimum NB ring size be on 32-bit platforms?

If a ring entry is written on average every M cycles, then a lap occurs every M*N cycles and each counter repeats every M*N*2^32 cycles. If M=100 on a 2GHz system, then the counter repeats every

N=1: 3.33 minutes
...
N=256: 14.22 hours
N=512: 28.44 hours
N=1024: 2.37 days
...
N=16384: 37.92 days

I think a minimum size of 1024 strikes a good balance between not too burdensome and sufficiently low odds of ABA occurring.

Thanks,
Gage

[snip]

author
#64 lore raw

PATCH v4 2/5 ring: add a non-blocking implementation

In reply to Gage Eads

This commit adds support for non-blocking circular ring enqueue and dequeue functions. The ring uses a 128-bit compare-and-swap instruction, and thus is currently limited to x86_64.

The algorithm is based on the original rte ring (derived from FreeBSD's bufring.h) and inspired by Michael and Scott's non-blocking concurrent queue. Importantly, it adds a modification counter to each ring entry to ensure only one thread can write to an unused entry.

-----
Algorithm:

Multi-producer non-blocking enqueue:
1. Move the producer head index 'n' locations forward, effectively reserving 'n' locations.
2. For each pointer:
a. Read the producer tail index, then ring[tail]. If ring[tail]'s
modification counter isn't 'tail', retry.
b. Construct the new entry: {pointer, tail + ring size}
c. Compare-and-swap the old entry with the new. If unsuccessful, the
next loop iteration will try to enqueue this pointer again.
d. Compare-and-swap the tail index with 'tail + 1', whether or not step 2c
succeeded. This guarantees threads can make forward progress.

Multi-consumer non-blocking dequeue:
1. Move the consumer head index 'n' locations forward, effectively reserving 'n' pointers to be dequeued.
2. Copy 'n' pointers into the caller's object table (ignoring the modification counter), starting from ring[tail], then compare-and-swap the tail index with 'tail + n'. If unsuccessful, repeat step 2.

-----
Discussion:

There are two cases where the ABA problem is mitigated:
1. Enqueueing a pointer to the ring: without a modification counter tied to the tail index, the index could become stale by the time the enqueue happens, causing it to overwrite valid data. Tying the counter to the tail index gives us an expected value (as opposed to, say, a monotonically incrementing counter).

Since the counter will eventually wrap, there is potential for the ABA problem. However, using a 64-bit counter makes this likelihood effectively zero.

2. Updating a tail index: the ABA problem can occur if the thread is preempted and the tail index wraps around. However, using 64-bit indexes makes this likelihood effectively zero.

With no contention, an enqueue of n pointers uses (1 + 2n) CAS operations and a dequeue of n pointers uses 2. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the non-blocking ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a non-blocking ring based mempool (a likely use case for this ring) with per-thread caching.

The non-blocking ring is enabled via a new flag, RING_F_NB. Because the ring's memsize is now a function of its flags (the non-blocking ring requires 128b for each entry), this commit adds a new argument ('flags') to rte_ring_get_memsize(). An API deprecation notice will be sent in a separate commit.

For ease-of-use, existing ring enqueue and dequeue functions work on both regular and non-blocking rings. This introduces an additional branch in the datapath, but this should be a highly predictable branch. ring_perf_autotest shows a negligible performance impact; it's hard to distinguish a real difference versus system noise.

| ring_perf_autotest cycles with branch -
Test | ring_perf_autotest cycles without
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | -4.00
SP/SC burst enq/dequeue (size 8)  | 0.00
MP/MC burst enq/dequeue (size 8)  | 0.00
SP/SC burst enq/dequeue (size 32) | 0.00
MP/MC burst enq/dequeue (size 32) | 0.00
SC empty dequeue                  | 1.00
MC empty dequeue                  | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | 0.49
MP/MC bulk enq/dequeue (size 8)   | 0.08
SP/SC bulk enq/dequeue (size 32)  | 0.07
MP/MC bulk enq/dequeue (size 32)  | 0.09

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | 0.19
MP/MC bulk enq/dequeue (size 8)   | -0.37
SP/SC bulk enq/dequeue (size 32)  | 0.09
MP/MC bulk enq/dequeue (size 32)  | -0.05

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | -1.96
MP/MC bulk enq/dequeue (size 8)   | 0.88
SP/SC bulk enq/dequeue (size 32)  | 0.10
MP/MC bulk enq/dequeue (size 32)  | 0.46

Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. Each test run three times and the results averaged.

---
 lib/librte_ring/rte_ring.c           |  72 ++++++--
 lib/librte_ring/rte_ring.h           | 313 +++++++++++++++++++++++++++++++----
 lib/librte_ring/rte_ring_c11_mem.h   | 282 ++++++++++++++++++++++++++++++-
 lib/librte_ring/rte_ring_generic.h   | 269 ++++++++++++++++++++++++++++++
 lib/librte_ring/rte_ring_version.map |   7 +
 5 files changed, 896 insertions(+), 47 deletions(-)
lib/librte_ring/rte_ring.c +60 -12
index d215acecc..f3378dccd 100644
@@ -45,9 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
 
 /* return the size of memory occupied by a ring */
 ssize_t
-rte_ring_get_memsize(unsigned count)
+rte_ring_get_memsize_v1905(unsigned int count, unsigned int flags)
 {
-	ssize_t sz;
+	ssize_t sz, elt_sz;
 
 	/* count must be a power of 2 */
 	if ((!POWEROF2(count)) || (count > RTE_RING_SZ_MASK )) {
@@ -57,10 +57,23 @@ rte_ring_get_memsize(unsigned count)
 		return -EINVAL;
 	}
 
-	sz = sizeof(struct rte_ring) + count * sizeof(void *);
+	elt_sz = (flags & RING_F_NB) ? 2 * sizeof(void *) : sizeof(void *);
+
+	sz = sizeof(struct rte_ring) + count * elt_sz;
 	sz = RTE_ALIGN(sz, RTE_CACHE_LINE_SIZE);
 	return sz;
 }
+BIND_DEFAULT_SYMBOL(rte_ring_get_memsize, _v1905, 19.05);
+MAP_STATIC_SYMBOL(ssize_t rte_ring_get_memsize(unsigned int count,
+					       unsigned int flags),
+		  rte_ring_get_memsize_v1905);
+
+ssize_t
+rte_ring_get_memsize_v20(unsigned int count)
+{
+	return rte_ring_get_memsize_v1905(count, 0);
+}
+VERSION_SYMBOL(rte_ring_get_memsize, _v20, 2.0);
 
 int
 rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
@@ -82,8 +95,6 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 	if (ret < 0 || ret >= (int)sizeof(r->name))
 		return -ENAMETOOLONG;
 	r->flags = flags;
-	r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
-	r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
 
 	if (flags & RING_F_EXACT_SZ) {
 		r->size = rte_align32pow2(count + 1);
@@ -100,8 +111,30 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 		r->mask = count - 1;
 		r->capacity = r->mask;
 	}
-	r->prod.head = r->cons.head = 0;
-	r->prod.tail = r->cons.tail = 0;
+
+	if (flags & RING_F_NB) {
+		uint64_t i;
+
+		r->prod_64.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
+		r->cons_64.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+		r->prod_64.head = r->cons_64.head = 0;
+		r->prod_64.tail = r->cons_64.tail = 0;
+
+		for (i = 0; i < r->size; i++) {
+			struct nb_ring_entry *ring_ptr, *base;
+
+			base = ((struct nb_ring_entry *)&r[1]);
+
+			ring_ptr = &base[i & r->mask];
+
+			ring_ptr->cnt = i;
+		}
+	} else {
+		r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
+		r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+		r->prod.head = r->cons.head = 0;
+		r->prod.tail = r->cons.tail = 0;
+	}
 
 	return 0;
 }
@@ -123,11 +156,19 @@ rte_ring_create(const char *name, unsigned count, int socket_id,
 
 	ring_list = RTE_TAILQ_CAST(rte_ring_tailq.head, rte_ring_list);
 
+#if !defined(RTE_ARCH_X86_64)
+	if (flags & RING_F_NB) {
+		printf("RING_F_NB is only supported on x86-64 platforms\n");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+#endif
+
 	/* for an exact size ring, round up from count to a power of two */
 	if (flags & RING_F_EXACT_SZ)
 		count = rte_align32pow2(count + 1);
 
-	ring_size = rte_ring_get_memsize(count);
+	ring_size = rte_ring_get_memsize(count, flags);
 	if (ring_size < 0) {
 		rte_errno = ring_size;
 		return NULL;
@@ -227,10 +268,17 @@ rte_ring_dump(FILE *f, const struct rte_ring *r)
 	fprintf(f, "  flags=%x\n", r->flags);
 	fprintf(f, "  size=%"PRIu32"\n", r->size);
 	fprintf(f, "  capacity=%"PRIu32"\n", r->capacity);
-	fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
-	fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
-	fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
-	fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	if (r->flags & RING_F_NB) {
+		fprintf(f, "  ct=%"PRIu64"\n", r->cons_64.tail);
+		fprintf(f, "  ch=%"PRIu64"\n", r->cons_64.head);
+		fprintf(f, "  pt=%"PRIu64"\n", r->prod_64.tail);
+		fprintf(f, "  ph=%"PRIu64"\n", r->prod_64.head);
+	} else {
+		fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
+		fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
+		fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
+		fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	}
 	fprintf(f, "  used=%u\n", rte_ring_count(r));
 	fprintf(f, "  avail=%u\n", rte_ring_free_count(r));
 }
lib/librte_ring/rte_ring.h +282 -31

453 lines not on this page · raw message

lib/librte_ring/rte_ring_c11_mem.h +278 -4

306 lines not on this page · raw message

lib/librte_ring/rte_ring_generic.h +269 -0

273 lines not on this page · raw message

lib/librte_ring/rte_ring_version.map +7 -0
index d935efd0d..8969467af 100644
@@ -17,3 +17,10 @@ DPDK_2.2 {
 	rte_ring_free;
 
 } DPDK_2.0;
+
+DPDK_19.05 {
+	global:
+
+	rte_ring_get_memsize;
+
+} DPDK_2.2;
Signature · 1 line
2.13.6
author
#65 lore raw

PATCH v4 3/5 test_ring: add non-blocking ring autotest

In reply to Gage Eads

ring_nb_autotest re-uses the ring_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 test/test/test_ring.c | 57 ++++++++++++++++++++++++++++++++-------------------
 1 file changed, 36 insertions(+), 21 deletions(-)
test/test/test_ring.c +36 -21
index aaf1e70ad..ff410d978 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 #include <string.h>
@@ -601,18 +601,20 @@ test_ring_burst_basic(struct rte_ring *r)
  * it will always fail to create ring with a wrong ring size number in this function
  */
 static int
-test_ring_creation_with_wrong_size(void)
+test_ring_creation_with_wrong_size(unsigned int flags)
 {
 	struct rte_ring * rp = NULL;
 
 	/* Test if ring size is not power of 2 */
-	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1,
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
 
 	/* Test if ring size is exceeding the limit */
-	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1), SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1),
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
@@ -623,11 +625,11 @@ test_ring_creation_with_wrong_size(void)
  * it tests if it would always fail to create ring with an used ring name
  */
 static int
-test_ring_creation_with_an_used_name(void)
+test_ring_creation_with_an_used_name(unsigned int flags)
 {
 	struct rte_ring * rp;
 
-	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (NULL != rp)
 		return -1;
 
@@ -639,10 +641,10 @@ test_ring_creation_with_an_used_name(void)
  * function to fail correctly
  */
 static int
-test_create_count_odd(void)
+test_create_count_odd(unsigned int flags)
 {
 	struct rte_ring *r = rte_ring_create("test_ring_count",
-			4097, SOCKET_ID_ANY, 0 );
+			4097, SOCKET_ID_ANY, flags);
 	if(r != NULL){
 		return -1;
 	}
@@ -665,7 +667,7 @@ test_lookup_null(void)
  * it tests some more basic ring operations
  */
 static int
-test_ring_basic_ex(void)
+test_ring_basic_ex(unsigned int flags)
 {
 	int ret = -1;
 	unsigned i;
@@ -679,7 +681,7 @@ test_ring_basic_ex(void)
 	}
 
 	rp = rte_ring_create("test_ring_basic_ex", RING_SIZE, SOCKET_ID_ANY,
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (rp == NULL) {
 		printf("test_ring_basic_ex fail to create ring\n");
 		goto fail_test;
@@ -737,7 +739,7 @@ test_ring_basic_ex(void)
 }
 
 static int
-test_ring_with_exact_size(void)
+test_ring_with_exact_size(unsigned int flags)
 {
 	struct rte_ring *std_ring = NULL, *exact_sz_ring = NULL;
 	void *ptr_array[16];
@@ -746,13 +748,13 @@ test_ring_with_exact_size(void)
 	int ret = -1;
 
 	std_ring = rte_ring_create("std", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (std_ring == NULL) {
 		printf("%s: error, can't create std ring\n", __func__);
 		goto end;
 	}
 	exact_sz_ring = rte_ring_create("exact sz", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ);
+		RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ | flags);
 	if (exact_sz_ring == NULL) {
 		printf("%s: error, can't create exact size ring\n", __func__);
 		goto end;
@@ -808,17 +810,17 @@ test_ring_with_exact_size(void)
 }
 
 static int
-test_ring(void)
+__test_ring(unsigned int flags)
 {
 	struct rte_ring *r = NULL;
 
 	/* some more basic operations */
-	if (test_ring_basic_ex() < 0)
+	if (test_ring_basic_ex(flags) < 0)
 		goto test_fail;
 
 	rte_atomic32_init(&synchro);
 
-	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (r == NULL)
 		goto test_fail;
 
@@ -837,27 +839,27 @@ test_ring(void)
 		goto test_fail;
 
 	/* basic operations */
-	if ( test_create_count_odd() < 0){
+	if (test_create_count_odd(flags) < 0) {
 		printf("Test failed to detect odd count\n");
 		goto test_fail;
 	} else
 		printf("Test detected odd count\n");
 
-	if ( test_lookup_null() < 0){
+	if (test_lookup_null() < 0) {
 		printf("Test failed to detect NULL ring lookup\n");
 		goto test_fail;
 	} else
 		printf("Test detected NULL ring lookup\n");
 
 	/* test of creating ring with wrong size */
-	if (test_ring_creation_with_wrong_size() < 0)
+	if (test_ring_creation_with_wrong_size(flags) < 0)
 		goto test_fail;
 
 	/* test of creation ring with an used name */
-	if (test_ring_creation_with_an_used_name() < 0)
+	if (test_ring_creation_with_an_used_name(flags) < 0)
 		goto test_fail;
 
-	if (test_ring_with_exact_size() < 0)
+	if (test_ring_with_exact_size(flags) < 0)
 		goto test_fail;
 
 	/* dump the ring status */
@@ -873,4 +875,17 @@ test_ring(void)
 	return -1;
 }
 
+static int
+test_ring(void)
+{
+	return __test_ring(0);
+}
+
+static int
+test_nb_ring(void)
+{
+	return __test_ring(RING_F_NB);
+}
+
 REGISTER_TEST_COMMAND(ring_autotest, test_ring);
+REGISTER_TEST_COMMAND(ring_nb_autotest, test_nb_ring);
Signature · 1 line
2.13.6
author
#66 lore raw

PATCH v4 4/5 test_ring_perf: add non-blocking ring perf test

In reply to Gage Eads

nb_ring_perf_autotest re-uses the ring_perf_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 test/test/test_ring_perf.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
test/test/test_ring_perf.c +16 -3
index ebb3939f5..380c4b4a1 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 
@@ -363,12 +363,12 @@ test_bulk_enqueue_dequeue(struct rte_ring *r)
 }
 
 static int
-test_ring_perf(void)
+__test_ring_perf(unsigned int flags)
 {
 	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
 
-	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), 0);
+	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), flags);
 	if (r == NULL)
 		return -1;
 
@@ -398,4 +398,17 @@ test_ring_perf(void)
 	return 0;
 }
 
+static int
+test_ring_perf(void)
+{
+	return __test_ring_perf(0);
+}
+
+static int
+test_nb_ring_perf(void)
+{
+	return __test_ring_perf(RING_F_NB);
+}
+
 REGISTER_TEST_COMMAND(ring_perf_autotest, test_ring_perf);
+REGISTER_TEST_COMMAND(ring_nb_perf_autotest, test_nb_ring_perf);
Signature · 1 line
2.13.6
author
#67 lore raw

PATCH v4 5/5 mempool/ring: add non-blocking ring handlers

In reply to Gage Eads

These handlers allow an application to create a mempool based on the non-blocking ring, with any combination of single/multi producer/consumer.

Also, add a note to the programmer's guide's "known issues" section.

---
 doc/guides/prog_guide/env_abstraction_layer.rst |  5 +++
 drivers/mempool/ring/Makefile                   |  1 +
 drivers/mempool/ring/meson.build                |  2 +
 drivers/mempool/ring/rte_mempool_ring.c         | 58 +++++++++++++++++++++++--
 4 files changed, 63 insertions(+), 3 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +5 -0
index 929d76dba..fcafd1cff 100644
@@ -541,6 +541,11 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
+  Alternatively, x86_64 applications can use the non-blocking ring mempool handler. When considering it, note that:
+
+  - it is currently limited to the x86_64 platform, because it uses a function (16-byte compare-and-swap) that is not yet available on other platforms.
+  - it has worse average-case performance than the non-preemptive rte_ring, but software caching (e.g. the mempool cache) can mitigate this by reducing the number of handler operations.
+
 + rte_timer
 
   Running  ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
drivers/mempool/ring/Makefile +1 -0
index ddab522fe..012ba6966 100644
@@ -10,6 +10,7 @@ LIB = librte_mempool_ring.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 LDLIBS += -lrte_eal -lrte_mempool -lrte_ring
 
 EXPORT_MAP := rte_mempool_ring_version.map
drivers/mempool/ring/meson.build +2 -0
index a021e908c..b1cb673cc 100644
@@ -1,4 +1,6 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
+allow_experimental_apis = true
+
 sources = files('rte_mempool_ring.c')
drivers/mempool/ring/rte_mempool_ring.c +55 -3

84 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#68 lore raw

v5 cover Add lock-free ring and mempool handler

In reply to Gage Eads

For some users, the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

This patchset introduces a lock-free ring and a mempool based on it. The lock-free algorithm relies on a double-pointer compare-and-swap, so for 64-bit architectures it is currently limited to x86_64.

The ring uses more compare-and-swap atomic operations than the regular rte ring:
With no contention, an enqueue of n pointers uses (1 + n) CAS operations and a dequeue of n pointers uses 1. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the lock-free ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a lock-free ring based mempool (a likely use case for this ring) with per-thread caching.

The lock-free ring is enabled via a new flag, RING_F_LF. For ease-of-use, existing ring enqueue/dequeue functions work with both standard and lock-free rings. This is also an experimental API, so RING_F_LF users must build with the ALLOW_EXPERIMENTAL_API flag.

This patchset also adds lock-free versions of ring_autotest and ring_perf_autotest, and a lock-free ring based mempool.

This patchset makes one API change; a deprecation notice was posted in a separate commit[1].

This patchset depends on the 128-bit compare-and-set patch[2].

[1] http://mails.dpdk.org/archives/dev/2019-February/124321.html
[2] http://mails.dpdk.org/archives/dev/2019-March/125751.html

v5:
- Incorporated lfring's enqueue and dequeue logic from
http://mails.dpdk.org/archives/dev/2019-January/124242.html
- Renamed non-blocking -> lock-free and NB -> LF to align with a similar change in the lock-free stack patchset:
http://mails.dpdk.org/archives/dev/2019-March/125797.html
- Added support for 32-bit architectures by using the full 32b of the modification counter and requiring LF rings on these architectures to be at least 1024 entries.
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Added ring start marker to struct rte_ring

v4:
- Split out nb_enqueue and nb_dequeue functions in generic and C11 versions, with the necessary memory ordering behavior for weakly consistent machines.
- Convert size_t variables (from v2) to uint64_t and no-longer-applicable comment about variably-sized ring indexes.
- Fix bug in nb_enqueue_mp that the breaks the non-blocking guarantee.
- Split the ring_ptr cast into two lines.
- Change the dependent patchset from the non-blocking stack patch series to one only containing the 128b CAS commit

v3:
- Avoid the ABI break by putting 64-bit head and tail values in the same cacheline as struct rte_ring's prod and cons members.
- Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of libraries.
- Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case someone tries to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
- Update the ring mempool to use experimental APIs
- Clarify that RINB_F_NB is only limited to x86_64 currently; e.g. ARMv8 has the ISA support for 128-bit CAS to eventually support it.

v2:
- Merge separate docs commit into patch #5
- Convert uintptr_t to size_t
- Add a compile-time check for the size of size_t
- Fix a space-after-typecast issue
- Fix an unnecessary-parentheses checkpatch warning
- Bump librte_ring's library version

Gage Eads (6):
ring: add a pointer-width headtail structure
ring: add a ring start marker
ring: add a lock-free implementation
test_ring: add lock-free ring autotest
test_ring_perf: add lock-free ring perf test
mempool/ring: add lock-free ring handlers

 doc/guides/prog_guide/env_abstraction_layer.rst |  10 +
 drivers/mempool/ring/Makefile                   |   1 +
 drivers/mempool/ring/meson.build                |   2 +
 drivers/mempool/ring/rte_mempool_ring.c         |  58 ++-
 lib/librte_ring/rte_ring.c                      |  92 ++++-
 lib/librte_ring/rte_ring.h                      | 334 ++++++++++++++--
 lib/librte_ring/rte_ring_c11_mem.h              | 501 ++++++++++++++++++++++++
 lib/librte_ring/rte_ring_generic.h              | 484 +++++++++++++++++++++++
 lib/librte_ring/rte_ring_version.map            |   7 +
 test/test/test_ring.c                           |  61 +--
 test/test/test_ring_perf.c                      |  19 +-
 11 files changed, 1492 insertions(+), 77 deletions(-)
Signature · 1 line
2.13.6
author
#69 lore raw

PATCH v5 1/6 ring: add a pointer-width headtail structure

For 64-bit systems, at current CPU speeds, 64-bit head and tail indexes will not wrap-around within the author's lifetime. This is important to avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming lock-free ring implementation. Using a 64-bit index makes the possibility of this occurring effectively zero. This commit uses pointer-width indexes so the lock-free ring can support 32-bit systems as well.

This commit places the new producer and consumer structures in the same location in struct rte_ring as their 32-bit counterparts. Since the 32-bit versions are padded out to a cache line, there is space for the new structure without affecting the layout of struct rte_ring. Thus, the ABI is preserved.

---
 lib/librte_ring/rte_ring.h         |  21 +++++-
 lib/librte_ring/rte_ring_c11_mem.h | 143 +++++++++++++++++++++++++++++++++++++
 lib/librte_ring/rte_ring_generic.h | 130 +++++++++++++++++++++++++++++++++
 3 files changed, 291 insertions(+), 3 deletions(-)
lib/librte_ring/rte_ring.h +18 -3
index af5444a9f..c78db6916 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007-2009 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -70,6 +70,13 @@ struct rte_ring_headtail {
 	uint32_t single;         /**< True if single prod/cons */
 };
 
+/* Structure to hold a pair of pointer-sized head/tail values and metadata */
+struct rte_ring_headtail_ptr {
+	volatile uintptr_t head; /**< Prod/consumer head. */
+	volatile uintptr_t tail; /**< Prod/consumer tail. */
+	uint32_t single;         /**< True if single prod/cons */
+};
+
 /**
  * An RTE ring structure.
  *
@@ -97,11 +104,19 @@ struct rte_ring {
 	char pad0 __rte_cache_aligned; /**< empty cache line */
 
 	/** Ring producer status. */
-	struct rte_ring_headtail prod __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail prod __rte_cache_aligned;
+		struct rte_ring_headtail_ptr prod_ptr __rte_cache_aligned;
+	};
 	char pad1 __rte_cache_aligned; /**< empty cache line */
 
 	/** Ring consumer status. */
-	struct rte_ring_headtail cons __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail cons __rte_cache_aligned;
+		struct rte_ring_headtail_ptr cons_ptr __rte_cache_aligned;
+	};
 	char pad2 __rte_cache_aligned; /**< empty cache line */
 };
 
lib/librte_ring/rte_ring_c11_mem.h +143 -0

147 lines not on this page · raw message

lib/librte_ring/rte_ring_generic.h +130 -0

134 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#70 lore raw

PATCH v5 2/6 ring: add a ring start marker

In reply to Gage Eads

This marker allows us to replace "&r[1]" with "&r->ring" to locate the start of the ring.

---
 lib/librte_ring/rte_ring.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
lib/librte_ring/rte_ring.h +3 -2
index c78db6916..f16d77b8a 100644
@@ -118,6 +118,7 @@ struct rte_ring {
 		struct rte_ring_headtail_ptr cons_ptr __rte_cache_aligned;
 	};
 	char pad2 __rte_cache_aligned; /**< empty cache line */
+	void *ring[] __rte_cache_aligned; /**< empty marker for ring start */
 };
 
 #define RING_F_SP_ENQ 0x0001 /**< The default enqueue is "single-producer". */
@@ -361,7 +362,7 @@ __rte_ring_do_enqueue(struct rte_ring *r, void * const *obj_table,
 	if (n == 0)
 		goto end;
 
-	ENQUEUE_PTRS(r, &r[1], prod_head, obj_table, n, void *);
+	ENQUEUE_PTRS(r, &r->ring, prod_head, obj_table, n, void *);
 
 	update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
 end:
@@ -403,7 +404,7 @@ __rte_ring_do_dequeue(struct rte_ring *r, void **obj_table,
 	if (n == 0)
 		goto end;
 
-	DEQUEUE_PTRS(r, &r[1], cons_head, obj_table, n, void *);
+	DEQUEUE_PTRS(r, &r->ring, cons_head, obj_table, n, void *);
 
 	update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
 
Signature · 1 line
2.13.6
author
#71 lore raw

PATCH v5 3/6 ring: add a lock-free implementation

In reply to Gage Eads

This commit adds support for lock-free circular ring enqueue and dequeue functions. The ring is supported on 32- and 64-bit architectures, however it uses a 128-bit compare-and-swap instruction when run on a 64-bit architecture, and thus is currently limited to x86_64.

The algorithm is based on Ola Liljedahl's lfring, modified to fit within the rte ring API. With no contention, an enqueue of n pointers uses (1 + n) CAS operations and a dequeue of n pointers uses 1. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the lock-free ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a lock-free ring based mempool (a likely use case for this ring) with per-thread caching.

To avoid the ABA problem, each ring entry contains a modification counter. On a 64-bit architecture, the chance of ABA occurring are effectively zero; a 64-bit counter will take many years to wrap at current CPU frequencies. On a 32-bit architectures, a lock-free ring must be at least 1024-entries deep; assuming 100 cycles per ring entry access, this guarantees the ring's modification counters will wrap on the order of days.

The lock-free ring is enabled via a new flag, RING_F_LF. Because the ring's memsize is now a function of its flags (the lock-free ring requires 128b for each entry), this commit adds a new argument ('flags') to rte_ring_get_memsize(). An API deprecation notice will be sent in a separate commit.

For ease-of-use, existing ring enqueue and dequeue functions work on both regular and lock-free rings. This introduces an additional branch in the datapath, but this should be a highly predictable branch. ring_perf_autotest shows a negligible performance impact; it's hard to distinguish a real difference versus system noise.

| ring_perf_autotest cycles with branch -
Test | ring_perf_autotest cycles without
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | -4.00
SP/SC burst enq/dequeue (size 8)  | 0.00
MP/MC burst enq/dequeue (size 8)  | 0.00
SP/SC burst enq/dequeue (size 32) | 0.00
MP/MC burst enq/dequeue (size 32) | 0.00
SC empty dequeue                  | 1.00
MC empty dequeue                  | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | 0.49
MP/MC bulk enq/dequeue (size 8)   | 0.08
SP/SC bulk enq/dequeue (size 32)  | 0.07
MP/MC bulk enq/dequeue (size 32)  | 0.09

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | 0.19
MP/MC bulk enq/dequeue (size 8)   | -0.37
SP/SC bulk enq/dequeue (size 32)  | 0.09
MP/MC bulk enq/dequeue (size 32)  | -0.05

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | -1.96
MP/MC bulk enq/dequeue (size 8)   | 0.88
SP/SC bulk enq/dequeue (size 32)  | 0.10
MP/MC bulk enq/dequeue (size 32)  | 0.46

Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. Each test run three times and the results averaged.

---
 lib/librte_ring/rte_ring.c           |  92 +++++++--
 lib/librte_ring/rte_ring.h           | 308 ++++++++++++++++++++++++++---
 lib/librte_ring/rte_ring_c11_mem.h   | 366 ++++++++++++++++++++++++++++++++++-
 lib/librte_ring/rte_ring_generic.h   | 354 +++++++++++++++++++++++++++++++++
 lib/librte_ring/rte_ring_version.map |   7 +
 5 files changed, 1080 insertions(+), 47 deletions(-)
lib/librte_ring/rte_ring.c +80 -12
index d215acecc..d4a176f57 100644
@@ -45,9 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
 
 /* return the size of memory occupied by a ring */
 ssize_t
-rte_ring_get_memsize(unsigned count)
+rte_ring_get_memsize_v1905(unsigned int count, unsigned int flags)
 {
-	ssize_t sz;
+	ssize_t sz, elt_sz;
 
 	/* count must be a power of 2 */
 	if ((!POWEROF2(count)) || (count > RTE_RING_SZ_MASK )) {
@@ -57,10 +57,23 @@ rte_ring_get_memsize(unsigned count)
 		return -EINVAL;
 	}
 
-	sz = sizeof(struct rte_ring) + count * sizeof(void *);
+	elt_sz = (flags & RING_F_LF) ? 2 * sizeof(void *) : sizeof(void *);
+
+	sz = sizeof(struct rte_ring) + count * elt_sz;
 	sz = RTE_ALIGN(sz, RTE_CACHE_LINE_SIZE);
 	return sz;
 }
+BIND_DEFAULT_SYMBOL(rte_ring_get_memsize, _v1905, 19.05);
+MAP_STATIC_SYMBOL(ssize_t rte_ring_get_memsize(unsigned int count,
+					       unsigned int flags),
+		  rte_ring_get_memsize_v1905);
+
+ssize_t
+rte_ring_get_memsize_v20(unsigned int count)
+{
+	return rte_ring_get_memsize_v1905(count, 0);
+}
+VERSION_SYMBOL(rte_ring_get_memsize, _v20, 2.0);
 
 int
 rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
@@ -75,6 +88,8 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 			  RTE_CACHE_LINE_MASK) != 0);
 	RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
 			  RTE_CACHE_LINE_MASK) != 0);
+	RTE_BUILD_BUG_ON(sizeof(struct rte_ring_lf_entry) !=
+			 2 * sizeof(void *));
 
 	/* init the ring structure */
 	memset(r, 0, sizeof(*r));
@@ -82,8 +97,6 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 	if (ret < 0 || ret >= (int)sizeof(r->name))
 		return -ENAMETOOLONG;
 	r->flags = flags;
-	r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
-	r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
 
 	if (flags & RING_F_EXACT_SZ) {
 		r->size = rte_align32pow2(count + 1);
@@ -100,12 +113,46 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 		r->mask = count - 1;
 		r->capacity = r->mask;
 	}
-	r->prod.head = r->cons.head = 0;
-	r->prod.tail = r->cons.tail = 0;
+
+	r->log2_size = rte_log2_u64(r->size);
+
+	if (flags & RING_F_LF) {
+		uint32_t i;
+
+		r->prod_ptr.single =
+			(flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
+		r->cons_ptr.single =
+			(flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+		r->prod_ptr.head = r->cons_ptr.head = 0;
+		r->prod_ptr.tail = r->cons_ptr.tail = 0;
+
+		for (i = 0; i < r->size; i++) {
+			struct rte_ring_lf_entry *ring_ptr, *base;
+
+			base = (struct rte_ring_lf_entry *)&r->ring;
+
+			ring_ptr = &base[i & r->mask];
+
+			ring_ptr->cnt = 0;
+		}
+	} else {
+		r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
+		r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+		r->prod.head = r->cons.head = 0;
+		r->prod.tail = r->cons.tail = 0;
+	}
 
 	return 0;
 }
 
+/* If a ring entry is written on average every M cycles, then a ring entry is
+ * reused every M*count cycles, and a ring entry's counter repeats every
+ * M*count*2^32 cycles. If M=100 on a 2GHz system, then a 1024-entry ring's
+ * counters would repeat every 2.37 days. The likelihood of ABA occurring is
+ * considered sufficiently low for 1024-entry and larger rings.
+ */
+#define MIN_32_BIT_LF_RING_SIZE 1024
+
 /* create the ring */
 struct rte_ring *
 rte_ring_create(const char *name, unsigned count, int socket_id,
@@ -123,11 +170,25 @@ rte_ring_create(const char *name, unsigned count, int socket_id,
 
 	ring_list = RTE_TAILQ_CAST(rte_ring_tailq.head, rte_ring_list);
 
+#ifdef RTE_ARCH_64
+#if !defined(RTE_ARCH_X86_64)
+	printf("This platform does not support the atomic operation required for RING_F_LF\n");
+	rte_errno = EINVAL;
+	return NULL;
+#endif
+#else
+	if ((flags & RING_F_LF) && count < MIN_32_BIT_LF_RING_SIZE) {
+		printf("RING_F_LF is only supported on 32-bit platforms for rings with at least 1024 entries.\n");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+#endif
+
 	/* for an exact size ring, round up from count to a power of two */
 	if (flags & RING_F_EXACT_SZ)
 		count = rte_align32pow2(count + 1);
 
-	ring_size = rte_ring_get_memsize(count);
+	ring_size = rte_ring_get_memsize(count, flags);
 	if (ring_size < 0) {
 		rte_errno = ring_size;
 		return NULL;
@@ -227,10 +288,17 @@ rte_ring_dump(FILE *f, const struct rte_ring *r)
 	fprintf(f, "  flags=%x\n", r->flags);
 	fprintf(f, "  size=%"PRIu32"\n", r->size);
 	fprintf(f, "  capacity=%"PRIu32"\n", r->capacity);
-	fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
-	fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
-	fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
-	fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	if (r->flags & RING_F_LF) {
+		fprintf(f, "  ct=%"PRIuPTR"\n", r->cons_ptr.tail);
+		fprintf(f, "  ch=%"PRIuPTR"\n", r->cons_ptr.head);
+		fprintf(f, "  pt=%"PRIuPTR"\n", r->prod_ptr.tail);
+		fprintf(f, "  ph=%"PRIuPTR"\n", r->prod_ptr.head);
+	} else {
+		fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
+		fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
+		fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
+		fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	}
 	fprintf(f, "  used=%u\n", rte_ring_count(r));
 	fprintf(f, "  avail=%u\n", rte_ring_free_count(r));
 }
lib/librte_ring/rte_ring.h +277 -31

449 lines not on this page · raw message

lib/librte_ring/rte_ring_c11_mem.h +362 -4

390 lines not on this page · raw message

lib/librte_ring/rte_ring_generic.h +354 -0

358 lines not on this page · raw message

lib/librte_ring/rte_ring_version.map +7 -0
index d935efd0d..8969467af 100644
@@ -17,3 +17,10 @@ DPDK_2.2 {
 	rte_ring_free;
 
 } DPDK_2.0;
+
+DPDK_19.05 {
+	global:
+
+	rte_ring_get_memsize;
+
+} DPDK_2.2;
Signature · 1 line
2.13.6
author
#72 lore raw

PATCH v5 4/6 test_ring: add lock-free ring autotest

In reply to Gage Eads

ring_nb_autotest re-uses the ring_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 test/test/test_ring.c | 61 ++++++++++++++++++++++++++++++++-------------------
 1 file changed, 38 insertions(+), 23 deletions(-)
test/test/test_ring.c +38 -23
index aaf1e70ad..400b1bffd 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 #include <string.h>
@@ -601,18 +601,20 @@ test_ring_burst_basic(struct rte_ring *r)
  * it will always fail to create ring with a wrong ring size number in this function
  */
 static int
-test_ring_creation_with_wrong_size(void)
+test_ring_creation_with_wrong_size(unsigned int flags)
 {
 	struct rte_ring * rp = NULL;
 
 	/* Test if ring size is not power of 2 */
-	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1,
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
 
 	/* Test if ring size is exceeding the limit */
-	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1), SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1),
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
@@ -623,11 +625,11 @@ test_ring_creation_with_wrong_size(void)
  * it tests if it would always fail to create ring with an used ring name
  */
 static int
-test_ring_creation_with_an_used_name(void)
+test_ring_creation_with_an_used_name(unsigned int flags)
 {
 	struct rte_ring * rp;
 
-	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (NULL != rp)
 		return -1;
 
@@ -639,10 +641,10 @@ test_ring_creation_with_an_used_name(void)
  * function to fail correctly
  */
 static int
-test_create_count_odd(void)
+test_create_count_odd(unsigned int flags)
 {
 	struct rte_ring *r = rte_ring_create("test_ring_count",
-			4097, SOCKET_ID_ANY, 0 );
+			4097, SOCKET_ID_ANY, flags);
 	if(r != NULL){
 		return -1;
 	}
@@ -665,7 +667,7 @@ test_lookup_null(void)
  * it tests some more basic ring operations
  */
 static int
-test_ring_basic_ex(void)
+test_ring_basic_ex(unsigned int flags)
 {
 	int ret = -1;
 	unsigned i;
@@ -679,7 +681,7 @@ test_ring_basic_ex(void)
 	}
 
 	rp = rte_ring_create("test_ring_basic_ex", RING_SIZE, SOCKET_ID_ANY,
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (rp == NULL) {
 		printf("test_ring_basic_ex fail to create ring\n");
 		goto fail_test;
@@ -737,22 +739,22 @@ test_ring_basic_ex(void)
 }
 
 static int
-test_ring_with_exact_size(void)
+test_ring_with_exact_size(unsigned int flags)
 {
 	struct rte_ring *std_ring = NULL, *exact_sz_ring = NULL;
-	void *ptr_array[16];
+	void *ptr_array[1024];
 	static const unsigned int ring_sz = RTE_DIM(ptr_array);
 	unsigned int i;
 	int ret = -1;
 
 	std_ring = rte_ring_create("std", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (std_ring == NULL) {
 		printf("%s: error, can't create std ring\n", __func__);
 		goto end;
 	}
 	exact_sz_ring = rte_ring_create("exact sz", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ);
+		RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ | flags);
 	if (exact_sz_ring == NULL) {
 		printf("%s: error, can't create exact size ring\n", __func__);
 		goto end;
@@ -770,7 +772,7 @@ test_ring_with_exact_size(void)
 	}
 	/*
 	 * check that the exact_sz_ring can hold one more element than the
-	 * standard ring. (16 vs 15 elements)
+	 * standard ring. (1024 vs 1023 elements)
 	 */
 	for (i = 0; i < ring_sz - 1; i++) {
 		rte_ring_enqueue(std_ring, NULL);
@@ -808,17 +810,17 @@ test_ring_with_exact_size(void)
 }
 
 static int
-test_ring(void)
+__test_ring(unsigned int flags)
 {
 	struct rte_ring *r = NULL;
 
 	/* some more basic operations */
-	if (test_ring_basic_ex() < 0)
+	if (test_ring_basic_ex(flags) < 0)
 		goto test_fail;
 
 	rte_atomic32_init(&synchro);
 
-	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (r == NULL)
 		goto test_fail;
 
@@ -837,27 +839,27 @@ test_ring(void)
 		goto test_fail;
 
 	/* basic operations */
-	if ( test_create_count_odd() < 0){
+	if (test_create_count_odd(flags) < 0) {
 		printf("Test failed to detect odd count\n");
 		goto test_fail;
 	} else
 		printf("Test detected odd count\n");
 
-	if ( test_lookup_null() < 0){
+	if (test_lookup_null() < 0) {
 		printf("Test failed to detect NULL ring lookup\n");
 		goto test_fail;
 	} else
 		printf("Test detected NULL ring lookup\n");
 
 	/* test of creating ring with wrong size */
-	if (test_ring_creation_with_wrong_size() < 0)
+	if (test_ring_creation_with_wrong_size(flags) < 0)
 		goto test_fail;
 
 	/* test of creation ring with an used name */
-	if (test_ring_creation_with_an_used_name() < 0)
+	if (test_ring_creation_with_an_used_name(flags) < 0)
 		goto test_fail;
 
-	if (test_ring_with_exact_size() < 0)
+	if (test_ring_with_exact_size(flags) < 0)
 		goto test_fail;
 
 	/* dump the ring status */
@@ -873,4 +875,17 @@ test_ring(void)
 	return -1;
 }
 
+static int
+test_ring(void)
+{
+	return __test_ring(0);
+}
+
+static int
+test_lf_ring(void)
+{
+	return __test_ring(RING_F_LF);
+}
+
 REGISTER_TEST_COMMAND(ring_autotest, test_ring);
+REGISTER_TEST_COMMAND(ring_lf_autotest, test_lf_ring);
Signature · 1 line
2.13.6
author
#73 lore raw

PATCH v5 5/6 test_ring_perf: add lock-free ring perf test

In reply to Gage Eads

nb_ring_perf_autotest re-uses the ring_perf_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 test/test/test_ring_perf.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
test/test/test_ring_perf.c +16 -3
index ebb3939f5..be465c758 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 
@@ -363,12 +363,12 @@ test_bulk_enqueue_dequeue(struct rte_ring *r)
 }
 
 static int
-test_ring_perf(void)
+__test_ring_perf(unsigned int flags)
 {
 	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
 
-	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), 0);
+	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), flags);
 	if (r == NULL)
 		return -1;
 
@@ -398,4 +398,17 @@ test_ring_perf(void)
 	return 0;
 }
 
+static int
+test_ring_perf(void)
+{
+	return __test_ring_perf(0);
+}
+
+static int
+test_lf_ring_perf(void)
+{
+	return __test_ring_perf(RING_F_LF);
+}
+
 REGISTER_TEST_COMMAND(ring_perf_autotest, test_ring_perf);
+REGISTER_TEST_COMMAND(ring_lf_perf_autotest, test_lf_ring_perf);
Signature · 1 line
2.13.6
author
#74 lore raw

PATCH v5 6/6 mempool/ring: add lock-free ring handlers

In reply to Gage Eads

These handlers allow an application to create a mempool based on the lock-free ring, with any combination of single/multi producer/consumer.

Also, add a note to the programmer's guide's "known issues" section.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 +++++
 drivers/mempool/ring/Makefile                   |  1 +
 drivers/mempool/ring/meson.build                |  2 +
 drivers/mempool/ring/rte_mempool_ring.c         | 58 +++++++++++++++++++++++--
 4 files changed, 68 insertions(+), 3 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
index 929d76dba..2e2516465 100644
@@ -541,6 +541,16 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
+  Alternatively, 32-bit and x86_64 applications can use the lock-free ring
+  mempool handler. When considering it, note that:
+
+  - Among 64-bit architectures it is currently limited to the x86_64 platform,
+    because it uses a function (16-byte compare-and-swap) that is not yet
+    available on other platforms.
+  - It has worse average-case performance than the non-preemptive rte_ring, but
+    software caching (e.g. the mempool cache) can mitigate this by reducing the
+    number of handler operations.
+
 + rte_timer
 
   Running  ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
drivers/mempool/ring/Makefile +1 -0
index ddab522fe..012ba6966 100644
@@ -10,6 +10,7 @@ LIB = librte_mempool_ring.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 LDLIBS += -lrte_eal -lrte_mempool -lrte_ring
 
 EXPORT_MAP := rte_mempool_ring_version.map
drivers/mempool/ring/meson.build +2 -0
index a021e908c..b1cb673cc 100644
@@ -1,4 +1,6 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
+allow_experimental_apis = true
+
 sources = files('rte_mempool_ring.c')
drivers/mempool/ring/rte_mempool_ring.c +55 -3

84 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#75 lore raw

v6 cover Add lock-free ring and mempool handler

In reply to Gage Eads

For some users, the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

This patchset introduces a lock-free ring and a mempool based on it. The lock-free algorithm relies on a double-pointer compare-and-swap, so for 64-bit architectures it is currently limited to x86_64.

The ring uses more compare-and-swap atomic operations than the regular rte ring:
With no contention, an enqueue of n pointers uses (1 + n) CAS operations and a dequeue of n pointers uses 1. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the lock-free ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a lock-free ring based mempool (a likely use case for this ring) with per-thread caching.

The lock-free ring is enabled via a new flag, RING_F_LF. For ease-of-use, existing ring enqueue/dequeue functions work with both standard and lock-free rings. This is also an experimental API, so RING_F_LF users must build with the ALLOW_EXPERIMENTAL_API flag.

This patchset also adds lock-free versions of ring_autotest and ring_perf_autotest, and a lock-free ring based mempool.

This patchset makes one API change; a deprecation notice was posted in a separate commit[1].

This patchset depends on the 128-bit compare-and-set patch[2].

[1] http://mails.dpdk.org/archives/dev/2019-February/124321.html
[2] http://mails.dpdk.org/archives/dev/2019-March/125751.html

v6:
- Rebase patchset onto master (test/test/ -> app/test/)

v5:
- Incorporated lfring's enqueue and dequeue logic from
http://mails.dpdk.org/archives/dev/2019-January/124242.html
- Renamed non-blocking -> lock-free and NB -> LF to align with a similar change in the lock-free stack patchset:
http://mails.dpdk.org/archives/dev/2019-March/125797.html
- Added support for 32-bit architectures by using the full 32b of the modification counter and requiring LF rings on these architectures to be at least 1024 entries.
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Added ring start marker to struct rte_ring

v4:
- Split out nb_enqueue and nb_dequeue functions in generic and C11 versions, with the necessary memory ordering behavior for weakly consistent machines.
- Convert size_t variables (from v2) to uint64_t and no-longer-applicable comment about variably-sized ring indexes.
- Fix bug in nb_enqueue_mp that the breaks the non-blocking guarantee.
- Split the ring_ptr cast into two lines.
- Change the dependent patchset from the non-blocking stack patch series to one only containing the 128b CAS commit

v3:
- Avoid the ABI break by putting 64-bit head and tail values in the same cacheline as struct rte_ring's prod and cons members.
- Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of libraries.
- Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case someone tries to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
- Update the ring mempool to use experimental APIs
- Clarify that RINB_F_NB is only limited to x86_64 currently; e.g. ARMv8 has the ISA support for 128-bit CAS to eventually support it.

v2:
- Merge separate docs commit into patch #5
- Convert uintptr_t to size_t
- Add a compile-time check for the size of size_t
- Fix a space-after-typecast issue
- Fix an unnecessary-parentheses checkpatch warning
- Bump librte_ring's library version

Gage Eads (6):
ring: add a pointer-width headtail structure
ring: add a ring start marker
ring: add a lock-free implementation
test_ring: add lock-free ring autotest
test_ring_perf: add lock-free ring perf test
mempool/ring: add lock-free ring handlers

 app/test/test_ring.c                            |  61 +--
 app/test/test_ring_perf.c                       |  19 +-
 doc/guides/prog_guide/env_abstraction_layer.rst |  10 +
 drivers/mempool/ring/Makefile                   |   1 +
 drivers/mempool/ring/meson.build                |   2 +
 drivers/mempool/ring/rte_mempool_ring.c         |  58 ++-
 lib/librte_ring/rte_ring.c                      |  92 ++++-
 lib/librte_ring/rte_ring.h                      | 334 ++++++++++++++--
 lib/librte_ring/rte_ring_c11_mem.h              | 501 ++++++++++++++++++++++++
 lib/librte_ring/rte_ring_generic.h              | 484 +++++++++++++++++++++++
 lib/librte_ring/rte_ring_version.map            |   7 +
 11 files changed, 1492 insertions(+), 77 deletions(-)
Signature · 1 line
2.13.6
author
#76 lore raw

PATCH v6 1/6 ring: add a pointer-width headtail structure

For 64-bit systems, at current CPU speeds, 64-bit head and tail indexes will not wrap-around within the author's lifetime. This is important to avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming lock-free ring implementation. Using a 64-bit index makes the possibility of this occurring effectively zero. This commit uses pointer-width indexes so the lock-free ring can support 32-bit systems as well.

This commit places the new producer and consumer structures in the same location in struct rte_ring as their 32-bit counterparts. Since the 32-bit versions are padded out to a cache line, there is space for the new structure without affecting the layout of struct rte_ring. Thus, the ABI is preserved.

---
 lib/librte_ring/rte_ring.h         |  21 +++++-
 lib/librte_ring/rte_ring_c11_mem.h | 143 +++++++++++++++++++++++++++++++++++++
 lib/librte_ring/rte_ring_generic.h | 130 +++++++++++++++++++++++++++++++++
 3 files changed, 291 insertions(+), 3 deletions(-)
lib/librte_ring/rte_ring.h +18 -3
index af5444a9f..c78db6916 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007-2009 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -70,6 +70,13 @@ struct rte_ring_headtail {
 	uint32_t single;         /**< True if single prod/cons */
 };
 
+/* Structure to hold a pair of pointer-sized head/tail values and metadata */
+struct rte_ring_headtail_ptr {
+	volatile uintptr_t head; /**< Prod/consumer head. */
+	volatile uintptr_t tail; /**< Prod/consumer tail. */
+	uint32_t single;         /**< True if single prod/cons */
+};
+
 /**
  * An RTE ring structure.
  *
@@ -97,11 +104,19 @@ struct rte_ring {
 	char pad0 __rte_cache_aligned; /**< empty cache line */
 
 	/** Ring producer status. */
-	struct rte_ring_headtail prod __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail prod __rte_cache_aligned;
+		struct rte_ring_headtail_ptr prod_ptr __rte_cache_aligned;
+	};
 	char pad1 __rte_cache_aligned; /**< empty cache line */
 
 	/** Ring consumer status. */
-	struct rte_ring_headtail cons __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail cons __rte_cache_aligned;
+		struct rte_ring_headtail_ptr cons_ptr __rte_cache_aligned;
+	};
 	char pad2 __rte_cache_aligned; /**< empty cache line */
 };
 
lib/librte_ring/rte_ring_c11_mem.h +143 -0

147 lines not on this page · raw message

lib/librte_ring/rte_ring_generic.h +130 -0

134 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#77 lore raw

PATCH v6 2/6 ring: add a ring start marker

In reply to Gage Eads

This marker allows us to replace "&r[1]" with "&r->ring" to locate the start of the ring.

---
 lib/librte_ring/rte_ring.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
lib/librte_ring/rte_ring.h +3 -2
index c78db6916..f16d77b8a 100644
@@ -118,6 +118,7 @@ struct rte_ring {
 		struct rte_ring_headtail_ptr cons_ptr __rte_cache_aligned;
 	};
 	char pad2 __rte_cache_aligned; /**< empty cache line */
+	void *ring[] __rte_cache_aligned; /**< empty marker for ring start */
 };
 
 #define RING_F_SP_ENQ 0x0001 /**< The default enqueue is "single-producer". */
@@ -361,7 +362,7 @@ __rte_ring_do_enqueue(struct rte_ring *r, void * const *obj_table,
 	if (n == 0)
 		goto end;
 
-	ENQUEUE_PTRS(r, &r[1], prod_head, obj_table, n, void *);
+	ENQUEUE_PTRS(r, &r->ring, prod_head, obj_table, n, void *);
 
 	update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
 end:
@@ -403,7 +404,7 @@ __rte_ring_do_dequeue(struct rte_ring *r, void **obj_table,
 	if (n == 0)
 		goto end;
 
-	DEQUEUE_PTRS(r, &r[1], cons_head, obj_table, n, void *);
+	DEQUEUE_PTRS(r, &r->ring, cons_head, obj_table, n, void *);
 
 	update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
 
Signature · 1 line
2.13.6
author
#78 lore raw

PATCH v6 3/6 ring: add a lock-free implementation

In reply to Gage Eads

This commit adds support for lock-free circular ring enqueue and dequeue functions. The ring is supported on 32- and 64-bit architectures, however it uses a 128-bit compare-and-swap instruction when run on a 64-bit architecture, and thus is currently limited to x86_64.

The algorithm is based on Ola Liljedahl's lfring, modified to fit within the rte ring API. With no contention, an enqueue of n pointers uses (1 + n) CAS operations and a dequeue of n pointers uses 1. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the lock-free ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a lock-free ring based mempool (a likely use case for this ring) with per-thread caching.

To avoid the ABA problem, each ring entry contains a modification counter. On a 64-bit architecture, the chance of ABA occurring are effectively zero; a 64-bit counter will take many years to wrap at current CPU frequencies. On a 32-bit architectures, a lock-free ring must be at least 1024-entries deep; assuming 100 cycles per ring entry access, this guarantees the ring's modification counters will wrap on the order of days.

The lock-free ring is enabled via a new flag, RING_F_LF. Because the ring's memsize is now a function of its flags (the lock-free ring requires 128b for each entry), this commit adds a new argument ('flags') to rte_ring_get_memsize(). An API deprecation notice will be sent in a separate commit.

For ease-of-use, existing ring enqueue and dequeue functions work on both regular and lock-free rings. This introduces an additional branch in the datapath, but this should be a highly predictable branch. ring_perf_autotest shows a negligible performance impact; it's hard to distinguish a real difference versus system noise.

| ring_perf_autotest cycles with branch -
Test | ring_perf_autotest cycles without
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | -4.00
SP/SC burst enq/dequeue (size 8)  | 0.00
MP/MC burst enq/dequeue (size 8)  | 0.00
SP/SC burst enq/dequeue (size 32) | 0.00
MP/MC burst enq/dequeue (size 32) | 0.00
SC empty dequeue                  | 1.00
MC empty dequeue                  | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | 0.49
MP/MC bulk enq/dequeue (size 8)   | 0.08
SP/SC bulk enq/dequeue (size 32)  | 0.07
MP/MC bulk enq/dequeue (size 32)  | 0.09

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | 0.19
MP/MC bulk enq/dequeue (size 8)   | -0.37
SP/SC bulk enq/dequeue (size 32)  | 0.09
MP/MC bulk enq/dequeue (size 32)  | -0.05

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | -1.96
MP/MC bulk enq/dequeue (size 8)   | 0.88
SP/SC bulk enq/dequeue (size 32)  | 0.10
MP/MC bulk enq/dequeue (size 32)  | 0.46

Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. Each test run three times and the results averaged.

---
 lib/librte_ring/rte_ring.c           |  92 +++++++--
 lib/librte_ring/rte_ring.h           | 308 ++++++++++++++++++++++++++---
 lib/librte_ring/rte_ring_c11_mem.h   | 366 ++++++++++++++++++++++++++++++++++-
 lib/librte_ring/rte_ring_generic.h   | 354 +++++++++++++++++++++++++++++++++
 lib/librte_ring/rte_ring_version.map |   7 +
 5 files changed, 1080 insertions(+), 47 deletions(-)
lib/librte_ring/rte_ring.c +80 -12
index d215acecc..d4a176f57 100644
@@ -45,9 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
 
 /* return the size of memory occupied by a ring */
 ssize_t
-rte_ring_get_memsize(unsigned count)
+rte_ring_get_memsize_v1905(unsigned int count, unsigned int flags)
 {
-	ssize_t sz;
+	ssize_t sz, elt_sz;
 
 	/* count must be a power of 2 */
 	if ((!POWEROF2(count)) || (count > RTE_RING_SZ_MASK )) {
@@ -57,10 +57,23 @@ rte_ring_get_memsize(unsigned count)
 		return -EINVAL;
 	}
 
-	sz = sizeof(struct rte_ring) + count * sizeof(void *);
+	elt_sz = (flags & RING_F_LF) ? 2 * sizeof(void *) : sizeof(void *);
+
+	sz = sizeof(struct rte_ring) + count * elt_sz;
 	sz = RTE_ALIGN(sz, RTE_CACHE_LINE_SIZE);
 	return sz;
 }
+BIND_DEFAULT_SYMBOL(rte_ring_get_memsize, _v1905, 19.05);
+MAP_STATIC_SYMBOL(ssize_t rte_ring_get_memsize(unsigned int count,
+					       unsigned int flags),
+		  rte_ring_get_memsize_v1905);
+
+ssize_t
+rte_ring_get_memsize_v20(unsigned int count)
+{
+	return rte_ring_get_memsize_v1905(count, 0);
+}
+VERSION_SYMBOL(rte_ring_get_memsize, _v20, 2.0);
 
 int
 rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
@@ -75,6 +88,8 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 			  RTE_CACHE_LINE_MASK) != 0);
 	RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
 			  RTE_CACHE_LINE_MASK) != 0);
+	RTE_BUILD_BUG_ON(sizeof(struct rte_ring_lf_entry) !=
+			 2 * sizeof(void *));
 
 	/* init the ring structure */
 	memset(r, 0, sizeof(*r));
@@ -82,8 +97,6 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 	if (ret < 0 || ret >= (int)sizeof(r->name))
 		return -ENAMETOOLONG;
 	r->flags = flags;
-	r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
-	r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
 
 	if (flags & RING_F_EXACT_SZ) {
 		r->size = rte_align32pow2(count + 1);
@@ -100,12 +113,46 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 		r->mask = count - 1;
 		r->capacity = r->mask;
 	}
-	r->prod.head = r->cons.head = 0;
-	r->prod.tail = r->cons.tail = 0;
+
+	r->log2_size = rte_log2_u64(r->size);
+
+	if (flags & RING_F_LF) {
+		uint32_t i;
+
+		r->prod_ptr.single =
+			(flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
+		r->cons_ptr.single =
+			(flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+		r->prod_ptr.head = r->cons_ptr.head = 0;
+		r->prod_ptr.tail = r->cons_ptr.tail = 0;
+
+		for (i = 0; i < r->size; i++) {
+			struct rte_ring_lf_entry *ring_ptr, *base;
+
+			base = (struct rte_ring_lf_entry *)&r->ring;
+
+			ring_ptr = &base[i & r->mask];
+
+			ring_ptr->cnt = 0;
+		}
+	} else {
+		r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
+		r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+		r->prod.head = r->cons.head = 0;
+		r->prod.tail = r->cons.tail = 0;
+	}
 
 	return 0;
 }
 
+/* If a ring entry is written on average every M cycles, then a ring entry is
+ * reused every M*count cycles, and a ring entry's counter repeats every
+ * M*count*2^32 cycles. If M=100 on a 2GHz system, then a 1024-entry ring's
+ * counters would repeat every 2.37 days. The likelihood of ABA occurring is
+ * considered sufficiently low for 1024-entry and larger rings.
+ */
+#define MIN_32_BIT_LF_RING_SIZE 1024
+
 /* create the ring */
 struct rte_ring *
 rte_ring_create(const char *name, unsigned count, int socket_id,
@@ -123,11 +170,25 @@ rte_ring_create(const char *name, unsigned count, int socket_id,
 
 	ring_list = RTE_TAILQ_CAST(rte_ring_tailq.head, rte_ring_list);
 
+#ifdef RTE_ARCH_64
+#if !defined(RTE_ARCH_X86_64)
+	printf("This platform does not support the atomic operation required for RING_F_LF\n");
+	rte_errno = EINVAL;
+	return NULL;
+#endif
+#else
+	if ((flags & RING_F_LF) && count < MIN_32_BIT_LF_RING_SIZE) {
+		printf("RING_F_LF is only supported on 32-bit platforms for rings with at least 1024 entries.\n");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+#endif
+
 	/* for an exact size ring, round up from count to a power of two */
 	if (flags & RING_F_EXACT_SZ)
 		count = rte_align32pow2(count + 1);
 
-	ring_size = rte_ring_get_memsize(count);
+	ring_size = rte_ring_get_memsize(count, flags);
 	if (ring_size < 0) {
 		rte_errno = ring_size;
 		return NULL;
@@ -227,10 +288,17 @@ rte_ring_dump(FILE *f, const struct rte_ring *r)
 	fprintf(f, "  flags=%x\n", r->flags);
 	fprintf(f, "  size=%"PRIu32"\n", r->size);
 	fprintf(f, "  capacity=%"PRIu32"\n", r->capacity);
-	fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
-	fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
-	fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
-	fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	if (r->flags & RING_F_LF) {
+		fprintf(f, "  ct=%"PRIuPTR"\n", r->cons_ptr.tail);
+		fprintf(f, "  ch=%"PRIuPTR"\n", r->cons_ptr.head);
+		fprintf(f, "  pt=%"PRIuPTR"\n", r->prod_ptr.tail);
+		fprintf(f, "  ph=%"PRIuPTR"\n", r->prod_ptr.head);
+	} else {
+		fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
+		fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
+		fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
+		fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	}
 	fprintf(f, "  used=%u\n", rte_ring_count(r));
 	fprintf(f, "  avail=%u\n", rte_ring_free_count(r));
 }
lib/librte_ring/rte_ring.h +277 -31

449 lines not on this page · raw message

lib/librte_ring/rte_ring_c11_mem.h +362 -4

390 lines not on this page · raw message

lib/librte_ring/rte_ring_generic.h +354 -0

358 lines not on this page · raw message

lib/librte_ring/rte_ring_version.map +7 -0
index d935efd0d..8969467af 100644
@@ -17,3 +17,10 @@ DPDK_2.2 {
 	rte_ring_free;
 
 } DPDK_2.0;
+
+DPDK_19.05 {
+	global:
+
+	rte_ring_get_memsize;
+
+} DPDK_2.2;
Signature · 1 line
2.13.6
author
#79 lore raw

PATCH v6 4/6 test_ring: add lock-free ring autotest

In reply to Gage Eads

ring_nb_autotest re-uses the ring_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 app/test/test_ring.c | 61 ++++++++++++++++++++++++++++++++--------------------
 1 file changed, 38 insertions(+), 23 deletions(-)
app/test/test_ring.c +38 -23
index aaf1e70ad..400b1bffd 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 #include <string.h>
@@ -601,18 +601,20 @@ test_ring_burst_basic(struct rte_ring *r)
  * it will always fail to create ring with a wrong ring size number in this function
  */
 static int
-test_ring_creation_with_wrong_size(void)
+test_ring_creation_with_wrong_size(unsigned int flags)
 {
 	struct rte_ring * rp = NULL;
 
 	/* Test if ring size is not power of 2 */
-	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1,
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
 
 	/* Test if ring size is exceeding the limit */
-	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1), SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1),
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
@@ -623,11 +625,11 @@ test_ring_creation_with_wrong_size(void)
  * it tests if it would always fail to create ring with an used ring name
  */
 static int
-test_ring_creation_with_an_used_name(void)
+test_ring_creation_with_an_used_name(unsigned int flags)
 {
 	struct rte_ring * rp;
 
-	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (NULL != rp)
 		return -1;
 
@@ -639,10 +641,10 @@ test_ring_creation_with_an_used_name(void)
  * function to fail correctly
  */
 static int
-test_create_count_odd(void)
+test_create_count_odd(unsigned int flags)
 {
 	struct rte_ring *r = rte_ring_create("test_ring_count",
-			4097, SOCKET_ID_ANY, 0 );
+			4097, SOCKET_ID_ANY, flags);
 	if(r != NULL){
 		return -1;
 	}
@@ -665,7 +667,7 @@ test_lookup_null(void)
  * it tests some more basic ring operations
  */
 static int
-test_ring_basic_ex(void)
+test_ring_basic_ex(unsigned int flags)
 {
 	int ret = -1;
 	unsigned i;
@@ -679,7 +681,7 @@ test_ring_basic_ex(void)
 	}
 
 	rp = rte_ring_create("test_ring_basic_ex", RING_SIZE, SOCKET_ID_ANY,
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (rp == NULL) {
 		printf("test_ring_basic_ex fail to create ring\n");
 		goto fail_test;
@@ -737,22 +739,22 @@ test_ring_basic_ex(void)
 }
 
 static int
-test_ring_with_exact_size(void)
+test_ring_with_exact_size(unsigned int flags)
 {
 	struct rte_ring *std_ring = NULL, *exact_sz_ring = NULL;
-	void *ptr_array[16];
+	void *ptr_array[1024];
 	static const unsigned int ring_sz = RTE_DIM(ptr_array);
 	unsigned int i;
 	int ret = -1;
 
 	std_ring = rte_ring_create("std", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (std_ring == NULL) {
 		printf("%s: error, can't create std ring\n", __func__);
 		goto end;
 	}
 	exact_sz_ring = rte_ring_create("exact sz", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ);
+		RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ | flags);
 	if (exact_sz_ring == NULL) {
 		printf("%s: error, can't create exact size ring\n", __func__);
 		goto end;
@@ -770,7 +772,7 @@ test_ring_with_exact_size(void)
 	}
 	/*
 	 * check that the exact_sz_ring can hold one more element than the
-	 * standard ring. (16 vs 15 elements)
+	 * standard ring. (1024 vs 1023 elements)
 	 */
 	for (i = 0; i < ring_sz - 1; i++) {
 		rte_ring_enqueue(std_ring, NULL);
@@ -808,17 +810,17 @@ test_ring_with_exact_size(void)
 }
 
 static int
-test_ring(void)
+__test_ring(unsigned int flags)
 {
 	struct rte_ring *r = NULL;
 
 	/* some more basic operations */
-	if (test_ring_basic_ex() < 0)
+	if (test_ring_basic_ex(flags) < 0)
 		goto test_fail;
 
 	rte_atomic32_init(&synchro);
 
-	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (r == NULL)
 		goto test_fail;
 
@@ -837,27 +839,27 @@ test_ring(void)
 		goto test_fail;
 
 	/* basic operations */
-	if ( test_create_count_odd() < 0){
+	if (test_create_count_odd(flags) < 0) {
 		printf("Test failed to detect odd count\n");
 		goto test_fail;
 	} else
 		printf("Test detected odd count\n");
 
-	if ( test_lookup_null() < 0){
+	if (test_lookup_null() < 0) {
 		printf("Test failed to detect NULL ring lookup\n");
 		goto test_fail;
 	} else
 		printf("Test detected NULL ring lookup\n");
 
 	/* test of creating ring with wrong size */
-	if (test_ring_creation_with_wrong_size() < 0)
+	if (test_ring_creation_with_wrong_size(flags) < 0)
 		goto test_fail;
 
 	/* test of creation ring with an used name */
-	if (test_ring_creation_with_an_used_name() < 0)
+	if (test_ring_creation_with_an_used_name(flags) < 0)
 		goto test_fail;
 
-	if (test_ring_with_exact_size() < 0)
+	if (test_ring_with_exact_size(flags) < 0)
 		goto test_fail;
 
 	/* dump the ring status */
@@ -873,4 +875,17 @@ test_ring(void)
 	return -1;
 }
 
+static int
+test_ring(void)
+{
+	return __test_ring(0);
+}
+
+static int
+test_lf_ring(void)
+{
+	return __test_ring(RING_F_LF);
+}
+
 REGISTER_TEST_COMMAND(ring_autotest, test_ring);
+REGISTER_TEST_COMMAND(ring_lf_autotest, test_lf_ring);
Signature · 1 line
2.13.6
author
#80 lore raw

PATCH v6 5/6 test_ring_perf: add lock-free ring perf test

In reply to Gage Eads

nb_ring_perf_autotest re-uses the ring_perf_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 app/test/test_ring_perf.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
app/test/test_ring_perf.c +16 -3
index ebb3939f5..be465c758 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 
@@ -363,12 +363,12 @@ test_bulk_enqueue_dequeue(struct rte_ring *r)
 }
 
 static int
-test_ring_perf(void)
+__test_ring_perf(unsigned int flags)
 {
 	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
 
-	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), 0);
+	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), flags);
 	if (r == NULL)
 		return -1;
 
@@ -398,4 +398,17 @@ test_ring_perf(void)
 	return 0;
 }
 
+static int
+test_ring_perf(void)
+{
+	return __test_ring_perf(0);
+}
+
+static int
+test_lf_ring_perf(void)
+{
+	return __test_ring_perf(RING_F_LF);
+}
+
 REGISTER_TEST_COMMAND(ring_perf_autotest, test_ring_perf);
+REGISTER_TEST_COMMAND(ring_lf_perf_autotest, test_lf_ring_perf);
Signature · 1 line
2.13.6
author
#81 lore raw

PATCH v6 6/6 mempool/ring: add lock-free ring handlers

In reply to Gage Eads

These handlers allow an application to create a mempool based on the lock-free ring, with any combination of single/multi producer/consumer.

Also, add a note to the programmer's guide's "known issues" section.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 +++++
 drivers/mempool/ring/Makefile                   |  1 +
 drivers/mempool/ring/meson.build                |  2 +
 drivers/mempool/ring/rte_mempool_ring.c         | 58 +++++++++++++++++++++++--
 4 files changed, 68 insertions(+), 3 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
index 929d76dba..2e2516465 100644
@@ -541,6 +541,16 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
+  Alternatively, 32-bit and x86_64 applications can use the lock-free ring
+  mempool handler. When considering it, note that:
+
+  - Among 64-bit architectures it is currently limited to the x86_64 platform,
+    because it uses a function (16-byte compare-and-swap) that is not yet
+    available on other platforms.
+  - It has worse average-case performance than the non-preemptive rte_ring, but
+    software caching (e.g. the mempool cache) can mitigate this by reducing the
+    number of handler operations.
+
 + rte_timer
 
   Running  ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
drivers/mempool/ring/Makefile +1 -0
index ddab522fe..012ba6966 100644
@@ -10,6 +10,7 @@ LIB = librte_mempool_ring.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 LDLIBS += -lrte_eal -lrte_mempool -lrte_ring
 
 EXPORT_MAP := rte_mempool_ring_version.map
drivers/mempool/ring/meson.build +2 -0
index a021e908c..b1cb673cc 100644
@@ -1,4 +1,6 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
+allow_experimental_apis = true
+
 sources = files('rte_mempool_ring.c')
drivers/mempool/ring/rte_mempool_ring.c +55 -3

84 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#82 lore raw

v7 cover Add lock-free ring and mempool handler

In reply to Gage Eads

For some users, the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

This patchset introduces a lock-free ring and a mempool based on it. The lock-free algorithm relies on a double-pointer compare-and-swap, so for 64-bit architectures it is currently limited to x86_64.

The ring uses more compare-and-swap atomic operations than the regular rte ring:
With no contention, an enqueue of n pointers uses (1 + n) CAS operations and a dequeue of n pointers uses 1. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the lock-free ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a lock-free ring based mempool (a likely use case for this ring) with per-thread caching.

The lock-free ring is enabled via a new flag, RING_F_LF. For ease-of-use, existing ring enqueue/dequeue functions work with both standard and lock-free rings. This is also an experimental API, so RING_F_LF users must build with the ALLOW_EXPERIMENTAL_API flag.

This patchset also adds lock-free versions of ring_autotest and ring_perf_autotest, and a lock-free ring based mempool.

This patchset makes one API change; a deprecation notice was posted in a separate commit[1].

This patchset depends on the 128-bit compare-and-set patch[2].

[1] http://mails.dpdk.org/archives/dev/2019-February/124321.html
[2] http://mails.dpdk.org/archives/dev/2019-March/125751.html

v7:
- Added ARM copyright to rte_ring_generic.h and rte_ring_c11_mem.h, since the lock-free algorithm is based on ARM's lfring (see v5 notes)
- Rename __rte_ring_reload_tail() -> __rte_ring_lf_load_tail()
- Remove the unused return value from __rte_ring_lf_load_tail()
- Rename 'prev_tail' to 'next_tail' in the multi-producer lock-free enqueue

v6:
- Rebase patchset onto master (test/test/ -> app/test/)

v5:
- Incorporated lfring's enqueue and dequeue logic from
http://mails.dpdk.org/archives/dev/2019-January/124242.html
- Renamed non-blocking -> lock-free and NB -> LF to align with a similar change in the lock-free stack patchset:
http://mails.dpdk.org/archives/dev/2019-March/125797.html
- Added support for 32-bit architectures by using the full 32b of the modification counter and requiring LF rings on these architectures to be at least 1024 entries.
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Added ring start marker to struct rte_ring

v4:
- Split out nb_enqueue and nb_dequeue functions in generic and C11 versions, with the necessary memory ordering behavior for weakly consistent machines.
- Convert size_t variables (from v2) to uint64_t and no-longer-applicable comment about variably-sized ring indexes.
- Fix bug in nb_enqueue_mp that the breaks the non-blocking guarantee.
- Split the ring_ptr cast into two lines.
- Change the dependent patchset from the non-blocking stack patch series to one only containing the 128b CAS commit

v3:
- Avoid the ABI break by putting 64-bit head and tail values in the same cacheline as struct rte_ring's prod and cons members.
- Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of libraries.
- Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case someone tries to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
- Update the ring mempool to use experimental APIs
- Clarify that RINB_F_NB is only limited to x86_64 currently; e.g. ARMv8 has the ISA support for 128-bit CAS to eventually support it.

v2:
- Merge separate docs commit into patch #5
- Convert uintptr_t to size_t
- Add a compile-time check for the size of size_t
- Fix a space-after-typecast issue
- Fix an unnecessary-parentheses checkpatch warning
- Bump librte_ring's library version

Gage Eads (6):
ring: add a pointer-width headtail structure
ring: add a ring start marker
ring: add a lock-free implementation
test_ring: add lock-free ring autotest
test_ring_perf: add lock-free ring perf test
mempool/ring: add lock-free ring handlers

 app/test/test_ring.c                            |  61 +--
 app/test/test_ring_perf.c                       |  19 +-
 doc/guides/prog_guide/env_abstraction_layer.rst |  10 +
 drivers/mempool/ring/Makefile                   |   1 +
 drivers/mempool/ring/meson.build                |   2 +
 drivers/mempool/ring/rte_mempool_ring.c         |  58 ++-
 lib/librte_ring/rte_ring.c                      |  92 ++++-
 lib/librte_ring/rte_ring.h                      | 334 ++++++++++++++--
 lib/librte_ring/rte_ring_c11_mem.h              | 501 ++++++++++++++++++++++++
 lib/librte_ring/rte_ring_generic.h              | 485 ++++++++++++++++++++++-
 lib/librte_ring/rte_ring_version.map            |   7 +
 11 files changed, 1492 insertions(+), 78 deletions(-)
Signature · 1 line
2.13.6
author
#83 lore raw

PATCH v7 1/6 ring: add a pointer-width headtail structure

For 64-bit systems, at current CPU speeds, 64-bit head and tail indexes will not wrap-around within the author's lifetime. This is important to avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming lock-free ring implementation. Using a 64-bit index makes the possibility of this occurring effectively zero. This commit uses pointer-width indexes so the lock-free ring can support 32-bit systems as well.

This commit places the new producer and consumer structures in the same location in struct rte_ring as their 32-bit counterparts. Since the 32-bit versions are padded out to a cache line, there is space for the new structure without affecting the layout of struct rte_ring. Thus, the ABI is preserved.

---
 lib/librte_ring/rte_ring.h         |  21 +++++-
 lib/librte_ring/rte_ring_c11_mem.h | 143 +++++++++++++++++++++++++++++++++++++
 lib/librte_ring/rte_ring_generic.h | 130 +++++++++++++++++++++++++++++++++
 3 files changed, 291 insertions(+), 3 deletions(-)
lib/librte_ring/rte_ring.h +18 -3
index af5444a9f..c78db6916 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007-2009 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -70,6 +70,13 @@ struct rte_ring_headtail {
 	uint32_t single;         /**< True if single prod/cons */
 };
 
+/* Structure to hold a pair of pointer-sized head/tail values and metadata */
+struct rte_ring_headtail_ptr {
+	volatile uintptr_t head; /**< Prod/consumer head. */
+	volatile uintptr_t tail; /**< Prod/consumer tail. */
+	uint32_t single;         /**< True if single prod/cons */
+};
+
 /**
  * An RTE ring structure.
  *
@@ -97,11 +104,19 @@ struct rte_ring {
 	char pad0 __rte_cache_aligned; /**< empty cache line */
 
 	/** Ring producer status. */
-	struct rte_ring_headtail prod __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail prod __rte_cache_aligned;
+		struct rte_ring_headtail_ptr prod_ptr __rte_cache_aligned;
+	};
 	char pad1 __rte_cache_aligned; /**< empty cache line */
 
 	/** Ring consumer status. */
-	struct rte_ring_headtail cons __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail cons __rte_cache_aligned;
+		struct rte_ring_headtail_ptr cons_ptr __rte_cache_aligned;
+	};
 	char pad2 __rte_cache_aligned; /**< empty cache line */
 };
 
lib/librte_ring/rte_ring_c11_mem.h +143 -0

147 lines not on this page · raw message

lib/librte_ring/rte_ring_generic.h +130 -0

134 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#84 lore raw

PATCH v7 2/6 ring: add a ring start marker

In reply to Gage Eads

This marker allows us to replace "&r[1]" with "&r->ring" to locate the start of the ring.

---
 lib/librte_ring/rte_ring.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
lib/librte_ring/rte_ring.h +3 -2
index c78db6916..f16d77b8a 100644
@@ -118,6 +118,7 @@ struct rte_ring {
 		struct rte_ring_headtail_ptr cons_ptr __rte_cache_aligned;
 	};
 	char pad2 __rte_cache_aligned; /**< empty cache line */
+	void *ring[] __rte_cache_aligned; /**< empty marker for ring start */
 };
 
 #define RING_F_SP_ENQ 0x0001 /**< The default enqueue is "single-producer". */
@@ -361,7 +362,7 @@ __rte_ring_do_enqueue(struct rte_ring *r, void * const *obj_table,
 	if (n == 0)
 		goto end;
 
-	ENQUEUE_PTRS(r, &r[1], prod_head, obj_table, n, void *);
+	ENQUEUE_PTRS(r, &r->ring, prod_head, obj_table, n, void *);
 
 	update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
 end:
@@ -403,7 +404,7 @@ __rte_ring_do_dequeue(struct rte_ring *r, void **obj_table,
 	if (n == 0)
 		goto end;
 
-	DEQUEUE_PTRS(r, &r[1], cons_head, obj_table, n, void *);
+	DEQUEUE_PTRS(r, &r->ring, cons_head, obj_table, n, void *);
 
 	update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
 
Signature · 1 line
2.13.6
author
#85 lore raw

PATCH v7 3/6 ring: add a lock-free implementation

In reply to Gage Eads

This commit adds support for lock-free circular ring enqueue and dequeue functions. The ring is supported on 32- and 64-bit architectures, however it uses a 128-bit compare-and-swap instruction when run on a 64-bit architecture, and thus is currently limited to x86_64.

The algorithm is based on Ola Liljedahl's lfring, modified to fit within the rte ring API. With no contention, an enqueue of n pointers uses (1 + n) CAS operations and a dequeue of n pointers uses 1. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the lock-free ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a lock-free ring based mempool (a likely use case for this ring) with per-thread caching.

To avoid the ABA problem, each ring entry contains a modification counter. On a 64-bit architecture, the chance of ABA occurring are effectively zero; a 64-bit counter will take many years to wrap at current CPU frequencies. On a 32-bit architectures, a lock-free ring must be at least 1024-entries deep; assuming 100 cycles per ring entry access, this guarantees the ring's modification counters will wrap on the order of days.

The lock-free ring is enabled via a new flag, RING_F_LF. Because the ring's memsize is now a function of its flags (the lock-free ring requires 128b for each entry), this commit adds a new argument ('flags') to rte_ring_get_memsize(). An API deprecation notice will be sent in a separate commit.

For ease-of-use, existing ring enqueue and dequeue functions work on both regular and lock-free rings. This introduces an additional branch in the datapath, but this should be a highly predictable branch. ring_perf_autotest shows a negligible performance impact; it's hard to distinguish a real difference versus system noise.

| ring_perf_autotest cycles with branch -
Test | ring_perf_autotest cycles without
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | -4.00
SP/SC burst enq/dequeue (size 8)  | 0.00
MP/MC burst enq/dequeue (size 8)  | 0.00
SP/SC burst enq/dequeue (size 32) | 0.00
MP/MC burst enq/dequeue (size 32) | 0.00
SC empty dequeue                  | 1.00
MC empty dequeue                  | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | 0.49
MP/MC bulk enq/dequeue (size 8)   | 0.08
SP/SC bulk enq/dequeue (size 32)  | 0.07
MP/MC bulk enq/dequeue (size 32)  | 0.09

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | 0.19
MP/MC bulk enq/dequeue (size 8)   | -0.37
SP/SC bulk enq/dequeue (size 32)  | 0.09
MP/MC bulk enq/dequeue (size 32)  | -0.05

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | -1.96
MP/MC bulk enq/dequeue (size 8)   | 0.88
SP/SC bulk enq/dequeue (size 32)  | 0.10
MP/MC bulk enq/dequeue (size 32)  | 0.46

Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. Each test run three times and the results averaged.

---
 lib/librte_ring/rte_ring.c           |  92 +++++++--
 lib/librte_ring/rte_ring.h           | 308 ++++++++++++++++++++++++++---
 lib/librte_ring/rte_ring_c11_mem.h   | 366 ++++++++++++++++++++++++++++++++++-
 lib/librte_ring/rte_ring_generic.h   | 355 ++++++++++++++++++++++++++++++++-
 lib/librte_ring/rte_ring_version.map |   7 +
 5 files changed, 1080 insertions(+), 48 deletions(-)
lib/librte_ring/rte_ring.c +80 -12
index d215acecc..d4a176f57 100644
@@ -45,9 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
 
 /* return the size of memory occupied by a ring */
 ssize_t
-rte_ring_get_memsize(unsigned count)
+rte_ring_get_memsize_v1905(unsigned int count, unsigned int flags)
 {
-	ssize_t sz;
+	ssize_t sz, elt_sz;
 
 	/* count must be a power of 2 */
 	if ((!POWEROF2(count)) || (count > RTE_RING_SZ_MASK )) {
@@ -57,10 +57,23 @@ rte_ring_get_memsize(unsigned count)
 		return -EINVAL;
 	}
 
-	sz = sizeof(struct rte_ring) + count * sizeof(void *);
+	elt_sz = (flags & RING_F_LF) ? 2 * sizeof(void *) : sizeof(void *);
+
+	sz = sizeof(struct rte_ring) + count * elt_sz;
 	sz = RTE_ALIGN(sz, RTE_CACHE_LINE_SIZE);
 	return sz;
 }
+BIND_DEFAULT_SYMBOL(rte_ring_get_memsize, _v1905, 19.05);
+MAP_STATIC_SYMBOL(ssize_t rte_ring_get_memsize(unsigned int count,
+					       unsigned int flags),
+		  rte_ring_get_memsize_v1905);
+
+ssize_t
+rte_ring_get_memsize_v20(unsigned int count)
+{
+	return rte_ring_get_memsize_v1905(count, 0);
+}
+VERSION_SYMBOL(rte_ring_get_memsize, _v20, 2.0);
 
 int
 rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
@@ -75,6 +88,8 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 			  RTE_CACHE_LINE_MASK) != 0);
 	RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
 			  RTE_CACHE_LINE_MASK) != 0);
+	RTE_BUILD_BUG_ON(sizeof(struct rte_ring_lf_entry) !=
+			 2 * sizeof(void *));
 
 	/* init the ring structure */
 	memset(r, 0, sizeof(*r));
@@ -82,8 +97,6 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 	if (ret < 0 || ret >= (int)sizeof(r->name))
 		return -ENAMETOOLONG;
 	r->flags = flags;
-	r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
-	r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
 
 	if (flags & RING_F_EXACT_SZ) {
 		r->size = rte_align32pow2(count + 1);
@@ -100,12 +113,46 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 		r->mask = count - 1;
 		r->capacity = r->mask;
 	}
-	r->prod.head = r->cons.head = 0;
-	r->prod.tail = r->cons.tail = 0;
+
+	r->log2_size = rte_log2_u64(r->size);
+
+	if (flags & RING_F_LF) {
+		uint32_t i;
+
+		r->prod_ptr.single =
+			(flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
+		r->cons_ptr.single =
+			(flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+		r->prod_ptr.head = r->cons_ptr.head = 0;
+		r->prod_ptr.tail = r->cons_ptr.tail = 0;
+
+		for (i = 0; i < r->size; i++) {
+			struct rte_ring_lf_entry *ring_ptr, *base;
+
+			base = (struct rte_ring_lf_entry *)&r->ring;
+
+			ring_ptr = &base[i & r->mask];
+
+			ring_ptr->cnt = 0;
+		}
+	} else {
+		r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
+		r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+		r->prod.head = r->cons.head = 0;
+		r->prod.tail = r->cons.tail = 0;
+	}
 
 	return 0;
 }
 
+/* If a ring entry is written on average every M cycles, then a ring entry is
+ * reused every M*count cycles, and a ring entry's counter repeats every
+ * M*count*2^32 cycles. If M=100 on a 2GHz system, then a 1024-entry ring's
+ * counters would repeat every 2.37 days. The likelihood of ABA occurring is
+ * considered sufficiently low for 1024-entry and larger rings.
+ */
+#define MIN_32_BIT_LF_RING_SIZE 1024
+
 /* create the ring */
 struct rte_ring *
 rte_ring_create(const char *name, unsigned count, int socket_id,
@@ -123,11 +170,25 @@ rte_ring_create(const char *name, unsigned count, int socket_id,
 
 	ring_list = RTE_TAILQ_CAST(rte_ring_tailq.head, rte_ring_list);
 
+#ifdef RTE_ARCH_64
+#if !defined(RTE_ARCH_X86_64)
+	printf("This platform does not support the atomic operation required for RING_F_LF\n");
+	rte_errno = EINVAL;
+	return NULL;
+#endif
+#else
+	if ((flags & RING_F_LF) && count < MIN_32_BIT_LF_RING_SIZE) {
+		printf("RING_F_LF is only supported on 32-bit platforms for rings with at least 1024 entries.\n");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+#endif
+
 	/* for an exact size ring, round up from count to a power of two */
 	if (flags & RING_F_EXACT_SZ)
 		count = rte_align32pow2(count + 1);
 
-	ring_size = rte_ring_get_memsize(count);
+	ring_size = rte_ring_get_memsize(count, flags);
 	if (ring_size < 0) {
 		rte_errno = ring_size;
 		return NULL;
@@ -227,10 +288,17 @@ rte_ring_dump(FILE *f, const struct rte_ring *r)
 	fprintf(f, "  flags=%x\n", r->flags);
 	fprintf(f, "  size=%"PRIu32"\n", r->size);
 	fprintf(f, "  capacity=%"PRIu32"\n", r->capacity);
-	fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
-	fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
-	fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
-	fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	if (r->flags & RING_F_LF) {
+		fprintf(f, "  ct=%"PRIuPTR"\n", r->cons_ptr.tail);
+		fprintf(f, "  ch=%"PRIuPTR"\n", r->cons_ptr.head);
+		fprintf(f, "  pt=%"PRIuPTR"\n", r->prod_ptr.tail);
+		fprintf(f, "  ph=%"PRIuPTR"\n", r->prod_ptr.head);
+	} else {
+		fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
+		fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
+		fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
+		fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	}
 	fprintf(f, "  used=%u\n", rte_ring_count(r));
 	fprintf(f, "  avail=%u\n", rte_ring_free_count(r));
 }
lib/librte_ring/rte_ring.h +277 -31

449 lines not on this page · raw message

lib/librte_ring/rte_ring_c11_mem.h +362 -4

395 lines not on this page · raw message

lib/librte_ring/rte_ring_generic.h +354 -1

364 lines not on this page · raw message

lib/librte_ring/rte_ring_version.map +7 -0
index d935efd0d..8969467af 100644
@@ -17,3 +17,10 @@ DPDK_2.2 {
 	rte_ring_free;
 
 } DPDK_2.0;
+
+DPDK_19.05 {
+	global:
+
+	rte_ring_get_memsize;
+
+} DPDK_2.2;
Signature · 1 line
2.13.6
#86 lore raw

Re: [PATCH v7 3/6] ring: add a lock-free implementation

Quoted 72 lines · On Mon, 18 Mar 2019 16:35:52 -0500 Gage Eads <[email protected]> wrote:
lib/librte_ring/rte_ring.h +64 -0
+/* The actual enqueue of pointers on the lock-free ring, used by the
+ * single-producer lock-free enqueue function.
+ */
+#define ENQUEUE_PTRS_LF(r, base, prod_head, obj_table, n) do { \
+	unsigned int i; \
+	const uint32_t size = (r)->size; \
+	size_t idx = prod_head & (r)->mask; \
+	size_t new_cnt = prod_head + size; \
+	struct rte_ring_lf_entry *ring = (struct rte_ring_lf_entry *)base; \
+	unsigned int mask = ~0x3; \
+	if (likely(idx + n < size)) { \
+		for (i = 0; i < (n & mask); i += 4, idx += 4) { \
+			ring[idx].ptr = obj_table[i]; \
+			ring[idx].cnt = (new_cnt + i) >> r->log2_size; \
+			ring[idx + 1].ptr = obj_table[i + 1]; \
+			ring[idx + 1].cnt = (new_cnt + i + 1) >> r->log2_size; \
+			ring[idx + 2].ptr = obj_table[i + 2]; \
+			ring[idx + 2].cnt = (new_cnt + i + 2) >> r->log2_size; \
+			ring[idx + 3].ptr = obj_table[i + 3]; \
+			ring[idx + 3].cnt = (new_cnt + i + 3) >> r->log2_size; \
+		} \
+		switch (n & 0x3) { \
+		case 3: \
+			ring[idx].cnt = (new_cnt + i) >> r->log2_size; \
+			ring[idx++].ptr = obj_table[i++]; /* fallthrough */ \
+		case 2: \
+			ring[idx].cnt = (new_cnt + i) >> r->log2_size; \
+			ring[idx++].ptr = obj_table[i++]; /* fallthrough */ \
+		case 1: \
+			ring[idx].cnt = (new_cnt + i) >> r->log2_size; \
+			ring[idx++].ptr = obj_table[i++]; \
+		} \
+	} else { \
+		for (i = 0; idx < size; i++, idx++) { \
+			ring[idx].cnt = (new_cnt + i) >> r->log2_size; \
+			ring[idx].ptr = obj_table[i]; \
+		} \
+		for (idx = 0; i < n; i++, idx++) {    \
+			ring[idx].cnt = (new_cnt + i) >> r->log2_size; \
+			ring[idx].ptr = obj_table[i]; \
+		} \
+	} \
+} while (0)
+
 /* the actual copy of pointers on the ring to obj_table.
  * Placed here since identical code needed in both
  * single and multi consumer dequeue functions */
@@ -314,6 +384,43 @@ void rte_ring_dump(FILE *f, const struct rte_ring *r);
 	} \
 } while (0)
 
+/* The actual copy of pointers on the lock-free ring to obj_table. */
+#define DEQUEUE_PTRS_LF(r, base, cons_head, obj_table, n) do { \
+	unsigned int i; \
+	size_t idx = cons_head & (r)->mask; \
+	const uint32_t size = (r)->size; \
+	struct rte_ring_lf_entry *ring = (struct rte_ring_lf_entry *)base; \
+	unsigned int mask = ~0x3; \
+	if (likely(idx + n < size)) { \
+		for (i = 0; i < (n & mask); i += 4, idx += 4) {\
+			obj_table[i] = ring[idx].ptr; \
+			obj_table[i + 1] = ring[idx + 1].ptr; \
+			obj_table[i + 2] = ring[idx + 2].ptr; \
+			obj_table[i + 3] = ring[idx + 3].ptr; \
+		} \
+		switch (n & 0x3) { \
+		case 3: \
+			obj_table[i++] = ring[idx++].ptr; /* fallthrough */ \
+		case 2: \
+			obj_table[i++] = ring[idx++].ptr; /* fallthrough */ \
+		case 1: \

lib/librte_ring/rte_ring.h void rte_ring_dump(FILE *f, const struct rte_ring *r); · line 416

Quoted 10 lines
lib/librte_ring/rte_ring.h +10 -0
+			obj_table[i++] = ring[idx++].ptr; \
+		} \
+	} else { \
+		for (i = 0; idx < size; i++, idx++) \
+			obj_table[i] = ring[idx].ptr; \
+		for (idx = 0; i < n; i++, idx++) \
+			obj_table[i] = ring[idx].ptr; \
+	} \
+} while (0)
+

Generic programming like this in C is hard to maintain. Please don't use large macros. It is better to have two copies than a single large macro. Better yet use a single inline function than can be expanded in different ways based on the pointer size.

author
#87 lore raw

PATCH v7 4/6 test_ring: add lock-free ring autotest

In reply to Gage Eads

ring_nb_autotest re-uses the ring_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 app/test/test_ring.c | 61 ++++++++++++++++++++++++++++++++--------------------
 1 file changed, 38 insertions(+), 23 deletions(-)
app/test/test_ring.c +38 -23
index aaf1e70ad..400b1bffd 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 #include <string.h>
@@ -601,18 +601,20 @@ test_ring_burst_basic(struct rte_ring *r)
  * it will always fail to create ring with a wrong ring size number in this function
  */
 static int
-test_ring_creation_with_wrong_size(void)
+test_ring_creation_with_wrong_size(unsigned int flags)
 {
 	struct rte_ring * rp = NULL;
 
 	/* Test if ring size is not power of 2 */
-	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1,
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
 
 	/* Test if ring size is exceeding the limit */
-	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1), SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1),
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
@@ -623,11 +625,11 @@ test_ring_creation_with_wrong_size(void)
  * it tests if it would always fail to create ring with an used ring name
  */
 static int
-test_ring_creation_with_an_used_name(void)
+test_ring_creation_with_an_used_name(unsigned int flags)
 {
 	struct rte_ring * rp;
 
-	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (NULL != rp)
 		return -1;
 
@@ -639,10 +641,10 @@ test_ring_creation_with_an_used_name(void)
  * function to fail correctly
  */
 static int
-test_create_count_odd(void)
+test_create_count_odd(unsigned int flags)
 {
 	struct rte_ring *r = rte_ring_create("test_ring_count",
-			4097, SOCKET_ID_ANY, 0 );
+			4097, SOCKET_ID_ANY, flags);
 	if(r != NULL){
 		return -1;
 	}
@@ -665,7 +667,7 @@ test_lookup_null(void)
  * it tests some more basic ring operations
  */
 static int
-test_ring_basic_ex(void)
+test_ring_basic_ex(unsigned int flags)
 {
 	int ret = -1;
 	unsigned i;
@@ -679,7 +681,7 @@ test_ring_basic_ex(void)
 	}
 
 	rp = rte_ring_create("test_ring_basic_ex", RING_SIZE, SOCKET_ID_ANY,
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (rp == NULL) {
 		printf("test_ring_basic_ex fail to create ring\n");
 		goto fail_test;
@@ -737,22 +739,22 @@ test_ring_basic_ex(void)
 }
 
 static int
-test_ring_with_exact_size(void)
+test_ring_with_exact_size(unsigned int flags)
 {
 	struct rte_ring *std_ring = NULL, *exact_sz_ring = NULL;
-	void *ptr_array[16];
+	void *ptr_array[1024];
 	static const unsigned int ring_sz = RTE_DIM(ptr_array);
 	unsigned int i;
 	int ret = -1;
 
 	std_ring = rte_ring_create("std", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (std_ring == NULL) {
 		printf("%s: error, can't create std ring\n", __func__);
 		goto end;
 	}
 	exact_sz_ring = rte_ring_create("exact sz", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ);
+		RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ | flags);
 	if (exact_sz_ring == NULL) {
 		printf("%s: error, can't create exact size ring\n", __func__);
 		goto end;
@@ -770,7 +772,7 @@ test_ring_with_exact_size(void)
 	}
 	/*
 	 * check that the exact_sz_ring can hold one more element than the
-	 * standard ring. (16 vs 15 elements)
+	 * standard ring. (1024 vs 1023 elements)
 	 */
 	for (i = 0; i < ring_sz - 1; i++) {
 		rte_ring_enqueue(std_ring, NULL);
@@ -808,17 +810,17 @@ test_ring_with_exact_size(void)
 }
 
 static int
-test_ring(void)
+__test_ring(unsigned int flags)
 {
 	struct rte_ring *r = NULL;
 
 	/* some more basic operations */
-	if (test_ring_basic_ex() < 0)
+	if (test_ring_basic_ex(flags) < 0)
 		goto test_fail;
 
 	rte_atomic32_init(&synchro);
 
-	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (r == NULL)
 		goto test_fail;
 
@@ -837,27 +839,27 @@ test_ring(void)
 		goto test_fail;
 
 	/* basic operations */
-	if ( test_create_count_odd() < 0){
+	if (test_create_count_odd(flags) < 0) {
 		printf("Test failed to detect odd count\n");
 		goto test_fail;
 	} else
 		printf("Test detected odd count\n");
 
-	if ( test_lookup_null() < 0){
+	if (test_lookup_null() < 0) {
 		printf("Test failed to detect NULL ring lookup\n");
 		goto test_fail;
 	} else
 		printf("Test detected NULL ring lookup\n");
 
 	/* test of creating ring with wrong size */
-	if (test_ring_creation_with_wrong_size() < 0)
+	if (test_ring_creation_with_wrong_size(flags) < 0)
 		goto test_fail;
 
 	/* test of creation ring with an used name */
-	if (test_ring_creation_with_an_used_name() < 0)
+	if (test_ring_creation_with_an_used_name(flags) < 0)
 		goto test_fail;
 
-	if (test_ring_with_exact_size() < 0)
+	if (test_ring_with_exact_size(flags) < 0)
 		goto test_fail;
 
 	/* dump the ring status */
@@ -873,4 +875,17 @@ test_ring(void)
 	return -1;
 }
 
+static int
+test_ring(void)
+{
+	return __test_ring(0);
+}
+
+static int
+test_lf_ring(void)
+{
+	return __test_ring(RING_F_LF);
+}
+
 REGISTER_TEST_COMMAND(ring_autotest, test_ring);
+REGISTER_TEST_COMMAND(ring_lf_autotest, test_lf_ring);
Signature · 1 line
2.13.6
author
#88 lore raw

PATCH v7 5/6 test_ring_perf: add lock-free ring perf test

In reply to Gage Eads

nb_ring_perf_autotest re-uses the ring_perf_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 app/test/test_ring_perf.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
app/test/test_ring_perf.c +16 -3
index ebb3939f5..be465c758 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 
@@ -363,12 +363,12 @@ test_bulk_enqueue_dequeue(struct rte_ring *r)
 }
 
 static int
-test_ring_perf(void)
+__test_ring_perf(unsigned int flags)
 {
 	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
 
-	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), 0);
+	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), flags);
 	if (r == NULL)
 		return -1;
 
@@ -398,4 +398,17 @@ test_ring_perf(void)
 	return 0;
 }
 
+static int
+test_ring_perf(void)
+{
+	return __test_ring_perf(0);
+}
+
+static int
+test_lf_ring_perf(void)
+{
+	return __test_ring_perf(RING_F_LF);
+}
+
 REGISTER_TEST_COMMAND(ring_perf_autotest, test_ring_perf);
+REGISTER_TEST_COMMAND(ring_lf_perf_autotest, test_lf_ring_perf);
Signature · 1 line
2.13.6
author
#89 lore raw

PATCH v7 6/6 mempool/ring: add lock-free ring handlers

In reply to Gage Eads

These handlers allow an application to create a mempool based on the lock-free ring, with any combination of single/multi producer/consumer.

Also, add a note to the programmer's guide's "known issues" section.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 +++++
 drivers/mempool/ring/Makefile                   |  1 +
 drivers/mempool/ring/meson.build                |  2 +
 drivers/mempool/ring/rte_mempool_ring.c         | 58 +++++++++++++++++++++++--
 4 files changed, 68 insertions(+), 3 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
index 2361c3b8f..18bc98c57 100644
@@ -563,6 +563,16 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
+  Alternatively, 32-bit and x86_64 applications can use the lock-free ring
+  mempool handler. When considering it, note that:
+
+  - Among 64-bit architectures it is currently limited to the x86_64 platform,
+    because it uses a function (16-byte compare-and-swap) that is not yet
+    available on other platforms.
+  - It has worse average-case performance than the non-preemptive rte_ring, but
+    software caching (e.g. the mempool cache) can mitigate this by reducing the
+    number of handler operations.
+
 + rte_timer
 
   Running  ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
drivers/mempool/ring/Makefile +1 -0
index ddab522fe..012ba6966 100644
@@ -10,6 +10,7 @@ LIB = librte_mempool_ring.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 LDLIBS += -lrte_eal -lrte_mempool -lrte_ring
 
 EXPORT_MAP := rte_mempool_ring_version.map
drivers/mempool/ring/meson.build +2 -0
index a021e908c..b1cb673cc 100644
@@ -1,4 +1,6 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
+allow_experimental_apis = true
+
 sources = files('rte_mempool_ring.c')
drivers/mempool/ring/rte_mempool_ring.c +55 -3

84 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#90 lore raw

Re: [PATCH v7 0/6] Add lock-free ring and mempool handler

In reply to Gage Eads

[Snip]

Quoted 9 lines

This patchset makes one API change; a deprecation notice was posted in a separate commit[1].

This patchset depends on the 128-bit compare-and-set patch[2].

[1] http://mails.dpdk.org/archives/dev/2019-February/124321.html
[2] http://mails.dpdk.org/archives/dev/2019-March/125751.html

Hi all,

Friendly reminder that in order to get this feature into 19.08 (assuming folks also want that :)), the API deprecation notice needs to be merged into 19.05.

Thanks,
Gage

#91 lore raw

Re: [PATCH v7 0/6] Add lock-free ring and mempool handler

Quoted 6 lines · Gage

Hi all,

Friendly reminder that in order to get this feature into 19.08 (assuming folks also want that :)), the API deprecation notice needs to be merged into 19.05.

Thanks,
Gage

Given the recent API/ABI stability discussion, this is the kind of patch that really needs to be examined and justified.

author
#92 lore raw

Re: [PATCH v7 0/6] Add lock-free ring and mempool handler

Quoted 7 lines
Quoted 3 lines · Gage

Hi all,

Friendly reminder that in order to get this feature into 19.08 (assuming

Quoted 8 lines

folks also want that :)), the API deprecation notice needs to be merged into 19.05.

Quoted 3 lines

Thanks,
Gage

Given the recent API/ABI stability discussion, this is the kind of patch that really needs to be examined and justified.

Can you point me to the discussion (assuming it was on the ML)? I'm aware of Ferruh's changes to the docs, but not the discussion you referenced.

The lock-free ring functionality itself is a valuable addition to DPDK, primarily because it lacks the standard ring's non-preemptive constraint. The non-preemptive constraint is important in an application with both high priority, performance-sensitive data-plane components and low-priority control-plane components. This was important enough to warrant further clarification recently[1], and has been a discussion topic for some time[2][3].

The modified API, rte_ring_get_memsize(), was added to allow users to initialize rings in separately allocated memory. This function isn't called in DPDK's examples/apps/drivers, and a quick google search didn't turn up any open source projects that call the function, so I suspect that a majority of ring code uses rte_ring_create() instead of rte_ring_get_memsize() + rte_ring_init(). So I suspect this interface change will affect a small percentage of DPDK users.

As a straw-man counter-proposal, we could instead introduce a lock-free specific function rte_lf_ring_get_memsize() that lock-free ring users would call instead of rte_ring_get_memsize(). This would avoid the API modification, but:
- It's awkward to have one rte_lf_ring_* function and otherwise access the lock-free ring through rte_ring_* functions.
- It's also easy to envision a user incorrectly calling rte_ring_get_memsize() rather than rte_lf_ring_get_memsize() for a lock-free ring, since otherwise rte_ring_* functions are used. DPDK would have no way to detect that the allocated memory is too small, and if such a bug occurs it would manifest itself as memory corruption.
- Changing rte_ring_get_memsize() to take a flags argument may be the better long-term design, if another flag is introduced that likewise uses a different ring size.

Another approach is to break out the lock-free ring into a fully separate API. One of the goals of my patchset was to allow applications to switch to lock-free rings with minimal code change; I think the value of the lock-free ring warrants such an approach.

(Unfortunately without hard numbers on the cost or benefit of such a change, these arguments are fairly subjective.)

Thanks,
Gage

[1] https://patches.dpdk.org/patch/43122/
[2] http://mails.dpdk.org/archives/dev/2013-November/000714.html
[3] http://mails.dpdk.org/archives/dev/2014-January/001163.html

#93 lore raw

Re: [PATCH v7 0/6] Add lock-free ring and mempool handler

Quoted 52 lines · Gage
Quoted 13 lines
Quoted 4 lines · Gage

Hi all,

Friendly reminder that in order to get this feature into 19.08 (assuming

folks also want that :)), the API deprecation notice needs to be merged into 19.05.

Quoted 2 lines
Quoted 5 lines
Quoted 2 lines

Thanks,
Gage

Given the recent API/ABI stability discussion, this is the kind of patch that
really needs to be examined and justified.

Can you point me to the discussion (assuming it was on the ML)? I'm aware of Ferruh's changes to the docs, but not the discussion you referenced.

The lock-free ring functionality itself is a valuable addition to DPDK, primarily because it lacks the standard ring's non-preemptive constraint. The non-preemptive constraint is important in an application with both high priority, performance-sensitive data-plane components and low-priority control-plane components. This was important enough to warrant further clarification recently[1], and has been a discussion topic for some time[2][3].

The modified API, rte_ring_get_memsize(), was added to allow users to initialize rings in separately allocated memory. This function isn't called in DPDK's examples/apps/drivers, and a quick google search didn't turn up any open source projects that call the function, so I suspect that a majority of ring code uses rte_ring_create() instead of rte_ring_get_memsize() + rte_ring_init(). So I suspect this interface change will affect a small percentage of DPDK users.

As a straw-man counter-proposal, we could instead introduce a lock-free specific function rte_lf_ring_get_memsize() that lock-free ring users would call instead of rte_ring_get_memsize(). This would avoid the API modification, but:
- It's awkward to have one rte_lf_ring_* function and otherwise access the lock-free ring through rte_ring_* functions.
- It's also easy to envision a user incorrectly calling rte_ring_get_memsize() rather than rte_lf_ring_get_memsize() for a lock-free ring, since otherwise rte_ring_* functions are used. DPDK would have no way to detect that the allocated memory is too small, and if such a bug occurs it would manifest itself as memory corruption.
- Changing rte_ring_get_memsize() to take a flags argument may be the better long-term design, if another flag is introduced that likewise uses a different ring size.

Quoted 1 line

Another approach is to break out the lock-free ring into a fully separate API.

As in the RFC I posted: http://patches.dpdk.org/patch/50095/
Cleaner API, simpler implementation.

One of the goals of my patchset was to allow applications to switch to lock-free rings with minimal code change; I think the value of the lock-free ring warrants such an approach.

A noble goal but personally I think DPDK API's and implementations are getting more and more messy and thus difficult to use and difficult to maintain. Is it so much worse to have separate but structurally equivalent API's? Yes, blocking vs non-blocking can no longer be selected at runtime (startup time), I think this is the biggest limitation.

-- Ola

(Unfortunately without hard numbers on the cost or benefit of such a change, these arguments are fairly subjective.)

Thanks,
Gage

[1] https://patches.dpdk.org/patch/43122/
[2] http://mails.dpdk.org/archives/dev/2013-November/000714.html
[3] http://mails.dpdk.org/archives/dev/2014-January/001163.html

Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
author
#94 lore raw

Re: [dpdk-dev] [PATCH v7 0/6] Add lock-free ring and mempool handler

Quoted 61 lines · Gage
Quoted 14 lines
Quoted 14 lines
Quoted 5 lines · Gage

Hi all,

Friendly reminder that in order to get this feature into 19.08 (assuming

folks also want that :)), the API deprecation notice needs to be merged into 19.05.

Quoted 2 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Thanks,
Gage

Given the recent API/ABI stability discussion, this is the kind of patch that really needs to be examined and justified.

Can you point me to the discussion (assuming it was on the ML)? I'm aware of Ferruh's changes to the docs, but not the discussion you

referenced.

Quoted 34 lines

The lock-free ring functionality itself is a valuable addition to DPDK, primarily because it lacks the standard ring's non-preemptive constraint. The non-preemptive constraint is important in an application with both high priority, performance-sensitive data-plane components and low-priority control-plane components. This was important enough to warrant further clarification recently[1], and has been a discussion topic for some time[2][3].

The modified API, rte_ring_get_memsize(), was added to allow users to initialize rings in separately allocated memory. This function isn't called in DPDK's examples/apps/drivers, and a quick google search didn't turn up any open source projects that call the function, so I suspect that a majority of ring code uses rte_ring_create() instead of rte_ring_get_memsize() + rte_ring_init(). So I suspect this interface change will affect a small percentage of DPDK users.

As a straw-man counter-proposal, we could instead introduce a lock-free specific function rte_lf_ring_get_memsize() that lock-free ring users would call instead of rte_ring_get_memsize(). This would avoid the API modification,
but:
- It's awkward to have one rte_lf_ring_* function and otherwise access the lock-free ring through rte_ring_* functions.
- It's also easy to envision a user incorrectly calling rte_ring_get_memsize() rather than rte_lf_ring_get_memsize() for a lock-free ring, since otherwise
rte_ring_* functions are used. DPDK would have no way to detect that the allocated memory is too small, and if such a bug occurs it would manifest itself as memory corruption.
- Changing rte_ring_get_memsize() to take a flags argument may be the better long-term design, if another flag is introduced that likewise uses a different ring size.

Quoted 1 line

Another approach is to break out the lock-free ring into a fully separate

API.
As in the RFC I posted: http://patches.dpdk.org/patch/50095/
Cleaner API, simpler implementation.

Quoted 8 lines
Quoted 3 lines

One of the goals of my patchset was to allow applications to switch to lock-
free rings with minimal code change; I think the value of the lock-free ring warrants such an approach.

A noble goal but personally I think DPDK API's and implementations are getting more and more messy and thus difficult to use and difficult to maintain.
Is it so much worse to have separate but structurally equivalent API's?

No -- I think both are valid options, with their own tradeoffs.

Quoted 2 lines

Yes, blocking vs non-blocking can no longer be selected at runtime (startup time), I think this is the biggest limitation.

Run-time selection with the LF flag means we can easily re-use a large amount of pre-existing ring code -- e.g. the way the ring tests are re-purposed for lock-free rings in this patchset.

The implementation may not be as easily maintained, but we don't have to maintain LF-specific tests/benchmarks/etc.

Maintainers/tech leads, do y'all have a preference?

Thanks,
Gage

author
#95 lore raw

v8 cover Add lock-free ring and mempool handler

In reply to Gage Eads

For some users, the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

This patchset introduces a lock-free ring and a mempool based on it. The lock-free algorithm relies on a double-pointer compare-and-swap, so for 64-bit architectures it is currently limited to x86_64.

The ring uses more compare-and-swap atomic operations than the regular rte ring:
With no contention, an enqueue of n pointers uses (1 + n) CAS operations and a dequeue of n pointers uses 1. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the lock-free ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a lock-free ring based mempool (a likely use case for this ring) with per-thread caching.

The lock-free ring is enabled via a new flag, RING_F_LF. For ease-of-use, existing ring enqueue/dequeue functions work with both standard and lock-free rings. This is also an experimental API, so RING_F_LF users must build with the ALLOW_EXPERIMENTAL_API flag.

This patchset also adds lock-free versions of ring_autotest and ring_perf_autotest, and a lock-free ring based mempool.

This patchset makes one API change; a deprecation notice was posted in a separate commit[1].

This patchset depends on the 128-bit compare-and-set patch[2].

[1] http://mails.dpdk.org/archives/dev/2019-February/124321.html
[2] http://mails.dpdk.org/archives/dev/2019-March/125751.html

v8:
- Fixed two bugs in the generic implementation written as though the compare-and-swap would update the expected value on failure.

v7:
- Added ARM copyright to rte_ring_generic.h and rte_ring_c11_mem.h, since the lock-free algorithm is based on ARM's lfring (see v5 notes)
- Rename __rte_ring_reload_tail() -> __rte_ring_lf_load_tail()
- Remove the unused return value from __rte_ring_lf_load_tail()
- Rename 'prev_tail' to 'next_tail' in the multi-producer lock-free enqueue

v6:
- Rebase patchset onto master (test/test/ -> app/test/)

v5:
- Incorporated lfring's enqueue and dequeue logic from
http://mails.dpdk.org/archives/dev/2019-January/124242.html
- Renamed non-blocking -> lock-free and NB -> LF to align with a similar change in the lock-free stack patchset:
http://mails.dpdk.org/archives/dev/2019-March/125797.html
- Added support for 32-bit architectures by using the full 32b of the modification counter and requiring LF rings on these architectures to be at least 1024 entries.
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Added ring start marker to struct rte_ring

v4:
- Split out nb_enqueue and nb_dequeue functions in generic and C11 versions, with the necessary memory ordering behavior for weakly consistent machines.
- Convert size_t variables (from v2) to uint64_t and no-longer-applicable comment about variably-sized ring indexes.
- Fix bug in nb_enqueue_mp that the breaks the non-blocking guarantee.
- Split the ring_ptr cast into two lines.
- Change the dependent patchset from the non-blocking stack patch series to one only containing the 128b CAS commit

v3:
- Avoid the ABI break by putting 64-bit head and tail values in the same cacheline as struct rte_ring's prod and cons members.
- Don't attempt to compile rte_atomic128_cmpset without ALLOW_EXPERIMENTAL_API, as this would break a large number of libraries.
- Add a helpful warning to __rte_ring_do_nb_enqueue_mp() in case someone tries to use RING_F_NB without the ALLOW_EXPERIMENTAL_API flag.
- Update the ring mempool to use experimental APIs
- Clarify that RINB_F_NB is only limited to x86_64 currently; e.g. ARMv8 has the ISA support for 128-bit CAS to eventually support it.

v2:
- Merge separate docs commit into patch #5
- Convert uintptr_t to size_t
- Add a compile-time check for the size of size_t
- Fix a space-after-typecast issue
- Fix an unnecessary-parentheses checkpatch warning
- Bump librte_ring's library version

Gage Eads (6):
ring: add a pointer-width headtail structure
ring: add a ring start marker
ring: add a lock-free implementation
test_ring: add lock-free ring autotest
test_ring_perf: add lock-free ring perf test
mempool/ring: add lock-free ring handlers

 app/test/test_ring.c                            |  61 +--
 app/test/test_ring_perf.c                       |  19 +-
 doc/guides/prog_guide/env_abstraction_layer.rst |  10 +
 drivers/mempool/ring/Makefile                   |   1 +
 drivers/mempool/ring/meson.build                |   2 +
 drivers/mempool/ring/rte_mempool_ring.c         |  58 ++-
 lib/librte_ring/rte_ring.c                      |  92 ++++-
 lib/librte_ring/rte_ring.h                      | 334 ++++++++++++++--
 lib/librte_ring/rte_ring_c11_mem.h              | 501 ++++++++++++++++++++++++
 lib/librte_ring/rte_ring_generic.h              | 487 ++++++++++++++++++++++-
 lib/librte_ring/rte_ring_version.map            |   7 +
 11 files changed, 1494 insertions(+), 78 deletions(-)
Signature · 1 line
2.13.6
author
#96 lore raw

PATCH v8 1/6 ring: add a pointer-width headtail structure

For 64-bit systems, at current CPU speeds, 64-bit head and tail indexes will not wrap-around within the author's lifetime. This is important to avoiding the ABA problem -- in which a thread mistakes reading the same tail index in two accesses to mean that the ring was not modified in the intervening time -- in the upcoming lock-free ring implementation. Using a 64-bit index makes the possibility of this occurring effectively zero. This commit uses pointer-width indexes so the lock-free ring can support 32-bit systems as well.

This commit places the new producer and consumer structures in the same location in struct rte_ring as their 32-bit counterparts. Since the 32-bit versions are padded out to a cache line, there is space for the new structure without affecting the layout of struct rte_ring. Thus, the ABI is preserved.

---
 lib/librte_ring/rte_ring.h         |  21 +++++-
 lib/librte_ring/rte_ring_c11_mem.h | 143 +++++++++++++++++++++++++++++++++++++
 lib/librte_ring/rte_ring_generic.h | 130 +++++++++++++++++++++++++++++++++
 3 files changed, 291 insertions(+), 3 deletions(-)
lib/librte_ring/rte_ring.h +18 -3
index af5444a9f..c78db6916 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  *
- * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2010-2019 Intel Corporation
  * Copyright (c) 2007-2009 Kip Macy [email protected]
  * All rights reserved.
  * Derived from FreeBSD's bufring.h
@@ -70,6 +70,13 @@ struct rte_ring_headtail {
 	uint32_t single;         /**< True if single prod/cons */
 };
 
+/* Structure to hold a pair of pointer-sized head/tail values and metadata */
+struct rte_ring_headtail_ptr {
+	volatile uintptr_t head; /**< Prod/consumer head. */
+	volatile uintptr_t tail; /**< Prod/consumer tail. */
+	uint32_t single;         /**< True if single prod/cons */
+};
+
 /**
  * An RTE ring structure.
  *
@@ -97,11 +104,19 @@ struct rte_ring {
 	char pad0 __rte_cache_aligned; /**< empty cache line */
 
 	/** Ring producer status. */
-	struct rte_ring_headtail prod __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail prod __rte_cache_aligned;
+		struct rte_ring_headtail_ptr prod_ptr __rte_cache_aligned;
+	};
 	char pad1 __rte_cache_aligned; /**< empty cache line */
 
 	/** Ring consumer status. */
-	struct rte_ring_headtail cons __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail cons __rte_cache_aligned;
+		struct rte_ring_headtail_ptr cons_ptr __rte_cache_aligned;
+	};
 	char pad2 __rte_cache_aligned; /**< empty cache line */
 };
 
lib/librte_ring/rte_ring_c11_mem.h +143 -0

147 lines not on this page · raw message

lib/librte_ring/rte_ring_generic.h +130 -0

134 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#97 lore raw

PATCH v8 2/6 ring: add a ring start marker

In reply to Gage Eads

This marker allows us to replace "&r[1]" with "&r->ring" to locate the start of the ring.

---
 lib/librte_ring/rte_ring.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
lib/librte_ring/rte_ring.h +3 -2
index c78db6916..f16d77b8a 100644
@@ -118,6 +118,7 @@ struct rte_ring {
 		struct rte_ring_headtail_ptr cons_ptr __rte_cache_aligned;
 	};
 	char pad2 __rte_cache_aligned; /**< empty cache line */
+	void *ring[] __rte_cache_aligned; /**< empty marker for ring start */
 };
 
 #define RING_F_SP_ENQ 0x0001 /**< The default enqueue is "single-producer". */
@@ -361,7 +362,7 @@ __rte_ring_do_enqueue(struct rte_ring *r, void * const *obj_table,
 	if (n == 0)
 		goto end;
 
-	ENQUEUE_PTRS(r, &r[1], prod_head, obj_table, n, void *);
+	ENQUEUE_PTRS(r, &r->ring, prod_head, obj_table, n, void *);
 
 	update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
 end:
@@ -403,7 +404,7 @@ __rte_ring_do_dequeue(struct rte_ring *r, void **obj_table,
 	if (n == 0)
 		goto end;
 
-	DEQUEUE_PTRS(r, &r[1], cons_head, obj_table, n, void *);
+	DEQUEUE_PTRS(r, &r->ring, cons_head, obj_table, n, void *);
 
 	update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
 
Signature · 1 line
2.13.6
author
#98 lore raw

PATCH v8 3/6 ring: add a lock-free implementation

In reply to Gage Eads

This commit adds support for lock-free circular ring enqueue and dequeue functions. The ring is supported on 32- and 64-bit architectures, however it uses a 128-bit compare-and-swap instruction when run on a 64-bit architecture, and thus is currently limited to x86_64.

The algorithm is based on Ola Liljedahl's lfring, modified to fit within the rte ring API. With no contention, an enqueue of n pointers uses (1 + n) CAS operations and a dequeue of n pointers uses 1. This algorithm has worse average-case performance than the regular rte ring (particularly a highly-contended ring with large bulk accesses), however:
- For applications with preemptible pthreads, the regular rte ring's worst-case performance (i.e. one thread being preempted in the update_tail() critical section) is much worse than the lock-free ring's.
- Software caching can mitigate the average case performance for ring-based algorithms. For example, a lock-free ring based mempool (a likely use case for this ring) with per-thread caching.

To avoid the ABA problem, each ring entry contains a modification counter. On a 64-bit architecture, the chance of ABA occurring are effectively zero; a 64-bit counter will take many years to wrap at current CPU frequencies. On a 32-bit architectures, a lock-free ring must be at least 1024-entries deep; assuming 100 cycles per ring entry access, this guarantees the ring's modification counters will wrap on the order of days.

The lock-free ring is enabled via a new flag, RING_F_LF. Because the ring's memsize is now a function of its flags (the lock-free ring requires 128b for each entry), this commit adds a new argument ('flags') to rte_ring_get_memsize(). An API deprecation notice will be sent in a separate commit.

For ease-of-use, existing ring enqueue and dequeue functions work on both regular and lock-free rings. This introduces an additional branch in the datapath, but this should be a highly predictable branch. ring_perf_autotest shows a negligible performance impact; it's hard to distinguish a real difference versus system noise.

| ring_perf_autotest cycles with branch -
Test | ring_perf_autotest cycles without
------------------------------------------------------------------

SP/SC single enq/dequeue          | 0.33
MP/MC single enq/dequeue          | -4.00
SP/SC burst enq/dequeue (size 8)  | 0.00
MP/MC burst enq/dequeue (size 8)  | 0.00
SP/SC burst enq/dequeue (size 32) | 0.00
MP/MC burst enq/dequeue (size 32) | 0.00
SC empty dequeue                  | 1.00
MC empty dequeue                  | 0.00

Single lcore:

SP/SC bulk enq/dequeue (size 8)   | 0.49
MP/MC bulk enq/dequeue (size 8)   | 0.08
SP/SC bulk enq/dequeue (size 32)  | 0.07
MP/MC bulk enq/dequeue (size 32)  | 0.09

Two physical cores:

SP/SC bulk enq/dequeue (size 8)   | 0.19
MP/MC bulk enq/dequeue (size 8)   | -0.37
SP/SC bulk enq/dequeue (size 32)  | 0.09
MP/MC bulk enq/dequeue (size 32)  | -0.05

Two NUMA nodes:

SP/SC bulk enq/dequeue (size 8)   | -1.96
MP/MC bulk enq/dequeue (size 8)   | 0.88
SP/SC bulk enq/dequeue (size 32)  | 0.10
MP/MC bulk enq/dequeue (size 32)  | 0.46

Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. Each test run three times and the results averaged.

---
 lib/librte_ring/rte_ring.c           |  92 +++++++--
 lib/librte_ring/rte_ring.h           | 308 ++++++++++++++++++++++++++---
 lib/librte_ring/rte_ring_c11_mem.h   | 366 ++++++++++++++++++++++++++++++++++-
 lib/librte_ring/rte_ring_generic.h   | 357 +++++++++++++++++++++++++++++++++-
 lib/librte_ring/rte_ring_version.map |   7 +
 5 files changed, 1082 insertions(+), 48 deletions(-)
lib/librte_ring/rte_ring.c +80 -12
index d215acecc..d4a176f57 100644
@@ -45,9 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
 
 /* return the size of memory occupied by a ring */
 ssize_t
-rte_ring_get_memsize(unsigned count)
+rte_ring_get_memsize_v1905(unsigned int count, unsigned int flags)
 {
-	ssize_t sz;
+	ssize_t sz, elt_sz;
 
 	/* count must be a power of 2 */
 	if ((!POWEROF2(count)) || (count > RTE_RING_SZ_MASK )) {
@@ -57,10 +57,23 @@ rte_ring_get_memsize(unsigned count)
 		return -EINVAL;
 	}
 
-	sz = sizeof(struct rte_ring) + count * sizeof(void *);
+	elt_sz = (flags & RING_F_LF) ? 2 * sizeof(void *) : sizeof(void *);
+
+	sz = sizeof(struct rte_ring) + count * elt_sz;
 	sz = RTE_ALIGN(sz, RTE_CACHE_LINE_SIZE);
 	return sz;
 }
+BIND_DEFAULT_SYMBOL(rte_ring_get_memsize, _v1905, 19.05);
+MAP_STATIC_SYMBOL(ssize_t rte_ring_get_memsize(unsigned int count,
+					       unsigned int flags),
+		  rte_ring_get_memsize_v1905);
+
+ssize_t
+rte_ring_get_memsize_v20(unsigned int count)
+{
+	return rte_ring_get_memsize_v1905(count, 0);
+}
+VERSION_SYMBOL(rte_ring_get_memsize, _v20, 2.0);
 
 int
 rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
@@ -75,6 +88,8 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 			  RTE_CACHE_LINE_MASK) != 0);
 	RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
 			  RTE_CACHE_LINE_MASK) != 0);
+	RTE_BUILD_BUG_ON(sizeof(struct rte_ring_lf_entry) !=
+			 2 * sizeof(void *));
 
 	/* init the ring structure */
 	memset(r, 0, sizeof(*r));
@@ -82,8 +97,6 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 	if (ret < 0 || ret >= (int)sizeof(r->name))
 		return -ENAMETOOLONG;
 	r->flags = flags;
-	r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
-	r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
 
 	if (flags & RING_F_EXACT_SZ) {
 		r->size = rte_align32pow2(count + 1);
@@ -100,12 +113,46 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 		r->mask = count - 1;
 		r->capacity = r->mask;
 	}
-	r->prod.head = r->cons.head = 0;
-	r->prod.tail = r->cons.tail = 0;
+
+	r->log2_size = rte_log2_u64(r->size);
+
+	if (flags & RING_F_LF) {
+		uint32_t i;
+
+		r->prod_ptr.single =
+			(flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
+		r->cons_ptr.single =
+			(flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+		r->prod_ptr.head = r->cons_ptr.head = 0;
+		r->prod_ptr.tail = r->cons_ptr.tail = 0;
+
+		for (i = 0; i < r->size; i++) {
+			struct rte_ring_lf_entry *ring_ptr, *base;
+
+			base = (struct rte_ring_lf_entry *)&r->ring;
+
+			ring_ptr = &base[i & r->mask];
+
+			ring_ptr->cnt = 0;
+		}
+	} else {
+		r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
+		r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+		r->prod.head = r->cons.head = 0;
+		r->prod.tail = r->cons.tail = 0;
+	}
 
 	return 0;
 }
 
+/* If a ring entry is written on average every M cycles, then a ring entry is
+ * reused every M*count cycles, and a ring entry's counter repeats every
+ * M*count*2^32 cycles. If M=100 on a 2GHz system, then a 1024-entry ring's
+ * counters would repeat every 2.37 days. The likelihood of ABA occurring is
+ * considered sufficiently low for 1024-entry and larger rings.
+ */
+#define MIN_32_BIT_LF_RING_SIZE 1024
+
 /* create the ring */
 struct rte_ring *
 rte_ring_create(const char *name, unsigned count, int socket_id,
@@ -123,11 +170,25 @@ rte_ring_create(const char *name, unsigned count, int socket_id,
 
 	ring_list = RTE_TAILQ_CAST(rte_ring_tailq.head, rte_ring_list);
 
+#ifdef RTE_ARCH_64
+#if !defined(RTE_ARCH_X86_64)
+	printf("This platform does not support the atomic operation required for RING_F_LF\n");
+	rte_errno = EINVAL;
+	return NULL;
+#endif
+#else
+	if ((flags & RING_F_LF) && count < MIN_32_BIT_LF_RING_SIZE) {
+		printf("RING_F_LF is only supported on 32-bit platforms for rings with at least 1024 entries.\n");
+		rte_errno = EINVAL;
+		return NULL;
+	}
+#endif
+
 	/* for an exact size ring, round up from count to a power of two */
 	if (flags & RING_F_EXACT_SZ)
 		count = rte_align32pow2(count + 1);
 
-	ring_size = rte_ring_get_memsize(count);
+	ring_size = rte_ring_get_memsize(count, flags);
 	if (ring_size < 0) {
 		rte_errno = ring_size;
 		return NULL;
@@ -227,10 +288,17 @@ rte_ring_dump(FILE *f, const struct rte_ring *r)
 	fprintf(f, "  flags=%x\n", r->flags);
 	fprintf(f, "  size=%"PRIu32"\n", r->size);
 	fprintf(f, "  capacity=%"PRIu32"\n", r->capacity);
-	fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
-	fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
-	fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
-	fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	if (r->flags & RING_F_LF) {
+		fprintf(f, "  ct=%"PRIuPTR"\n", r->cons_ptr.tail);
+		fprintf(f, "  ch=%"PRIuPTR"\n", r->cons_ptr.head);
+		fprintf(f, "  pt=%"PRIuPTR"\n", r->prod_ptr.tail);
+		fprintf(f, "  ph=%"PRIuPTR"\n", r->prod_ptr.head);
+	} else {
+		fprintf(f, "  ct=%"PRIu32"\n", r->cons.tail);
+		fprintf(f, "  ch=%"PRIu32"\n", r->cons.head);
+		fprintf(f, "  pt=%"PRIu32"\n", r->prod.tail);
+		fprintf(f, "  ph=%"PRIu32"\n", r->prod.head);
+	}
 	fprintf(f, "  used=%u\n", rte_ring_count(r));
 	fprintf(f, "  avail=%u\n", rte_ring_free_count(r));
 }
lib/librte_ring/rte_ring.h +277 -31

449 lines not on this page · raw message

lib/librte_ring/rte_ring_c11_mem.h +362 -4

395 lines not on this page · raw message

lib/librte_ring/rte_ring_generic.h +356 -1

366 lines not on this page · raw message

lib/librte_ring/rte_ring_version.map +7 -0
index d935efd0d..8969467af 100644
@@ -17,3 +17,10 @@ DPDK_2.2 {
 	rte_ring_free;
 
 } DPDK_2.0;
+
+DPDK_19.05 {
+	global:
+
+	rte_ring_get_memsize;
+
+} DPDK_2.2;
Signature · 1 line
2.13.6
author
#99 lore raw

PATCH v8 4/6 test_ring: add lock-free ring autotest

In reply to Gage Eads

ring_nb_autotest re-uses the ring_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 app/test/test_ring.c | 61 ++++++++++++++++++++++++++++++++--------------------
 1 file changed, 38 insertions(+), 23 deletions(-)
app/test/test_ring.c +38 -23
index aaf1e70ad..400b1bffd 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 #include <string.h>
@@ -601,18 +601,20 @@ test_ring_burst_basic(struct rte_ring *r)
  * it will always fail to create ring with a wrong ring size number in this function
  */
 static int
-test_ring_creation_with_wrong_size(void)
+test_ring_creation_with_wrong_size(unsigned int flags)
 {
 	struct rte_ring * rp = NULL;
 
 	/* Test if ring size is not power of 2 */
-	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", RING_SIZE + 1,
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
 
 	/* Test if ring size is exceeding the limit */
-	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1), SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test_bad_ring_size", (RTE_RING_SZ_MASK + 1),
+			     SOCKET_ID_ANY, flags);
 	if (NULL != rp) {
 		return -1;
 	}
@@ -623,11 +625,11 @@ test_ring_creation_with_wrong_size(void)
  * it tests if it would always fail to create ring with an used ring name
  */
 static int
-test_ring_creation_with_an_used_name(void)
+test_ring_creation_with_an_used_name(unsigned int flags)
 {
 	struct rte_ring * rp;
 
-	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	rp = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (NULL != rp)
 		return -1;
 
@@ -639,10 +641,10 @@ test_ring_creation_with_an_used_name(void)
  * function to fail correctly
  */
 static int
-test_create_count_odd(void)
+test_create_count_odd(unsigned int flags)
 {
 	struct rte_ring *r = rte_ring_create("test_ring_count",
-			4097, SOCKET_ID_ANY, 0 );
+			4097, SOCKET_ID_ANY, flags);
 	if(r != NULL){
 		return -1;
 	}
@@ -665,7 +667,7 @@ test_lookup_null(void)
  * it tests some more basic ring operations
  */
 static int
-test_ring_basic_ex(void)
+test_ring_basic_ex(unsigned int flags)
 {
 	int ret = -1;
 	unsigned i;
@@ -679,7 +681,7 @@ test_ring_basic_ex(void)
 	}
 
 	rp = rte_ring_create("test_ring_basic_ex", RING_SIZE, SOCKET_ID_ANY,
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (rp == NULL) {
 		printf("test_ring_basic_ex fail to create ring\n");
 		goto fail_test;
@@ -737,22 +739,22 @@ test_ring_basic_ex(void)
 }
 
 static int
-test_ring_with_exact_size(void)
+test_ring_with_exact_size(unsigned int flags)
 {
 	struct rte_ring *std_ring = NULL, *exact_sz_ring = NULL;
-	void *ptr_array[16];
+	void *ptr_array[1024];
 	static const unsigned int ring_sz = RTE_DIM(ptr_array);
 	unsigned int i;
 	int ret = -1;
 
 	std_ring = rte_ring_create("std", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ);
+			RING_F_SP_ENQ | RING_F_SC_DEQ | flags);
 	if (std_ring == NULL) {
 		printf("%s: error, can't create std ring\n", __func__);
 		goto end;
 	}
 	exact_sz_ring = rte_ring_create("exact sz", ring_sz, rte_socket_id(),
-			RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ);
+		RING_F_SP_ENQ | RING_F_SC_DEQ | RING_F_EXACT_SZ | flags);
 	if (exact_sz_ring == NULL) {
 		printf("%s: error, can't create exact size ring\n", __func__);
 		goto end;
@@ -770,7 +772,7 @@ test_ring_with_exact_size(void)
 	}
 	/*
 	 * check that the exact_sz_ring can hold one more element than the
-	 * standard ring. (16 vs 15 elements)
+	 * standard ring. (1024 vs 1023 elements)
 	 */
 	for (i = 0; i < ring_sz - 1; i++) {
 		rte_ring_enqueue(std_ring, NULL);
@@ -808,17 +810,17 @@ test_ring_with_exact_size(void)
 }
 
 static int
-test_ring(void)
+__test_ring(unsigned int flags)
 {
 	struct rte_ring *r = NULL;
 
 	/* some more basic operations */
-	if (test_ring_basic_ex() < 0)
+	if (test_ring_basic_ex(flags) < 0)
 		goto test_fail;
 
 	rte_atomic32_init(&synchro);
 
-	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, 0);
+	r = rte_ring_create("test", RING_SIZE, SOCKET_ID_ANY, flags);
 	if (r == NULL)
 		goto test_fail;
 
@@ -837,27 +839,27 @@ test_ring(void)
 		goto test_fail;
 
 	/* basic operations */
-	if ( test_create_count_odd() < 0){
+	if (test_create_count_odd(flags) < 0) {
 		printf("Test failed to detect odd count\n");
 		goto test_fail;
 	} else
 		printf("Test detected odd count\n");
 
-	if ( test_lookup_null() < 0){
+	if (test_lookup_null() < 0) {
 		printf("Test failed to detect NULL ring lookup\n");
 		goto test_fail;
 	} else
 		printf("Test detected NULL ring lookup\n");
 
 	/* test of creating ring with wrong size */
-	if (test_ring_creation_with_wrong_size() < 0)
+	if (test_ring_creation_with_wrong_size(flags) < 0)
 		goto test_fail;
 
 	/* test of creation ring with an used name */
-	if (test_ring_creation_with_an_used_name() < 0)
+	if (test_ring_creation_with_an_used_name(flags) < 0)
 		goto test_fail;
 
-	if (test_ring_with_exact_size() < 0)
+	if (test_ring_with_exact_size(flags) < 0)
 		goto test_fail;
 
 	/* dump the ring status */
@@ -873,4 +875,17 @@ test_ring(void)
 	return -1;
 }
 
+static int
+test_ring(void)
+{
+	return __test_ring(0);
+}
+
+static int
+test_lf_ring(void)
+{
+	return __test_ring(RING_F_LF);
+}
+
 REGISTER_TEST_COMMAND(ring_autotest, test_ring);
+REGISTER_TEST_COMMAND(ring_lf_autotest, test_lf_ring);
Signature · 1 line
2.13.6
author
#100 lore raw

PATCH v8 5/6 test_ring_perf: add lock-free ring perf test

In reply to Gage Eads

nb_ring_perf_autotest re-uses the ring_perf_autotest code by wrapping its top-level function with one that takes a 'flags' argument.

---
 app/test/test_ring_perf.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
app/test/test_ring_perf.c +16 -3
index ebb3939f5..be465c758 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2019 Intel Corporation
  */
 
 
@@ -363,12 +363,12 @@ test_bulk_enqueue_dequeue(struct rte_ring *r)
 }
 
 static int
-test_ring_perf(void)
+__test_ring_perf(unsigned int flags)
 {
 	struct lcore_pair cores;
 	struct rte_ring *r = NULL;
 
-	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), 0);
+	r = rte_ring_create(RING_NAME, RING_SIZE, rte_socket_id(), flags);
 	if (r == NULL)
 		return -1;
 
@@ -398,4 +398,17 @@ test_ring_perf(void)
 	return 0;
 }
 
+static int
+test_ring_perf(void)
+{
+	return __test_ring_perf(0);
+}
+
+static int
+test_lf_ring_perf(void)
+{
+	return __test_ring_perf(RING_F_LF);
+}
+
 REGISTER_TEST_COMMAND(ring_perf_autotest, test_ring_perf);
+REGISTER_TEST_COMMAND(ring_lf_perf_autotest, test_lf_ring_perf);
Signature · 1 line
2.13.6
author
#101 lore raw

PATCH v8 6/6 mempool/ring: add lock-free ring handlers

In reply to Gage Eads

These handlers allow an application to create a mempool based on the lock-free ring, with any combination of single/multi producer/consumer.

Also, add a note to the programmer's guide's "known issues" section.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 +++++
 drivers/mempool/ring/Makefile                   |  1 +
 drivers/mempool/ring/meson.build                |  2 +
 drivers/mempool/ring/rte_mempool_ring.c         | 58 +++++++++++++++++++++++--
 4 files changed, 68 insertions(+), 3 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
index 2361c3b8f..18bc98c57 100644
@@ -563,6 +563,16 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
+  Alternatively, 32-bit and x86_64 applications can use the lock-free ring
+  mempool handler. When considering it, note that:
+
+  - Among 64-bit architectures it is currently limited to the x86_64 platform,
+    because it uses a function (16-byte compare-and-swap) that is not yet
+    available on other platforms.
+  - It has worse average-case performance than the non-preemptive rte_ring, but
+    software caching (e.g. the mempool cache) can mitigate this by reducing the
+    number of handler operations.
+
 + rte_timer
 
   Running  ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
drivers/mempool/ring/Makefile +1 -0
index ddab522fe..012ba6966 100644
@@ -10,6 +10,7 @@ LIB = librte_mempool_ring.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 LDLIBS += -lrte_eal -lrte_mempool -lrte_ring
 
 EXPORT_MAP := rte_mempool_ring_version.map
drivers/mempool/ring/meson.build +2 -0
index a021e908c..b1cb673cc 100644
@@ -1,4 +1,6 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
+allow_experimental_apis = true
+
 sources = files('rte_mempool_ring.c')
drivers/mempool/ring/rte_mempool_ring.c +55 -3

84 lines not on this page · raw message

Signature · 1 line
2.13.6
#102 lore raw

Re: [PATCH v8 0/6] Add lock-free ring and mempool handler

In reply to Gage Eads

19/03/2019 02:20, Gage Eads:

Quoted 2 lines

This patchset makes one API change; a deprecation notice was posted in a separate commit[1].

It means this patchset targets the release 19.08.
In such case, it is better to use --subject-prefix='PATCH 19.08'