Skip to contentSkip to conversation

Subject: [PATCH ...] Add stack library and new mempool handler

PATCH Superseded Gage Eads 4 participants 133 messages Feb 22, 2019 – Apr 4, 2019 1 message not in the archive Page 1 of 2

Subsystems (2)
lore
Share & tools
lore

Gage Eads

author
#0 lore raw

cover Subject: [PATCH ...] Add stack library and new mempool handler

This patchset introduces a stack library, supporting both lock-based and non-blocking stacks, and a non-blocking stack mempool handler.

The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.

The non-blocking stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

Note that the non-blocking algorithm relies on a 128-bit compare-and-swap, so it is currently limited to the x86_64 platform.

This patchset is the successor to a patchset containing only the new mempool handler[1].

[1] http://mails.dpdk.org/archives/dev/2019-January/123555.html

Gage Eads (7):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add non-blocking stack implementation test/stack: add non-blocking stack tests mempool/stack: add non-blocking stack mempool handler

 MAINTAINERS                                     |   9 +-
 config/common_base                              |   5 +
 doc/api/doxy-api-index.md                       |   1 +
 doc/api/doxy-api.conf.in                        |   1 +
 doc/guides/prog_guide/env_abstraction_layer.rst |   5 +
 doc/guides/prog_guide/index.rst                 |   1 +
 doc/guides/prog_guide/stack_lib.rst             |  66 ++++
 doc/guides/rel_notes/release_19_05.rst          |  13 +
 drivers/mempool/stack/Makefile                  |   3 +-
 drivers/mempool/stack/meson.build               |   6 +-
 drivers/mempool/stack/rte_mempool_stack.c       | 115 +++----
 lib/Makefile                                    |   2 +
 lib/librte_stack/Makefile                       |  25 ++
 lib/librte_stack/meson.build                    |  10 +
 lib/librte_stack/rte_stack.c                    | 220 +++++++++++++
 lib/librte_stack/rte_stack.h                    | 406 +++++++++++++++++++++++
 lib/librte_stack/rte_stack_c11_mem.h            | 173 ++++++++++
 lib/librte_stack/rte_stack_generic.h            | 157 +++++++++
 lib/librte_stack/rte_stack_pvt.h                |  34 ++
 lib/librte_stack/rte_stack_version.map          |   9 +
 lib/meson.build                                 |   2 +-
 mk/rte.app.mk                                   |   1 +
 test/test/Makefile                              |   3 +
 test/test/meson.build                           |   7 +
 test/test/test_stack.c                          | 407 ++++++++++++++++++++++++
 test/test/test_stack_perf.c                     | 356 +++++++++++++++++++++
 26 files changed, 1965 insertions(+), 72 deletions(-)
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_c11_mem.h
 create mode 100644 lib/librte_stack/rte_stack_generic.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
 create mode 100644 test/test/test_stack.c
 create mode 100644 test/test/test_stack_perf.c
Signature · 1 line
2.13.6
author
#1 lore raw

PATCH 1/7 stack: introduce rte stack library

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

---
 MAINTAINERS                            |   6 +
 config/common_base                     |   5 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/index.rst        |   1 +
 doc/guides/prog_guide/stack_lib.rst    |  26 ++++
 doc/guides/rel_notes/release_19_05.rst |   5 +
 lib/Makefile                           |   2 +
 lib/librte_stack/Makefile              |  23 +++
 lib/librte_stack/meson.build           |   8 +
 lib/librte_stack/rte_stack.c           | 194 +++++++++++++++++++++++
 lib/librte_stack/rte_stack.h           | 277 +++++++++++++++++++++++++++++++++
 lib/librte_stack/rte_stack_pvt.h       |  34 ++++
 lib/librte_stack/rte_stack_version.map |   9 ++
 lib/meson.build                        |   2 +-
 mk/rte.app.mk                          |   1 +
 16 files changed, 594 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
index eef480ab5..237f05eb2 100644
@@ -407,6 +407,12 @@ F: drivers/raw/skeleton_rawdev/
 F: test/test/test_rawdev.c
 F: doc/guides/prog_guide/rawdev.rst
 
+Stack API - EXPERIMENTAL
+M: Gage Eads <[email protected]>
+M: Olivier Matz <[email protected]>
+F: lib/librte_stack/
+F: doc/guides/prog_guide/stack_lib.rst
+
 
 Memory Pool Drivers
 -------------------
config/common_base +5 -0
index 7c6da5165..5861eb09c 100644
@@ -980,3 +980,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
 # Compile the eventdev application
 #
 CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Compile librte_stack
+#
+CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
index d95ad566c..0df8848c0 100644
@@ -124,6 +124,7 @@ The public API headers are grouped by topics:
   [mbuf]               (@ref rte_mbuf.h),
   [mbuf pool ops]      (@ref rte_mbuf_pool_ops.h),
   [ring]               (@ref rte_ring.h),
+  [stack]              (@ref rte_stack.h),
   [tailq]              (@ref rte_tailq.h),
   [bitmap]             (@ref rte_bitmap.h)
 
doc/api/doxy-api.conf.in +1 -0
index bef9320c0..dd972a3fe 100644
@@ -56,6 +56,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/librte_ring \
                           @TOPDIR@/lib/librte_sched \
                           @TOPDIR@/lib/librte_security \
+                          @TOPDIR@/lib/librte_stack \
                           @TOPDIR@/lib/librte_table \
                           @TOPDIR@/lib/librte_telemetry \
                           @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
index 6726b1e8d..f4f60862f 100644
@@ -55,6 +55,7 @@ Programmer's Guide
     metrics_lib
     bpf_lib
     ipsec_lib
+    stack_lib
     source_org
     dev_kit_build_system
     dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +26 -0
new file mode 100644
index 000000000..51689cfe1
@@ -0,0 +1,26 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Stack Library
+=============
+
+DPDK's stack library provides an API for configuration and use of a bounded stack of
+pointers.
+
+The stack library provides the following basic operations:
+
+*  Create a uniquely named stack of a user-specified size and using a user-specified socket.
+
+*  Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe.
+
+*  Free a previously created stack.
+
+*  Lookup a pointer to a stack by its name.
+
+*  Query a stack's current depth and number of free entries.
+
+Implementation
+~~~~~~~~~~~~~~
+
+The stack consists of a contiguous array of pointers, a current index, and a
+spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
index 2b0f60d3d..04394f8cf 100644
@@ -65,6 +65,11 @@ New Features
     process.
   * Added support for Rx packet types list in a secondary process.
 
+* **Added Stack API.**
+
+  Added a new stack API for configuration and use of a bounded stack of
+  pointers. The API provides MT-safe push and pop operations that can operate
+  on one or more pointers per operation.
 
 Removed Items
 -------------
lib/Makefile +2 -0
index d6239d27c..d22e2072b 100644
@@ -111,6 +111,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec
 DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security
 DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry
 DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev
+DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack
+DEPDIRS-librte_stack := librte_eal
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +23 -0
new file mode 100644
index 000000000..e956b6535
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_stack.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+LDLIBS += -lrte_eal
+
+EXPORT_MAP := rte_stack_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c
+
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
new file mode 100644
index 000000000..99f43710e
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+allow_experimental_apis = true
+
+version = 1
+sources = files('rte_stack.c')
+headers = files('rte_stack.h')
lib/librte_stack/rte_stack.c new file +194 -0

194 lines not on this page · raw message

lib/librte_stack/rte_stack.h new file +277 -0

277 lines not on this page · raw message

lib/librte_stack/rte_stack_pvt.h new file +34 -0
new file mode 100644
index 000000000..4a6a7bdb3
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_STACK_PVT_H_
+#define _RTE_STACK_PVT_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_log.h>
+
+extern int stack_logtype;
+
+#define STACK_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \
+		__func__, ##args)
+
+#define STACK_LOG_ERR(fmt, args...) \
+	STACK_LOG(ERR, fmt, ## args)
+
+#define STACK_LOG_WARN(fmt, args...) \
+	STACK_LOG(WARNING, fmt, ## args)
+
+#define STACK_LOG_INFO(fmt, args...) \
+	STACK_LOG(INFO, fmt, ## args)
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_version.map new file +9 -0
new file mode 100644
index 000000000..6662679c3
@@ -0,0 +1,9 @@
+EXPERIMENTAL {
+	global:
+
+	rte_stack_create;
+	rte_stack_free;
+	rte_stack_lookup;
+
+	local: *;
+};
lib/meson.build +1 -1
index e8b40f546..0f0e589bc 100644
@@ -21,7 +21,7 @@ libraries = [ 'compat', # just a header, used for versioning
 	'gro', 'gso', 'ip_frag', 'jobstats',
 	'kni', 'latencystats', 'lpm', 'member',
 	'power', 'pdump', 'rawdev',
-	'reorder', 'sched', 'security', 'vhost',
+	'reorder', 'sched', 'security', 'stack', 'vhost',
 	#ipsec lib depends on crypto and security
 	'ipsec',
 	# add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
index 8a4f0f4e5..55568c603 100644
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -lrte_security
 _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV)         += -lrte_rawdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER)          += -lrte_timer
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL)        += -lrte_mempool
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
Signature · 1 line
2.13.6
#2 lore raw

Re: [PATCH 1/7] stack: introduce rte stack library

Hi Gage,

Please find few comments below.

Quoted 11 lines · Gage Eads

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

[...]

Quoted 13 lines
doc/guides/prog_guide/stack_lib.rst new file +10 -0
@@ -0,0 +1,26 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Stack Library
+=============
+
+DPDK's stack library provides an API for configuration and use of a bounded stack of
+pointers.
+
+The stack library provides the following basic operations:

doc/guides/prog_guide/stack_lib.rst line 20

Quoted 10 lines
doc/guides/prog_guide/stack_lib.rst new file +10 -0
+
+*  Create a uniquely named stack of a user-specified size and using a user-specified socket.
+
+*  Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe.
+
+*  Free a previously created stack.
+
+*  Lookup a pointer to a stack by its name.
+
+*  Query a stack's current depth and number of free entries.

It seems the 80-cols limitation also applies to documentation:
https://mails.dpdk.org/archives/dev/2019-February/124917.html

[...]

Quoted 61 lines
lib/librte_stack/rte_stack.h new file +58 -0
@@ -0,0 +1,277 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+/**
+ * @file rte_stack.h
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * RTE Stack.
+ * librte_stack provides an API for configuration and use of a bounded stack of
+ * pointers. Push and pop operations are MT-safe, allowing concurrent access,
+ * and the interface supports pushing and popping multiple pointers at a time.
+ */
+
+#ifndef _RTE_STACK_H_
+#define _RTE_STACK_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_errno.h>
+#include <rte_memzone.h>
+#include <rte_spinlock.h>
+
+#define RTE_TAILQ_STACK_NAME "RTE_STACK"
+#define RTE_STACK_MZ_PREFIX "STK_"
+/**< The maximum length of a stack name. */
+#define RTE_STACK_NAMESIZE (RTE_MEMZONE_NAMESIZE - \
+			   sizeof(RTE_STACK_MZ_PREFIX) + 1)
+
+/* Structure containing the LIFO, its current length, and a lock for mutual
+ * exclusion.
+ */
+struct rte_lifo {
+	rte_spinlock_t lock; /**< LIFO lock */
+	uint32_t len; /**< LIFO len */
+	void *objs[]; /**< LIFO pointer table */
+};
+
+/* The RTE stack structure contains the LIFO structure itself, plus metadata
+ * such as its name and memzone pointer.
+ */
+struct rte_stack {
+	/** Name of the stack. */
+	char name[RTE_STACK_NAMESIZE] __rte_cache_aligned;
+	/** Memzone containing the rte_stack structure */
+	const struct rte_memzone *memzone;
+	uint32_t capacity; /**< Usable size of the stack */
+	uint32_t flags; /**< Flags supplied at creation */
+	struct rte_lifo lifo; /**< LIFO structure */
+} __rte_cache_aligned;
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * @internal Push several objects on the stack (MT-safe)

lib/librte_stack/rte_stack.h line 68

Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10 -0
+ *
+ * @param s
+ *   A pointer to the stack structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to push on the stack from the obj_table.
+ * @return
+ *   Actual number of objects pushed (either 0 or *n*).
+ */

Minor: a dot is missing at the end of the title. There are few in this patch, and maybe in next ones.

[...]

Quoted 6 lines
lib/librte_stack/rte_stack.h new file +6 -0
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Return the number of used entries in a stack.
+ *

lib/librte_stack/rte_stack.h line 194

Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10 -0
+ * @param s
+ *   A pointer to the stack structure.
+ * @return
+ *   The number of used entries in the stack.
+ */
+static __rte_always_inline unsigned int __rte_experimental
+rte_stack_count(struct rte_stack *s)
+{
+	return (unsigned int)s->lifo.len;
+}

The argument can be const.

Quoted 7 lines
lib/librte_stack/rte_stack.h new file +7 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Return the number of free entries in a stack.
+ *

lib/librte_stack/rte_stack.h line 211

Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10 -0
+ * @param s
+ *   A pointer to the stack structure.
+ * @return
+ *   The number of free entries in the stack.
+ */
+static __rte_always_inline unsigned int __rte_experimental
+rte_stack_free_count(struct rte_stack *s)
+{
+	return s->capacity - rte_stack_count(s);
+}

Same here.

author
#3 lore raw

Re: [PATCH 1/7] stack: introduce rte stack library

Quoted 50 lines

-----Original Message-----
From: Olivier Matz [mailto:[email protected]]
Sent: Monday, February 25, 2019 4:43 AM
To: Eads, Gage <[email protected]>
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 1/7] stack: introduce rte stack library

Hi Gage,

Please find few comments below.

Quoted 11 lines · Gage Eads

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

[...]

Quoted 5 lines
doc/guides/prog_guide/stack_lib.rst new file +2 -0
@@ -0,0 +1,26 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.

doc/guides/prog_guide/stack_lib.rst line 12

Quoted 10 lines
doc/guides/prog_guide/stack_lib.rst new file +10 -0
+
+Stack Library
+=============
+
+DPDK's stack library provides an API for configuration and use of a
+bounded stack of pointers.
+
+The stack library provides the following basic operations:
+
+*  Create a uniquely named stack of a user-specified size and using a user-

specified socket.

doc/guides/prog_guide/stack_lib.rst line 14

Quoted 2 lines
doc/guides/prog_guide/stack_lib.rst new file +2 -0
+
+*  Push and pop a burst of one or more stack objects (pointers). These

function are multi-threading safe.

Quoted 10 lines

doc/guides/prog_guide/stack_lib.rst line 20

Quoted 6 lines
doc/guides/prog_guide/stack_lib.rst new file +6 -0
+
+*  Free a previously created stack.
+
+*  Lookup a pointer to a stack by its name.
+
+*  Query a stack's current depth and number of free entries.

It seems the 80-cols limitation also applies to documentation:
https://mails.dpdk.org/archives/dev/2019-February/124917.html

Sure, will fix in v2.

Quoted 69 lines

[...]

Quoted 67 lines
lib/librte_stack/rte_stack.h new file +64 -0
@@ -0,0 +1,277 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation  */
+
+/**
+ * @file rte_stack.h
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * RTE Stack.
+ * librte_stack provides an API for configuration and use of a
+bounded stack of
+ * pointers. Push and pop operations are MT-safe, allowing concurrent
+access,
+ * and the interface supports pushing and popping multiple pointers at a time.
+ */
+
+#ifndef _RTE_STACK_H_
+#define _RTE_STACK_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_errno.h>
+#include <rte_memzone.h>
+#include <rte_spinlock.h>
+
+#define RTE_TAILQ_STACK_NAME "RTE_STACK"
+#define RTE_STACK_MZ_PREFIX "STK_"
+/**< The maximum length of a stack name. */ #define
+RTE_STACK_NAMESIZE (RTE_MEMZONE_NAMESIZE - \
+			   sizeof(RTE_STACK_MZ_PREFIX) + 1)
+
+/* Structure containing the LIFO, its current length, and a lock for
+mutual
+ * exclusion.
+ */
+struct rte_lifo {
+	rte_spinlock_t lock; /**< LIFO lock */
+	uint32_t len; /**< LIFO len */
+	void *objs[]; /**< LIFO pointer table */ };
+
+/* The RTE stack structure contains the LIFO structure itself, plus
+metadata
+ * such as its name and memzone pointer.
+ */
+struct rte_stack {
+	/** Name of the stack. */
+	char name[RTE_STACK_NAMESIZE] __rte_cache_aligned;
+	/** Memzone containing the rte_stack structure */
+	const struct rte_memzone *memzone;
+	uint32_t capacity; /**< Usable size of the stack */
+	uint32_t flags; /**< Flags supplied at creation */
+	struct rte_lifo lifo; /**< LIFO structure */ } __rte_cache_aligned;
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * @internal Push several objects on the stack (MT-safe)
+ *
+ * @param s
+ *   A pointer to the stack structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
Quoted 9 lines

lib/librte_stack/rte_stack.h line 68

Quoted 5 lines
lib/librte_stack/rte_stack.h new file +5 -0
+ * @param n
+ *   The number of objects to push on the stack from the obj_table.
+ * @return
+ *   Actual number of objects pushed (either 0 or *n*).
+ */

Minor: a dot is missing at the end of the title. There are few in this patch, and maybe in next ones.

Will fix.

Quoted 29 lines

[...]

Quoted 5 lines
lib/librte_stack/rte_stack.h new file +5 -0
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Return the number of used entries in a stack.

lib/librte_stack/rte_stack.h line 194

Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10 -0
+ *
+ * @param s
+ *   A pointer to the stack structure.
+ * @return
+ *   The number of used entries in the stack.
+ */
+static __rte_always_inline unsigned int __rte_experimental
+rte_stack_count(struct rte_stack *s) {
+	return (unsigned int)s->lifo.len;
+}

The argument can be const.

Quoted 9 lines
lib/librte_stack/rte_stack.h new file +9 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Return the number of free entries in a stack.
+ *
+ * @param s
+ *   A pointer to the stack structure.
Quoted 8 lines

lib/librte_stack/rte_stack.h

Quoted 6 lines

+ * @return
+ * The number of free entries in the stack.
+ */

lib/librte_stack/rte_stack.h new file +3 -0
+static __rte_always_inline unsigned int __rte_experimental
+rte_stack_free_count(struct rte_stack *s) {
+	return s->capacity - rte_stack_count(s); }

Same here.

Unfortunately the const keyword causes a discarded-qualifiers warning in the non-blocking implementation, due to the call to rte_atomic64_read().

Thanks,
Gage

author
#4 lore raw

PATCH 2/7 mempool/stack: convert mempool to use rte stack

In reply to Gage Eads

The new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.

---
 MAINTAINERS                               |  2 +-
 drivers/mempool/stack/Makefile            |  3 +-
 drivers/mempool/stack/meson.build         |  6 +-
 drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++----------------------
 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
index 237f05eb2..7e64f63b6 100644
@@ -284,7 +284,6 @@ M: Andrew Rybchenko <[email protected]>
 F: lib/librte_mempool/
 F: drivers/mempool/Makefile
 F: drivers/mempool/ring/
-F: drivers/mempool/stack/
 F: doc/guides/prog_guide/mempool_lib.rst
 F: test/test/test_mempool*
 F: test/test/test_func_reentrancy.c
@@ -412,6 +411,7 @@ M: Gage Eads <[email protected]>
 M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
+F: drivers/mempool/stack/
 
 
 Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
index 0444aedad..1681a62bc 100644
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # Headers
 CFLAGS += -I$(RTE_SDK)/lib/librte_mempool
-LDLIBS += -lrte_eal -lrte_mempool -lrte_ring
+LDLIBS += -lrte_eal -lrte_mempool -lrte_stack
 
 EXPORT_MAP := rte_mempool_stack_version.map
 
drivers/mempool/stack/meson.build +5 -1
index b75a3bb56..03e369a41 100644
@@ -1,4 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
+# Copyright(c) 2017-2019 Intel Corporation
+
+allow_experimental_apis = true
 
 sources = files('rte_mempool_stack.c')
+
+deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68

133 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#6 lore raw

PATCH 3/7 test/stack: add stack test

In reply to Gage Eads

stack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.

---
 MAINTAINERS            |   1 +
 test/test/Makefile     |   2 +
 test/test/meson.build  |   3 +
 test/test/test_stack.c | 394 +++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 400 insertions(+)
 create mode 100644 test/test/test_stack.c
MAINTAINERS +1 -0
index 7e64f63b6..58b438414 100644
@@ -412,6 +412,7 @@ M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
 F: drivers/mempool/stack/
+F: test/test/*stack*
 
 
 Memory Pool Drivers
test/test/Makefile +2 -0
index 89949c2bb..47cf98a3a 100644
@@ -89,6 +89,8 @@ endif
 
 SRCS-y += test_rwlock.c
 
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
test/test/meson.build +3 -0
index 05e5ddeb0..b00e1201a 100644
@@ -95,6 +95,7 @@ test_sources = files('commands.c',
 	'test_sched.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
+	'test_stack.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -132,6 +133,7 @@ test_deps = ['acl',
 	'port',
 	'reorder',
 	'ring',
+	'stack',
 	'timer'
 ]
 
@@ -173,6 +175,7 @@ fast_parallel_test_names = [
         'rwlock_autotest',
         'sched_autotest',
         'spinlock_autotest',
+        'stack_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
test/test/test_stack.c new file +394 -0

394 lines not on this page · raw message

Signature · 1 line
2.13.6
#7 lore raw

Re: [PATCH 3/7] test/stack: add stack test

Quoted 4 lines · Gage Eads

stack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.

[...]

Quoted 14 lines
test/test/test_stack.c new file +11 -0
@@ -0,0 +1,394 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include <string.h>
+
+#include <rte_lcore.h>
+#include <rte_malloc.h>
+#include <rte_random.h>
+#include <rte_stack.h>
+

test/test/test_stack.c line 21

Quoted 10 lines
test/test/test_stack.c new file +10 -0
+#include "test.h"
+
+#define STACK_SIZE 4096
+#define MAX_BULK 32
+
+static int
+test_stack_push_pop(struct rte_stack *s, void **obj_table, unsigned int bulk_sz)
+{
+	void *popped_objs[STACK_SIZE];
+	unsigned int i, ret;

Here, a dynamic sized table is used. In test_stack_basic() below, it uses a heap-based allocation for the same purpose. I think it would be more consistent to have the same method for both. I suggest to allocate in heap to avoid a stack overflow if STACK_SIZE is increased in the future.

[...]

Quoted 68 lines
test/test/test_stack.c new file +68 -0
+static int
+test_stack_basic(void)
+{
+	struct rte_stack *s = NULL;
+	void **obj_table = NULL;
+	int i, ret = -1;
+
+	obj_table = rte_calloc(NULL, STACK_SIZE, sizeof(void *), 0);
+	if (obj_table == NULL) {
+		printf("[%s():%u] failed to calloc %lu bytes\n",
+		       __func__, __LINE__, STACK_SIZE * sizeof(void *));
+		goto fail_test;
+	}
+
+	for (i = 0; i < STACK_SIZE; i++)
+		obj_table[i] = (void *)(uintptr_t)i;
+
+	s = rte_stack_create(__func__, STACK_SIZE, rte_socket_id(), 0);
+	if (s == NULL) {
+		printf("[%s():%u] failed to create a stack\n",
+		       __func__, __LINE__);
+		goto fail_test;
+	}
+
+	if (rte_stack_lookup(__func__) != s) {
+		printf("[%s():%u] failed to lookup a stack\n",
+		       __func__, __LINE__);
+		goto fail_test;
+	}
+
+	if (rte_stack_count(s) != 0) {
+		printf("[%s():%u] stack count: %u (expected 0)\n",
+		       __func__, __LINE__, rte_stack_count(s));
+		goto fail_test;
+	}
+
+	if (rte_stack_free_count(s) != STACK_SIZE) {
+		printf("[%s():%u] stack free count: %u (expected %u)\n",
+		       __func__, __LINE__, rte_stack_count(s), STACK_SIZE);
+		goto fail_test;
+	}
+
+	ret = test_stack_push_pop(s, obj_table, 1);
+	if (ret) {
+		printf("[%s():%u] Single object push/pop failed\n",
+		       __func__, __LINE__);
+		goto fail_test;
+	}
+
+	ret = test_stack_push_pop(s, obj_table, MAX_BULK);
+	if (ret) {
+		printf("[%s():%u] Bulk object push/pop failed\n",
+		       __func__, __LINE__);
+		goto fail_test;
+	}
+
+	ret = rte_stack_push(s, obj_table, 2 * STACK_SIZE);
+	if (ret != 0) {
+		printf("[%s():%u] Excess objects push succeeded\n",
+		       __func__, __LINE__);
+		goto fail_test;
+	}
+
+	ret = rte_stack_pop(s, obj_table, 1);
+	if (ret != 0) {
+		printf("[%s():%u] Empty stack pop succeeded\n",
+		       __func__, __LINE__);
+		goto fail_test;

test/test/test_stack.c line 160

Quoted 10 lines
test/test/test_stack.c new file +10 -0
+	}
+
+	ret = 0;
+
+fail_test:
+	rte_stack_free(s);
+
+	if (obj_table != NULL)
+		rte_free(obj_table);
+

The if can be removed.

Quoted 4 lines
test/test/test_stack.c new file +4 -0
+static int
+test_stack_name_length(void)
+{
+	char name[RTE_STACK_NAMESIZE + 1];

test/test/test_stack.c line 201

Quoted 10 lines
test/test/test_stack.c new file +10 -0
+	struct rte_stack *s;
+
+	memset(name, 's', sizeof(name));
+
+	s = rte_stack_create(name, STACK_SIZE, rte_socket_id(), 0);
+	if (s != NULL) {
+		printf("[%s():%u] Failed to prevent long name\n",
+		       __func__, __LINE__);
+		return -1;
+	}

Here, "name" is not a valid string (no \0 at the end). It does not hurt because the length check is properly done in the lib, but we could imagine that the wrong name is logged by the library on error, which would trigger a crash here. So I suggest to pass a valid string instead.

author
#8 lore raw

Re: [PATCH 3/7] test/stack: add stack test

Quoted 41 lines

-----Original Message-----
From: Olivier Matz [mailto:[email protected]]
Sent: Monday, February 25, 2019 4:59 AM
To: Eads, Gage <[email protected]>
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 3/7] test/stack: add stack test

Quoted 2 lines · Gage Eads

stack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available

lcores.

Quoted 2 lines

[...]

Quoted 20 lines
test/test/test_stack.c new file +17 -0
@@ -0,0 +1,394 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation  */
+
+#include <string.h>
+
+#include <rte_lcore.h>
+#include <rte_malloc.h>
+#include <rte_random.h>
+#include <rte_stack.h>
+
+#include "test.h"
+
+#define STACK_SIZE 4096
+#define MAX_BULK 32
+
+static int
+test_stack_push_pop(struct rte_stack *s, void **obj_table, unsigned
Quoted 9 lines

test/test/test_stack.c line 21

Quoted 3 lines
test/test/test_stack.c new file +3 -0
+int bulk_sz) {
+	void *popped_objs[STACK_SIZE];
+	unsigned int i, ret;

Here, a dynamic sized table is used. In test_stack_basic() below, it uses a heap- based allocation for the same purpose. I think it would be more consistent to have the same method for both. I suggest to allocate in heap to avoid a stack overflow if STACK_SIZE is increased in the future.

Sure, I'll make popped_objs dynamically allocated.

Quoted 74 lines

[...]

Quoted 72 lines
test/test/test_stack.c new file +72 -0
+static int
+test_stack_basic(void)
+{
+	struct rte_stack *s = NULL;
+	void **obj_table = NULL;
+	int i, ret = -1;
+
+	obj_table = rte_calloc(NULL, STACK_SIZE, sizeof(void *), 0);
+	if (obj_table == NULL) {
+		printf("[%s():%u] failed to calloc %lu bytes\n",
+		       __func__, __LINE__, STACK_SIZE * sizeof(void *));
+		goto fail_test;
+	}
+
+	for (i = 0; i < STACK_SIZE; i++)
+		obj_table[i] = (void *)(uintptr_t)i;
+
+	s = rte_stack_create(__func__, STACK_SIZE, rte_socket_id(), 0);
+	if (s == NULL) {
+		printf("[%s():%u] failed to create a stack\n",
+		       __func__, __LINE__);
+		goto fail_test;
+	}
+
+	if (rte_stack_lookup(__func__) != s) {
+		printf("[%s():%u] failed to lookup a stack\n",
+		       __func__, __LINE__);
+		goto fail_test;
+	}
+
+	if (rte_stack_count(s) != 0) {
+		printf("[%s():%u] stack count: %u (expected 0)\n",
+		       __func__, __LINE__, rte_stack_count(s));
+		goto fail_test;
+	}
+
+	if (rte_stack_free_count(s) != STACK_SIZE) {
+		printf("[%s():%u] stack free count: %u (expected %u)\n",
+		       __func__, __LINE__, rte_stack_count(s), STACK_SIZE);
+		goto fail_test;
+	}
+
+	ret = test_stack_push_pop(s, obj_table, 1);
+	if (ret) {
+		printf("[%s():%u] Single object push/pop failed\n",
+		       __func__, __LINE__);
+		goto fail_test;
+	}
+
+	ret = test_stack_push_pop(s, obj_table, MAX_BULK);
+	if (ret) {
+		printf("[%s():%u] Bulk object push/pop failed\n",
+		       __func__, __LINE__);
+		goto fail_test;
+	}
+
+	ret = rte_stack_push(s, obj_table, 2 * STACK_SIZE);
+	if (ret != 0) {
+		printf("[%s():%u] Excess objects push succeeded\n",
+		       __func__, __LINE__);
+		goto fail_test;
+	}
+
+	ret = rte_stack_pop(s, obj_table, 1);
+	if (ret != 0) {
+		printf("[%s():%u] Empty stack pop succeeded\n",
+		       __func__, __LINE__);
+		goto fail_test;
+	}
+
+	ret = 0;
+
Quoted 8 lines

test/test/test_stack.c line 160

Quoted 6 lines
test/test/test_stack.c new file +6 -0
+fail_test:
+	rte_stack_free(s);
+
+	if (obj_table != NULL)
+		rte_free(obj_table);
+

The if can be removed.

Ah, I didn't know rte_free() checks for NULL. Will remove.

Quoted 12 lines
Quoted 12 lines
test/test/test_stack.c new file +11 -0
+static int
+test_stack_name_length(void)
+{
+	char name[RTE_STACK_NAMESIZE + 1];
+	struct rte_stack *s;
+
+	memset(name, 's', sizeof(name));
+
+	s = rte_stack_create(name, STACK_SIZE, rte_socket_id(), 0);
+	if (s != NULL) {
+		printf("[%s():%u] Failed to prevent long name\n",
Quoted 8 lines

test/test/test_stack.c line 174

Quoted 3 lines
test/test/test_stack.c new file +3 -0
+		       __func__, __LINE__);
+		return -1;
+	}

Here, "name" is not a valid string (no \0 at the end). It does not hurt because the length check is properly done in the lib, but we could imagine that the wrong name is logged by the library on error, which would trigger a crash here. So I suggest to pass a valid string instead.

Good catch. Will fix.

Thanks,
Gage

author
#9 lore raw

PATCH 4/7 test/stack: add stack perf test

In reply to Gage Eads

stack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects

It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores

---
 test/test/Makefile          |   1 +
 test/test/meson.build       |   2 +
 test/test/test_stack_perf.c | 343 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 346 insertions(+)
 create mode 100644 test/test/test_stack_perf.c
test/test/Makefile +1 -0
index 47cf98a3a..f9536fb31 100644
@@ -90,6 +90,7 @@ endif
 SRCS-y += test_rwlock.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
test/test/meson.build +2 -0
index b00e1201a..ba3cb6261 100644
@@ -96,6 +96,7 @@ test_sources = files('commands.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
 	'test_stack.c',
+	'test_stack_perf.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -240,6 +241,7 @@ perf_test_names = [
         'distributor_perf_autotest',
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
+        'stack_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
test/test/test_stack_perf.c new file +343 -0

343 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v1 4/7
#10 lore raw

Re: [PATCH 4/7] test/stack: add stack perf test

Quoted 5 lines · Gage Eads

stack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects

Quoted 8 lines

It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores

author
#11 lore raw

PATCH 5/7 stack: add non-blocking stack implementation

In reply to Gage Eads

This commit adds support for a non-blocking (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, STACK_F_NB.

The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.

The non-blocking push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.

The non-blocking pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.

This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.

The linked list elements themselves are maintained in a non-blocking LIFO, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.

---
 doc/guides/prog_guide/stack_lib.rst    |  46 ++++++++-
 doc/guides/rel_notes/release_19_05.rst |   3 +
 lib/librte_stack/Makefile              |   4 +-
 lib/librte_stack/meson.build           |   4 +-
 lib/librte_stack/rte_stack.c           |  42 ++++++--
 lib/librte_stack/rte_stack.h           | 139 +++++++++++++++++++++++++-
 lib/librte_stack/rte_stack_c11_mem.h   | 173 +++++++++++++++++++++++++++++++++
 lib/librte_stack/rte_stack_generic.h   | 157 ++++++++++++++++++++++++++++++
 8 files changed, 550 insertions(+), 18 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_c11_mem.h
 create mode 100644 lib/librte_stack/rte_stack_generic.h
doc/guides/prog_guide/stack_lib.rst +43 -3
index 51689cfe1..86fdc0a9b 100644
@@ -9,7 +9,7 @@ pointers.
 
 The stack library provides the following basic operations:
 
-*  Create a uniquely named stack of a user-specified size and using a user-specified socket.
+*  Create a uniquely named stack of a user-specified size and using a user-specified socket, with either lock-based or non-blocking behavior.
 
 *  Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe.
 
@@ -22,5 +22,45 @@ The stack library provides the following basic operations:
 Implementation
 ~~~~~~~~~~~~~~
 
-The stack consists of a contiguous array of pointers, a current index, and a
-spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
+The library supports two types of stacks: lock-based and non-blocking.
+Both types use the same set of interfaces, but their implementations differ.
+
+Lock-based Stack
+----------------
+
+The lock-based stack consists of a contiguous array of pointers, a current index, and a spinlock.
+Accesses to the stack are made multi-thread safe by the spinlock.
+
+The lock-based stack consists of a contiguous array of pointers, a current
+index, and a spinlock. Accesses to the stack are made multi-thread safe by the
+spinlock.
+
+Non-blocking Stack
+------------------
+
+The non-blocking stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.
+The non-blocking property means that multiple threads can push and pop simultaneously, and one thread being preempted/delayed in a push or pop operation will not impede the forward progress of any other thread.
+
+The non-blocking push operation enqueues a linked list of pointers by pointing the list's tail to the current stack head, and using a CAS to swing the stack head pointer to the head of the list.
+The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.
+
+The non-blocking pop operation first reserves one or more list elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking.
+It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well).
+While walking the list, the data pointers are recorded in an object table.
+
+The linked list elements themselves are maintained in a non-blocking LIFO, and are allocated before stack pushes and freed after stack pops.
+Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.
+
+The non-blocking behavior is selected by passing the *STACK_F_NB* flag to rte_stack_create().
+
+Preventing the ABA Problem
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To prevent the ABA problem, this algorithm stack uses a 128-bit compare-and-swap instruction to atomically update both the stack top pointer and a modification counter. The ABA problem can occur without a modification counter if, for example:
+
+1. Thread A reads head pointer X and stores the pointed-to list element.
+2. Other threads modify the list such that the head pointer is once again X, but its pointed-to data is different than what thread A read.
+3. Thread A changes the head pointer with a compare-and-swap and succeeds.
+
+In this case thread A would not detect that the list had changed, and would both pop stale data and incorrect change the head pointer.
+By adding a modification counter that is updated on every push and pop as part of the compare-and-swap, the algorithm can detect when the list changes even if the head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
index 04394f8cf..52c5ba78e 100644
@@ -71,6 +71,9 @@ New Features
   pointers. The API provides MT-safe push and pop operations that can operate
   on one or more pointers per operation.
 
+  The library supports two stack implementations: lock-based and non-blocking.
+  The non-blocking implementation is currently limited to x86-64 platforms.
+
 Removed Items
 -------------
 
lib/librte_stack/Makefile +3 -1
index e956b6535..94a7c1476 100644
@@ -18,6 +18,8 @@ LIBABIVER := 1
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c
 
 # install includes
-SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h
+SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
+					      rte_stack_generic.h \
+					      rte_stack_c11_mem.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +3 -1
index 99f43710e..dec527966 100644
@@ -5,4 +5,6 @@ allow_experimental_apis = true
 
 version = 1
 sources = files('rte_stack.c')
-headers = files('rte_stack.h')
+headers = files('rte_stack.h',
+		'rte_stack_c11_mem.h',
+		'rte_stack_generic.h')
lib/librte_stack/rte_stack.c +34 -8

77 lines not on this page · raw message

lib/librte_stack/rte_stack.h +134 -5

178 lines not on this page · raw message

lib/librte_stack/rte_stack_c11_mem.h new file +173 -0

173 lines not on this page · raw message

lib/librte_stack/rte_stack_generic.h new file +157 -0

157 lines not on this page · raw message

Signature · 1 line
2.13.6
#12 lore raw

Re: [PATCH 5/7] stack: add non-blocking stack implementation

Quoted 29 lines · Gage Eads

This commit adds support for a non-blocking (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, STACK_F_NB.

The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.

The non-blocking push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.

The non-blocking pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.

This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.

The linked list elements themselves are maintained in a non-blocking LIFO, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.

[...]

doc/guides/prog_guide/stack_lib.rst pointers. · line 15

Quoted 13 lines
doc/guides/prog_guide/stack_lib.rst +1 -1
index 51689cfe1..86fdc0a9b 100644
@@ -9,7 +9,7 @@ pointers.
 
 The stack library provides the following basic operations:
 
-*  Create a uniquely named stack of a user-specified size and using a user-specified socket.
+*  Create a uniquely named stack of a user-specified size and using a user-specified socket, with either lock-based or non-blocking behavior.
 
 *  Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe.
 

Same comment about 80-cols than in the first patch.

[...]

Quoted 5 lines
lib/librte_stack/rte_stack.c +0 -0
@@ -26,27 +26,46 @@ static struct rte_tailq_elem rte_stack_tailq = {
 EAL_REGISTER_TAILQ(rte_stack_tailq)
 

lib/librte_stack/rte_stack.c static struct rte_tailq_elem rte_stack_tailq = { · line 37

Quoted 10 lines
lib/librte_stack/rte_stack.c +9 -0
 static void
+nb_lifo_init(struct rte_stack *s, unsigned int count)
+{
+	struct rte_nb_lifo_elem *elems;
+	unsigned int i;
+
+	elems = (struct rte_nb_lifo_elem *)&s[1];
+	for (i = 0; i < count; i++)
+		__nb_lifo_push(&s->nb_lifo.free, &elems[i], &elems[i], 1);
+}

Would it be possible to add:

	 struct rte_nb_lifo {
	 	/** LIFO list of elements */
	 	struct rte_nb_lifo_list used __rte_cache_aligned;
	 	/** LIFO list of free elements */
	 	struct rte_nb_lifo_list free __rte_cache_aligned;
	+	struct rte_nb_lifo_elem elems[];
	 };

I think it is more consistent with the non-blocking structure.

[...]

Quoted 23 lines
lib/librte_stack/rte_stack.h +17 -0
@@ -29,6 +29,33 @@ extern "C" {
 #define RTE_STACK_NAMESIZE (RTE_MEMZONE_NAMESIZE - \
 			   sizeof(RTE_STACK_MZ_PREFIX) + 1)
 
+struct rte_nb_lifo_elem {
+	void *data;			/**< Data pointer */
+	struct rte_nb_lifo_elem *next;	/**< Next pointer */
+};
+
+struct rte_nb_lifo_head {
+	struct rte_nb_lifo_elem *top; /**< Stack top */
+	uint64_t cnt; /**< Modification counter for avoiding ABA problem */
+};
+
+struct rte_nb_lifo_list {
+	/** List head */
+	struct rte_nb_lifo_head head __rte_aligned(16);
+	/** List len */
+	rte_atomic64_t len;
+};
+

lib/librte_stack/rte_stack.h extern "C" { · line 58

Quoted 10 lines
lib/librte_stack/rte_stack.h +10 -0
+/* Structure containing two non-blocking LIFO lists: the stack itself and a
+ * list of free linked-list elements.
+ */
+struct rte_nb_lifo {
+	/** LIFO list of elements */
+	struct rte_nb_lifo_list used __rte_cache_aligned;
+	/** LIFO list of free elements */
+	struct rte_nb_lifo_list free __rte_cache_aligned;
+};
+

The names "rte_nb_lifo*" bothers me a bit. I think a more usual name format is "rte_<module_name>_<struct_name>".

What would you think about names like this?
rte_nb_lifo -> rte_stack_nb
rte_nb_lifo_elem -> rte_stack_nb_elem
rte_nb_lifo_head -> rte_stack_nb_head
rte_nb_lifo_list -> rte_stack_nb_list
rte_lifo -> rte_stack_std

I even wonder if "nonblock", "noblk", or "lockless" shouldn't be used in place of "nb" (which is also a common abbreviation for number). This can also applies to the STACK_F_NB flag name.

[...]

Quoted 10 lines
lib/librte_stack/rte_stack.h +2 -1
 /* Structure containing the LIFO, its current length, and a lock for mutual
  * exclusion.
  */
@@ -48,10 +75,69 @@ struct rte_stack {
 	const struct rte_memzone *memzone;
 	uint32_t capacity; /**< Usable size of the stack */
 	uint32_t flags; /**< Flags supplied at creation */
-	struct rte_lifo lifo; /**< LIFO structure */
+	RTE_STD_C11
+	union {

lib/librte_stack/rte_stack.h struct rte_stack { · line 89

Quoted 10 lines
lib/librte_stack/rte_stack.h +7 -0
+		struct rte_nb_lifo nb_lifo; /**< Non-blocking LIFO structure */
+		struct rte_lifo lifo;	    /**< LIFO structure */
+	};
 } __rte_cache_aligned;
 
 /**
+ * The stack uses non-blocking push and pop functions. This flag is only
+ * supported on x86_64 platforms, currently.
+ */
+#define STACK_F_NB 0x0001

What about adding the RTE_ prefix?

Quoted 17 lines
lib/librte_stack/rte_stack.h +17 -0
+static __rte_always_inline unsigned int __rte_experimental
+rte_nb_lifo_push(struct rte_stack *s, void * const *obj_table, unsigned int n)
+{
+	struct rte_nb_lifo_elem *tmp, *first, *last = NULL;
+	unsigned int i;
+
+	if (unlikely(n == 0))
+		return 0;
+
+	/* Pop n free elements */
+	first = __nb_lifo_pop(&s->nb_lifo.free, n, NULL, NULL);
+	if (unlikely(first == NULL))
+		return 0;
+
+	/* Construct the list elements */
+	tmp = first;
+	for (i = 0; i < n; i++) {

lib/librte_stack/rte_stack.h struct rte_stack { · line 138

Quoted 10 lines
lib/librte_stack/rte_stack.h +10 -0
+		tmp->data = obj_table[n - i - 1];
+		last = tmp;
+		tmp = tmp->next;
+	}
+
+	/* Push them to the used list */
+	__nb_lifo_push(&s->nb_lifo.used, first, last, n);
+
+	return n;
+}

Here, I didn't get why "last" is not retrieved through __nb_lifo_pop(), like it's done in rte_nb_lifo_pop(). Is there a reason for that?

[...]

lib/librte_stack/rte_stack_c11_mem.h

Quoted 2 lines
lib/librte_stack/rte_stack_c11_mem.h new file +0 -0

For the c11 memory model, please consider having an additional reviewer ;)

[...]

Quoted 99 lines
lib/librte_stack/rte_stack_generic.h new file +96 -0
@@ -0,0 +1,157 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _NB_LIFO_GENERIC_H_
+#define _NB_LIFO_GENERIC_H_
+
+#include <rte_branch_prediction.h>
+#include <rte_prefetch.h>
+
+static __rte_always_inline unsigned int
+rte_nb_lifo_len(struct rte_stack *s)
+{
+	/* nb_lifo_push() and nb_lifo_pop() do not update the list's contents
+	 * and nb_lifo->len atomically, which can cause the list to appear
+	 * shorter than it actually is if this function is called while other
+	 * threads are modifying the list.
+	 *
+	 * However, given the inherently approximate nature of the get_count
+	 * callback -- even if the list and its size were updated atomically,
+	 * the size could change between when get_count executes and when the
+	 * value is returned to the caller -- this is acceptable.
+	 *
+	 * The nb_lifo->len updates are placed such that the list may appear to
+	 * have fewer elements than it does, but will never appear to have more
+	 * elements. If the mempool is near-empty to the point that this is a
+	 * concern, the user should consider increasing the mempool size.
+	 */
+	return (unsigned int)rte_atomic64_read(&s->nb_lifo.used.len);
+}
+
+static __rte_always_inline void
+__nb_lifo_push(struct rte_nb_lifo_list *lifo,
+	       struct rte_nb_lifo_elem *first,
+	       struct rte_nb_lifo_elem *last,
+	       unsigned int num)
+{
+#ifndef RTE_ARCH_X86_64
+	RTE_SET_USED(first);
+	RTE_SET_USED(last);
+	RTE_SET_USED(lifo);
+	RTE_SET_USED(num);
+#else
+	struct rte_nb_lifo_head old_head;
+	int success;
+
+	old_head = lifo->head;
+
+	do {
+		struct rte_nb_lifo_head new_head;
+
+		/* Swing the top pointer to the first element in the list and
+		 * make the last element point to the old top.
+		 */
+		new_head.top = first;
+		new_head.cnt = old_head.cnt + 1;
+
+		last->next = old_head.top;
+
+		/* Ensure the list entry writes are visible before pushing them
+		 * to the stack.
+		 */
+		rte_wmb();
+
+		/* old_head is updated on failure */
+		success = rte_atomic128_cmpxchg((rte_int128_t *)&lifo->head,
+					       (rte_int128_t *)&old_head,
+					       (rte_int128_t *)&new_head,
+					       1, __ATOMIC_RELEASE,
+					       __ATOMIC_RELAXED);
+	} while (success == 0);
+
+	rte_atomic64_add(&lifo->len, num);
+#endif
+}
+
+static __rte_always_inline struct rte_nb_lifo_elem *
+__nb_lifo_pop(struct rte_nb_lifo_list *lifo,
+	      unsigned int num,
+	      void **obj_table,
+	      struct rte_nb_lifo_elem **last)
+{
+#ifndef RTE_ARCH_X86_64
+	RTE_SET_USED(obj_table);
+	RTE_SET_USED(last);
+	RTE_SET_USED(lifo);
+	RTE_SET_USED(num);
+
+	return NULL;
+#else
+	struct rte_nb_lifo_head old_head;
+	int success;
+
+	/* Reserve num elements, if available */
+	while (1) {
+		uint64_t len = rte_atomic64_read(&lifo->len);

lib/librte_stack/rte_stack_generic.h line 106

Quoted 10 lines
lib/librte_stack/rte_stack_generic.h new file +10 -0
+
+		/* Does the list contain enough elements? */
+		if (unlikely(len < num))
+			return NULL;
+
+		if (rte_atomic64_cmpset((volatile uint64_t *)&lifo->len,
+					len, len - num))
+			break;
+	}
+

Here, accessing the length with a compare and set costs probably more than a standard atomic sub fonction. I understand that was done for the reason described above:

The nb_lifo->len updates are placed such that the list may appear to have fewer elements than it does, but will never appear to have more elements.

Another strategy could be to use a rte_atomic64_sub() after the effective pop and change rte_nb_lifo_len() to bound the result to [0:size].

A message that is not in the archive (sent off-list or not yet received): 2EC44CCD3517A842B44C82651A5557A14AF13386@fmsmsx118.amr.corp.intel.com

author
#14 lore raw

PATCH 5/7 stack: add non-blocking stack implementation

Quoted 17 lines

-----Original Message-----
From: Olivier Matz [mailto:[email protected]]
Sent: Monday, February 25, 2019 5:28 AM
To: Eads, Gage <[email protected]>
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 5/7] stack: add non-blocking stack implementation

Quoted 4 lines · Gage Eads

This commit adds support for a non-blocking (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, STACK_F_NB.

Quoted 8 lines
Quoted 4 lines

The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.

The non-blocking push operation enqueues a linked list of pointers

by pointing the tail of the list to the current stack head, and

Quoted 3 lines

using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length

and returns.

Quoted 71 lines
Quoted 7 lines

The non-blocking pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in

an object table.

Quoted 11 lines

This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.

The linked list elements themselves are maintained in a non-blocking LIFO, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.

[...]

Quoted 11 lines
doc/guides/prog_guide/stack_lib.rst +0 -1
index 51689cfe1..86fdc0a9b 100644
@@ -9,7 +9,7 @@ pointers.
 
 The stack library provides the following basic operations:
 
-*  Create a uniquely named stack of a user-specified size and using

a user-

specified socket.

doc/guides/prog_guide/stack_lib.rst

Quoted 2 lines
doc/guides/prog_guide/stack_lib.rst +2 -0
+*  Create a uniquely named stack of a user-specified size and using
+a user-

specified socket, with either lock-based or non-blocking behavior.

Quoted 3 lines

* Push and pop a burst of one or more stack objects (pointers).
These

function are multi-threading safe.

Quoted 1 line

Same comment about 80-cols than in the first patch.

[...]

lib/librte_stack/rte_stack.c static struct rte_tailq_elem rte_stack_tailq = { · line 35

Quoted 13 lines
lib/librte_stack/rte_stack.c +7 -0
@@ -26,27 +26,46 @@ static struct rte_tailq_elem rte_stack_tailq = {
 EAL_REGISTER_TAILQ(rte_stack_tailq)
 
 static void
+nb_lifo_init(struct rte_stack *s, unsigned int count) {
+	struct rte_nb_lifo_elem *elems;
+	unsigned int i;
+
+	elems = (struct rte_nb_lifo_elem *)&s[1];
+	for (i = 0; i < count; i++)
+		__nb_lifo_push(&s->nb_lifo.free, &elems[i], &elems[i], 1); }

Would it be possible to add:

	 struct rte_nb_lifo {
	 	/** LIFO list of elements */
	 	struct rte_nb_lifo_list used __rte_cache_aligned;
	 	/** LIFO list of free elements */
	 	struct rte_nb_lifo_list free __rte_cache_aligned;
	+	struct rte_nb_lifo_elem elems[];
	 };
Quoted 2 lines

I think it is more consistent with the non-blocking structure.

Will do.

Quoted 46 lines

[...]

Quoted 23 lines
lib/librte_stack/rte_stack.h +17 -0
@@ -29,6 +29,33 @@ extern "C" {
 #define RTE_STACK_NAMESIZE (RTE_MEMZONE_NAMESIZE - \
 			   sizeof(RTE_STACK_MZ_PREFIX) + 1)
 
+struct rte_nb_lifo_elem {
+	void *data;			/**< Data pointer */
+	struct rte_nb_lifo_elem *next;	/**< Next pointer */
+};
+
+struct rte_nb_lifo_head {
+	struct rte_nb_lifo_elem *top; /**< Stack top */
+	uint64_t cnt; /**< Modification counter for avoiding ABA problem
+*/ };
+
+struct rte_nb_lifo_list {
+	/** List head */
+	struct rte_nb_lifo_head head __rte_aligned(16);
+	/** List len */
+	rte_atomic64_t len;
+};
+

lib/librte_stack/rte_stack.h extern "C" { · line 58

Quoted 10 lines
lib/librte_stack/rte_stack.h +10 -0
+/* Structure containing two non-blocking LIFO lists: the stack
+itself and a
+ * list of free linked-list elements.
+ */
+struct rte_nb_lifo {
+	/** LIFO list of elements */
+	struct rte_nb_lifo_list used __rte_cache_aligned;
+	/** LIFO list of free elements */
+	struct rte_nb_lifo_list free __rte_cache_aligned; };
+

The names "rte_nb_lifo*" bothers me a bit. I think a more usual name format is "rte_<module_name>_<struct_name>".

What would you think about names like this?
rte_nb_lifo -> rte_stack_nb
rte_nb_lifo_elem -> rte_stack_nb_elem
rte_nb_lifo_head -> rte_stack_nb_head
rte_nb_lifo_list -> rte_stack_nb_list
rte_lifo -> rte_stack_std

Quoted 4 lines

I even wonder if "nonblock", "noblk", or "lockless" shouldn't be used in place of "nb" (which is also a common abbreviation for number). This can also applies to the STACK_F_NB flag name.

How about std and lf (lock-free)?

Quoted 17 lines

[...]

Quoted 15 lines

/* Structure containing the LIFO, its current length, and a lock for mutual
* exclusion.
*/

lib/librte_stack/rte_stack.h +5 -1
@@ -48,10 +75,69 @@ struct rte_stack {
 	const struct rte_memzone *memzone;
 	uint32_t capacity; /**< Usable size of the stack */
 	uint32_t flags; /**< Flags supplied at creation */
-	struct rte_lifo lifo; /**< LIFO structure */
+	RTE_STD_C11
+	union {
+		struct rte_nb_lifo nb_lifo; /**< Non-blocking LIFO structure */
+		struct rte_lifo lifo;	    /**< LIFO structure */
+	};
 } __rte_cache_aligned;
 
Quoted 8 lines

lib/librte_stack/rte_stack.h struct rte_stack { · line 89

Quoted 6 lines
lib/librte_stack/rte_stack.h +5 -0
 /**
+ * The stack uses non-blocking push and pop functions. This flag is
+only
+ * supported on x86_64 platforms, currently.
+ */
+#define STACK_F_NB 0x0001

What about adding the RTE_ prefix?

I'm fine with either, but there's precedent for flag macros named <module_name>_*. E.g. RING_F_*, MEMPOOL_F_*, ETH_*, and SERVICE_F_*.

Quoted 23 lines
Quoted 23 lines
lib/librte_stack/rte_stack.h +22 -0
+static __rte_always_inline unsigned int __rte_experimental
+rte_nb_lifo_push(struct rte_stack *s, void * const *obj_table,
+unsigned int n) {
+	struct rte_nb_lifo_elem *tmp, *first, *last = NULL;
+	unsigned int i;
+
+	if (unlikely(n == 0))
+		return 0;
+
+	/* Pop n free elements */
+	first = __nb_lifo_pop(&s->nb_lifo.free, n, NULL, NULL);
+	if (unlikely(first == NULL))
+		return 0;
+
+	/* Construct the list elements */
+	tmp = first;
+	for (i = 0; i < n; i++) {
+		tmp->data = obj_table[n - i - 1];
+		last = tmp;
+		tmp = tmp->next;
+	}
+
Quoted 9 lines

lib/librte_stack/rte_stack.h struct rte_stack { · line 138

Quoted 5 lines
lib/librte_stack/rte_stack.h +5 -0
+	/* Push them to the used list */
+	__nb_lifo_push(&s->nb_lifo.used, first, last, n);
+
+	return n;
+}

Here, I didn't get why "last" is not retrieved through __nb_lifo_pop(), like it's done in rte_nb_lifo_pop(). Is there a reason for that?

Just an simple oversight -- that works, and I'll change it for v2.

Quoted 7 lines

[...]

lib/librte_stack/rte_stack_c11_mem.h

Quoted 2 lines
lib/librte_stack/rte_stack_c11_mem.h new file +0 -0

For the c11 memory model, please consider having an additional reviewer ;)

No problem, and I'll break out the C11 implementation into a separate patch in case that makes reviewing it easier.

Quoted 112 lines

[...]

Quoted 12 lines
lib/librte_stack/rte_stack_generic.h new file +9 -0
@@ -0,0 +1,157 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation  */
+
+#ifndef _NB_LIFO_GENERIC_H_
+#define _NB_LIFO_GENERIC_H_
+
+#include <rte_branch_prediction.h>
+#include <rte_prefetch.h>
+

lib/librte_stack/rte_stack_generic.h line 19

Quoted 10 lines
lib/librte_stack/rte_stack_generic.h new file +10 -0
+static __rte_always_inline unsigned int rte_nb_lifo_len(struct
+rte_stack *s) {
+	/* nb_lifo_push() and nb_lifo_pop() do not update the list's contents
+	 * and nb_lifo->len atomically, which can cause the list to appear
+	 * shorter than it actually is if this function is called while other
+	 * threads are modifying the list.
+	 *
+	 * However, given the inherently approximate nature of the get_count
+	 * callback -- even if the list and its size were updated atomically,
+	 * the size could change between when get_count executes and when

the

Quoted 71 lines
lib/librte_stack/rte_stack_generic.h new file +71 -0
+	 * value is returned to the caller -- this is acceptable.
+	 *
+	 * The nb_lifo->len updates are placed such that the list may appear to
+	 * have fewer elements than it does, but will never appear to have more
+	 * elements. If the mempool is near-empty to the point that this is a
+	 * concern, the user should consider increasing the mempool size.
+	 */
+	return (unsigned int)rte_atomic64_read(&s->nb_lifo.used.len);
+}
+
+static __rte_always_inline void
+__nb_lifo_push(struct rte_nb_lifo_list *lifo,
+	       struct rte_nb_lifo_elem *first,
+	       struct rte_nb_lifo_elem *last,
+	       unsigned int num)
+{
+#ifndef RTE_ARCH_X86_64
+	RTE_SET_USED(first);
+	RTE_SET_USED(last);
+	RTE_SET_USED(lifo);
+	RTE_SET_USED(num);
+#else
+	struct rte_nb_lifo_head old_head;
+	int success;
+
+	old_head = lifo->head;
+
+	do {
+		struct rte_nb_lifo_head new_head;
+
+		/* Swing the top pointer to the first element in the list and
+		 * make the last element point to the old top.
+		 */
+		new_head.top = first;
+		new_head.cnt = old_head.cnt + 1;
+
+		last->next = old_head.top;
+
+		/* Ensure the list entry writes are visible before pushing them
+		 * to the stack.
+		 */
+		rte_wmb();
+
+		/* old_head is updated on failure */
+		success = rte_atomic128_cmpxchg((rte_int128_t *)&lifo->head,
+					       (rte_int128_t *)&old_head,
+					       (rte_int128_t *)&new_head,
+					       1, __ATOMIC_RELEASE,
+					       __ATOMIC_RELAXED);
+	} while (success == 0);
+
+	rte_atomic64_add(&lifo->len, num); #endif }
+
+static __rte_always_inline struct rte_nb_lifo_elem *
+__nb_lifo_pop(struct rte_nb_lifo_list *lifo,
+	      unsigned int num,
+	      void **obj_table,
+	      struct rte_nb_lifo_elem **last) { #ifndef RTE_ARCH_X86_64
+	RTE_SET_USED(obj_table);
+	RTE_SET_USED(last);
+	RTE_SET_USED(lifo);
+	RTE_SET_USED(num);
+
+	return NULL;
+#else
+	struct rte_nb_lifo_head old_head;
+	int success;
+
+	/* Reserve num elements, if available */
+	while (1) {
+		uint64_t len = rte_atomic64_read(&lifo->len);

lib/librte_stack/rte_stack_generic.h line 106

Quoted 10 lines
lib/librte_stack/rte_stack_generic.h new file +10 -0
+
+		/* Does the list contain enough elements? */
+		if (unlikely(len < num))
+			return NULL;
+
+		if (rte_atomic64_cmpset((volatile uint64_t *)&lifo->len,
+					len, len - num))
+			break;
+	}
+

Here, accessing the length with a compare and set costs probably more than a standard atomic sub fonction. I understand that was done for the reason described above:

Quoted 6 lines

The nb_lifo->len updates are placed such that the list may appear to have fewer elements than it does, but will never appear to have more elements.

Another strategy could be to use a rte_atomic64_sub() after the effective pop and change rte_nb_lifo_len() to bound the result to [0:size].

It serves a second purpose: if the CAS succeeds, the subsequent do-while loop is guaranteed to (eventually) succeed because we've effectively reserved num elements. Otherwise there's the chance that the list runs empty after popping fewer than num elements.

If folks are interested in this patchset, please also consider reviewing the 128-bit CAS patch here: http://mails.dpdk.org/archives/dev/2019-February/125059.html

Thanks!
Gage

#15 lore raw

Re: FW: [PATCH 5/7] stack: add non-blocking stack implementation

01/03/2019 21:53, Eads, Gage:

Quoted 8 lines

From: Olivier Matz [mailto:[email protected]]

Quoted 4 lines · Gage Eads
Quoted 1 line

+#define STACK_F_NB 0x0001

What about adding the RTE_ prefix?

I'm fine with either, but there's precedent for flag macros named <module_name>_*. E.g. RING_F_*, MEMPOOL_F_*, ETH_*, and SERVICE_F_*.

They should be fixed.
Every public symbols should be prefixed to avoid namespace conflict. At first, we should rename them and keep the old name as an alias. Later, non-prefixed names should be removed after a deprecation notice.

author
#16 lore raw

Re: FW: [PATCH 5/7] stack: add non-blocking stack implementation

Quoted 19 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Friday, March 1, 2019 3:13 PM
To: Eads, Gage <[email protected]>
Cc: Olivier Matz <[email protected]>; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: FW: [PATCH 5/7] stack: add non-blocking stack implementation

01/03/2019 21:53, Eads, Gage:

Quoted 6 lines

From: Olivier Matz [mailto:[email protected]]

Quoted 4 lines · Gage Eads
Quoted 1 line

+#define STACK_F_NB 0x0001

What about adding the RTE_ prefix?

Quoted 9 lines
Quoted 2 lines

I'm fine with either, but there's precedent for flag macros named <module_name>_*. E.g. RING_F_*, MEMPOOL_F_*, ETH_*, and

SERVICE_F_*.

They should be fixed.
Every public symbols should be prefixed to avoid namespace conflict. At first, we should rename them and keep the old name as an alias. Later, non-prefixed names should be removed after a deprecation notice.

Ok, will fix.

Thanks,
Gage

author
#17 lore raw

PATCH 6/7 test/stack: add non-blocking stack tests

In reply to Gage Eads

This commit adds non-blocking stack variants of stack_autotest (stack_nb_autotest) and stack_perf_autotest (stack_nb_perf_autotest), which differ only in that the non-blocking versions pass the STACK_F_NB flag to all rte_stack_create() calls.

---
 test/test/meson.build       |  2 ++
 test/test/test_stack.c      | 41 +++++++++++++++++++++++++++--------------
 test/test/test_stack_perf.c | 17 +++++++++++++++--
 3 files changed, 44 insertions(+), 16 deletions(-)
test/test/meson.build +2 -0
index ba3cb6261..474611291 100644
@@ -177,6 +177,7 @@ fast_parallel_test_names = [
         'sched_autotest',
         'spinlock_autotest',
         'stack_autotest',
+        'stack_nb_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
@@ -242,6 +243,7 @@ perf_test_names = [
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
         'stack_perf_autotest',
+        'stack_nb_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
test/test/test_stack.c +27 -14

112 lines not on this page · raw message

test/test/test_stack_perf.c +15 -2
index 484370d30..57a0e806b 100644
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s)
 }
 
 static int
-test_stack_perf(void)
+__test_stack_perf(uint32_t flags)
 {
 	struct lcore_pair cores;
 	struct rte_stack *s;
 
 	rte_atomic32_init(&lcore_barrier);
 
-	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), 0);
+	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), flags);
 	if (s == NULL) {
 		printf("[%s():%u] failed to create a stack\n",
 		       __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)
 	return 0;
 }
 
+static int
+test_stack_perf(void)
+{
+	return __test_stack_perf(0);
+}
+
+static int
+test_nb_stack_perf(void)
+{
+	return __test_stack_perf(STACK_F_NB);
+}
+
 REGISTER_TEST_COMMAND(stack_perf_autotest, test_stack_perf);
+REGISTER_TEST_COMMAND(stack_nb_perf_autotest, test_nb_stack_perf);
Signature · 1 line
2.13.6
Reviewed-by on v1 6/7
#18 lore raw

Re: [PATCH 6/7] test/stack: add non-blocking stack tests

Quoted 6 lines · Gage Eads

This commit adds non-blocking stack variants of stack_autotest (stack_nb_autotest) and stack_perf_autotest (stack_nb_perf_autotest), which differ only in that the non-blocking versions pass the STACK_F_NB flag to all rte_stack_create() calls.

author
#19 lore raw

PATCH 7/7 mempool/stack: add non-blocking stack mempool handler

In reply to Gage Eads

This commit adds support for non-blocking (linked list based) stack mempool handler.

In mempool_perf_autotest the lock-based stack outperforms the non-blocking handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a lock-based stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the non-blocking stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.

*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the non-blocking stack's.

---
 doc/guides/prog_guide/env_abstraction_layer.rst |  5 +++++
 doc/guides/rel_notes/release_19_05.rst          |  5 +++++
 drivers/mempool/stack/rte_mempool_stack.c       | 26 +++++++++++++++++++++++--
 3 files changed, 34 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +5 -0
index 929d76dba..5c2dbc706 100644
@@ -541,6 +541,11 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
+  Alternatively, applications can use the non-blocking stack mempool handler. When considering this handler, note that:
+
+  - it is currently limited to the x86_64 platform, because it uses an instruction (16-byte compare-and-swap) that is not yet available on other platforms.
+  - it has worse average-case performance than the non-preemptive rte_ring, but software caching (e.g. the mempool cache) can mitigate this by reducing the number of stack accesses.
+
 + rte_timer
 
   Running  ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
index 52c5ba78e..111a93ea6 100644
@@ -74,6 +74,11 @@ New Features
   The library supports two stack implementations: lock-based and non-blocking.
   The non-blocking implementation is currently limited to x86-64 platforms.
 
+* **Added Non-blocking Stack Mempool Handler.**
+
+  Added a new non-blocking stack handler, which uses the newly added stack
+  library.
+
 Removed Items
 -------------
 
drivers/mempool/stack/rte_mempool_stack.c +24 -2
index 25ccdb9af..eae71aa0c 100644
@@ -7,7 +7,7 @@
 #include <rte_stack.h>
 
 static int
-stack_alloc(struct rte_mempool *mp)
+__stack_alloc(struct rte_mempool *mp, uint32_t flags)
 {
 	char name[RTE_STACK_NAMESIZE];
 	struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)
 		return -rte_errno;
 	}
 
-	s = rte_stack_create(name, mp->size, mp->socket_id, 0);
+	s = rte_stack_create(name, mp->size, mp->socket_id, flags);
 	if (s == NULL)
 		return -rte_errno;
 
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp)
 }
 
 static int
+stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, 0);
+}
+
+static int
+nb_stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, STACK_F_NB);
+}
+
+static int
 stack_enqueue(struct rte_mempool *mp, void * const *obj_table,
 	      unsigned int n)
 {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = {
 	.get_count = stack_get_count
 };
 
+static struct rte_mempool_ops ops_nb_stack = {
+	.name = "nb_stack",
+	.alloc = nb_stack_alloc,
+	.free = stack_free,
+	.enqueue = stack_enqueue,
+	.dequeue = stack_dequeue,
+	.get_count = stack_get_count
+};
+
 MEMPOOL_REGISTER_OPS(ops_stack);
+MEMPOOL_REGISTER_OPS(ops_nb_stack);
Signature · 1 line
2.13.6
Reviewed-by on v1 7/7
#20 lore raw

Re: [PATCH 7/7] mempool/stack: add non-blocking stack mempool handler

Quoted 12 lines · Gage Eads

This commit adds support for non-blocking (linked list based) stack mempool handler.

In mempool_perf_autotest the lock-based stack outperforms the non-blocking handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a lock-based stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the non-blocking stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.

Quoted 5 lines

*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the non-blocking stack's.

author
#21 lore raw

v2 cover Add stack library and new mempool handler

In reply to Gage Eads

This patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.

The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.

The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.

This patchset is the successor to a patchset containing only the new mempool handler[2].

[1] http://mails.dpdk.org/archives/dev/2019-March/125751.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html

---
v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset

Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler

 MAINTAINERS                                     |   9 +-
 config/common_base                              |   5 +
 doc/api/doxy-api-index.md                       |   1 +
 doc/api/doxy-api.conf.in                        |   1 +
 doc/guides/prog_guide/env_abstraction_layer.rst |  10 +
 doc/guides/prog_guide/index.rst                 |   1 +
 doc/guides/prog_guide/stack_lib.rst             |  83 +++++
 doc/guides/rel_notes/release_19_05.rst          |  13 +
 drivers/mempool/stack/Makefile                  |   3 +-
 drivers/mempool/stack/meson.build               |   6 +-
 drivers/mempool/stack/rte_mempool_stack.c       | 115 +++----
 lib/Makefile                                    |   2 +
 lib/librte_stack/Makefile                       |  25 ++
 lib/librte_stack/meson.build                    |  10 +
 lib/librte_stack/rte_stack.c                    | 219 ++++++++++++
 lib/librte_stack/rte_stack.h                    | 395 ++++++++++++++++++++++
 lib/librte_stack/rte_stack_c11_mem.h            | 175 ++++++++++
 lib/librte_stack/rte_stack_generic.h            | 151 +++++++++
 lib/librte_stack/rte_stack_pvt.h                |  34 ++
 lib/librte_stack/rte_stack_version.map          |   9 +
 lib/meson.build                                 |   2 +-
 mk/rte.app.mk                                   |   1 +
 test/test/Makefile                              |   3 +
 test/test/meson.build                           |   7 +
 test/test/test_stack.c                          | 423 ++++++++++++++++++++++++
 test/test/test_stack_perf.c                     | 356 ++++++++++++++++++++
 26 files changed, 1987 insertions(+), 72 deletions(-)
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_c11_mem.h
 create mode 100644 lib/librte_stack/rte_stack_generic.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
 create mode 100644 test/test/test_stack.c
 create mode 100644 test/test/test_stack_perf.c
Signature · 1 line
2.13.6
author
#22 lore raw

PATCH v2 1/8 stack: introduce rte stack library

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

---
 MAINTAINERS                            |   6 +
 config/common_base                     |   5 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/index.rst        |   1 +
 doc/guides/prog_guide/stack_lib.rst    |  28 ++++
 doc/guides/rel_notes/release_19_05.rst |   5 +
 lib/Makefile                           |   2 +
 lib/librte_stack/Makefile              |  23 +++
 lib/librte_stack/meson.build           |   8 +
 lib/librte_stack/rte_stack.c           | 194 +++++++++++++++++++++++
 lib/librte_stack/rte_stack.h           | 274 +++++++++++++++++++++++++++++++++
 lib/librte_stack/rte_stack_pvt.h       |  34 ++++
 lib/librte_stack/rte_stack_version.map |   9 ++
 lib/meson.build                        |   2 +-
 mk/rte.app.mk                          |   1 +
 16 files changed, 593 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
index eef480ab5..237f05eb2 100644
@@ -407,6 +407,12 @@ F: drivers/raw/skeleton_rawdev/
 F: test/test/test_rawdev.c
 F: doc/guides/prog_guide/rawdev.rst
 
+Stack API - EXPERIMENTAL
+M: Gage Eads <[email protected]>
+M: Olivier Matz <[email protected]>
+F: lib/librte_stack/
+F: doc/guides/prog_guide/stack_lib.rst
+
 
 Memory Pool Drivers
 -------------------
config/common_base +5 -0
index 7c6da5165..5861eb09c 100644
@@ -980,3 +980,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
 # Compile the eventdev application
 #
 CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Compile librte_stack
+#
+CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
index d95ad566c..0df8848c0 100644
@@ -124,6 +124,7 @@ The public API headers are grouped by topics:
   [mbuf]               (@ref rte_mbuf.h),
   [mbuf pool ops]      (@ref rte_mbuf_pool_ops.h),
   [ring]               (@ref rte_ring.h),
+  [stack]              (@ref rte_stack.h),
   [tailq]              (@ref rte_tailq.h),
   [bitmap]             (@ref rte_bitmap.h)
 
doc/api/doxy-api.conf.in +1 -0
index bef9320c0..dd972a3fe 100644
@@ -56,6 +56,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/librte_ring \
                           @TOPDIR@/lib/librte_sched \
                           @TOPDIR@/lib/librte_security \
+                          @TOPDIR@/lib/librte_stack \
                           @TOPDIR@/lib/librte_table \
                           @TOPDIR@/lib/librte_telemetry \
                           @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
index 6726b1e8d..f4f60862f 100644
@@ -55,6 +55,7 @@ Programmer's Guide
     metrics_lib
     bpf_lib
     ipsec_lib
+    stack_lib
     source_org
     dev_kit_build_system
     dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +28 -0
new file mode 100644
index 000000000..25a8cc38a
@@ -0,0 +1,28 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Stack Library
+=============
+
+DPDK's stack library provides an API for configuration and use of a bounded
+stack of pointers.
+
+The stack library provides the following basic operations:
+
+*  Create a uniquely named stack of a user-specified size and using a
+   user-specified socket.
+
+*  Push and pop a burst of one or more stack objects (pointers). These function
+   are multi-threading safe.
+
+*  Free a previously created stack.
+
+*  Lookup a pointer to a stack by its name.
+
+*  Query a stack's current depth and number of free entries.
+
+Implementation
+~~~~~~~~~~~~~~
+
+The stack consists of a contiguous array of pointers, a current index, and a
+spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
index 2b0f60d3d..04394f8cf 100644
@@ -65,6 +65,11 @@ New Features
     process.
   * Added support for Rx packet types list in a secondary process.
 
+* **Added Stack API.**
+
+  Added a new stack API for configuration and use of a bounded stack of
+  pointers. The API provides MT-safe push and pop operations that can operate
+  on one or more pointers per operation.
 
 Removed Items
 -------------
lib/Makefile +2 -0
index d6239d27c..d22e2072b 100644
@@ -111,6 +111,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec
 DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security
 DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry
 DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev
+DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack
+DEPDIRS-librte_stack := librte_eal
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +23 -0
new file mode 100644
index 000000000..e956b6535
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_stack.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+LDLIBS += -lrte_eal
+
+EXPORT_MAP := rte_stack_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c
+
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
new file mode 100644
index 000000000..99f43710e
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+allow_experimental_apis = true
+
+version = 1
+sources = files('rte_stack.c')
+headers = files('rte_stack.h')
lib/librte_stack/rte_stack.c new file +194 -0

194 lines not on this page · raw message

lib/librte_stack/rte_stack.h new file +274 -0

274 lines not on this page · raw message

lib/librte_stack/rte_stack_pvt.h new file +34 -0
new file mode 100644
index 000000000..4a6a7bdb3
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_STACK_PVT_H_
+#define _RTE_STACK_PVT_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_log.h>
+
+extern int stack_logtype;
+
+#define STACK_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \
+		__func__, ##args)
+
+#define STACK_LOG_ERR(fmt, args...) \
+	STACK_LOG(ERR, fmt, ## args)
+
+#define STACK_LOG_WARN(fmt, args...) \
+	STACK_LOG(WARNING, fmt, ## args)
+
+#define STACK_LOG_INFO(fmt, args...) \
+	STACK_LOG(INFO, fmt, ## args)
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_version.map new file +9 -0
new file mode 100644
index 000000000..6662679c3
@@ -0,0 +1,9 @@
+EXPERIMENTAL {
+	global:
+
+	rte_stack_create;
+	rte_stack_free;
+	rte_stack_lookup;
+
+	local: *;
+};
lib/meson.build +1 -1
index e8b40f546..0f0e589bc 100644
@@ -21,7 +21,7 @@ libraries = [ 'compat', # just a header, used for versioning
 	'gro', 'gso', 'ip_frag', 'jobstats',
 	'kni', 'latencystats', 'lpm', 'member',
 	'power', 'pdump', 'rawdev',
-	'reorder', 'sched', 'security', 'vhost',
+	'reorder', 'sched', 'security', 'stack', 'vhost',
 	#ipsec lib depends on crypto and security
 	'ipsec',
 	# add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
index 8a4f0f4e5..55568c603 100644
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -lrte_security
 _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV)         += -lrte_rawdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER)          += -lrte_timer
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL)        += -lrte_mempool
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
Signature · 1 line
2.13.6
author
#23 lore raw

PATCH v2 2/8 mempool/stack: convert mempool to use rte stack

In reply to Gage Eads

The new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.

---
 MAINTAINERS                               |  2 +-
 drivers/mempool/stack/Makefile            |  3 +-
 drivers/mempool/stack/meson.build         |  6 +-
 drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++----------------------
 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
index 237f05eb2..7e64f63b6 100644
@@ -284,7 +284,6 @@ M: Andrew Rybchenko <[email protected]>
 F: lib/librte_mempool/
 F: drivers/mempool/Makefile
 F: drivers/mempool/ring/
-F: drivers/mempool/stack/
 F: doc/guides/prog_guide/mempool_lib.rst
 F: test/test/test_mempool*
 F: test/test/test_func_reentrancy.c
@@ -412,6 +411,7 @@ M: Gage Eads <[email protected]>
 M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
+F: drivers/mempool/stack/
 
 
 Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
index 0444aedad..1681a62bc 100644
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # Headers
 CFLAGS += -I$(RTE_SDK)/lib/librte_mempool
-LDLIBS += -lrte_eal -lrte_mempool -lrte_ring
+LDLIBS += -lrte_eal -lrte_mempool -lrte_stack
 
 EXPORT_MAP := rte_mempool_stack_version.map
 
drivers/mempool/stack/meson.build +5 -1
index b75a3bb56..03e369a41 100644
@@ -1,4 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
+# Copyright(c) 2017-2019 Intel Corporation
+
+allow_experimental_apis = true
 
 sources = files('rte_mempool_stack.c')
+
+deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68

133 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#24 lore raw

PATCH v2 3/8 test/stack: add stack test

In reply to Gage Eads

stack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.

---
 MAINTAINERS            |   1 +
 test/test/Makefile     |   2 +
 test/test/meson.build  |   3 +
 test/test/test_stack.c | 410 +++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 416 insertions(+)
 create mode 100644 test/test/test_stack.c
MAINTAINERS +1 -0
index 7e64f63b6..58b438414 100644
@@ -412,6 +412,7 @@ M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
 F: drivers/mempool/stack/
+F: test/test/*stack*
 
 
 Memory Pool Drivers
test/test/Makefile +2 -0
index 89949c2bb..47cf98a3a 100644
@@ -89,6 +89,8 @@ endif
 
 SRCS-y += test_rwlock.c
 
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
test/test/meson.build +3 -0
index 05e5ddeb0..b00e1201a 100644
@@ -95,6 +95,7 @@ test_sources = files('commands.c',
 	'test_sched.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
+	'test_stack.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -132,6 +133,7 @@ test_deps = ['acl',
 	'port',
 	'reorder',
 	'ring',
+	'stack',
 	'timer'
 ]
 
@@ -173,6 +175,7 @@ fast_parallel_test_names = [
         'rwlock_autotest',
         'sched_autotest',
         'spinlock_autotest',
+        'stack_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
test/test/test_stack.c new file +410 -0

410 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#25 lore raw

PATCH v2 4/8 test/stack: add stack perf test

In reply to Gage Eads

stack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects

It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores

---
 test/test/Makefile          |   1 +
 test/test/meson.build       |   2 +
 test/test/test_stack_perf.c | 343 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 346 insertions(+)
 create mode 100644 test/test/test_stack_perf.c
test/test/Makefile +1 -0
index 47cf98a3a..f9536fb31 100644
@@ -90,6 +90,7 @@ endif
 SRCS-y += test_rwlock.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
test/test/meson.build +2 -0
index b00e1201a..ba3cb6261 100644
@@ -96,6 +96,7 @@ test_sources = files('commands.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
 	'test_stack.c',
+	'test_stack_perf.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -240,6 +241,7 @@ perf_test_names = [
         'distributor_perf_autotest',
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
+        'stack_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
test/test/test_stack_perf.c new file +343 -0

343 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#26 lore raw

PATCH v2 5/8 stack: add lock-free stack implementation

In reply to Gage Eads

This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.

The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.

The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.

The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.

This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.

The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.

---
 doc/guides/prog_guide/stack_lib.rst    |  61 ++++++++++++-
 doc/guides/rel_notes/release_19_05.rst |   3 +
 lib/librte_stack/Makefile              |   3 +-
 lib/librte_stack/meson.build           |   3 +-
 lib/librte_stack/rte_stack.c           |  41 +++++++--
 lib/librte_stack/rte_stack.h           | 127 +++++++++++++++++++++++++--
 lib/librte_stack/rte_stack_generic.h   | 151 +++++++++++++++++++++++++++++++++
 7 files changed, 371 insertions(+), 18 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_generic.h
doc/guides/prog_guide/stack_lib.rst +58 -3
index 25a8cc38a..8fe8804e3 100644
@@ -10,7 +10,8 @@ stack of pointers.
 The stack library provides the following basic operations:
 
 *  Create a uniquely named stack of a user-specified size and using a
-   user-specified socket.
+   user-specified socket, with either standard (lock-based) or lock-free
+   behavior.
 
 *  Push and pop a burst of one or more stack objects (pointers). These function
    are multi-threading safe.
@@ -24,5 +25,59 @@ The stack library provides the following basic operations:
 Implementation
 ~~~~~~~~~~~~~~
 
-The stack consists of a contiguous array of pointers, a current index, and a
-spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
+The library supports two types of stacks: standard (lock-based) and lock-free.
+Both types use the same set of interfaces, but their implementations differ.
+
+Lock-based Stack
+----------------
+
+The lock-based stack consists of a contiguous array of pointers, a current
+index, and a spinlock. Accesses to the stack are made multi-thread safe by the
+spinlock.
+
+Lock-free Stack
+------------------
+
+The lock-free stack consists of a linked list of elements, each containing a
+data pointer and a next pointer, and an atomic stack depth counter. The
+lock-free property means that multiple threads can push and pop simultaneously,
+and one thread being preempted/delayed in a push or pop operation will not
+impede the forward progress of any other thread.
+
+The lock-free push operation enqueues a linked list of pointers by pointing the
+list's tail to the current stack head, and using a CAS to swing the stack head
+pointer to the head of the list. The operation retries if it is unsuccessful
+(i.e. the list changed between reading the head and modifying it), else it
+adjusts the stack length and returns.
+
+The lock-free pop operation first reserves one or more list elements by
+adjusting the stack length, to ensure the dequeue operation will succeed
+without blocking. It then dequeues pointers by walking the list -- starting
+from the head -- then swinging the head pointer (using a CAS as well). While
+walking the list, the data pointers are recorded in an object table.
+
+The linked list elements themselves are maintained in a lock-free LIFO, and are
+allocated before stack pushes and freed after stack pops. Since the stack has a
+fixed maximum depth, these elements do not need to be dynamically created.
+
+The lock-free behavior is selected by passing the *RTE_STACK_F_LF* flag to
+rte_stack_create().
+
+Preventing the ABA Problem
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To prevent the ABA problem, this algorithm stack uses a 128-bit
+compare-and-swap instruction to atomically update both the stack top pointer
+and a modification counter. The ABA problem can occur without a modification
+counter if, for example:
+
+1. Thread A reads head pointer X and stores the pointed-to list element.
+2. Other threads modify the list such that the head pointer is once again X,
+   but its pointed-to data is different than what thread A read.
+3. Thread A changes the head pointer with a compare-and-swap and succeeds.
+
+In this case thread A would not detect that the list had changed, and would
+both pop stale data and incorrect change the head pointer. By adding a
+modification counter that is updated on every push and pop as part of the
+compare-and-swap, the algorithm can detect when the list changes even if the
+head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
index 04394f8cf..51f0d2121 100644
@@ -71,6 +71,9 @@ New Features
   pointers. The API provides MT-safe push and pop operations that can operate
   on one or more pointers per operation.
 
+  The library supports two stack implementations: standard (lock-based) and lock-free.
+  The lock-free implementation is currently limited to x86-64 platforms.
+
 Removed Items
 -------------
 
lib/librte_stack/Makefile +2 -1
index e956b6535..3ecddf033 100644
@@ -18,6 +18,7 @@ LIBABIVER := 1
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c
 
 # install includes
-SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h
+SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
+					      rte_stack_generic.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
index 99f43710e..99d7f9ec5 100644
@@ -5,4 +5,5 @@ allow_experimental_apis = true
 
 version = 1
 sources = files('rte_stack.c')
-headers = files('rte_stack.h')
+headers = files('rte_stack.h',
+		'rte_stack_generic.h')
lib/librte_stack/rte_stack.c +33 -8

76 lines not on this page · raw message

lib/librte_stack/rte_stack.h +122 -5

166 lines not on this page · raw message

lib/librte_stack/rte_stack_generic.h new file +151 -0

151 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#27 lore raw

PATCH v2 6/8 stack: add C11 atomic implementation

In reply to Gage Eads

This commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.

---
 lib/librte_stack/Makefile            |   3 +-
 lib/librte_stack/meson.build         |   3 +-
 lib/librte_stack/rte_stack.h         |   4 +
 lib/librte_stack/rte_stack_c11_mem.h | 175 +++++++++++++++++++++++++++++++++++
 4 files changed, 183 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_c11_mem.h
lib/librte_stack/Makefile +2 -1
index 3ecddf033..94a7c1476 100644
@@ -19,6 +19,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c
 
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
-					      rte_stack_generic.h
+					      rte_stack_generic.h \
+					      rte_stack_c11_mem.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
index 99d7f9ec5..7e2d1dbb8 100644
@@ -6,4 +6,5 @@ allow_experimental_apis = true
 version = 1
 sources = files('rte_stack.c')
 headers = files('rte_stack.h',
-		'rte_stack_generic.h')
+		'rte_stack_generic.h',
+		'rte_stack_c11_mem.h')
lib/librte_stack/rte_stack.h +4 -0
index e576fb9ce..01a6ae281 100644
@@ -91,7 +91,11 @@ struct rte_stack {
  */
 #define RTE_STACK_F_LF 0x0001
 
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "rte_stack_c11_mem.h"
+#else
 #include "rte_stack_generic.h"
+#endif
 
 /**
  * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_c11_mem.h new file +175 -0

175 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#28 lore raw

PATCH v2 7/8 test/stack: add lock-free stack tests

In reply to Gage Eads

This commit adds lock-free stack variants of stack_autotest (stack_lf_autotest) and stack_perf_autotest (stack_lf_perf_autotest), which differ only in that the lock-free versions pass the RTE_STACK_F_LF flag to all rte_stack_create() calls.

---
 test/test/meson.build       |  2 ++
 test/test/test_stack.c      | 41 +++++++++++++++++++++++++++--------------
 test/test/test_stack_perf.c | 17 +++++++++++++++--
 3 files changed, 44 insertions(+), 16 deletions(-)
test/test/meson.build +2 -0
index ba3cb6261..474611291 100644
@@ -177,6 +177,7 @@ fast_parallel_test_names = [
         'sched_autotest',
         'spinlock_autotest',
         'stack_autotest',
+        'stack_nb_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
@@ -242,6 +243,7 @@ perf_test_names = [
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
         'stack_perf_autotest',
+        'stack_nb_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
test/test/test_stack.c +27 -14

112 lines not on this page · raw message

test/test/test_stack_perf.c +15 -2
index 484370d30..e09d5384c 100644
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s)
 }
 
 static int
-test_stack_perf(void)
+__test_stack_perf(uint32_t flags)
 {
 	struct lcore_pair cores;
 	struct rte_stack *s;
 
 	rte_atomic32_init(&lcore_barrier);
 
-	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), 0);
+	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), flags);
 	if (s == NULL) {
 		printf("[%s():%u] failed to create a stack\n",
 		       __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)
 	return 0;
 }
 
+static int
+test_stack_perf(void)
+{
+	return __test_stack_perf(0);
+}
+
+static int
+test_lf_stack_perf(void)
+{
+	return __test_stack_perf(RTE_STACK_F_LF);
+}
+
 REGISTER_TEST_COMMAND(stack_perf_autotest, test_stack_perf);
+REGISTER_TEST_COMMAND(stack_lf_perf_autotest, test_lf_stack_perf);
Signature · 1 line
2.13.6
author
#29 lore raw

PATCH v2 8/8 mempool/stack: add lock-free stack mempool handler

In reply to Gage Eads

This commit adds support for lock-free (linked list based) stack mempool handler.

In mempool_perf_autotest the lock-based stack outperforms the lock-free handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a standard (lock-based) stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the lock-free stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.

*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the lock-free stack's.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
 doc/guides/rel_notes/release_19_05.rst          |  5 +++++
 drivers/mempool/stack/rte_mempool_stack.c       | 26 +++++++++++++++++++++++--
 3 files changed, 39 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
index 929d76dba..dbcfc328e 100644
@@ -541,6 +541,16 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
+  Alternatively, applications can use the lock-free stack mempool handler. When
+  considering this handler, note that:
+
+  - It is currently limited to the x86_64 platform, because it uses an
+    instruction (16-byte compare-and-swap) that is not yet available on other
+    platforms.
+  - It has worse average-case performance than the non-preemptive rte_ring, but
+    software caching (e.g. the mempool cache) can mitigate this by reducing the
+    number of stack accesses.
+
 + rte_timer
 
   Running  ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
index 51f0d2121..f916f34c9 100644
@@ -74,6 +74,11 @@ New Features
   The library supports two stack implementations: standard (lock-based) and lock-free.
   The lock-free implementation is currently limited to x86-64 platforms.
 
+* **Added Lock-Free Stack Mempool Handler.**
+
+  Added a new lock-free stack handler, which uses the newly added stack
+  library.
+
 Removed Items
 -------------
 
drivers/mempool/stack/rte_mempool_stack.c +24 -2
index 25ccdb9af..7e85c8d6b 100644
@@ -7,7 +7,7 @@
 #include <rte_stack.h>
 
 static int
-stack_alloc(struct rte_mempool *mp)
+__stack_alloc(struct rte_mempool *mp, uint32_t flags)
 {
 	char name[RTE_STACK_NAMESIZE];
 	struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)
 		return -rte_errno;
 	}
 
-	s = rte_stack_create(name, mp->size, mp->socket_id, 0);
+	s = rte_stack_create(name, mp->size, mp->socket_id, flags);
 	if (s == NULL)
 		return -rte_errno;
 
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp)
 }
 
 static int
+stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, 0);
+}
+
+static int
+lf_stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, RTE_STACK_F_LF);
+}
+
+static int
 stack_enqueue(struct rte_mempool *mp, void * const *obj_table,
 	      unsigned int n)
 {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = {
 	.get_count = stack_get_count
 };
 
+static struct rte_mempool_ops ops_lf_stack = {
+	.name = "lf_stack",
+	.alloc = lf_stack_alloc,
+	.free = stack_free,
+	.enqueue = stack_enqueue,
+	.dequeue = stack_dequeue,
+	.get_count = stack_get_count
+};
+
 MEMPOOL_REGISTER_OPS(ops_stack);
+MEMPOOL_REGISTER_OPS(ops_lf_stack);
Signature · 1 line
2.13.6
author
#30 lore raw

v3 cover Add stack library and new mempool handler

In reply to Gage Eads

This patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.

The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.

The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.

This patchset is the successor to a patchset containing only the new mempool handler[2].

[1] http://mails.dpdk.org/archives/dev/2019-March/125751.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html

---
v3:
- Rebase patchset onto master (test/test/ -> app/test/)
- Fix rte_stack_std_push() segfault introduced in v2

v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset

Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler

 MAINTAINERS                                     |   9 +-
 app/test/Makefile                               |   3 +
 app/test/meson.build                            |   7 +
 app/test/test_stack.c                           | 423 ++++++++++++++++++++++++
 app/test/test_stack_perf.c                      | 356 ++++++++++++++++++++
 config/common_base                              |   5 +
 doc/api/doxy-api-index.md                       |   1 +
 doc/api/doxy-api.conf.in                        |   1 +
 doc/guides/prog_guide/env_abstraction_layer.rst |  10 +
 doc/guides/prog_guide/index.rst                 |   1 +
 doc/guides/prog_guide/stack_lib.rst             |  83 +++++
 doc/guides/rel_notes/release_19_05.rst          |  13 +
 drivers/mempool/stack/Makefile                  |   3 +-
 drivers/mempool/stack/meson.build               |   6 +-
 drivers/mempool/stack/rte_mempool_stack.c       | 115 +++----
 lib/Makefile                                    |   2 +
 lib/librte_stack/Makefile                       |  25 ++
 lib/librte_stack/meson.build                    |  10 +
 lib/librte_stack/rte_stack.c                    | 219 ++++++++++++
 lib/librte_stack/rte_stack.h                    | 395 ++++++++++++++++++++++
 lib/librte_stack/rte_stack_c11_mem.h            | 175 ++++++++++
 lib/librte_stack/rte_stack_generic.h            | 151 +++++++++
 lib/librte_stack/rte_stack_pvt.h                |  34 ++
 lib/librte_stack/rte_stack_version.map          |   9 +
 lib/meson.build                                 |   2 +-
 mk/rte.app.mk                                   |   1 +
 26 files changed, 1987 insertions(+), 72 deletions(-)
 create mode 100644 app/test/test_stack.c
 create mode 100644 app/test/test_stack_perf.c
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_c11_mem.h
 create mode 100644 lib/librte_stack/rte_stack_generic.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
Signature · 1 line
2.13.6
author
#31 lore raw

PATCH v3 1/8 stack: introduce rte stack library

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

---
 MAINTAINERS                            |   6 +
 config/common_base                     |   5 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/index.rst        |   1 +
 doc/guides/prog_guide/stack_lib.rst    |  28 ++++
 doc/guides/rel_notes/release_19_05.rst |   5 +
 lib/Makefile                           |   2 +
 lib/librte_stack/Makefile              |  23 +++
 lib/librte_stack/meson.build           |   8 +
 lib/librte_stack/rte_stack.c           | 194 +++++++++++++++++++++++
 lib/librte_stack/rte_stack.h           | 274 +++++++++++++++++++++++++++++++++
 lib/librte_stack/rte_stack_pvt.h       |  34 ++++
 lib/librte_stack/rte_stack_version.map |   9 ++
 lib/meson.build                        |   2 +-
 mk/rte.app.mk                          |   1 +
 16 files changed, 593 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
index 097cfb4f3..5fca30823 100644
@@ -405,6 +405,12 @@ F: drivers/raw/skeleton_rawdev/
 F: app/test/test_rawdev.c
 F: doc/guides/prog_guide/rawdev.rst
 
+Stack API - EXPERIMENTAL
+M: Gage Eads <[email protected]>
+M: Olivier Matz <[email protected]>
+F: lib/librte_stack/
+F: doc/guides/prog_guide/stack_lib.rst
+
 
 Memory Pool Drivers
 -------------------
config/common_base +5 -0
index 0b09a9348..1b45dea6c 100644
@@ -980,3 +980,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
 # Compile the eventdev application
 #
 CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Compile librte_stack
+#
+CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
index d95ad566c..0df8848c0 100644
@@ -124,6 +124,7 @@ The public API headers are grouped by topics:
   [mbuf]               (@ref rte_mbuf.h),
   [mbuf pool ops]      (@ref rte_mbuf_pool_ops.h),
   [ring]               (@ref rte_ring.h),
+  [stack]              (@ref rte_stack.h),
   [tailq]              (@ref rte_tailq.h),
   [bitmap]             (@ref rte_bitmap.h)
 
doc/api/doxy-api.conf.in +1 -0
index a365e669b..7722fc3e9 100644
@@ -55,6 +55,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/librte_ring \
                           @TOPDIR@/lib/librte_sched \
                           @TOPDIR@/lib/librte_security \
+                          @TOPDIR@/lib/librte_stack \
                           @TOPDIR@/lib/librte_table \
                           @TOPDIR@/lib/librte_telemetry \
                           @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
index 6726b1e8d..f4f60862f 100644
@@ -55,6 +55,7 @@ Programmer's Guide
     metrics_lib
     bpf_lib
     ipsec_lib
+    stack_lib
     source_org
     dev_kit_build_system
     dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +28 -0
new file mode 100644
index 000000000..25a8cc38a
@@ -0,0 +1,28 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Stack Library
+=============
+
+DPDK's stack library provides an API for configuration and use of a bounded
+stack of pointers.
+
+The stack library provides the following basic operations:
+
+*  Create a uniquely named stack of a user-specified size and using a
+   user-specified socket.
+
+*  Push and pop a burst of one or more stack objects (pointers). These function
+   are multi-threading safe.
+
+*  Free a previously created stack.
+
+*  Lookup a pointer to a stack by its name.
+
+*  Query a stack's current depth and number of free entries.
+
+Implementation
+~~~~~~~~~~~~~~
+
+The stack consists of a contiguous array of pointers, a current index, and a
+spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
index 4a3e2a7f3..8c649a954 100644
@@ -77,6 +77,11 @@ New Features
   which includes the directory name, lib name, filenames, makefile, docs,
   macros, functions, structs and any other strings in the code.
 
+* **Added Stack API.**
+
+  Added a new stack API for configuration and use of a bounded stack of
+  pointers. The API provides MT-safe push and pop operations that can operate
+  on one or more pointers per operation.
 
 Removed Items
 -------------
lib/Makefile +2 -0
index ffbfd0d94..d941bd849 100644
@@ -109,6 +109,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec
 DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security
 DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry
 DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev
+DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack
+DEPDIRS-librte_stack := librte_eal
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +23 -0
new file mode 100644
index 000000000..e956b6535
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_stack.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+LDLIBS += -lrte_eal
+
+EXPORT_MAP := rte_stack_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c
+
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
new file mode 100644
index 000000000..99f43710e
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+allow_experimental_apis = true
+
+version = 1
+sources = files('rte_stack.c')
+headers = files('rte_stack.h')
lib/librte_stack/rte_stack.c new file +194 -0

194 lines not on this page · raw message

lib/librte_stack/rte_stack.h new file +274 -0

274 lines not on this page · raw message

lib/librte_stack/rte_stack_pvt.h new file +34 -0
new file mode 100644
index 000000000..4a6a7bdb3
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_STACK_PVT_H_
+#define _RTE_STACK_PVT_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_log.h>
+
+extern int stack_logtype;
+
+#define STACK_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \
+		__func__, ##args)
+
+#define STACK_LOG_ERR(fmt, args...) \
+	STACK_LOG(ERR, fmt, ## args)
+
+#define STACK_LOG_WARN(fmt, args...) \
+	STACK_LOG(WARNING, fmt, ## args)
+
+#define STACK_LOG_INFO(fmt, args...) \
+	STACK_LOG(INFO, fmt, ## args)
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_version.map new file +9 -0
new file mode 100644
index 000000000..6662679c3
@@ -0,0 +1,9 @@
+EXPERIMENTAL {
+	global:
+
+	rte_stack_create;
+	rte_stack_free;
+	rte_stack_lookup;
+
+	local: *;
+};
lib/meson.build +1 -1
index 99957ba7d..90115477f 100644
@@ -22,7 +22,7 @@ libraries = [
 	'gro', 'gso', 'ip_frag', 'jobstats',
 	'kni', 'latencystats', 'lpm', 'member',
 	'power', 'pdump', 'rawdev',
-	'reorder', 'sched', 'security', 'vhost',
+	'reorder', 'sched', 'security', 'stack', 'vhost',
 	#ipsec lib depends on crypto and security
 	'ipsec',
 	# add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
index 3c40f9df2..8decfb851 100644
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -lrte_security
 _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV)         += -lrte_rawdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER)          += -lrte_timer
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL)        += -lrte_mempool
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
Signature · 1 line
2.13.6
Reviewed-by on v3 1/8
#32 lore raw

Re: [PATCH v3 1/8] stack: introduce rte stack library

Quoted 11 lines · Gage Eads

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

#33 lore raw

Re: [PATCH v3 1/8] stack: introduce rte stack library

In reply to Gage Eads

Hi Gage,
Apologies for the late comments.

Quoted 262 lines

-----Original Message-----
From: Gage Eads <[email protected]>
Sent: Wednesday, March 6, 2019 8:46 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Gavin Hu (Arm Technology China) <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>; [email protected]
Subject: [PATCH v3 1/8] stack: introduce rte stack library

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

---
 MAINTAINERS                            |   6 +
 config/common_base                     |   5 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/index.rst        |   1 +
 doc/guides/prog_guide/stack_lib.rst    |  28 ++++
 doc/guides/rel_notes/release_19_05.rst |   5 +
 lib/Makefile                           |   2 +
 lib/librte_stack/Makefile              |  23 +++
 lib/librte_stack/meson.build           |   8 +
 lib/librte_stack/rte_stack.c           | 194 +++++++++++++++++++++++
 lib/librte_stack/rte_stack.h           | 274
+++++++++++++++++++++++++++++++++
 lib/librte_stack/rte_stack_pvt.h       |  34 ++++
 lib/librte_stack/rte_stack_version.map |   9 ++
 lib/meson.build                        |   2 +-
 mk/rte.app.mk                          |   1 +
 16 files changed, 593 insertions(+), 1 deletion(-)  create mode 100644
doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile  create mode 100644

lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_version.map

MAINTAINERS +6 -0
index 097cfb4f3..5fca30823 100644
@@ -405,6 +405,12 @@ F: drivers/raw/skeleton_rawdev/
 F: app/test/test_rawdev.c
 F: doc/guides/prog_guide/rawdev.rst
 
+Stack API - EXPERIMENTAL
+M: Gage Eads <[email protected]>
+M: Olivier Matz <[email protected]>
+F: lib/librte_stack/
+F: doc/guides/prog_guide/stack_lib.rst
+
 
 Memory Pool Drivers
 -------------------
config/common_base index renamed from config/common_base +0 -0

0b09a9348..1b45dea6c 100644

config/common_base +0 -0
@@ -980,3 +980,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y  # Compile the

eventdev application # CONFIG_RTE_APP_EVENTDEV=y

config/common_base +5 -0
+
+#
+# Compile librte_stack
+#
+CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md index renamed from doc/api/doxy-api-index.md +0 -0

d95ad566c..0df8848c0 100644

doc/api/doxy-api-index.md +1 -0
@@ -124,6 +124,7 @@ The public API headers are grouped by topics:
   [mbuf]               (@ref rte_mbuf.h),
   [mbuf pool ops]      (@ref rte_mbuf_pool_ops.h),
   [ring]               (@ref rte_ring.h),
+  [stack]              (@ref rte_stack.h),
   [tailq]              (@ref rte_tailq.h),
   [bitmap]             (@ref rte_bitmap.h)
 
doc/api/doxy-api.conf.in index renamed from doc/api/doxy-api.conf.in +0 -0

a365e669b..7722fc3e9 100644

doc/api/doxy-api.conf.in +1 -0
@@ -55,6 +55,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-
 index.md \
                           @TOPDIR@/lib/librte_ring \
                           @TOPDIR@/lib/librte_sched \
                           @TOPDIR@/lib/librte_security \
+                          @TOPDIR@/lib/librte_stack \
                           @TOPDIR@/lib/librte_table \
                           @TOPDIR@/lib/librte_telemetry \
                           @TOPDIR@/lib/librte_timer \ diff --git

a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst index 6726b1e8d..f4f60862f 100644

doc/guides/prog_guide/index.rst +1 -0
@@ -55,6 +55,7 @@ Programmer's Guide
     metrics_lib
     bpf_lib
     ipsec_lib
+    stack_lib
     source_org
     dev_kit_build_system
     dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +15 -0
new file mode 100644
index 000000000..25a8cc38a
@@ -0,0 +1,28 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Stack Library
+=============
+
+DPDK's stack library provides an API for configuration and use of a
+bounded stack of pointers.
+
+The stack library provides the following basic operations:
+
+*  Create a uniquely named stack of a user-specified size and using a
+   user-specified socket.
+
+*  Push and pop a burst of one or more stack objects (pointers). These

function

doc/guides/prog_guide/stack_lib.rst new file +13 -0
+   are multi-threading safe.
+
+*  Free a previously created stack.
+
+*  Lookup a pointer to a stack by its name.
+
+*  Query a stack's current depth and number of free entries.
+
+Implementation
+~~~~~~~~~~~~~~
+
+The stack consists of a contiguous array of pointers, a current index,
+and a spinlock. Accesses to the stack are made multi-thread safe by the

spinlock.

doc/guides/rel_notes/release_19_05.rst +5 -0
index 4a3e2a7f3..8c649a954 100644
@@ -77,6 +77,11 @@ New Features
   which includes the directory name, lib name, filenames, makefile, docs,
   macros, functions, structs and any other strings in the code.
 
+* **Added Stack API.**
+
+  Added a new stack API for configuration and use of a bounded stack of
+ pointers. The API provides MT-safe push and pop operations that can
+ operate  on one or more pointers per operation.
 
 Removed Items
 -------------
lib/Makefile +0 -0
@@ -109,6 +109,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec

DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security
DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry DEPDIRS-
librte_telemetry := librte_eal librte_metrics librte_ethdev

lib/Makefile +2 -0
+DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack DEPDIRS-librte_stack :=
+librte_eal
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni diff --git

a/lib/librte_stack/Makefile b/lib/librte_stack/Makefile new file mode 100644 index 000000000..e956b6535

lib/librte_stack/Makefile new file +22 -0
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2019 Intel
+Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_stack.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 CFLAGS +=
+-DALLOW_EXPERIMENTAL_API LDLIBS += -lrte_eal
+
+EXPORT_MAP := rte_stack_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c
+
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new renamed from lib/librte_stack/meson.build +0 -0

file mode 100644 index 000000000..99f43710e

lib/librte_stack/meson.build new file +8 -0
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2019 Intel
+Corporation
+
+allow_experimental_apis = true
+
+version = 1
+sources = files('rte_stack.c')
+headers = files('rte_stack.h')
lib/librte_stack/rte_stack.c new file renamed from lib/librte_stack/rte_stack.c +0 -0

mode 100644 index 000000000..96dffdf44

lib/librte_stack/rte_stack.c new file +37 -0
@@ -0,0 +1,194 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include <string.h>
+
+#include <rte_atomic.h>
+#include <rte_eal.h>
+#include <rte_eal_memconfig.h>
+#include <rte_errno.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+#include <rte_rwlock.h>
+#include <rte_tailq.h>
+
+#include "rte_stack.h"
+#include "rte_stack_pvt.h"
+
+int stack_logtype;
+
+TAILQ_HEAD(rte_stack_list, rte_tailq_entry);
+
+static struct rte_tailq_elem rte_stack_tailq = {
+	.name = RTE_TAILQ_STACK_NAME,
+};
+EAL_REGISTER_TAILQ(rte_stack_tailq)
+
+static void
+rte_stack_std_init(struct rte_stack *s) {
+	rte_spinlock_init(&s->stack_std.lock);
+}
+
+static void
+rte_stack_init(struct rte_stack *s)
+{
+	memset(s, 0, sizeof(*s));
+

lib/librte_stack/rte_stack.c line 47

Quoted 10 lines
lib/librte_stack/rte_stack.c new file +10 -0
+	rte_stack_std_init(s);
+}
+
+static ssize_t
+rte_stack_get_memsize(unsigned int count) {
+	ssize_t sz = sizeof(struct rte_stack);
+
+	/* Add padding to avoid false sharing conflicts */
+	sz += RTE_CACHE_LINE_ROUNDUP(count * sizeof(void *)) +
+		2 * RTE_CACHE_LINE_SIZE;

I did not understand how the false sharing is caused and how this padding is solving the issue. Verbose comments would help.

Quoted 26 lines
lib/librte_stack/rte_stack.c new file +26 -0
+
+	return sz;
+}
+
+struct rte_stack *
+rte_stack_create(const char *name, unsigned int count, int socket_id,
+		 uint32_t flags)
+{
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	struct rte_stack_list *stack_list;
+	const struct rte_memzone *mz;
+	struct rte_tailq_entry *te;
+	struct rte_stack *s;
+	unsigned int sz;
+	int ret;
+
+	RTE_SET_USED(flags);
+
+	sz = rte_stack_get_memsize(count);
+
+	ret = snprintf(mz_name, sizeof(mz_name), "%s%s",
+		       RTE_STACK_MZ_PREFIX, name);
+	if (ret < 0 || ret >= (int)sizeof(mz_name)) {
+		rte_errno = ENAMETOOLONG;
+		return NULL;
+	}

lib/librte_stack/rte_stack.c line 85

Quoted 10 lines
lib/librte_stack/rte_stack.c new file +10 -0
+
+	te = rte_zmalloc("STACK_TAILQ_ENTRY", sizeof(*te), 0);
+	if (te == NULL) {
+		STACK_LOG_ERR("Cannot reserve memory for tailq\n");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+
+	rte_rwlock_write_lock(RTE_EAL_TAILQ_RWLOCK);
+

I think there is a need to check if a stack with the same name exists already.

Quoted 38 lines
lib/librte_stack/rte_stack.c new file +38 -0
+	mz = rte_memzone_reserve_aligned(mz_name, sz, socket_id,
+					 0, __alignof__(*s));
+	if (mz == NULL) {
+		STACK_LOG_ERR("Cannot reserve stack memzone!\n");
+		rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+		rte_free(te);
+		return NULL;
+	}
+
+	s = mz->addr;
+
+	rte_stack_init(s);
+
+	/* Store the name for later lookups */
+	ret = snprintf(s->name, sizeof(s->name), "%s", name);
+	if (ret < 0 || ret >= (int)sizeof(s->name)) {
+		rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+
+		rte_errno = ENAMETOOLONG;
+		rte_free(te);
+		rte_memzone_free(mz);
+		return NULL;
+	}
+
+	s->memzone = mz;
+	s->capacity = count;
+	s->flags = flags;
+
+	te->data = s;
+
+	stack_list = RTE_TAILQ_CAST(rte_stack_tailq.head, rte_stack_list);
+
+	TAILQ_INSERT_TAIL(stack_list, te, next);
+
+	rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+
+	return s;
+}

lib/librte_stack/rte_stack.c line 133

Quoted 10 lines
lib/librte_stack/rte_stack.c new file +10 -0
+
+void
+rte_stack_free(struct rte_stack *s)
+{
+	struct rte_stack_list *stack_list;
+	struct rte_tailq_entry *te;
+
+	if (s == NULL)
+		return;
+

Adding a check to make sure the length of the stack is 0 would help catch issues?

Quoted 86 lines
lib/librte_stack/rte_stack.c new file +60 -0
+	stack_list = RTE_TAILQ_CAST(rte_stack_tailq.head, rte_stack_list);
+	rte_rwlock_write_lock(RTE_EAL_TAILQ_RWLOCK);
+
+	/* find out tailq entry */
+	TAILQ_FOREACH(te, stack_list, next) {
+		if (te->data == s)
+			break;
+	}
+
+	if (te == NULL) {
+		rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+		return;
+	}
+
+	TAILQ_REMOVE(stack_list, te, next);
+
+	rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+
+	rte_free(te);
+
+	rte_memzone_free(s->memzone);
+}
+
+struct rte_stack *
+rte_stack_lookup(const char *name)
+{
+	struct rte_stack_list *stack_list;
+	struct rte_tailq_entry *te;
+	struct rte_stack *r = NULL;
+
+	if (name == NULL) {
+		rte_errno = EINVAL;
+		return NULL;
+	}
+
+	stack_list = RTE_TAILQ_CAST(rte_stack_tailq.head, rte_stack_list);
+
+	rte_rwlock_read_lock(RTE_EAL_TAILQ_RWLOCK);
+
+	TAILQ_FOREACH(te, stack_list, next) {
+		r = (struct rte_stack *) te->data;
+		if (strncmp(name, r->name, RTE_STACK_NAMESIZE) == 0)
+			break;
+	}
+
+	rte_rwlock_read_unlock(RTE_EAL_TAILQ_RWLOCK);
+
+	if (te == NULL) {
+		rte_errno = ENOENT;
+		return NULL;
+	}
+
+	return r;
+}
+
+RTE_INIT(librte_stack_init_log)
+{
+	stack_logtype = rte_log_register("lib.stack");
+	if (stack_logtype >= 0)
+		rte_log_set_level(stack_logtype, RTE_LOG_NOTICE); }
lib/librte_stack/rte_stack.h new file renamed from lib/librte_stack/rte_stack.h +0 -0

mode 100644 index 000000000..7a633deb5

lib/librte_stack/rte_stack.h new file +15 -0
@@ -0,0 +1,274 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+/**
+ * @file rte_stack.h
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * RTE Stack
+ *
+ * librte_stack provides an API for configuration and use of a bounded
+stack of
+ * pointers. Push and pop operations are MT-safe, allowing concurrent
+access,
+ * and the interface supports pushing and popping multiple pointers at a

time.
+ */

lib/librte_stack/rte_stack.h new file +4 -0
+
+#ifndef _RTE_STACK_H_
+#define _RTE_STACK_H_
+

lib/librte_stack/rte_stack.h line 28

Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10 -0
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_errno.h>
+#include <rte_memzone.h>
+#include <rte_spinlock.h>
+
+#define RTE_TAILQ_STACK_NAME "RTE_STACK"
+#define RTE_STACK_MZ_PREFIX "STK_"

Nit, "STACK_" would be easier to debug

Quoted 30 lines
lib/librte_stack/rte_stack.h new file +30 -0
+/** The maximum length of a stack name. */ #define RTE_STACK_NAMESIZE
+(RTE_MEMZONE_NAMESIZE - \
+			   sizeof(RTE_STACK_MZ_PREFIX) + 1)
+
+/* Structure containing the LIFO, its current length, and a lock for
+mutual
+ * exclusion.
+ */
+struct rte_stack_std {
+	rte_spinlock_t lock; /**< LIFO lock */
+	uint32_t len; /**< LIFO len */
+	void *objs[]; /**< LIFO pointer table */ };
+
+/* The RTE stack structure contains the LIFO structure itself, plus
+metadata
+ * such as its name and memzone pointer.
+ */
+struct rte_stack {
+	/** Name of the stack. */
+	char name[RTE_STACK_NAMESIZE] __rte_cache_aligned;
+	/** Memzone containing the rte_stack structure. */
+	const struct rte_memzone *memzone;
+	uint32_t capacity; /**< Usable size of the stack. */
+	uint32_t flags; /**< Flags supplied at creation. */
+	struct rte_stack_std stack_std; /**< LIFO structure. */ }
+__rte_cache_aligned;
+
+/**
+ * @internal Push several objects on the stack (MT-safe).
+ *

lib/librte_stack/rte_stack.h line 67

Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10 -0
+ * @param s
+ *   A pointer to the stack structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to push on the stack from the obj_table.
+ * @return
+ *   Actual number of objects pushed (either 0 or *n*).
+ */
+static __rte_always_inline unsigned int __rte_experimental

This is an internal function. Is '__rte_experimental' tag required?

lib/librte_stack/rte_stack.h

Quoted 2 lines
lib/librte_stack/rte_stack.h new file +2 -0
+rte_stack_std_push(struct rte_stack *s, void * const *obj_table,
+unsigned int n) {

Since this is an internal function, does it make sense to add '__' to the beginning of the function name (similar to what is done in rte_ring?).

Quoted 46 lines
lib/librte_stack/rte_stack.h new file +46 -0
+	struct rte_stack_std *stack = &s->stack_std;
+	unsigned int index;
+	void **cache_objs;
+
+	rte_spinlock_lock(&stack->lock);
+	cache_objs = &stack->objs[stack->len];
+
+	/* Is there sufficient space in the stack? */
+	if ((stack->len + n) > s->capacity) {
+		rte_spinlock_unlock(&stack->lock);
+		return 0;
+	}
+
+	/* Add elements back into the cache */
+	for (index = 0; index < n; ++index, obj_table++)
+		cache_objs[index] = *obj_table;
+
+	stack->len += n;
+
+	rte_spinlock_unlock(&stack->lock);
+	return n;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Push several objects on the stack (MT-safe).
+ *
+ * @param s
+ *   A pointer to the stack structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to push on the stack from the obj_table.
+ * @return
+ *   Actual number of objects pushed (either 0 or *n*).
+ */
+static __rte_always_inline unsigned int __rte_experimental
+rte_stack_push(struct rte_stack *s, void * const *obj_table, unsigned
+int n) {
+	return rte_stack_std_push(s, obj_table, n); }
+
+/**
+ * @internal Pop several objects from the stack (MT-safe).
+ *

lib/librte_stack/rte_stack.h line 126

Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10 -0
+ * @param s
+ *   A pointer to the stack structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to pull from the stack.
+ * @return
+ *   Actual number of objects popped (either 0 or *n*).
+ */
+static __rte_always_inline unsigned int __rte_experimental

This is an internal function. Is '__rte_experimental' tag required?

Quoted 34 lines
lib/librte_stack/rte_stack.h new file +34 -0
+rte_stack_std_pop(struct rte_stack *s, void **obj_table, unsigned int
+n) {
+	struct rte_stack_std *stack = &s->stack_std;
+	unsigned int index, len;
+	void **cache_objs;
+
+	rte_spinlock_lock(&stack->lock);
+
+	if (unlikely(n > stack->len)) {
+		rte_spinlock_unlock(&stack->lock);
+		return 0;
+	}
+
+	cache_objs = stack->objs;
+
+	for (index = 0, len = stack->len - 1; index < n;
+			++index, len--, obj_table++)
+		*obj_table = cache_objs[len];
+
+	stack->len -= n;
+	rte_spinlock_unlock(&stack->lock);
+
+	return n;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Pop several objects from the stack (MT-safe).
+ *
+ * @param s
+ *   A pointer to the stack structure.
+ * @param obj_table

lib/librte_stack/rte_stack.h line 171

Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10 -0
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to pull from the stack.
+ * @return
+ *   Actual number of objects popped (either 0 or *n*).
+ */
+static __rte_always_inline unsigned int __rte_experimental
+rte_stack_pop(struct rte_stack *s, void **obj_table, unsigned int n) {
+	if (unlikely(n == 0 || obj_table == NULL))
+		return 0;

's == NULL' can be added as well. Similar check is missing in 'rte_stack_push'. Since these are data-path APIs, RTE_ASSERT would be better.

Quoted 57 lines
lib/librte_stack/rte_stack.h new file +57 -0
+
+	return rte_stack_std_pop(s, obj_table, n); }
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Return the number of used entries in a stack.
+ *
+ * @param s
+ *   A pointer to the stack structure.
+ * @return
+ *   The number of used entries in the stack.
+ */
+static __rte_always_inline unsigned int __rte_experimental
+rte_stack_count(struct rte_stack *s) {
+	return (unsigned int)s->stack_std.len; }
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Return the number of free entries in a stack.
+ *
+ * @param s
+ *   A pointer to the stack structure.
+ * @return
+ *   The number of free entries in the stack.
+ */
+static __rte_always_inline unsigned int __rte_experimental
+rte_stack_free_count(struct rte_stack *s) {
+	return s->capacity - rte_stack_count(s); }
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Create a new stack named *name* in memory.
+ *
+ * This function uses ``memzone_reserve()`` to allocate memory for a
+stack of
+ * size *count*. The behavior of the stack is controlled by the *flags*.
+ *
+ * @param name
+ *   The name of the stack.
+ * @param count
+ *   The size of the stack.
+ * @param socket_id
+ *   The *socket_id* argument is the socket identifier in case of
+ *   NUMA. The value can be *SOCKET_ID_ANY* if there is no NUMA
+ *   constraint for the reserved zone.
+ * @param flags
+ *   Reserved for future use.
+ * @return
+ *   On success, the pointer to the new allocated stack. NULL on error with
+ *    rte_errno set appropriately. Possible errno values include:
+ *    - ENOSPC - the maximum number of memzones has already been
Quoted 2 lines

allocated
+ * - EEXIST - a stack with the same name already exists

This is not implemented currently

Quoted 124 lines

+ * - ENOMEM - insufficient memory to create the stack
+ * - ENAMETOOLONG - name size exceeds RTE_STACK_NAMESIZE
+ */

lib/librte_stack/rte_stack.h new file +38 -0
+struct rte_stack *__rte_experimental
+rte_stack_create(const char *name, unsigned int count, int socket_id,
+		 uint32_t flags);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Free all memory used by the stack.
+ *
+ * @param s
+ *   Stack to free
+ */
+void __rte_experimental
+rte_stack_free(struct rte_stack *s);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Lookup a stack by its name.
+ *
+ * @param name
+ *   The name of the stack.
+ * @return
+ *   The pointer to the stack matching the name, or NULL if not found,
+ *   with rte_errno set appropriately. Possible rte_errno values include:
+ *    - ENOENT - Stack with name *name* not found.
+ *    - EINVAL - *name* pointer is NULL.
+ */
+struct rte_stack * __rte_experimental
+rte_stack_lookup(const char *name);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_STACK_H_ */
lib/librte_stack/rte_stack_pvt.h new file +34 -0
new file mode 100644
index 000000000..4a6a7bdb3
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_STACK_PVT_H_
+#define _RTE_STACK_PVT_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_log.h>
+
+extern int stack_logtype;
+
+#define STACK_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \
+		__func__, ##args)
+
+#define STACK_LOG_ERR(fmt, args...) \
+	STACK_LOG(ERR, fmt, ## args)
+
+#define STACK_LOG_WARN(fmt, args...) \
+	STACK_LOG(WARNING, fmt, ## args)
+
+#define STACK_LOG_INFO(fmt, args...) \
+	STACK_LOG(INFO, fmt, ## args)
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_version.map new file +9 -0
new file mode 100644
index 000000000..6662679c3
@@ -0,0 +1,9 @@
+EXPERIMENTAL {
+	global:
+
+	rte_stack_create;
+	rte_stack_free;
+	rte_stack_lookup;
+
+	local: *;
+};
lib/meson.build index 99957ba7d..90115477f renamed from lib/meson.build +0 -0

100644

lib/meson.build +1 -1
@@ -22,7 +22,7 @@ libraries = [
 	'gro', 'gso', 'ip_frag', 'jobstats',
 	'kni', 'latencystats', 'lpm', 'member',
 	'power', 'pdump', 'rawdev',
-	'reorder', 'sched', 'security', 'vhost',
+	'reorder', 'sched', 'security', 'stack', 'vhost',
 	#ipsec lib depends on crypto and security
 	'ipsec',
 	# add pkt framework libs which use other libs from above diff --git

a/mk/rte.app.mk b/mk/rte.app.mk index 3c40f9df2..8decfb851 100644

mk/rte.app.mk +0 -0
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -

lrte_security

 _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV)         += -lrte_rawdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER)          += -lrte_timer
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL)        += -lrte_mempool
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring

--
2.13.6

author
#34 lore raw

Re: [PATCH v3 1/8] stack: introduce rte stack library

@Thomas: I expect I can address Honnappa's feedback within a day or two. Since today is the 19.05 merge deadline, what do you think about these options for merging?
1. Merge V4 now and address these comments during RC1.
2. Delay merge until RC2, with all comments addressed.

In terms of risk, Honnappa identified an incorrect memory ordering argument (patch 6/8), but that doesn't affect the one platform (x86-64) that can (currently) use this library. His other comments address readability, error-checking, and performance, but aren't critical. Beyond that, this patchset is isolated from the rest of DPDK. So, I think the risk to the project is very low.

(Also, note that I accidentally left off Olivier's Reviewed-by tag in V4's patches 1, 3, 5, and 6 -- I'll address that as well)

Quoted 11 lines

-----Original Message-----
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Thursday, March 28, 2019 6:27 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v3 1/8] stack: introduce rte stack library

Quoted 3 lines

Hi Gage,
Apologies for the late comments.

No problem, I appreciate the feedback.

[snip]

Quoted 9 lines

lib/librte_stack/rte_stack.c line 49

Quoted 7 lines
lib/librte_stack/rte_stack.c new file +7 -0
+static ssize_t
+rte_stack_get_memsize(unsigned int count) {
+	ssize_t sz = sizeof(struct rte_stack);
+
+	/* Add padding to avoid false sharing conflicts */
+	sz += RTE_CACHE_LINE_ROUNDUP(count * sizeof(void *)) +
+		2 * RTE_CACHE_LINE_SIZE;

I did not understand how the false sharing is caused and how this padding is solving the issue. Verbose comments would help.

The additional padding (beyond the CACHE_LINE_ROUNDUP) is to prevent false sharing caused by adjacent/next-line hardware prefetchers. I'll address this.

[snip]

Quoted 25 lines
Quoted 25 lines
lib/librte_stack/rte_stack.c new file +25 -0
+struct rte_stack *
+rte_stack_create(const char *name, unsigned int count, int socket_id,
+		 uint32_t flags)
+{
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	struct rte_stack_list *stack_list;
+	const struct rte_memzone *mz;
+	struct rte_tailq_entry *te;
+	struct rte_stack *s;
+	unsigned int sz;
+	int ret;
+
+	RTE_SET_USED(flags);
+
+	sz = rte_stack_get_memsize(count);
+
+	ret = snprintf(mz_name, sizeof(mz_name), "%s%s",
+		       RTE_STACK_MZ_PREFIX, name);
+	if (ret < 0 || ret >= (int)sizeof(mz_name)) {
+		rte_errno = ENAMETOOLONG;
+		return NULL;
+	}
+
+	te = rte_zmalloc("STACK_TAILQ_ENTRY", sizeof(*te), 0);
+	if (te == NULL) {
Quoted 8 lines

lib/librte_stack/rte_stack.c line 85

Quoted 7 lines
lib/librte_stack/rte_stack.c new file +7 -0
+		STACK_LOG_ERR("Cannot reserve memory for tailq\n");
+		rte_errno = ENOMEM;
+		return NULL;
+	}
+
+	rte_rwlock_write_lock(RTE_EAL_TAILQ_RWLOCK);
+

I think there is a need to check if a stack with the same name exists already.

rte_memzone_reserve_aligned() does just that. This behavior is tested in the function test_stack_name_reuse(), added in commit " test/stack: add stack test".

Quoted 8 lines
Quoted 8 lines
lib/librte_stack/rte_stack.c new file +8 -0
+	mz = rte_memzone_reserve_aligned(mz_name, sz, socket_id,
+					 0, __alignof__(*s));
+	if (mz == NULL) {
+		STACK_LOG_ERR("Cannot reserve stack memzone!\n");
+		rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+		rte_free(te);
+		return NULL;
+	}

[snip]

Quoted 11 lines

lib/librte_stack/rte_stack.c line 133

Quoted 9 lines
lib/librte_stack/rte_stack.c new file +9 -0
+void
+rte_stack_free(struct rte_stack *s)
+{
+	struct rte_stack_list *stack_list;
+	struct rte_tailq_entry *te;
+
+	if (s == NULL)
+		return;
+

Adding a check to make sure the length of the stack is 0 would help catch issues?

My preference is to leave that check to the user, for any apps that want to/can safely free non-empty stacks.

[snip]

Quoted 3 lines

lib/librte_stack/rte_stack.h line 28

Quoted 2 lines
lib/librte_stack/rte_stack.h new file +2 -0
+#define RTE_TAILQ_STACK_NAME "RTE_STACK"
+#define RTE_STACK_MZ_PREFIX "STK_"

Nit, "STACK_" would be easier to debug

Since RTE_MEMZONE_NAMESIZE (32) doesn't give us a lot of space, I kept the prefix short. Adding 2 more characters *probably* won't make a difference...but I'd prefer the shortened name.

Quoted 5 lines
Quoted 1 line

+/** The maximum length of a stack name. */ #define

RTE_STACK_NAMESIZE

Quoted 3 lines
lib/librte_stack/rte_stack.h new file +3 -0
+(RTE_MEMZONE_NAMESIZE - \
+			   sizeof(RTE_STACK_MZ_PREFIX) + 1)
+

[snip]

Quoted 6 lines
Quoted 6 lines

+/**
+ * @internal Push several objects on the stack (MT-safe).
+ *
+ * @param s
+ * A pointer to the stack structure.
+ * @param obj_table

Quoted 8 lines
Quoted 7 lines

+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to push on the stack from the obj_table.
+ * @return
+ * Actual number of objects pushed (either 0 or *n*).
+ */
+static __rte_always_inline unsigned int __rte_experimental

This is an internal function. Is '__rte_experimental' tag required?

I don't think so, but I erred on the side of caution. I don't think the tag causes any problems.

Quoted 5 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+rte_stack_std_push(struct rte_stack *s, void * const *obj_table,
+unsigned int n) {

Since this is an internal function, does it make sense to add '__' to the beginning of the function name (similar to what is done in rte_ring?).

Makes sense. I'll address this.

[snip]

Quoted 6 lines
Quoted 6 lines

+/**
+ * @internal Pop several objects from the stack (MT-safe).
+ *
+ * @param s
+ * A pointer to the stack structure.
+ * @param obj_table

Quoted 8 lines
Quoted 7 lines

+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to pull from the stack.
+ * @return
+ * Actual number of objects popped (either 0 or *n*).
+ */
+static __rte_always_inline unsigned int __rte_experimental

This is an internal function. Is '__rte_experimental' tag required?

(see above)

[snip]

Quoted 7 lines

lib/librte_stack/rte_stack.h line 171

Quoted 4 lines
lib/librte_stack/rte_stack.h new file +4 -0
+static __rte_always_inline unsigned int __rte_experimental
+rte_stack_pop(struct rte_stack *s, void **obj_table, unsigned int n) {
+	if (unlikely(n == 0 || obj_table == NULL))
+		return 0;

's == NULL' can be added as well. Similar check is missing in 'rte_stack_push'. Since these are data-path APIs, RTE_ASSERT would be better.

Good point. I'll add RTE_ASSERT for obj_table and s. That won't work for "n == 0" -- the pop code assumes n > 0, so we can't allow that check to be compiled out.

[snip]

Quoted 19 lines
Quoted 19 lines
lib/librte_stack/rte_stack.h new file +19 -0
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Create a new stack named *name* in memory.
+ *
+ * This function uses ``memzone_reserve()`` to allocate memory for a
+stack of
+ * size *count*. The behavior of the stack is controlled by the *flags*.
+ *
+ * @param name
+ *   The name of the stack.
+ * @param count
+ *   The size of the stack.
+ * @param socket_id
+ *   The *socket_id* argument is the socket identifier in case of
+ *   NUMA. The value can be *SOCKET_ID_ANY* if there is no NUMA
+ *   constraint for the reserved zone.
+ * @param flags
Quoted 8 lines
Quoted 7 lines

+ * Reserved for future use.
+ * @return
+ * On success, the pointer to the new allocated stack. NULL on error with
+ * rte_errno set appropriately. Possible errno values include:
+ * - ENOSPC - the maximum number of memzones has already been
allocated
+ * - EEXIST - a stack with the same name already exists

This is not implemented currently

It is -- see above.

Thanks,
Gage

#35 lore raw

Re: [PATCH v3 1/8] stack: introduce rte stack library

29/03/2019 20:23, Eads, Gage:

Quoted 3 lines

@Thomas: I expect I can address Honnappa's feedback within a day or two. Since today is the 19.05 merge deadline, what do you think about these options for merging?
1. Merge V4 now and address these comments during RC1.
2. Delay merge until RC2, with all comments addressed.

I plan to release RC1 on Wednesday,
allowing last revision to be sent on Tuesday.

If it does not impact the rest of DPDK, the RC2 is also an option to consider.

#36 lore raw

Re: [PATCH v3 1/8] stack: introduce rte stack library

In reply to Eads, Gage
Quoted 11 lines
Quoted 10 lines

lib/librte_stack/rte_stack.c line 49

Quoted 8 lines
lib/librte_stack/rte_stack.c new file +7 -0
+static ssize_t
+rte_stack_get_memsize(unsigned int count) {
+	ssize_t sz = sizeof(struct rte_stack);
+
+	/* Add padding to avoid false sharing conflicts */
+	sz += RTE_CACHE_LINE_ROUNDUP(count * sizeof(void *)) +
+		2 * RTE_CACHE_LINE_SIZE;

I did not understand how the false sharing is caused and how this padding is solving the issue. Verbose comments would help.

Quoted 4 lines

The additional padding (beyond the CACHE_LINE_ROUNDUP) is to prevent false sharing caused by adjacent/next-line hardware prefetchers. I'll address this.

Is it not a generic problem? Or is it specific to this library?

author
#37 lore raw

Re: [PATCH v3 1/8] stack: introduce rte stack library

Quoted 22 lines

-----Original Message-----
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Monday, April 1, 2019 12:41 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v3 1/8] stack: introduce rte stack library

Quoted 11 lines
Quoted 10 lines

lib/librte_stack/rte_stack.c line 49

Quoted 8 lines
lib/librte_stack/rte_stack.c new file +7 -0
+static ssize_t
+rte_stack_get_memsize(unsigned int count) {
+	ssize_t sz = sizeof(struct rte_stack);
+
+	/* Add padding to avoid false sharing conflicts */
+	sz += RTE_CACHE_LINE_ROUNDUP(count * sizeof(void *)) +
+		2 * RTE_CACHE_LINE_SIZE;

I did not understand how the false sharing is caused and how this padding is solving the issue. Verbose comments would help.

Quoted 5 lines
Quoted 4 lines

The additional padding (beyond the CACHE_LINE_ROUNDUP) is to prevent false sharing caused by adjacent/next-line hardware prefetchers. I'll address this.

Is it not a generic problem? Or is it specific to this library?

This is not limited to this library, but it only affects systems with (enabled) next-line prefetchers, for example Intel products with an L2 adjacent cache line prefetcher[1]. For those systems, additional padding can potentially improve performance. As I understand it, this was the reason behind the 128B alignment added to rte_ring a couple years ago[2].

[1] https://software.intel.com/en-us/articles/disclosure-of-hw-prefetcher-control-on-some-intel-processors
[2] http://mails.dpdk.org/archives/dev/2017-February/058613.html

author
#38 lore raw

PATCH v3 2/8 mempool/stack: convert mempool to use rte stack

In reply to Gage Eads

The new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.

---
 MAINTAINERS                               |  2 +-
 drivers/mempool/stack/Makefile            |  3 +-
 drivers/mempool/stack/meson.build         |  6 +-
 drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++----------------------
 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
index 5fca30823..4e088d2bd 100644
@@ -282,7 +282,6 @@ M: Andrew Rybchenko <[email protected]>
 F: lib/librte_mempool/
 F: drivers/mempool/Makefile
 F: drivers/mempool/ring/
-F: drivers/mempool/stack/
 F: doc/guides/prog_guide/mempool_lib.rst
 F: app/test/test_mempool*
 F: app/test/test_func_reentrancy.c
@@ -410,6 +409,7 @@ M: Gage Eads <[email protected]>
 M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
+F: drivers/mempool/stack/
 
 
 Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
index 0444aedad..1681a62bc 100644
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # Headers
 CFLAGS += -I$(RTE_SDK)/lib/librte_mempool
-LDLIBS += -lrte_eal -lrte_mempool -lrte_ring
+LDLIBS += -lrte_eal -lrte_mempool -lrte_stack
 
 EXPORT_MAP := rte_mempool_stack_version.map
 
drivers/mempool/stack/meson.build +5 -1
index b75a3bb56..03e369a41 100644
@@ -1,4 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
+# Copyright(c) 2017-2019 Intel Corporation
+
+allow_experimental_apis = true
 
 sources = files('rte_mempool_stack.c')
+
+deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68

133 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#39 lore raw

PATCH v3 3/8 test/stack: add stack test

In reply to Gage Eads

stack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.

---
 MAINTAINERS           |   1 +
 app/test/Makefile     |   2 +
 app/test/meson.build  |   3 +
 app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 416 insertions(+)
 create mode 100644 app/test/test_stack.c
MAINTAINERS +1 -0
index 4e088d2bd..f6593fa9c 100644
@@ -410,6 +410,7 @@ M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
 F: drivers/mempool/stack/
+F: test/test/*stack*
 
 
 Memory Pool Drivers
app/test/Makefile +2 -0
index 89949c2bb..47cf98a3a 100644
@@ -89,6 +89,8 @@ endif
 
 SRCS-y += test_rwlock.c
 
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
app/test/meson.build +3 -0
index 05e5ddeb0..b00e1201a 100644
@@ -95,6 +95,7 @@ test_sources = files('commands.c',
 	'test_sched.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
+	'test_stack.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -132,6 +133,7 @@ test_deps = ['acl',
 	'port',
 	'reorder',
 	'ring',
+	'stack',
 	'timer'
 ]
 
@@ -173,6 +175,7 @@ fast_parallel_test_names = [
         'rwlock_autotest',
         'sched_autotest',
         'spinlock_autotest',
+        'stack_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
app/test/test_stack.c new file +410 -0

410 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#41 lore raw

PATCH v3 4/8 test/stack: add stack perf test

In reply to Gage Eads

stack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects

It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores

---
 app/test/Makefile          |   1 +
 app/test/meson.build       |   2 +
 app/test/test_stack_perf.c | 343 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 346 insertions(+)
 create mode 100644 app/test/test_stack_perf.c
app/test/Makefile +1 -0
index 47cf98a3a..f9536fb31 100644
@@ -90,6 +90,7 @@ endif
 SRCS-y += test_rwlock.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
app/test/meson.build +2 -0
index b00e1201a..ba3cb6261 100644
@@ -96,6 +96,7 @@ test_sources = files('commands.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
 	'test_stack.c',
+	'test_stack_perf.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -240,6 +241,7 @@ perf_test_names = [
         'distributor_perf_autotest',
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
+        'stack_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
app/test/test_stack_perf.c new file +343 -0

343 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#42 lore raw

PATCH v3 5/8 stack: add lock-free stack implementation

In reply to Gage Eads

This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.

The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.

The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.

The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.

This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.

The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.

---
 doc/guides/prog_guide/stack_lib.rst    |  61 ++++++++++++-
 doc/guides/rel_notes/release_19_05.rst |   3 +
 lib/librte_stack/Makefile              |   3 +-
 lib/librte_stack/meson.build           |   3 +-
 lib/librte_stack/rte_stack.c           |  41 +++++++--
 lib/librte_stack/rte_stack.h           | 127 +++++++++++++++++++++++++--
 lib/librte_stack/rte_stack_generic.h   | 151 +++++++++++++++++++++++++++++++++
 7 files changed, 371 insertions(+), 18 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_generic.h
doc/guides/prog_guide/stack_lib.rst +58 -3
index 25a8cc38a..8fe8804e3 100644
@@ -10,7 +10,8 @@ stack of pointers.
 The stack library provides the following basic operations:
 
 *  Create a uniquely named stack of a user-specified size and using a
-   user-specified socket.
+   user-specified socket, with either standard (lock-based) or lock-free
+   behavior.
 
 *  Push and pop a burst of one or more stack objects (pointers). These function
    are multi-threading safe.
@@ -24,5 +25,59 @@ The stack library provides the following basic operations:
 Implementation
 ~~~~~~~~~~~~~~
 
-The stack consists of a contiguous array of pointers, a current index, and a
-spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
+The library supports two types of stacks: standard (lock-based) and lock-free.
+Both types use the same set of interfaces, but their implementations differ.
+
+Lock-based Stack
+----------------
+
+The lock-based stack consists of a contiguous array of pointers, a current
+index, and a spinlock. Accesses to the stack are made multi-thread safe by the
+spinlock.
+
+Lock-free Stack
+------------------
+
+The lock-free stack consists of a linked list of elements, each containing a
+data pointer and a next pointer, and an atomic stack depth counter. The
+lock-free property means that multiple threads can push and pop simultaneously,
+and one thread being preempted/delayed in a push or pop operation will not
+impede the forward progress of any other thread.
+
+The lock-free push operation enqueues a linked list of pointers by pointing the
+list's tail to the current stack head, and using a CAS to swing the stack head
+pointer to the head of the list. The operation retries if it is unsuccessful
+(i.e. the list changed between reading the head and modifying it), else it
+adjusts the stack length and returns.
+
+The lock-free pop operation first reserves one or more list elements by
+adjusting the stack length, to ensure the dequeue operation will succeed
+without blocking. It then dequeues pointers by walking the list -- starting
+from the head -- then swinging the head pointer (using a CAS as well). While
+walking the list, the data pointers are recorded in an object table.
+
+The linked list elements themselves are maintained in a lock-free LIFO, and are
+allocated before stack pushes and freed after stack pops. Since the stack has a
+fixed maximum depth, these elements do not need to be dynamically created.
+
+The lock-free behavior is selected by passing the *RTE_STACK_F_LF* flag to
+rte_stack_create().
+
+Preventing the ABA Problem
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To prevent the ABA problem, this algorithm stack uses a 128-bit
+compare-and-swap instruction to atomically update both the stack top pointer
+and a modification counter. The ABA problem can occur without a modification
+counter if, for example:
+
+1. Thread A reads head pointer X and stores the pointed-to list element.
+2. Other threads modify the list such that the head pointer is once again X,
+   but its pointed-to data is different than what thread A read.
+3. Thread A changes the head pointer with a compare-and-swap and succeeds.
+
+In this case thread A would not detect that the list had changed, and would
+both pop stale data and incorrect change the head pointer. By adding a
+modification counter that is updated on every push and pop as part of the
+compare-and-swap, the algorithm can detect when the list changes even if the
+head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
index 8c649a954..5294ace3b 100644
@@ -83,6 +83,9 @@ New Features
   pointers. The API provides MT-safe push and pop operations that can operate
   on one or more pointers per operation.
 
+  The library supports two stack implementations: standard (lock-based) and lock-free.
+  The lock-free implementation is currently limited to x86-64 platforms.
+
 Removed Items
 -------------
 
lib/librte_stack/Makefile +2 -1
index e956b6535..3ecddf033 100644
@@ -18,6 +18,7 @@ LIBABIVER := 1
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c
 
 # install includes
-SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h
+SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
+					      rte_stack_generic.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
index 99f43710e..99d7f9ec5 100644
@@ -5,4 +5,5 @@ allow_experimental_apis = true
 
 version = 1
 sources = files('rte_stack.c')
-headers = files('rte_stack.h')
+headers = files('rte_stack.h',
+		'rte_stack_generic.h')
lib/librte_stack/rte_stack.c +33 -8

76 lines not on this page · raw message

lib/librte_stack/rte_stack.h +122 -5

166 lines not on this page · raw message

lib/librte_stack/rte_stack_generic.h new file +151 -0

151 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v3 5/8
#43 lore raw

Re: [PATCH v3 5/8] stack: add lock-free stack implementation

Quoted 23 lines · Gage Eads

This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.

The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.

The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.

The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.

This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.

Quoted 6 lines

The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.

#44 lore raw

Re: [PATCH v3 5/8] stack: add lock-free stack implementation

In reply to Gage Eads

<snip>

Quoted 4 lines
lib/librte_stack/rte_stack.c index renamed from lib/librte_stack/rte_stack.c +0 -0

96dffdf44..8f0361ea1 100644

lib/librte_stack/rte_stack.c +0 -0

<snip>

lib/librte_stack/rte_stack.c rte_stack_create(const char *name, unsigned int · line 86

Quoted 8 lines
lib/librte_stack/rte_stack.c +2 -1
@@ -63,9 +81,16 @@ rte_stack_create(const char *name, unsigned int
 count, int socket_id,
 	unsigned int sz;
 	int ret;
 
-	RTE_SET_USED(flags);
+#ifdef RTE_ARCH_X86_64
+	RTE_BUILD_BUG_ON(sizeof(struct rte_stack_lf_head) != 16);

This check should be independent of the platform.

#else

Quoted 27 lines
lib/librte_stack/rte_stack.c +7 -2
+	if (flags & RTE_STACK_F_LF) {
+		STACK_LOG_ERR("Lock-free stack is not supported on your
 platform\n");
+		return NULL;
+	}
+#endif
 
-	sz = rte_stack_get_memsize(count);
+	sz = rte_stack_get_memsize(count, flags);
 
 	ret = snprintf(mz_name, sizeof(mz_name), "%s%s",
 		       RTE_STACK_MZ_PREFIX, name);
@@ -94,7 +119,7 @@ rte_stack_create(const char *name, unsigned int
 count, int socket_id,
 
 	s = mz->addr;
 
-	rte_stack_init(s);
+	rte_stack_init(s, count, flags);
 
 	/* Store the name for later lookups */

ret = snprintf(s->name, sizeof(s->name), "%s", name); diff --git
a/lib/librte_stack/rte_stack.h b/lib/librte_stack/rte_stack.h index 7a633deb5..b484313bb 100644

lib/librte_stack/rte_stack.h +0 -0
@@ -30,6 +30,35 @@ extern "C" {

<snip>

lib/librte_stack/rte_stack.h struct rte_stack { · line 108

Quoted 13 lines
lib/librte_stack/rte_stack.h +13 -0
+/**
+ * @internal Push several objects on the lock-free stack (MT-safe).
+ *
+ * @param s
+ *   A pointer to the stack structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to push on the stack from the obj_table.
+ * @return
+ *   Actual number of objects enqueued.
+ */
+static __rte_always_inline unsigned int __rte_experimental

This is an internal function. Is '__rte_experimental' tag required? (applies to other instances in this patch)

Quoted 23 lines
lib/librte_stack/rte_stack.h +23 -0
+rte_stack_lf_push(struct rte_stack *s, void * const *obj_table,
+unsigned int n) {
+	struct rte_stack_lf_elem *tmp, *first, *last = NULL;
+	unsigned int i;
+
+	if (unlikely(n == 0))
+		return 0;
+
+	/* Pop n free elements */
+	first = __rte_stack_lf_pop(&s->stack_lf.free, n, NULL, &last);
+	if (unlikely(first == NULL))
+		return 0;
+
+	/* Construct the list elements */
+	for (tmp = first, i = 0; i < n; i++, tmp = tmp->next)
+		tmp->data = obj_table[n - i - 1];
+
+	/* Push them to the used list */
+	__rte_stack_lf_push(&s->stack_lf.used, first, last, n);
+
+	return n;
+}
+

<snip>

lib/librte_stack/rte_stack_generic.h

Quoted 20 lines

/**

lib/librte_stack/rte_stack.h +4 -1
@@ -225,7 +339,10 @@ rte_stack_free_count(struct rte_stack *s)
  *   NUMA. The value can be *SOCKET_ID_ANY* if there is no NUMA
  *   constraint for the reserved zone.
  * @param flags
- *   Reserved for future use.
+ *   An OR of the following:
+ *    - RTE_STACK_F_LF: If this flag is set, the stack uses lock-free
+ *      variants of the push and pop functions. Otherwise, it achieves
+ *      thread-safety using a lock.
  * @return
  *   On success, the pointer to the new allocated stack. NULL on error with
  *    rte_errno set appropriately. Possible errno values include:
lib/librte_stack/rte_stack_generic.h new file +0 -0
new file mode 100644
index 000000000..5e4cbc38e

The name "...stack_generic.h" is confusing. It is implementing LF algorithm.
IMO, the code should be re-organized differently.
rte_stack.h, rte_stack.c - Contain the APIs (calling std or LF based on the flag) and top level structure definition rte_stack_std.c, rte_stack_std.h - Contain the standard implementation
rte_stack_lf.c, rte_stack_lf.h - Contain the LF implementation
rte_stack_lf_c11.h - Contain the LF C11 implementation

Quoted 11 lines
lib/librte_stack/rte_stack_generic.h new file +10 -0
@@ -0,0 +1,151 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_STACK_GENERIC_H_
+#define _RTE_STACK_GENERIC_H_
+
+#include <rte_branch_prediction.h>
+#include <rte_prefetch.h>
+

<snip>

Quoted 64 lines
lib/librte_stack/rte_stack_generic.h new file +46 -0
+
+static __rte_always_inline struct rte_stack_lf_elem *
+__rte_stack_lf_pop(struct rte_stack_lf_list *list,
+		   unsigned int num,
+		   void **obj_table,
+		   struct rte_stack_lf_elem **last)
+{
+#ifndef RTE_ARCH_X86_64
+	RTE_SET_USED(obj_table);
+	RTE_SET_USED(last);
+	RTE_SET_USED(list);
+	RTE_SET_USED(num);
+
+	return NULL;
+#else
+	struct rte_stack_lf_head old_head;
+	int success;
+
+	/* Reserve num elements, if available */
+	while (1) {
+		uint64_t len = rte_atomic64_read(&list->len);
+
+		/* Does the list contain enough elements? */
+		if (unlikely(len < num))
+			return NULL;
+
+		if (rte_atomic64_cmpset((volatile uint64_t *)&list->len,
+					len, len - num))
+			break;
+	}
+
+	old_head = list->head;
+
+	/* Pop num elements */
+	do {
+		struct rte_stack_lf_head new_head;
+		struct rte_stack_lf_elem *tmp;
+		unsigned int i;
+
+		rte_prefetch0(old_head.top);
+
+		tmp = old_head.top;
+
+		/* Traverse the list to find the new head. A next pointer will
+		 * either point to another element or NULL; if a thread
+		 * encounters a pointer that has already been popped, the

CAS

lib/librte_stack/rte_stack_generic.h new file +17 -0
+		 * will fail.
+		 */
+		for (i = 0; i < num && tmp != NULL; i++) {
+			rte_prefetch0(tmp->next);
+			if (obj_table)
+				obj_table[i] = tmp->data;
+			if (last)
+				*last = tmp;
+			tmp = tmp->next;
+		}
+
+		/* If NULL was encountered, the list was modified while
+		 * traversing it. Retry.
+		 */
+		if (i != num)
+			continue;
+

lib/librte_stack/rte_stack_generic.h line 144

Quoted 10 lines
lib/librte_stack/rte_stack_generic.h new file +10 -0
+		new_head.top = tmp;
+		new_head.cnt = old_head.cnt + 1;
+
+		/* old_head is updated on failure */
+		success = rte_atomic128_cmp_exchange(
+				(rte_int128_t *)&list->head,
+				(rte_int128_t *)&old_head,
+				(rte_int128_t *)&new_head,
+				1, __ATOMIC_ACQUIRE,
+				__ATOMIC_ACQUIRE);

Just wondering if 'rte_atomic128_cmp_exchange' for x86 should have compiler barriers based on the memory order passed? C++11 memory model is getting mixed with barrier based model. I think this is something that needs to be discussed at a wider level.

Quoted 9 lines
lib/librte_stack/rte_stack_generic.h new file +7 -1
+	} while (success == 0);
+
+	return old_head.top;
+#endif
+}
+
+#endif /* _RTE_STACK_GENERIC_H_ */
--

2.13.6

author
#45 lore raw

Re: [PATCH v3 5/8] stack: add lock-free stack implementation

Quoted 28 lines

-----Original Message-----
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Thursday, March 28, 2019 6:27 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v3 5/8] stack: add lock-free stack implementation

<snip>

Quoted 5 lines
(quoted diff) +0 -0

b/lib/librte_stack/rte_stack.c index
96dffdf44..8f0361ea1 100644

lib/librte_stack/rte_stack.c +0 -0

<snip>

Quoted 5 lines
lib/librte_stack/rte_stack.c +0 -0
@@ -63,9 +81,16 @@ rte_stack_create(const char *name, unsigned int
 count, int socket_id,
 	unsigned int sz;
 	int ret;
 
Quoted 4 lines

lib/librte_stack/rte_stack.c rte_stack_create(const char *name, unsigned int count, int socket_id, · line 85

Quoted 3 lines
lib/librte_stack/rte_stack.c +2 -1
-	RTE_SET_USED(flags);
+#ifdef RTE_ARCH_X86_64
+	RTE_BUILD_BUG_ON(sizeof(struct rte_stack_lf_head) != 16);

This check should be independent of the platform.

Good catch. Will change the ifdef to RTE_ARCH_64.

[snip]

Quoted 7 lines
Quoted 7 lines
lib/librte_stack/rte_stack.c +7 -0
+/**
+ * @internal Push several objects on the lock-free stack (MT-safe).
+ *
+ * @param s
+ *   A pointer to the stack structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
Quoted 8 lines

lib/librte_stack/rte_stack.c rte_stack_create(const char *name, unsigned int

Quoted 6 lines
lib/librte_stack/rte_stack.c +6 -0
+ * @param n
+ *   The number of objects to push on the stack from the obj_table.
+ * @return
+ *   Actual number of objects enqueued.
+ */
+static __rte_always_inline unsigned int __rte_experimental

This is an internal function. Is '__rte_experimental' tag required? (applies to other instances in this patch)

(Addressed in comments to patch 1/8)

[snip]

Quoted 18 lines
Quoted 18 lines

/**

lib/librte_stack/rte_stack.c +4 -1
@@ -225,7 +339,10 @@ rte_stack_free_count(struct rte_stack *s)
  *   NUMA. The value can be *SOCKET_ID_ANY* if there is no NUMA
  *   constraint for the reserved zone.
  * @param flags
- *   Reserved for future use.
+ *   An OR of the following:
+ *    - RTE_STACK_F_LF: If this flag is set, the stack uses lock-free
+ *      variants of the push and pop functions. Otherwise, it achieves
+ *      thread-safety using a lock.
  * @return
  *   On success, the pointer to the new allocated stack. NULL on error with
  *    rte_errno set appropriately. Possible errno values include:
lib/librte_stack/rte_stack_generic.h new file +0 -0
new file mode 100644
index 000000000..5e4cbc38e
Quoted 9 lines

lib/librte_stack/rte_stack_generic.h

Quoted 2 lines
lib/librte_stack/rte_stack_generic.h new file +0 -0

The name "...stack_generic.h" is confusing. It is implementing LF algorithm. IMO, the code should be re-organized differently.
rte_stack.h, rte_stack.c - Contain the APIs (calling std or LF based on the flag) and top level structure definition rte_stack_std.c, rte_stack_std.h - Contain the standard implementation rte_stack_lf.c, rte_stack_lf.h - Contain the LF implementation rte_stack_lf_c11.h - Contain the LF C11 implementation

'generic' here refers to the "generic API for atomic operations" (generic/rte_atomic.h:12), but I see how that can be misleading.

Yeah, I like this proposal, but with one tweak: use three lock-free header files: rte_stack_lf.h (common inline lock-free functions like rte_stack_lf_pop()), rte_stack_lf_c11.h (C11 implementation), rte_stack_lf_generic.h (generic atomic implementation). Since the name is *_lf_generic.h, it should be clear that it implements the lock-free functions, and this naming matches rte ring's (easier to pick up for those already used to the ring organization).

[snip]

Quoted 11 lines

lib/librte_stack/rte_stack_generic.h line 144

Quoted 7 lines
lib/librte_stack/rte_stack_generic.h new file +7 -0
+		/* old_head is updated on failure */
+		success = rte_atomic128_cmp_exchange(
+				(rte_int128_t *)&list->head,
+				(rte_int128_t *)&old_head,
+				(rte_int128_t *)&new_head,
+				1, __ATOMIC_ACQUIRE,
+				__ATOMIC_ACQUIRE);

Just wondering if 'rte_atomic128_cmp_exchange' for x86 should have compiler barriers based on the memory order passed?
C++11 memory model is getting mixed with barrier based model. I think this is something that needs to be discussed at a wider level.

The x86 implementation uses a compiler barrier (i.e. the inline assembly clobber list contains "memory") regardless of the memory order, so we're (conservatively) adhering to the C11 model, which guarantees ordering at both the compiler and processor levels. Whether/how we relax the x86 implementation (e.g. no compiler barrier if ordering == relaxed) is an interesting question.

Thanks,
Gage

author
#46 lore raw

PATCH v3 6/8 stack: add C11 atomic implementation

In reply to Gage Eads

This commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.

---
 lib/librte_stack/Makefile            |   3 +-
 lib/librte_stack/meson.build         |   3 +-
 lib/librte_stack/rte_stack.h         |   4 +
 lib/librte_stack/rte_stack_c11_mem.h | 175 +++++++++++++++++++++++++++++++++++
 4 files changed, 183 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_c11_mem.h
lib/librte_stack/Makefile +2 -1
index 3ecddf033..94a7c1476 100644
@@ -19,6 +19,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c
 
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
-					      rte_stack_generic.h
+					      rte_stack_generic.h \
+					      rte_stack_c11_mem.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
index 99d7f9ec5..7e2d1dbb8 100644
@@ -6,4 +6,5 @@ allow_experimental_apis = true
 version = 1
 sources = files('rte_stack.c')
 headers = files('rte_stack.h',
-		'rte_stack_generic.h')
+		'rte_stack_generic.h',
+		'rte_stack_c11_mem.h')
lib/librte_stack/rte_stack.h +4 -0
index b484313bb..de16f8fff 100644
@@ -91,7 +91,11 @@ struct rte_stack {
  */
 #define RTE_STACK_F_LF 0x0001
 
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "rte_stack_c11_mem.h"
+#else
 #include "rte_stack_generic.h"
+#endif
 
 /**
  * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_c11_mem.h new file +175 -0

175 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v3 6/8
#47 lore raw

Re: [PATCH v3 6/8] stack: add C11 atomic implementation

Quoted 5 lines · Gage Eads

This commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.

#48 lore raw

Re: [PATCH v3 6/8] stack: add C11 atomic implementation

In reply to Gage Eads
Quoted 102 lines

This commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.

---
 lib/librte_stack/Makefile            |   3 +-
 lib/librte_stack/meson.build         |   3 +-
 lib/librte_stack/rte_stack.h         |   4 +
 lib/librte_stack/rte_stack_c11_mem.h | 175
+++++++++++++++++++++++++++++++++++
 4 files changed, 183 insertions(+), 2 deletions(-)  create mode 100644

lib/librte_stack/rte_stack_c11_mem.h

lib/librte_stack/Makefile index renamed from lib/librte_stack/Makefile +0 -0

3ecddf033..94a7c1476 100644

lib/librte_stack/Makefile +2 -1
@@ -19,6 +19,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c
 
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
-					      rte_stack_generic.h
+					      rte_stack_generic.h \
+					      rte_stack_c11_mem.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build index renamed from lib/librte_stack/meson.build +0 -0

99d7f9ec5..7e2d1dbb8 100644

lib/librte_stack/meson.build +2 -1
@@ -6,4 +6,5 @@ allow_experimental_apis = true  version = 1  sources =
 files('rte_stack.c')  headers = files('rte_stack.h',
-		'rte_stack_generic.h')
+		'rte_stack_generic.h',
+		'rte_stack_c11_mem.h')
lib/librte_stack/rte_stack.h index renamed from lib/librte_stack/rte_stack.h +0 -0

b484313bb..de16f8fff 100644

lib/librte_stack/rte_stack.h +4 -0
@@ -91,7 +91,11 @@ struct rte_stack {
  */
 #define RTE_STACK_F_LF 0x0001
 
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "rte_stack_c11_mem.h"
+#else
 #include "rte_stack_generic.h"
+#endif
 
 /**
  * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_c11_mem.h new file +13 -0
new file mode 100644
index 000000000..44f9ece6e
@@ -0,0 +1,175 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_STACK_C11_MEM_H_
+#define _RTE_STACK_C11_MEM_H_
+
+#include <rte_branch_prediction.h>
+#include <rte_prefetch.h>
+
+static __rte_always_inline unsigned int rte_stack_lf_len(struct
+rte_stack *s) {
+	/* stack_lf_push() and stack_lf_pop() do not update the list's

contents

lib/librte_stack/rte_stack_c11_mem.h new file +5 -0
+	 * and stack_lf->len atomically, which can cause the list to appear
+	 * shorter than it actually is if this function is called while other
+	 * threads are modifying the list.
+	 *
+	 * However, given the inherently approximate nature of the

get_count

lib/librte_stack/rte_stack_c11_mem.h new file +2 -0
+	 * callback -- even if the list and its size were updated atomically,
+	 * the size could change between when get_count executes and when

the

lib/librte_stack/rte_stack_c11_mem.h new file +3 -0
+	 * value is returned to the caller -- this is acceptable.
+	 *
+	 * The stack_lf->len updates are placed such that the list may appear

to

lib/librte_stack/rte_stack_c11_mem.h new file +1 -0
+	 * have fewer elements than it does, but will never appear to have

more

lib/librte_stack/rte_stack_c11_mem.h new file +13 -0
+	 * elements. If the mempool is near-empty to the point that this is a
+	 * concern, the user should consider increasing the mempool size.
+	 */
+	return (unsigned int)__atomic_load_n(&s->stack_lf.used.len.cnt,
+					     __ATOMIC_RELAXED);
+}
+
+static __rte_always_inline void
+__rte_stack_lf_push(struct rte_stack_lf_list *list,
+		    struct rte_stack_lf_elem *first,
+		    struct rte_stack_lf_elem *last,
+		    unsigned int num)
+{

lib/librte_stack/rte_stack_c11_mem.h line 48

Quoted 10 lines
lib/librte_stack/rte_stack_c11_mem.h new file +10 -0
+#ifndef RTE_ARCH_X86_64
+	RTE_SET_USED(first);
+	RTE_SET_USED(last);
+	RTE_SET_USED(list);
+	RTE_SET_USED(num);
+#else
+	struct rte_stack_lf_head old_head;
+	int success;
+
+	old_head = list->head;

This can be a torn read (same as you have mentioned in __rte_stack_lf_pop). I suggest we use acquire thread fence here as well (please see the comments in __rte_stack_lf_pop).

lib/librte_stack/rte_stack_c11_mem.h line 52

Quoted 4 lines
lib/librte_stack/rte_stack_c11_mem.h new file +4 -0
+
+	do {
+		struct rte_stack_lf_head new_head;
+

We can add __atomic_thread_fence(__ATOMIC_ACQUIRE) here (please see the comments in __rte_stack_lf_pop).

Quoted 10 lines
lib/librte_stack/rte_stack_c11_mem.h new file +9 -0
+		/* Swing the top pointer to the first element in the list and
+		 * make the last element point to the old top.
+		 */
+		new_head.top = first;
+		new_head.cnt = old_head.cnt + 1;
+
+		last->next = old_head.top;
+
+		/* Use the release memmodel to ensure the writes to the LF

LIFO

lib/librte_stack/rte_stack_c11_mem.h line 69

Quoted 8 lines
lib/librte_stack/rte_stack_c11_mem.h new file +8 -0
+		 * elements are visible before the head pointer write.
+		 */
+		success = rte_atomic128_cmp_exchange(
+				(rte_int128_t *)&list->head,
+				(rte_int128_t *)&old_head,
+				(rte_int128_t *)&new_head,
+				1, __ATOMIC_RELEASE,
+				__ATOMIC_RELAXED);

Success memory order can be RELAXED as the store to list->len.cnt is RELEASE.

Quoted 19 lines
lib/librte_stack/rte_stack_c11_mem.h new file +6 -0
+	} while (success == 0);
+
+	/* Ensure the stack modifications are not reordered with respect
+	 * to the LIFO len update.
+	 */
+	__atomic_add_fetch(&list->len.cnt, num, __ATOMIC_RELEASE);

#endif }

lib/librte_stack/rte_stack_c11_mem.h new file +12 -0
+
+static __rte_always_inline struct rte_stack_lf_elem *
+__rte_stack_lf_pop(struct rte_stack_lf_list *list,
+		   unsigned int num,
+		   void **obj_table,
+		   struct rte_stack_lf_elem **last)
+{
+#ifndef RTE_ARCH_X86_64
+	RTE_SET_USED(obj_table);
+	RTE_SET_USED(last);
+	RTE_SET_USED(list);
+	RTE_SET_USED(num);

lib/librte_stack/rte_stack_c11_mem.h line 99

Quoted 10 lines
lib/librte_stack/rte_stack_c11_mem.h new file +10 -0
+
+	return NULL;
+#else
+	struct rte_stack_lf_head old_head;
+	int success;
+
+	/* Reserve num elements, if available */
+	while (1) {
+		uint64_t len = __atomic_load_n(&list->len.cnt,
+					       __ATOMIC_ACQUIRE);

This can be outside the loop.

Quoted 19 lines
lib/librte_stack/rte_stack_c11_mem.h new file +14 -0
+
+		/* Does the list contain enough elements? */
+		if (unlikely(len < num))
+			return NULL;
+
+		if (__atomic_compare_exchange_n(&list->len.cnt,
+						&len, len - num,
+						0, __ATOMIC_RELAXED,
+						__ATOMIC_RELAXED))
+			break;
+	}
+
+#ifndef RTE_ARCH_X86_64
+	/* Use the acquire memmodel to ensure the reads to the LF LIFO

elements

lib/librte_stack/rte_stack_c11_mem.h new file +3 -0
+	 * are properly ordered with respect to the head pointer read.
+	 *
+	 * Note that for aarch64, GCC's implementation of __atomic_load_16

in

lib/librte_stack/rte_stack_c11_mem.h line 118

Quoted 2 lines
lib/librte_stack/rte_stack_c11_mem.h new file +2 -0
+	 * libatomic uses locks, and so this function should be replaced by
+	 * a new function (e.g. "rte_atomic128_load()").

aarch64 does not have 'pure' atomic 128b load instructions. They have to be implemented using load/store exclusives.

lib/librte_stack/rte_stack_c11_mem.h line 122

Quoted 4 lines
lib/librte_stack/rte_stack_c11_mem.h new file +4 -0
+	 */
+	__atomic_load((volatile __int128 *)&list->head,
+		      &old_head,
+		      __ATOMIC_ACQUIRE);

Since, we know of x86/aarch64 (power?) that cannot implement pure atomic 128b loads, should we just use relaxed reads and assume the possibility of torn reads for all architectures? Then, we can use acquire fence to prevent the reordering (see below)

lib/librte_stack/rte_stack_c11_mem.h

Quoted 2 lines
lib/librte_stack/rte_stack_c11_mem.h new file +2 -0
+#else
+	/* x86-64 does not require an atomic load here; if a torn read occurs,

IMO, we should not make architecture specific distinctions as this algorithm is based on C11 memory model.

lib/librte_stack/rte_stack_c11_mem.h line 135

Quoted 11 lines
lib/librte_stack/rte_stack_c11_mem.h new file +11 -0
+	 * the CAS will fail and set old_head to the correct/latest value.
+	 */
+	old_head = list->head;
+#endif
+
+	/* Pop num elements */
+	do {
+		struct rte_stack_lf_head new_head;
+		struct rte_stack_lf_elem *tmp;
+		unsigned int i;
+

We can add __atomic_thread_fence(__ATOMIC_ACQUIRE) here.

Quoted 24 lines
lib/librte_stack/rte_stack_c11_mem.h new file +7 -0
+		rte_prefetch0(old_head.top);
+
+		tmp = old_head.top;
+
+		/* Traverse the list to find the new head. A next pointer will
+		 * either point to another element or NULL; if a thread
+		 * encounters a pointer that has already been popped, the

CAS

lib/librte_stack/rte_stack_c11_mem.h new file +16 -0
+		 * will fail.
+		 */
+		for (i = 0; i < num && tmp != NULL; i++) {
+			rte_prefetch0(tmp->next);
+			if (obj_table)
+				obj_table[i] = tmp->data;
+			if (last)
+				*last = tmp;
+			tmp = tmp->next;
+		}
+
+		/* If NULL was encountered, the list was modified while
+		 * traversing it. Retry.
+		 */
+		if (i != num)
+			continue;

lib/librte_stack/rte_stack_c11_mem.h line 168

Quoted 10 lines
lib/librte_stack/rte_stack_c11_mem.h new file +10 -0
+
+		new_head.top = tmp;
+		new_head.cnt = old_head.cnt + 1;
+
+		success = rte_atomic128_cmp_exchange(
+				(rte_int128_t *)&list->head,
+				(rte_int128_t *)&old_head,
+				(rte_int128_t *)&new_head,
+				1, __ATOMIC_ACQUIRE,
+				__ATOMIC_ACQUIRE);

The success order should be __ATOMIC_RELEASE as the write to list->len.cnt is relaxed. The failure order can be __ATOMIC_RELAXED if the thread fence is added.

Quoted 9 lines
lib/librte_stack/rte_stack_c11_mem.h new file +7 -1
+	} while (success == 0);
+
+	return old_head.top;
+#endif
+}
+
+#endif /* _RTE_STACK_C11_MEM_H_ */
--

2.13.6

author
#49 lore raw

Re: [PATCH v3 6/8] stack: add C11 atomic implementation

[snip]

Quoted 11 lines
Quoted 11 lines
lib/librte_stack/rte_stack_c11_mem.h new file +11 -0
+static __rte_always_inline void
+__rte_stack_lf_push(struct rte_stack_lf_list *list,
+		    struct rte_stack_lf_elem *first,
+		    struct rte_stack_lf_elem *last,
+		    unsigned int num)
+{
+#ifndef RTE_ARCH_X86_64
+	RTE_SET_USED(first);
+	RTE_SET_USED(last);
+	RTE_SET_USED(list);
+	RTE_SET_USED(num);
Quoted 8 lines

lib/librte_stack/rte_stack_c11_mem.h line 48

Quoted 5 lines
lib/librte_stack/rte_stack_c11_mem.h new file +5 -0
+#else
+	struct rte_stack_lf_head old_head;
+	int success;
+
+	old_head = list->head;

This can be a torn read (same as you have mentioned in __rte_stack_lf_pop). I suggest we use acquire thread fence here as well (please see the comments in __rte_stack_lf_pop).

Agreed. I'll add the acquire fence.

Quoted 6 lines

lib/librte_stack/rte_stack_c11_mem.h line 52

Quoted 4 lines
lib/librte_stack/rte_stack_c11_mem.h new file +4 -0
+
+	do {
+		struct rte_stack_lf_head new_head;
+

We can add __atomic_thread_fence(__ATOMIC_ACQUIRE) here (please see the comments in __rte_stack_lf_pop).

Will add the fence here.

Quoted 12 lines
Quoted 12 lines
lib/librte_stack/rte_stack_c11_mem.h new file +9 -0
+		/* Swing the top pointer to the first element in the list and
+		 * make the last element point to the old top.
+		 */
+		new_head.top = first;
+		new_head.cnt = old_head.cnt + 1;
+
+		last->next = old_head.top;
+
+		/* Use the release memmodel to ensure the writes to the LF

LIFO

(quoted diff) +2 -0
+		 * elements are visible before the head pointer write.
+		 */
Quoted 8 lines

lib/librte_stack/rte_stack_c11_mem.h line 69

Quoted 6 lines
lib/librte_stack/rte_stack_c11_mem.h new file +6 -0
+		success = rte_atomic128_cmp_exchange(
+				(rte_int128_t *)&list->head,
+				(rte_int128_t *)&old_head,
+				(rte_int128_t *)&new_head,
+				1, __ATOMIC_RELEASE,
+				__ATOMIC_RELAXED);

Success memory order can be RELAXED as the store to list->len.cnt is RELEASE.

The RELEASE success order here ensures that the store to 'last->next' is visible before the head update. The RELEASE in the list->len.cnt store only guarantees that the preceding stores are visible before list->len.cnt's store, but doesn't guarantee any ordering between the 'last->next' store and the head update, so we can't rely on that.

Quoted 14 lines

lib/librte_stack/rte_stack_c11_mem.h

Quoted 13 lines
lib/librte_stack/rte_stack_c11_mem.h new file +6 -0
+	} while (success == 0);
+
+	/* Ensure the stack modifications are not reordered with respect
+	 * to the LIFO len update.
+	 */
+	__atomic_add_fetch(&list->len.cnt, num, __ATOMIC_RELEASE);

#endif }

lib/librte_stack/rte_stack_c11_mem.h new file +6 -0
+
+static __rte_always_inline struct rte_stack_lf_elem *
+__rte_stack_lf_pop(struct rte_stack_lf_list *list,
+		   unsigned int num,
+		   void **obj_table,
+		   struct rte_stack_lf_elem **last) { #ifndef

RTE_ARCH_X86_64

Quoted 15 lines
Quoted 4 lines
lib/librte_stack/rte_stack_c11_mem.h new file +4 -0
+	RTE_SET_USED(obj_table);
+	RTE_SET_USED(last);
+	RTE_SET_USED(list);
+	RTE_SET_USED(num);

lib/librte_stack/rte_stack_c11_mem.h line 99

Quoted 10 lines
lib/librte_stack/rte_stack_c11_mem.h new file +10 -0
+
+	return NULL;
+#else
+	struct rte_stack_lf_head old_head;
+	int success;
+
+	/* Reserve num elements, if available */
+	while (1) {
+		uint64_t len = __atomic_load_n(&list->len.cnt,
+					       __ATOMIC_ACQUIRE);

This can be outside the loop.

Good idea. I'll move this out of the loop and change the atomic_compare_exchange_n's failure memory order to ACQUIRE.

Quoted 24 lines
Quoted 19 lines
lib/librte_stack/rte_stack_c11_mem.h new file +14 -0
+
+		/* Does the list contain enough elements? */
+		if (unlikely(len < num))
+			return NULL;
+
+		if (__atomic_compare_exchange_n(&list->len.cnt,
+						&len, len - num,
+						0, __ATOMIC_RELAXED,
+						__ATOMIC_RELAXED))
+			break;
+	}
+
+#ifndef RTE_ARCH_X86_64
+	/* Use the acquire memmodel to ensure the reads to the LF LIFO

elements

lib/librte_stack/rte_stack_c11_mem.h new file +3 -0
+	 * are properly ordered with respect to the head pointer read.
+	 *
+	 * Note that for aarch64, GCC's implementation of __atomic_load_16

in

lib/librte_stack/rte_stack_c11_mem.h line 118

Quoted 2 lines
lib/librte_stack/rte_stack_c11_mem.h new file +2 -0
+	 * libatomic uses locks, and so this function should be replaced by
+	 * a new function (e.g. "rte_atomic128_load()").

aarch64 does not have 'pure' atomic 128b load instructions. They have to be implemented using load/store exclusives.

Quoted 8 lines

lib/librte_stack/rte_stack_c11_mem.h line 122

Quoted 4 lines
lib/librte_stack/rte_stack_c11_mem.h new file +4 -0
+	 */
+	__atomic_load((volatile __int128 *)&list->head,
+		      &old_head,
+		      __ATOMIC_ACQUIRE);

Since, we know of x86/aarch64 (power?) that cannot implement pure atomic 128b loads, should we just use relaxed reads and assume the possibility of torn reads for all architectures? Then, we can use acquire fence to prevent the reordering (see below)

That's a cleaner solution. I'll implement that, dropping the architecture distinction.

Quoted 6 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+#else
+	/* x86-64 does not require an atomic load here; if a torn read
+occurs,

IMO, we should not make architecture specific distinctions as this algorithm is based on C11 memory model.

Quoted 12 lines

lib/librte_stack/rte_stack_c11_mem.h line 135

Quoted 11 lines
lib/librte_stack/rte_stack_c11_mem.h new file +11 -0
+	 * the CAS will fail and set old_head to the correct/latest value.
+	 */
+	old_head = list->head;
+#endif
+
+	/* Pop num elements */
+	do {
+		struct rte_stack_lf_head new_head;
+		struct rte_stack_lf_elem *tmp;
+		unsigned int i;
+

We can add __atomic_thread_fence(__ATOMIC_ACQUIRE) here.

Will do.

Quoted 29 lines
Quoted 29 lines
lib/librte_stack/rte_stack_c11_mem.h new file +7 -0
+		rte_prefetch0(old_head.top);
+
+		tmp = old_head.top;
+
+		/* Traverse the list to find the new head. A next pointer will
+		 * either point to another element or NULL; if a thread
+		 * encounters a pointer that has already been popped, the

CAS

lib/librte_stack/rte_stack_c11_mem.h new file +21 -0
+		 * will fail.
+		 */
+		for (i = 0; i < num && tmp != NULL; i++) {
+			rte_prefetch0(tmp->next);
+			if (obj_table)
+				obj_table[i] = tmp->data;
+			if (last)
+				*last = tmp;
+			tmp = tmp->next;
+		}
+
+		/* If NULL was encountered, the list was modified while
+		 * traversing it. Retry.
+		 */
+		if (i != num)
+			continue;
+
+		new_head.top = tmp;
+		new_head.cnt = old_head.cnt + 1;
+
+		success = rte_atomic128_cmp_exchange(
Quoted 8 lines

lib/librte_stack/rte_stack_c11_mem.h line 168

Quoted 5 lines
lib/librte_stack/rte_stack_c11_mem.h new file +5 -0
+				(rte_int128_t *)&list->head,
+				(rte_int128_t *)&old_head,
+				(rte_int128_t *)&new_head,
+				1, __ATOMIC_ACQUIRE,
+				__ATOMIC_ACQUIRE);

The success order should be __ATOMIC_RELEASE as the write to list->len.cnt is relaxed.
The failure order can be __ATOMIC_RELAXED if the thread fence is added.

Agreed on both counts. Will address.

Quoted 9 lines
Quoted 9 lines
lib/librte_stack/rte_stack_c11_mem.h new file +7 -1
+	} while (success == 0);
+
+	return old_head.top;
+#endif
+}
+
+#endif /* _RTE_STACK_C11_MEM_H_ */
--

2.13.6

author
#50 lore raw

Re: [PATCH v3 6/8] stack: add C11 atomic implementation

Quoted 36 lines

-----Original Message-----
From: Eads, Gage
Sent: Friday, March 29, 2019 2:25 PM
To: Honnappa Nagarahalli <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>; Thomas Monjalon <[email protected]>
Subject: RE: [PATCH v3 6/8] stack: add C11 atomic implementation

[snip]

Quoted 11 lines
Quoted 11 lines
lib/librte_stack/rte_stack_c11_mem.h new file +11 -0
+static __rte_always_inline void
+__rte_stack_lf_push(struct rte_stack_lf_list *list,
+		    struct rte_stack_lf_elem *first,
+		    struct rte_stack_lf_elem *last,
+		    unsigned int num)
+{
+#ifndef RTE_ARCH_X86_64
+	RTE_SET_USED(first);
+	RTE_SET_USED(last);
+	RTE_SET_USED(list);
+	RTE_SET_USED(num);
Quoted 8 lines

lib/librte_stack/rte_stack_c11_mem.h line 48

Quoted 5 lines
lib/librte_stack/rte_stack_c11_mem.h new file +5 -0
+#else
+	struct rte_stack_lf_head old_head;
+	int success;
+
+	old_head = list->head;

This can be a torn read (same as you have mentioned in __rte_stack_lf_pop). I suggest we use acquire thread fence here as well (please see the comments in __rte_stack_lf_pop).

Quoted 2 lines

Agreed. I'll add the acquire fence.

On second thought, an acquire fence isn't necessary. The acquire fence in __rte_stack_lf_pop() ensures the list->head is ordered before the list element reads. That isn't necessary here; we need to ensure that the last->next write occurs (and is observed) before the list->head write, which the CAS's RELEASE success memorder accomplishes.

If a torn read occurs, the CAS will fail and will atomically re-load &old_head.

#51 lore raw

Re: [PATCH v3 6/8] stack: add C11 atomic implementation

Quoted 27 lines
Quoted 24 lines

Subject: RE: [PATCH v3 6/8] stack: add C11 atomic implementation

[snip]

Quoted 11 lines
Quoted 11 lines
lib/librte_stack/rte_stack_c11_mem.h new file +11 -0
+static __rte_always_inline void
+__rte_stack_lf_push(struct rte_stack_lf_list *list,
+		    struct rte_stack_lf_elem *first,
+		    struct rte_stack_lf_elem *last,
+		    unsigned int num)
+{
+#ifndef RTE_ARCH_X86_64
+	RTE_SET_USED(first);
+	RTE_SET_USED(last);
+	RTE_SET_USED(list);
+	RTE_SET_USED(num);
Quoted 8 lines

lib/librte_stack/rte_stack_c11_mem.h line 48

Quoted 5 lines
lib/librte_stack/rte_stack_c11_mem.h new file +5 -0
+#else
+	struct rte_stack_lf_head old_head;
+	int success;
+
+	old_head = list->head;

This can be a torn read (same as you have mentioned in __rte_stack_lf_pop). I suggest we use acquire thread fence here as well (please see the comments in __rte_stack_lf_pop).

Quoted 2 lines

Agreed. I'll add the acquire fence.

Quoted 7 lines

On second thought, an acquire fence isn't necessary. The acquire fence in __rte_stack_lf_pop() ensures the list->head is ordered before the list element reads. That isn't necessary here; we need to ensure that the last->next write occurs (and is observed) before the list->head write, which the CAS's RELEASE success memorder accomplishes.

If a torn read occurs, the CAS will fail and will atomically re-load &old_head.

Following is my understanding:
The general guideline is there should be a load-acquire for every store-release. In both xxx_lf_pop and xxx_lf_push, the head is store-released, hence the load of the head should be load-acquire.

Quoted 1 line

From the code (for ex: in function _xxx_lf_push), you can notice that there is dependency from 'old_head to new_head to list->head(in compare_exchange)'. When such a dependency exists, if the memory orderings have to be avoided, one needs to use __ATOMIC_CONSUME. Currently, the compilers will use a stronger memory order (which is __ATOMIC_ACQUIRE) as __ATOMIC_CONSUME is not well defined. Please refer to [1] and [2] for more info.

IMO, since, for 128b, we do not have a pure load-acquire, I suggest we use thread_fence with acquire semantics. It is a heavier barrier, but I think it is a safer code which will adhere to C11 memory model.

[1] https://preshing.com/20140709/the-purpose-of-memory_order_consume-in-cpp11/
[2] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0750r1.html

author
#52 lore raw

Re: [PATCH v3 6/8] stack: add C11 atomic implementation

Quoted 62 lines

-----Original Message-----
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Monday, April 1, 2019 2:07 PM
To: Eads, Gage <[email protected]>; '[email protected]' <[email protected]>
Cc: '[email protected]' <[email protected]>; '[email protected]' <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v3 6/8] stack: add C11 atomic implementation

Quoted 27 lines
Quoted 24 lines

Subject: RE: [PATCH v3 6/8] stack: add C11 atomic implementation

[snip]

Quoted 11 lines
Quoted 11 lines
lib/librte_stack/rte_stack_c11_mem.h new file +11 -0
+static __rte_always_inline void __rte_stack_lf_push(struct
+rte_stack_lf_list *list,
+		    struct rte_stack_lf_elem *first,
+		    struct rte_stack_lf_elem *last,
+		    unsigned int num)
+{
+#ifndef RTE_ARCH_X86_64
+	RTE_SET_USED(first);
+	RTE_SET_USED(last);
+	RTE_SET_USED(list);
+	RTE_SET_USED(num);
Quoted 8 lines

lib/librte_stack/rte_stack_c11_mem.h line 48

Quoted 5 lines
lib/librte_stack/rte_stack_c11_mem.h new file +5 -0
+#else
+	struct rte_stack_lf_head old_head;
+	int success;
+
+	old_head = list->head;

This can be a torn read (same as you have mentioned in __rte_stack_lf_pop). I suggest we use acquire thread fence here as well (please see the comments in __rte_stack_lf_pop).

Quoted 2 lines

Agreed. I'll add the acquire fence.

Quoted 8 lines

On second thought, an acquire fence isn't necessary. The acquire fence in
__rte_stack_lf_pop() ensures the list->head is ordered before the list element reads. That isn't necessary here; we need to ensure that the last->next write occurs (and is observed) before the list->head write, which the CAS's RELEASE success memorder accomplishes.

If a torn read occurs, the CAS will fail and will atomically re-load &old_head.

Following is my understanding:
The general guideline is there should be a load-acquire for every store- release. In both xxx_lf_pop and xxx_lf_push, the head is store-released, hence the load of the head should be load-acquire.
From the code (for ex: in function _xxx_lf_push), you can notice that there is dependency from 'old_head to new_head to list->head(in compare_exchange)'. When such a dependency exists, if the memory orderings have to be avoided, one needs to use __ATOMIC_CONSUME. Currently, the compilers will use a stronger memory order (which is __ATOMIC_ACQUIRE) as __ATOMIC_CONSUME is not well defined. Please refer to [1] and [2] for more info.

Quoted 8 lines

IMO, since, for 128b, we do not have a pure load-acquire, I suggest we use thread_fence with acquire semantics. It is a heavier barrier, but I think it is a safer code which will adhere to C11 memory model.

[1] https://preshing.com/20140709/the-purpose-of-
memory_order_consume-in-cpp11/
[2] http://www.open-
std.org/jtc1/sc22/wg21/docs/papers/2018/p0750r1.html

Thanks for those two links, they're good resources.

I agree with your understanding. I admit I'm not fully convinced the synchronized-with relationship is needed between pop's list->head store and push's list->head load (or between push's list->head store and its list->head load), but it's better to err on the side of caution to ensure it's functionally correct...at least until I can manage to convince you :).

I'll send out a V6 with the acquire thread fence.

Thanks,
Gage

author
#53 lore raw

PATCH v3 7/8 test/stack: add lock-free stack tests

In reply to Gage Eads

This commit adds lock-free stack variants of stack_autotest (stack_lf_autotest) and stack_perf_autotest (stack_lf_perf_autotest), which differ only in that the lock-free versions pass the RTE_STACK_F_LF flag to all rte_stack_create() calls.

---
 app/test/meson.build       |  2 ++
 app/test/test_stack.c      | 41 +++++++++++++++++++++++++++--------------
 app/test/test_stack_perf.c | 17 +++++++++++++++--
 3 files changed, 44 insertions(+), 16 deletions(-)
app/test/meson.build +2 -0
index ba3cb6261..474611291 100644
@@ -177,6 +177,7 @@ fast_parallel_test_names = [
         'sched_autotest',
         'spinlock_autotest',
         'stack_autotest',
+        'stack_nb_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
@@ -242,6 +243,7 @@ perf_test_names = [
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
         'stack_perf_autotest',
+        'stack_nb_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
app/test/test_stack.c +27 -14

112 lines not on this page · raw message

app/test/test_stack_perf.c +15 -2
index 484370d30..e09d5384c 100644
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s)
 }
 
 static int
-test_stack_perf(void)
+__test_stack_perf(uint32_t flags)
 {
 	struct lcore_pair cores;
 	struct rte_stack *s;
 
 	rte_atomic32_init(&lcore_barrier);
 
-	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), 0);
+	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), flags);
 	if (s == NULL) {
 		printf("[%s():%u] failed to create a stack\n",
 		       __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)
 	return 0;
 }
 
+static int
+test_stack_perf(void)
+{
+	return __test_stack_perf(0);
+}
+
+static int
+test_lf_stack_perf(void)
+{
+	return __test_stack_perf(RTE_STACK_F_LF);
+}
+
 REGISTER_TEST_COMMAND(stack_perf_autotest, test_stack_perf);
+REGISTER_TEST_COMMAND(stack_lf_perf_autotest, test_lf_stack_perf);
Signature · 1 line
2.13.6
author
#54 lore raw

PATCH v3 8/8 mempool/stack: add lock-free stack mempool handler

In reply to Gage Eads

This commit adds support for lock-free (linked list based) stack mempool handler.

In mempool_perf_autotest the lock-based stack outperforms the lock-free handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a standard (lock-based) stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the lock-free stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.

*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the lock-free stack's.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
 doc/guides/rel_notes/release_19_05.rst          |  5 +++++
 drivers/mempool/stack/rte_mempool_stack.c       | 26 +++++++++++++++++++++++--
 3 files changed, 39 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
index 929d76dba..dbcfc328e 100644
@@ -541,6 +541,16 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
+  Alternatively, applications can use the lock-free stack mempool handler. When
+  considering this handler, note that:
+
+  - It is currently limited to the x86_64 platform, because it uses an
+    instruction (16-byte compare-and-swap) that is not yet available on other
+    platforms.
+  - It has worse average-case performance than the non-preemptive rte_ring, but
+    software caching (e.g. the mempool cache) can mitigate this by reducing the
+    number of stack accesses.
+
 + rte_timer
 
   Running  ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
index 5294ace3b..90e9bbaa6 100644
@@ -86,6 +86,11 @@ New Features
   The library supports two stack implementations: standard (lock-based) and lock-free.
   The lock-free implementation is currently limited to x86-64 platforms.
 
+* **Added Lock-Free Stack Mempool Handler.**
+
+  Added a new lock-free stack handler, which uses the newly added stack
+  library.
+
 Removed Items
 -------------
 
drivers/mempool/stack/rte_mempool_stack.c +24 -2
index 25ccdb9af..7e85c8d6b 100644
@@ -7,7 +7,7 @@
 #include <rte_stack.h>
 
 static int
-stack_alloc(struct rte_mempool *mp)
+__stack_alloc(struct rte_mempool *mp, uint32_t flags)
 {
 	char name[RTE_STACK_NAMESIZE];
 	struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)
 		return -rte_errno;
 	}
 
-	s = rte_stack_create(name, mp->size, mp->socket_id, 0);
+	s = rte_stack_create(name, mp->size, mp->socket_id, flags);
 	if (s == NULL)
 		return -rte_errno;
 
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp)
 }
 
 static int
+stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, 0);
+}
+
+static int
+lf_stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, RTE_STACK_F_LF);
+}
+
+static int
 stack_enqueue(struct rte_mempool *mp, void * const *obj_table,
 	      unsigned int n)
 {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = {
 	.get_count = stack_get_count
 };
 
+static struct rte_mempool_ops ops_lf_stack = {
+	.name = "lf_stack",
+	.alloc = lf_stack_alloc,
+	.free = stack_free,
+	.enqueue = stack_enqueue,
+	.dequeue = stack_dequeue,
+	.get_count = stack_get_count
+};
+
 MEMPOOL_REGISTER_OPS(ops_stack);
+MEMPOOL_REGISTER_OPS(ops_lf_stack);
Signature · 1 line
2.13.6
author
#55 lore raw

v4 cover Add stack library and new mempool handler

In reply to Gage Eads

This patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.

The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.

The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.

This patchset is the successor to a patchset containing only the new mempool handler[2].

[1] http://mails.dpdk.org/archives/dev/2019-March/125751.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html

---
v4:
- Fix test_stack.c 32-bit build by using %zu format specifier for size_t
- Rebase onto master

v3:
- Rebase patchset onto master (test/test/ -> app/test/)
- Fix rte_stack_std_push() segfault introduced in v2

v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset

Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler

 MAINTAINERS                                     |   9 +-
 app/test/Makefile                               |   3 +
 app/test/meson.build                            |   7 +
 app/test/test_stack.c                           | 423 ++++++++++++++++++++++++
 app/test/test_stack_perf.c                      | 356 ++++++++++++++++++++
 config/common_base                              |   5 +
 doc/api/doxy-api-index.md                       |   1 +
 doc/api/doxy-api.conf.in                        |   1 +
 doc/guides/prog_guide/env_abstraction_layer.rst |  10 +
 doc/guides/prog_guide/index.rst                 |   1 +
 doc/guides/prog_guide/stack_lib.rst             |  83 +++++
 doc/guides/rel_notes/release_19_05.rst          |  13 +
 drivers/mempool/stack/Makefile                  |   3 +-
 drivers/mempool/stack/meson.build               |   6 +-
 drivers/mempool/stack/rte_mempool_stack.c       | 115 +++----
 lib/Makefile                                    |   2 +
 lib/librte_stack/Makefile                       |  25 ++
 lib/librte_stack/meson.build                    |  10 +
 lib/librte_stack/rte_stack.c                    | 219 ++++++++++++
 lib/librte_stack/rte_stack.h                    | 395 ++++++++++++++++++++++
 lib/librte_stack/rte_stack_c11_mem.h            | 175 ++++++++++
 lib/librte_stack/rte_stack_generic.h            | 151 +++++++++
 lib/librte_stack/rte_stack_pvt.h                |  34 ++
 lib/librte_stack/rte_stack_version.map          |   9 +
 lib/meson.build                                 |   2 +-
 mk/rte.app.mk                                   |   1 +
 26 files changed, 1987 insertions(+), 72 deletions(-)
 create mode 100644 app/test/test_stack.c
 create mode 100644 app/test/test_stack_perf.c
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_c11_mem.h
 create mode 100644 lib/librte_stack/rte_stack_generic.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
Signature · 1 line
2.13.6
author
#56 lore raw

PATCH v4 1/8 stack: introduce rte stack library

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

---
 MAINTAINERS                            |   6 +
 config/common_base                     |   5 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/index.rst        |   1 +
 doc/guides/prog_guide/stack_lib.rst    |  28 ++++
 doc/guides/rel_notes/release_19_05.rst |   5 +
 lib/Makefile                           |   2 +
 lib/librte_stack/Makefile              |  23 +++
 lib/librte_stack/meson.build           |   8 +
 lib/librte_stack/rte_stack.c           | 194 +++++++++++++++++++++++
 lib/librte_stack/rte_stack.h           | 274 +++++++++++++++++++++++++++++++++
 lib/librte_stack/rte_stack_pvt.h       |  34 ++++
 lib/librte_stack/rte_stack_version.map |   9 ++
 lib/meson.build                        |   2 +-
 mk/rte.app.mk                          |   1 +
 16 files changed, 593 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
index e9ff2b4c2..09fd99dbf 100644
@@ -416,6 +416,12 @@ F: drivers/raw/skeleton_rawdev/
 F: app/test/test_rawdev.c
 F: doc/guides/prog_guide/rawdev.rst
 
+Stack API - EXPERIMENTAL
+M: Gage Eads <[email protected]>
+M: Olivier Matz <[email protected]>
+F: lib/librte_stack/
+F: doc/guides/prog_guide/stack_lib.rst
+
 
 Memory Pool Drivers
 -------------------
config/common_base +5 -0
index 6292bc4af..fc8dba69d 100644
@@ -994,3 +994,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
 # Compile the eventdev application
 #
 CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Compile librte_stack
+#
+CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
index aacc66bd8..de1e215dd 100644
@@ -125,6 +125,7 @@ The public API headers are grouped by topics:
   [mbuf]               (@ref rte_mbuf.h),
   [mbuf pool ops]      (@ref rte_mbuf_pool_ops.h),
   [ring]               (@ref rte_ring.h),
+  [stack]              (@ref rte_stack.h),
   [tailq]              (@ref rte_tailq.h),
   [bitmap]             (@ref rte_bitmap.h)
 
doc/api/doxy-api.conf.in +1 -0
index a365e669b..7722fc3e9 100644
@@ -55,6 +55,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/librte_ring \
                           @TOPDIR@/lib/librte_sched \
                           @TOPDIR@/lib/librte_security \
+                          @TOPDIR@/lib/librte_stack \
                           @TOPDIR@/lib/librte_table \
                           @TOPDIR@/lib/librte_telemetry \
                           @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
index 6726b1e8d..f4f60862f 100644
@@ -55,6 +55,7 @@ Programmer's Guide
     metrics_lib
     bpf_lib
     ipsec_lib
+    stack_lib
     source_org
     dev_kit_build_system
     dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +28 -0
new file mode 100644
index 000000000..25a8cc38a
@@ -0,0 +1,28 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Stack Library
+=============
+
+DPDK's stack library provides an API for configuration and use of a bounded
+stack of pointers.
+
+The stack library provides the following basic operations:
+
+*  Create a uniquely named stack of a user-specified size and using a
+   user-specified socket.
+
+*  Push and pop a burst of one or more stack objects (pointers). These function
+   are multi-threading safe.
+
+*  Free a previously created stack.
+
+*  Lookup a pointer to a stack by its name.
+
+*  Query a stack's current depth and number of free entries.
+
+Implementation
+~~~~~~~~~~~~~~
+
+The stack consists of a contiguous array of pointers, a current index, and a
+spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
index d11bb5a2b..525ae616f 100644
@@ -105,6 +105,11 @@ New Features
   Improved testpmd application performance on ARM platform. For ``macswap``
   forwarding mode, NEON intrinsics were used to do swap to save CPU cycles.
 
+* **Added Stack API.**
+
+  Added a new stack API for configuration and use of a bounded stack of
+  pointers. The API provides MT-safe push and pop operations that can operate
+  on one or more pointers per operation.
 
 Removed Items
 -------------
lib/Makefile +2 -0
index a358f1c19..9f90e80ad 100644
@@ -109,6 +109,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec
 DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security
 DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry
 DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev
+DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack
+DEPDIRS-librte_stack := librte_eal
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +23 -0
new file mode 100644
index 000000000..e956b6535
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_stack.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+LDLIBS += -lrte_eal
+
+EXPORT_MAP := rte_stack_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c
+
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
new file mode 100644
index 000000000..99f43710e
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+allow_experimental_apis = true
+
+version = 1
+sources = files('rte_stack.c')
+headers = files('rte_stack.h')
lib/librte_stack/rte_stack.c new file +194 -0

194 lines not on this page · raw message

lib/librte_stack/rte_stack.h new file +274 -0

274 lines not on this page · raw message

lib/librte_stack/rte_stack_pvt.h new file +34 -0
new file mode 100644
index 000000000..4a6a7bdb3
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_STACK_PVT_H_
+#define _RTE_STACK_PVT_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_log.h>
+
+extern int stack_logtype;
+
+#define STACK_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \
+		__func__, ##args)
+
+#define STACK_LOG_ERR(fmt, args...) \
+	STACK_LOG(ERR, fmt, ## args)
+
+#define STACK_LOG_WARN(fmt, args...) \
+	STACK_LOG(WARNING, fmt, ## args)
+
+#define STACK_LOG_INFO(fmt, args...) \
+	STACK_LOG(INFO, fmt, ## args)
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_version.map new file +9 -0
new file mode 100644
index 000000000..6662679c3
@@ -0,0 +1,9 @@
+EXPERIMENTAL {
+	global:
+
+	rte_stack_create;
+	rte_stack_free;
+	rte_stack_lookup;
+
+	local: *;
+};
lib/meson.build +1 -1
index 99957ba7d..90115477f 100644
@@ -22,7 +22,7 @@ libraries = [
 	'gro', 'gso', 'ip_frag', 'jobstats',
 	'kni', 'latencystats', 'lpm', 'member',
 	'power', 'pdump', 'rawdev',
-	'reorder', 'sched', 'security', 'vhost',
+	'reorder', 'sched', 'security', 'stack', 'vhost',
 	#ipsec lib depends on crypto and security
 	'ipsec',
 	# add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
index 262132fc6..7e033e78c 100644
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -lrte_security
 _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV)         += -lrte_rawdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER)          += -lrte_timer
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL)        += -lrte_mempool
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
Signature · 1 line
2.13.6
author
#57 lore raw

PATCH v4 2/8 mempool/stack: convert mempool to use rte stack

In reply to Gage Eads

The new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.

---
 MAINTAINERS                               |  2 +-
 drivers/mempool/stack/Makefile            |  3 +-
 drivers/mempool/stack/meson.build         |  6 +-
 drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++----------------------
 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
index 09fd99dbf..13fe49e2b 100644
@@ -293,7 +293,6 @@ M: Andrew Rybchenko <[email protected]>
 F: lib/librte_mempool/
 F: drivers/mempool/Makefile
 F: drivers/mempool/ring/
-F: drivers/mempool/stack/
 F: doc/guides/prog_guide/mempool_lib.rst
 F: app/test/test_mempool*
 F: app/test/test_func_reentrancy.c
@@ -421,6 +420,7 @@ M: Gage Eads <[email protected]>
 M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
+F: drivers/mempool/stack/
 
 
 Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
index 0444aedad..1681a62bc 100644
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # Headers
 CFLAGS += -I$(RTE_SDK)/lib/librte_mempool
-LDLIBS += -lrte_eal -lrte_mempool -lrte_ring
+LDLIBS += -lrte_eal -lrte_mempool -lrte_stack
 
 EXPORT_MAP := rte_mempool_stack_version.map
 
drivers/mempool/stack/meson.build +5 -1
index b75a3bb56..03e369a41 100644
@@ -1,4 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
+# Copyright(c) 2017-2019 Intel Corporation
+
+allow_experimental_apis = true
 
 sources = files('rte_mempool_stack.c')
+
+deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68

133 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#58 lore raw

PATCH v4 3/8 test/stack: add stack test

In reply to Gage Eads

stack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.

---
 MAINTAINERS           |   1 +
 app/test/Makefile     |   2 +
 app/test/meson.build  |   3 +
 app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 416 insertions(+)
 create mode 100644 app/test/test_stack.c
MAINTAINERS +1 -0
index 13fe49e2b..2842f07ab 100644
@@ -421,6 +421,7 @@ M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
 F: drivers/mempool/stack/
+F: test/test/*stack*
 
 
 Memory Pool Drivers
app/test/Makefile +2 -0
index d6aa28bad..e5bde81af 100644
@@ -90,6 +90,8 @@ endif
 
 SRCS-y += test_rwlock.c
 
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
app/test/meson.build +3 -0
index ddb4d09ae..29e88106b 100644
@@ -95,6 +95,7 @@ test_sources = files('commands.c',
 	'test_sched.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
+	'test_stack.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -133,6 +134,7 @@ test_deps = ['acl',
 	'port',
 	'reorder',
 	'ring',
+	'stack',
 	'timer'
 ]
 
@@ -174,6 +176,7 @@ fast_parallel_test_names = [
         'rwlock_autotest',
         'sched_autotest',
         'spinlock_autotest',
+        'stack_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
app/test/test_stack.c new file +410 -0

410 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#59 lore raw

PATCH v4 4/8 test/stack: add stack perf test

In reply to Gage Eads

stack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects

It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores

---
 app/test/Makefile          |   1 +
 app/test/meson.build       |   2 +
 app/test/test_stack_perf.c | 343 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 346 insertions(+)
 create mode 100644 app/test/test_stack_perf.c
app/test/Makefile +1 -0
index e5bde81af..b28bed2d4 100644
@@ -91,6 +91,7 @@ endif
 SRCS-y += test_rwlock.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
app/test/meson.build +2 -0
index 29e88106b..2699a3f5b 100644
@@ -96,6 +96,7 @@ test_sources = files('commands.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
 	'test_stack.c',
+	'test_stack_perf.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -241,6 +242,7 @@ perf_test_names = [
         'distributor_perf_autotest',
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
+        'stack_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
app/test/test_stack_perf.c new file +343 -0

343 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#60 lore raw

PATCH v4 5/8 stack: add lock-free stack implementation

In reply to Gage Eads

This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.

The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.

The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.

The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.

This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.

The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.

---
 doc/guides/prog_guide/stack_lib.rst    |  61 ++++++++++++-
 doc/guides/rel_notes/release_19_05.rst |   3 +
 lib/librte_stack/Makefile              |   3 +-
 lib/librte_stack/meson.build           |   3 +-
 lib/librte_stack/rte_stack.c           |  41 +++++++--
 lib/librte_stack/rte_stack.h           | 127 +++++++++++++++++++++++++--
 lib/librte_stack/rte_stack_generic.h   | 151 +++++++++++++++++++++++++++++++++
 7 files changed, 371 insertions(+), 18 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_generic.h
doc/guides/prog_guide/stack_lib.rst +58 -3
index 25a8cc38a..8fe8804e3 100644
@@ -10,7 +10,8 @@ stack of pointers.
 The stack library provides the following basic operations:
 
 *  Create a uniquely named stack of a user-specified size and using a
-   user-specified socket.
+   user-specified socket, with either standard (lock-based) or lock-free
+   behavior.
 
 *  Push and pop a burst of one or more stack objects (pointers). These function
    are multi-threading safe.
@@ -24,5 +25,59 @@ The stack library provides the following basic operations:
 Implementation
 ~~~~~~~~~~~~~~
 
-The stack consists of a contiguous array of pointers, a current index, and a
-spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
+The library supports two types of stacks: standard (lock-based) and lock-free.
+Both types use the same set of interfaces, but their implementations differ.
+
+Lock-based Stack
+----------------
+
+The lock-based stack consists of a contiguous array of pointers, a current
+index, and a spinlock. Accesses to the stack are made multi-thread safe by the
+spinlock.
+
+Lock-free Stack
+------------------
+
+The lock-free stack consists of a linked list of elements, each containing a
+data pointer and a next pointer, and an atomic stack depth counter. The
+lock-free property means that multiple threads can push and pop simultaneously,
+and one thread being preempted/delayed in a push or pop operation will not
+impede the forward progress of any other thread.
+
+The lock-free push operation enqueues a linked list of pointers by pointing the
+list's tail to the current stack head, and using a CAS to swing the stack head
+pointer to the head of the list. The operation retries if it is unsuccessful
+(i.e. the list changed between reading the head and modifying it), else it
+adjusts the stack length and returns.
+
+The lock-free pop operation first reserves one or more list elements by
+adjusting the stack length, to ensure the dequeue operation will succeed
+without blocking. It then dequeues pointers by walking the list -- starting
+from the head -- then swinging the head pointer (using a CAS as well). While
+walking the list, the data pointers are recorded in an object table.
+
+The linked list elements themselves are maintained in a lock-free LIFO, and are
+allocated before stack pushes and freed after stack pops. Since the stack has a
+fixed maximum depth, these elements do not need to be dynamically created.
+
+The lock-free behavior is selected by passing the *RTE_STACK_F_LF* flag to
+rte_stack_create().
+
+Preventing the ABA Problem
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To prevent the ABA problem, this algorithm stack uses a 128-bit
+compare-and-swap instruction to atomically update both the stack top pointer
+and a modification counter. The ABA problem can occur without a modification
+counter if, for example:
+
+1. Thread A reads head pointer X and stores the pointed-to list element.
+2. Other threads modify the list such that the head pointer is once again X,
+   but its pointed-to data is different than what thread A read.
+3. Thread A changes the head pointer with a compare-and-swap and succeeds.
+
+In this case thread A would not detect that the list had changed, and would
+both pop stale data and incorrect change the head pointer. By adding a
+modification counter that is updated on every push and pop as part of the
+compare-and-swap, the algorithm can detect when the list changes even if the
+head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
index 525ae616f..96e851e13 100644
@@ -111,6 +111,9 @@ New Features
   pointers. The API provides MT-safe push and pop operations that can operate
   on one or more pointers per operation.
 
+  The library supports two stack implementations: standard (lock-based) and lock-free.
+  The lock-free implementation is currently limited to x86-64 platforms.
+
 Removed Items
 -------------
 
lib/librte_stack/Makefile +2 -1
index e956b6535..3ecddf033 100644
@@ -18,6 +18,7 @@ LIBABIVER := 1
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c
 
 # install includes
-SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h
+SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
+					      rte_stack_generic.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
index 99f43710e..99d7f9ec5 100644
@@ -5,4 +5,5 @@ allow_experimental_apis = true
 
 version = 1
 sources = files('rte_stack.c')
-headers = files('rte_stack.h')
+headers = files('rte_stack.h',
+		'rte_stack_generic.h')
lib/librte_stack/rte_stack.c +33 -8

76 lines not on this page · raw message

lib/librte_stack/rte_stack.h +122 -5

166 lines not on this page · raw message

lib/librte_stack/rte_stack_generic.h new file +151 -0

151 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#61 lore raw

PATCH v4 6/8 stack: add C11 atomic implementation

In reply to Gage Eads

This commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.

---
 lib/librte_stack/Makefile            |   3 +-
 lib/librte_stack/meson.build         |   3 +-
 lib/librte_stack/rte_stack.h         |   4 +
 lib/librte_stack/rte_stack_c11_mem.h | 175 +++++++++++++++++++++++++++++++++++
 4 files changed, 183 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_c11_mem.h
lib/librte_stack/Makefile +2 -1
index 3ecddf033..94a7c1476 100644
@@ -19,6 +19,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c
 
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
-					      rte_stack_generic.h
+					      rte_stack_generic.h \
+					      rte_stack_c11_mem.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
index 99d7f9ec5..7e2d1dbb8 100644
@@ -6,4 +6,5 @@ allow_experimental_apis = true
 version = 1
 sources = files('rte_stack.c')
 headers = files('rte_stack.h',
-		'rte_stack_generic.h')
+		'rte_stack_generic.h',
+		'rte_stack_c11_mem.h')
lib/librte_stack/rte_stack.h +4 -0
index b484313bb..de16f8fff 100644
@@ -91,7 +91,11 @@ struct rte_stack {
  */
 #define RTE_STACK_F_LF 0x0001
 
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "rte_stack_c11_mem.h"
+#else
 #include "rte_stack_generic.h"
+#endif
 
 /**
  * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_c11_mem.h new file +175 -0

175 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#62 lore raw

PATCH v4 7/8 test/stack: add lock-free stack tests

In reply to Gage Eads

This commit adds lock-free stack variants of stack_autotest (stack_lf_autotest) and stack_perf_autotest (stack_lf_perf_autotest), which differ only in that the lock-free versions pass the RTE_STACK_F_LF flag to all rte_stack_create() calls.

---
 app/test/meson.build       |  2 ++
 app/test/test_stack.c      | 41 +++++++++++++++++++++++++++--------------
 app/test/test_stack_perf.c | 17 +++++++++++++++--
 3 files changed, 44 insertions(+), 16 deletions(-)
app/test/meson.build +2 -0
index 2699a3f5b..57d8a5b55 100644
@@ -178,6 +178,7 @@ fast_parallel_test_names = [
         'sched_autotest',
         'spinlock_autotest',
         'stack_autotest',
+        'stack_nb_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
@@ -243,6 +244,7 @@ perf_test_names = [
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
         'stack_perf_autotest',
+        'stack_nb_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
app/test/test_stack.c +27 -14

112 lines not on this page · raw message

app/test/test_stack_perf.c +15 -2
index 484370d30..e09d5384c 100644
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s)
 }
 
 static int
-test_stack_perf(void)
+__test_stack_perf(uint32_t flags)
 {
 	struct lcore_pair cores;
 	struct rte_stack *s;
 
 	rte_atomic32_init(&lcore_barrier);
 
-	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), 0);
+	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), flags);
 	if (s == NULL) {
 		printf("[%s():%u] failed to create a stack\n",
 		       __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)
 	return 0;
 }
 
+static int
+test_stack_perf(void)
+{
+	return __test_stack_perf(0);
+}
+
+static int
+test_lf_stack_perf(void)
+{
+	return __test_stack_perf(RTE_STACK_F_LF);
+}
+
 REGISTER_TEST_COMMAND(stack_perf_autotest, test_stack_perf);
+REGISTER_TEST_COMMAND(stack_lf_perf_autotest, test_lf_stack_perf);
Signature · 1 line
2.13.6
author
#63 lore raw

PATCH v4 8/8 mempool/stack: add lock-free stack mempool handler

In reply to Gage Eads

This commit adds support for lock-free (linked list based) stack mempool handler.

In mempool_perf_autotest the lock-based stack outperforms the lock-free handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a standard (lock-based) stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the lock-free stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.

*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the lock-free stack's.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
 doc/guides/rel_notes/release_19_05.rst          |  5 +++++
 drivers/mempool/stack/rte_mempool_stack.c       | 26 +++++++++++++++++++++++--
 3 files changed, 39 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
index 2361c3b8f..d22f72f65 100644
@@ -563,6 +563,16 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
+  Alternatively, applications can use the lock-free stack mempool handler. When
+  considering this handler, note that:
+
+  - It is currently limited to the x86_64 platform, because it uses an
+    instruction (16-byte compare-and-swap) that is not yet available on other
+    platforms.
+  - It has worse average-case performance than the non-preemptive rte_ring, but
+    software caching (e.g. the mempool cache) can mitigate this by reducing the
+    number of stack accesses.
+
 + rte_timer
 
   Running  ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
index 96e851e13..9e56d1058 100644
@@ -114,6 +114,11 @@ New Features
   The library supports two stack implementations: standard (lock-based) and lock-free.
   The lock-free implementation is currently limited to x86-64 platforms.
 
+* **Added Lock-Free Stack Mempool Handler.**
+
+  Added a new lock-free stack handler, which uses the newly added stack
+  library.
+
 Removed Items
 -------------
 
drivers/mempool/stack/rte_mempool_stack.c +24 -2
index 25ccdb9af..7e85c8d6b 100644
@@ -7,7 +7,7 @@
 #include <rte_stack.h>
 
 static int
-stack_alloc(struct rte_mempool *mp)
+__stack_alloc(struct rte_mempool *mp, uint32_t flags)
 {
 	char name[RTE_STACK_NAMESIZE];
 	struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)
 		return -rte_errno;
 	}
 
-	s = rte_stack_create(name, mp->size, mp->socket_id, 0);
+	s = rte_stack_create(name, mp->size, mp->socket_id, flags);
 	if (s == NULL)
 		return -rte_errno;
 
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp)
 }
 
 static int
+stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, 0);
+}
+
+static int
+lf_stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, RTE_STACK_F_LF);
+}
+
+static int
 stack_enqueue(struct rte_mempool *mp, void * const *obj_table,
 	      unsigned int n)
 {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = {
 	.get_count = stack_get_count
 };
 
+static struct rte_mempool_ops ops_lf_stack = {
+	.name = "lf_stack",
+	.alloc = lf_stack_alloc,
+	.free = stack_free,
+	.enqueue = stack_enqueue,
+	.dequeue = stack_dequeue,
+	.get_count = stack_get_count
+};
+
 MEMPOOL_REGISTER_OPS(ops_stack);
+MEMPOOL_REGISTER_OPS(ops_lf_stack);
Signature · 1 line
2.13.6
author
#64 lore raw

v5 cover Add stack library and new mempool handler

In reply to Gage Eads

This patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.

The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.

The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.

This patchset is the successor to a patchset containing only the new mempool handler[2].

[1] http://mails.dpdk.org/archives/dev/2019-March/125751.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html

---
v5:
- Add comment to explain padding in *_get_memsize() functions
- Prefix internal functions with '__'
- Use RTE_ASSERT for performance critical run-time checks
- Don't use __atomic_load in the C11 pop_elems function, and put an acquire thread fence at the start of the 2nd do-while loop
- Change pop_elems 128b CAS success memorder to RELEASE and failure memorder to RELAXED
- Change compile-time assertion to run for all 64-bit architectures
- Reorganize the code with standard and lock-free .c and .h files

v4:
- Fix 32-bit build error in test_stack.c by using %zu format specifier for size_t
- Rebase onto master

v3:
- Rebase patchset onto master (test/test/ -> app/test/)
- Fix rte_stack_std_push() segfault introduced in v2

v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset

Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler

 MAINTAINERS                                     |   9 +-
 app/test/Makefile                               |   3 +
 app/test/meson.build                            |   7 +
 app/test/test_stack.c                           | 423 ++++++++++++++++++++++++
 app/test/test_stack_perf.c                      | 356 ++++++++++++++++++++
 config/common_base                              |   5 +
 doc/api/doxy-api-index.md                       |   1 +
 doc/api/doxy-api.conf.in                        |   1 +
 doc/guides/prog_guide/env_abstraction_layer.rst |  10 +
 doc/guides/prog_guide/index.rst                 |   1 +
 doc/guides/prog_guide/stack_lib.rst             |  83 +++++
 doc/guides/rel_notes/release_19_05.rst          |  13 +
 drivers/mempool/stack/Makefile                  |   3 +-
 drivers/mempool/stack/meson.build               |   6 +-
 drivers/mempool/stack/rte_mempool_stack.c       | 115 +++----
 lib/Makefile                                    |   2 +
 lib/librte_stack/Makefile                       |  29 ++
 lib/librte_stack/meson.build                    |  12 +
 lib/librte_stack/rte_stack.c                    | 196 +++++++++++
 lib/librte_stack/rte_stack.h                    | 259 +++++++++++++++
 lib/librte_stack/rte_stack_lf.c                 |  31 ++
 lib/librte_stack/rte_stack_lf.h                 | 106 ++++++
 lib/librte_stack/rte_stack_lf_c11.h             | 169 ++++++++++
 lib/librte_stack/rte_stack_lf_generic.h         | 151 +++++++++
 lib/librte_stack/rte_stack_pvt.h                |  34 ++
 lib/librte_stack/rte_stack_std.c                |  26 ++
 lib/librte_stack/rte_stack_std.h                | 119 +++++++
 lib/librte_stack/rte_stack_version.map          |   9 +
 lib/meson.build                                 |   2 +-
 mk/rte.app.mk                                   |   1 +
 30 files changed, 2110 insertions(+), 72 deletions(-)
 create mode 100644 app/test/test_stack.c
 create mode 100644 app/test/test_stack_perf.c
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_lf.c
 create mode 100644 lib/librte_stack/rte_stack_lf.h
 create mode 100644 lib/librte_stack/rte_stack_lf_c11.h
 create mode 100644 lib/librte_stack/rte_stack_lf_generic.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_std.c
 create mode 100644 lib/librte_stack/rte_stack_std.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
Signature · 1 line
2.13.6
author
#65 lore raw

PATCH v5 1/8 stack: introduce rte stack library

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

---
 MAINTAINERS                            |   6 +
 config/common_base                     |   5 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/index.rst        |   1 +
 doc/guides/prog_guide/stack_lib.rst    |  28 +++++
 doc/guides/rel_notes/release_19_05.rst |   5 +
 lib/Makefile                           |   2 +
 lib/librte_stack/Makefile              |  25 ++++
 lib/librte_stack/meson.build           |   8 ++
 lib/librte_stack/rte_stack.c           | 182 +++++++++++++++++++++++++++++
 lib/librte_stack/rte_stack.h           | 207 +++++++++++++++++++++++++++++++++
 lib/librte_stack/rte_stack_pvt.h       |  34 ++++++
 lib/librte_stack/rte_stack_std.c       |  26 +++++
 lib/librte_stack/rte_stack_std.h       | 119 +++++++++++++++++++
 lib/librte_stack/rte_stack_version.map |   9 ++
 lib/meson.build                        |   2 +-
 mk/rte.app.mk                          |   1 +
 18 files changed, 661 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_std.c
 create mode 100644 lib/librte_stack/rte_stack_std.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
index e9ff2b4c2..09fd99dbf 100644
@@ -416,6 +416,12 @@ F: drivers/raw/skeleton_rawdev/
 F: app/test/test_rawdev.c
 F: doc/guides/prog_guide/rawdev.rst
 
+Stack API - EXPERIMENTAL
+M: Gage Eads <[email protected]>
+M: Olivier Matz <[email protected]>
+F: lib/librte_stack/
+F: doc/guides/prog_guide/stack_lib.rst
+
 
 Memory Pool Drivers
 -------------------
config/common_base +5 -0
index 6292bc4af..fc8dba69d 100644
@@ -994,3 +994,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
 # Compile the eventdev application
 #
 CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Compile librte_stack
+#
+CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
index aacc66bd8..de1e215dd 100644
@@ -125,6 +125,7 @@ The public API headers are grouped by topics:
   [mbuf]               (@ref rte_mbuf.h),
   [mbuf pool ops]      (@ref rte_mbuf_pool_ops.h),
   [ring]               (@ref rte_ring.h),
+  [stack]              (@ref rte_stack.h),
   [tailq]              (@ref rte_tailq.h),
   [bitmap]             (@ref rte_bitmap.h)
 
doc/api/doxy-api.conf.in +1 -0
index a365e669b..7722fc3e9 100644
@@ -55,6 +55,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/librte_ring \
                           @TOPDIR@/lib/librte_sched \
                           @TOPDIR@/lib/librte_security \
+                          @TOPDIR@/lib/librte_stack \
                           @TOPDIR@/lib/librte_table \
                           @TOPDIR@/lib/librte_telemetry \
                           @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
index 6726b1e8d..f4f60862f 100644
@@ -55,6 +55,7 @@ Programmer's Guide
     metrics_lib
     bpf_lib
     ipsec_lib
+    stack_lib
     source_org
     dev_kit_build_system
     dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +28 -0
new file mode 100644
index 000000000..25a8cc38a
@@ -0,0 +1,28 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Stack Library
+=============
+
+DPDK's stack library provides an API for configuration and use of a bounded
+stack of pointers.
+
+The stack library provides the following basic operations:
+
+*  Create a uniquely named stack of a user-specified size and using a
+   user-specified socket.
+
+*  Push and pop a burst of one or more stack objects (pointers). These function
+   are multi-threading safe.
+
+*  Free a previously created stack.
+
+*  Lookup a pointer to a stack by its name.
+
+*  Query a stack's current depth and number of free entries.
+
+Implementation
+~~~~~~~~~~~~~~
+
+The stack consists of a contiguous array of pointers, a current index, and a
+spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
index bdad1ddbe..ebfbe36e5 100644
@@ -121,6 +121,11 @@ New Features
   Improved testpmd application performance on ARM platform. For ``macswap``
   forwarding mode, NEON intrinsics were used to do swap to save CPU cycles.
 
+* **Added Stack API.**
+
+  Added a new stack API for configuration and use of a bounded stack of
+  pointers. The API provides MT-safe push and pop operations that can operate
+  on one or more pointers per operation.
 
 Removed Items
 -------------
lib/Makefile +2 -0
index a358f1c19..9f90e80ad 100644
@@ -109,6 +109,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec
 DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security
 DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry
 DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev
+DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack
+DEPDIRS-librte_stack := librte_eal
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +25 -0
new file mode 100644
index 000000000..6db540073
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_stack.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+LDLIBS += -lrte_eal
+
+EXPORT_MAP := rte_stack_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \
+				   rte_stack_std.c
+
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
+					      rte_stack_std.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
new file mode 100644
index 000000000..d2e60ce9b
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+allow_experimental_apis = true
+
+version = 1
+sources = files('rte_stack.c', 'rte_stack_std.c')
+headers = files('rte_stack.h', 'rte_stack_std.h')
lib/librte_stack/rte_stack.c new file +182 -0

182 lines not on this page · raw message

lib/librte_stack/rte_stack.h new file +207 -0

207 lines not on this page · raw message

lib/librte_stack/rte_stack_pvt.h new file +34 -0
new file mode 100644
index 000000000..4a6a7bdb3
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_STACK_PVT_H_
+#define _RTE_STACK_PVT_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_log.h>
+
+extern int stack_logtype;
+
+#define STACK_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \
+		__func__, ##args)
+
+#define STACK_LOG_ERR(fmt, args...) \
+	STACK_LOG(ERR, fmt, ## args)
+
+#define STACK_LOG_WARN(fmt, args...) \
+	STACK_LOG(WARNING, fmt, ## args)
+
+#define STACK_LOG_INFO(fmt, args...) \
+	STACK_LOG(INFO, fmt, ## args)
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_std.c new file +26 -0
new file mode 100644
index 000000000..0a310d7c6
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include "rte_stack.h"
+
+void
+rte_stack_std_init(struct rte_stack *s)
+{
+	rte_spinlock_init(&s->stack_std.lock);
+}
+
+ssize_t
+rte_stack_std_get_memsize(unsigned int count)
+{
+	ssize_t sz = sizeof(struct rte_stack);
+
+	sz += RTE_CACHE_LINE_ROUNDUP(count * sizeof(void *));
+
+	/* Add padding to avoid false sharing conflicts caused by
+	 * next-line hardware prefetchers.
+	 */
+	sz += 2 * RTE_CACHE_LINE_SIZE;
+
+	return sz;
+}
lib/librte_stack/rte_stack_std.h new file +119 -0

119 lines not on this page · raw message

lib/librte_stack/rte_stack_version.map new file +9 -0
new file mode 100644
index 000000000..6662679c3
@@ -0,0 +1,9 @@
+EXPERIMENTAL {
+	global:
+
+	rte_stack_create;
+	rte_stack_free;
+	rte_stack_lookup;
+
+	local: *;
+};
lib/meson.build +1 -1
index 99957ba7d..90115477f 100644
@@ -22,7 +22,7 @@ libraries = [
 	'gro', 'gso', 'ip_frag', 'jobstats',
 	'kni', 'latencystats', 'lpm', 'member',
 	'power', 'pdump', 'rawdev',
-	'reorder', 'sched', 'security', 'vhost',
+	'reorder', 'sched', 'security', 'stack', 'vhost',
 	#ipsec lib depends on crypto and security
 	'ipsec',
 	# add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
index 262132fc6..7e033e78c 100644
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -lrte_security
 _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV)         += -lrte_rawdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER)          += -lrte_timer
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL)        += -lrte_mempool
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
Signature · 1 line
2.13.6
author
#66 lore raw

PATCH v5 2/8 mempool/stack: convert mempool to use rte stack

In reply to Gage Eads

The new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.

---
 MAINTAINERS                               |  2 +-
 drivers/mempool/stack/Makefile            |  3 +-
 drivers/mempool/stack/meson.build         |  6 +-
 drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++----------------------
 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
index 09fd99dbf..13fe49e2b 100644
@@ -293,7 +293,6 @@ M: Andrew Rybchenko <[email protected]>
 F: lib/librte_mempool/
 F: drivers/mempool/Makefile
 F: drivers/mempool/ring/
-F: drivers/mempool/stack/
 F: doc/guides/prog_guide/mempool_lib.rst
 F: app/test/test_mempool*
 F: app/test/test_func_reentrancy.c
@@ -421,6 +420,7 @@ M: Gage Eads <[email protected]>
 M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
+F: drivers/mempool/stack/
 
 
 Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
index 0444aedad..1681a62bc 100644
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # Headers
 CFLAGS += -I$(RTE_SDK)/lib/librte_mempool
-LDLIBS += -lrte_eal -lrte_mempool -lrte_ring
+LDLIBS += -lrte_eal -lrte_mempool -lrte_stack
 
 EXPORT_MAP := rte_mempool_stack_version.map
 
drivers/mempool/stack/meson.build +5 -1
index b75a3bb56..03e369a41 100644
@@ -1,4 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
+# Copyright(c) 2017-2019 Intel Corporation
+
+allow_experimental_apis = true
 
 sources = files('rte_mempool_stack.c')
+
+deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68

133 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#67 lore raw

PATCH v5 3/8 test/stack: add stack test

In reply to Gage Eads

stack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.

---
 MAINTAINERS           |   1 +
 app/test/Makefile     |   2 +
 app/test/meson.build  |   3 +
 app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 416 insertions(+)
 create mode 100644 app/test/test_stack.c
MAINTAINERS +1 -0
index 13fe49e2b..2842f07ab 100644
@@ -421,6 +421,7 @@ M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
 F: drivers/mempool/stack/
+F: test/test/*stack*
 
 
 Memory Pool Drivers
app/test/Makefile +2 -0
index d6aa28bad..e5bde81af 100644
@@ -90,6 +90,8 @@ endif
 
 SRCS-y += test_rwlock.c
 
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
app/test/meson.build +3 -0
index c5e65fe66..56ea13f53 100644
@@ -95,6 +95,7 @@ test_sources = files('commands.c',
 	'test_sched.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
+	'test_stack.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -133,6 +134,7 @@ test_deps = ['acl',
 	'port',
 	'reorder',
 	'ring',
+	'stack',
 	'timer'
 ]
 
@@ -174,6 +176,7 @@ fast_parallel_test_names = [
         'rwlock_autotest',
         'sched_autotest',
         'spinlock_autotest',
+        'stack_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
app/test/test_stack.c new file +410 -0

410 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#68 lore raw

PATCH v5 4/8 test/stack: add stack perf test

In reply to Gage Eads

stack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects

It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores

---
 app/test/Makefile          |   1 +
 app/test/meson.build       |   2 +
 app/test/test_stack_perf.c | 343 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 346 insertions(+)
 create mode 100644 app/test/test_stack_perf.c
app/test/Makefile +1 -0
index e5bde81af..b28bed2d4 100644
@@ -91,6 +91,7 @@ endif
 SRCS-y += test_rwlock.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
app/test/meson.build +2 -0
index 56ea13f53..02eb788a4 100644
@@ -96,6 +96,7 @@ test_sources = files('commands.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
 	'test_stack.c',
+	'test_stack_perf.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -241,6 +242,7 @@ perf_test_names = [
         'distributor_perf_autotest',
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
+        'stack_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
app/test/test_stack_perf.c new file +343 -0

343 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#69 lore raw

PATCH v5 5/8 stack: add lock-free stack implementation

In reply to Gage Eads

This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.

The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.

The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.

The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.

This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.

The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.

---
 doc/guides/prog_guide/stack_lib.rst     |  61 ++++++++++++-
 doc/guides/rel_notes/release_19_05.rst  |   3 +
 lib/librte_stack/Makefile               |   7 +-
 lib/librte_stack/meson.build            |   7 +-
 lib/librte_stack/rte_stack.c            |  28 ++++--
 lib/librte_stack/rte_stack.h            |  62 +++++++++++--
 lib/librte_stack/rte_stack_lf.c         |  31 +++++++
 lib/librte_stack/rte_stack_lf.h         | 102 +++++++++++++++++++++
 lib/librte_stack/rte_stack_lf_generic.h | 151 ++++++++++++++++++++++++++++++++
 9 files changed, 433 insertions(+), 19 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_lf.c
 create mode 100644 lib/librte_stack/rte_stack_lf.h
 create mode 100644 lib/librte_stack/rte_stack_lf_generic.h
doc/guides/prog_guide/stack_lib.rst +58 -3
index 25a8cc38a..8fe8804e3 100644
@@ -10,7 +10,8 @@ stack of pointers.
 The stack library provides the following basic operations:
 
 *  Create a uniquely named stack of a user-specified size and using a
-   user-specified socket.
+   user-specified socket, with either standard (lock-based) or lock-free
+   behavior.
 
 *  Push and pop a burst of one or more stack objects (pointers). These function
    are multi-threading safe.
@@ -24,5 +25,59 @@ The stack library provides the following basic operations:
 Implementation
 ~~~~~~~~~~~~~~
 
-The stack consists of a contiguous array of pointers, a current index, and a
-spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
+The library supports two types of stacks: standard (lock-based) and lock-free.
+Both types use the same set of interfaces, but their implementations differ.
+
+Lock-based Stack
+----------------
+
+The lock-based stack consists of a contiguous array of pointers, a current
+index, and a spinlock. Accesses to the stack are made multi-thread safe by the
+spinlock.
+
+Lock-free Stack
+------------------
+
+The lock-free stack consists of a linked list of elements, each containing a
+data pointer and a next pointer, and an atomic stack depth counter. The
+lock-free property means that multiple threads can push and pop simultaneously,
+and one thread being preempted/delayed in a push or pop operation will not
+impede the forward progress of any other thread.
+
+The lock-free push operation enqueues a linked list of pointers by pointing the
+list's tail to the current stack head, and using a CAS to swing the stack head
+pointer to the head of the list. The operation retries if it is unsuccessful
+(i.e. the list changed between reading the head and modifying it), else it
+adjusts the stack length and returns.
+
+The lock-free pop operation first reserves one or more list elements by
+adjusting the stack length, to ensure the dequeue operation will succeed
+without blocking. It then dequeues pointers by walking the list -- starting
+from the head -- then swinging the head pointer (using a CAS as well). While
+walking the list, the data pointers are recorded in an object table.
+
+The linked list elements themselves are maintained in a lock-free LIFO, and are
+allocated before stack pushes and freed after stack pops. Since the stack has a
+fixed maximum depth, these elements do not need to be dynamically created.
+
+The lock-free behavior is selected by passing the *RTE_STACK_F_LF* flag to
+rte_stack_create().
+
+Preventing the ABA Problem
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To prevent the ABA problem, this algorithm stack uses a 128-bit
+compare-and-swap instruction to atomically update both the stack top pointer
+and a modification counter. The ABA problem can occur without a modification
+counter if, for example:
+
+1. Thread A reads head pointer X and stores the pointed-to list element.
+2. Other threads modify the list such that the head pointer is once again X,
+   but its pointed-to data is different than what thread A read.
+3. Thread A changes the head pointer with a compare-and-swap and succeeds.
+
+In this case thread A would not detect that the list had changed, and would
+both pop stale data and incorrect change the head pointer. By adding a
+modification counter that is updated on every push and pop as part of the
+compare-and-swap, the algorithm can detect when the list changes even if the
+head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
index ebfbe36e5..3b115b5f6 100644
@@ -127,6 +127,9 @@ New Features
   pointers. The API provides MT-safe push and pop operations that can operate
   on one or more pointers per operation.
 
+  The library supports two stack implementations: standard (lock-based) and lock-free.
+  The lock-free implementation is currently limited to x86-64 platforms.
+
 Removed Items
 -------------
 
lib/librte_stack/Makefile +5 -2
index 6db540073..311edd997 100644
@@ -16,10 +16,13 @@ LIBABIVER := 1
 
 # all source are stored in SRCS-y
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \
-				   rte_stack_std.c
+				   rte_stack_std.c \
+				   rte_stack_lf.c
 
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
-					      rte_stack_std.h
+					      rte_stack_std.h \
+					      rte_stack_lf.h \
+					      rte_stack_lf_generic.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +5 -2
index d2e60ce9b..7a09a5d66 100644
@@ -4,5 +4,8 @@
 allow_experimental_apis = true
 
 version = 1
-sources = files('rte_stack.c', 'rte_stack_std.c')
-headers = files('rte_stack.h', 'rte_stack_std.h')
+sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c')
+headers = files('rte_stack.h',
+		'rte_stack_std.h',
+		'rte_stack_lf.h',
+		'rte_stack_lf_generic.h')
lib/librte_stack/rte_stack.c +21 -7
index 610014b6c..1a4d9bd1e 100644
@@ -25,18 +25,25 @@ static struct rte_tailq_elem rte_stack_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_stack_tailq)
 
+
 static void
-rte_stack_init(struct rte_stack *s)
+rte_stack_init(struct rte_stack *s, unsigned int count, uint32_t flags)
 {
 	memset(s, 0, sizeof(*s));
 
-	rte_stack_std_init(s);
+	if (flags & RTE_STACK_F_LF)
+		rte_stack_lf_init(s, count);
+	else
+		rte_stack_std_init(s);
 }
 
 static ssize_t
-rte_stack_get_memsize(unsigned int count)
+rte_stack_get_memsize(unsigned int count, uint32_t flags)
 {
-	return rte_stack_std_get_memsize(count);
+	if (flags & RTE_STACK_F_LF)
+		return rte_stack_lf_get_memsize(count);
+	else
+		return rte_stack_std_get_memsize(count);
 }
 
 struct rte_stack *
@@ -51,9 +58,16 @@ rte_stack_create(const char *name, unsigned int count, int socket_id,
 	unsigned int sz;
 	int ret;
 
-	RTE_SET_USED(flags);
+#ifdef RTE_ARCH_64
+	RTE_BUILD_BUG_ON(sizeof(struct rte_stack_lf_head) != 16);
+#else
+	if (flags & RTE_STACK_F_LF) {
+		STACK_LOG_ERR("Lock-free stack is not supported on your platform\n");
+		return NULL;
+	}
+#endif
 
-	sz = rte_stack_get_memsize(count);
+	sz = rte_stack_get_memsize(count, flags);
 
 	ret = snprintf(mz_name, sizeof(mz_name), "%s%s",
 		       RTE_STACK_MZ_PREFIX, name);
@@ -82,7 +96,7 @@ rte_stack_create(const char *name, unsigned int count, int socket_id,
 
 	s = mz->addr;
 
-	rte_stack_init(s);
+	rte_stack_init(s, count, flags);
 
 	/* Store the name for later lookups */
 	ret = snprintf(s->name, sizeof(s->name), "%s", name);
lib/librte_stack/rte_stack.h +57 -5

101 lines not on this page · raw message

lib/librte_stack/rte_stack_lf.c new file +31 -0
new file mode 100644
index 000000000..0adcc263e
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include "rte_stack.h"
+
+void
+rte_stack_lf_init(struct rte_stack *s, unsigned int count)
+{
+	struct rte_stack_lf_elem *elems = s->stack_lf.elems;
+	unsigned int i;
+
+	for (i = 0; i < count; i++)
+		__rte_stack_lf_push_elems(&s->stack_lf.free,
+					  &elems[i], &elems[i], 1);
+}
+
+ssize_t
+rte_stack_lf_get_memsize(unsigned int count)
+{
+	ssize_t sz = sizeof(struct rte_stack);
+
+	sz += RTE_CACHE_LINE_ROUNDUP(count * sizeof(struct rte_stack_lf_elem));
+
+	/* Add padding to avoid false sharing conflicts caused by
+	 * next-line hardware prefetchers.
+	 */
+	sz += 2 * RTE_CACHE_LINE_SIZE;
+
+	return sz;
+}
lib/librte_stack/rte_stack_lf.h new file +102 -0

102 lines not on this page · raw message

lib/librte_stack/rte_stack_lf_generic.h new file +151 -0

151 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v5 5/8
#70 lore raw

Re: [PATCH v5 5/8] stack: add lock-free stack implementation

Quoted 25 lines

Subject: [PATCH v5 5/8] stack: add lock-free stack implementation

This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.

The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.

The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.

The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.

This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.

Quoted 8 lines

The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.

---

author
#71 lore raw

PATCH v5 6/8 stack: add C11 atomic implementation

In reply to Gage Eads

This commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.

---
 lib/librte_stack/Makefile           |   3 +-
 lib/librte_stack/meson.build        |   3 +-
 lib/librte_stack/rte_stack_lf.h     |   4 +
 lib/librte_stack/rte_stack_lf_c11.h | 169 ++++++++++++++++++++++++++++++++++++
 4 files changed, 177 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_lf_c11.h
lib/librte_stack/Makefile +2 -1
index 311edd997..8d18ce520 100644
@@ -23,6 +23,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \
 SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
 					      rte_stack_std.h \
 					      rte_stack_lf.h \
-					      rte_stack_lf_generic.h
+					      rte_stack_lf_generic.h \
+					      rte_stack_lf_c11.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
index 7a09a5d66..46fce0c20 100644
@@ -8,4 +8,5 @@ sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c')
 headers = files('rte_stack.h',
 		'rte_stack_std.h',
 		'rte_stack_lf.h',
-		'rte_stack_lf_generic.h')
+		'rte_stack_lf_generic.h',
+		'rte_stack_lf_c11.h')
lib/librte_stack/rte_stack_lf.h +4 -0
index bfd680133..518889a05 100644
@@ -5,7 +5,11 @@
 #ifndef _RTE_STACK_LF_H_
 #define _RTE_STACK_LF_H_
 
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "rte_stack_lf_c11.h"
+#else
 #include "rte_stack_lf_generic.h"
+#endif
 
 /**
  * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_lf_c11.h new file +169 -0

169 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#72 lore raw

PATCH v5 7/8 test/stack: add lock-free stack tests

In reply to Gage Eads

This commit adds lock-free stack variants of stack_autotest (stack_lf_autotest) and stack_perf_autotest (stack_lf_perf_autotest), which differ only in that the lock-free versions pass the RTE_STACK_F_LF flag to all rte_stack_create() calls.

---
 app/test/meson.build       |  2 ++
 app/test/test_stack.c      | 41 +++++++++++++++++++++++++++--------------
 app/test/test_stack_perf.c | 17 +++++++++++++++--
 3 files changed, 44 insertions(+), 16 deletions(-)
app/test/meson.build +2 -0
index 02eb788a4..867cc5863 100644
@@ -178,6 +178,7 @@ fast_parallel_test_names = [
         'sched_autotest',
         'spinlock_autotest',
         'stack_autotest',
+        'stack_nb_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
@@ -243,6 +244,7 @@ perf_test_names = [
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
         'stack_perf_autotest',
+        'stack_nb_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
app/test/test_stack.c +27 -14

112 lines not on this page · raw message

app/test/test_stack_perf.c +15 -2
index 484370d30..e09d5384c 100644
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s)
 }
 
 static int
-test_stack_perf(void)
+__test_stack_perf(uint32_t flags)
 {
 	struct lcore_pair cores;
 	struct rte_stack *s;
 
 	rte_atomic32_init(&lcore_barrier);
 
-	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), 0);
+	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), flags);
 	if (s == NULL) {
 		printf("[%s():%u] failed to create a stack\n",
 		       __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)
 	return 0;
 }
 
+static int
+test_stack_perf(void)
+{
+	return __test_stack_perf(0);
+}
+
+static int
+test_lf_stack_perf(void)
+{
+	return __test_stack_perf(RTE_STACK_F_LF);
+}
+
 REGISTER_TEST_COMMAND(stack_perf_autotest, test_stack_perf);
+REGISTER_TEST_COMMAND(stack_lf_perf_autotest, test_lf_stack_perf);
Signature · 1 line
2.13.6
author
#73 lore raw

PATCH v5 8/8 mempool/stack: add lock-free stack mempool handler

In reply to Gage Eads

This commit adds support for lock-free (linked list based) stack mempool handler.

In mempool_perf_autotest the lock-based stack outperforms the lock-free handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a standard (lock-based) stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the lock-free stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.

*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the lock-free stack's.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
 doc/guides/rel_notes/release_19_05.rst          |  5 +++++
 drivers/mempool/stack/rte_mempool_stack.c       | 26 +++++++++++++++++++++++--
 3 files changed, 39 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
index c1346363b..1a4391898 100644
@@ -563,6 +563,16 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
+  Alternatively, applications can use the lock-free stack mempool handler. When
+  considering this handler, note that:
+
+  - It is currently limited to the x86_64 platform, because it uses an
+    instruction (16-byte compare-and-swap) that is not yet available on other
+    platforms.
+  - It has worse average-case performance than the non-preemptive rte_ring, but
+    software caching (e.g. the mempool cache) can mitigate this by reducing the
+    number of stack accesses.
+
 + rte_timer
 
   Running  ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
index 3b115b5f6..f873984ad 100644
@@ -130,6 +130,11 @@ New Features
   The library supports two stack implementations: standard (lock-based) and lock-free.
   The lock-free implementation is currently limited to x86-64 platforms.
 
+* **Added Lock-Free Stack Mempool Handler.**
+
+  Added a new lock-free stack handler, which uses the newly added stack
+  library.
+
 Removed Items
 -------------
 
drivers/mempool/stack/rte_mempool_stack.c +24 -2
index 25ccdb9af..7e85c8d6b 100644
@@ -7,7 +7,7 @@
 #include <rte_stack.h>
 
 static int
-stack_alloc(struct rte_mempool *mp)
+__stack_alloc(struct rte_mempool *mp, uint32_t flags)
 {
 	char name[RTE_STACK_NAMESIZE];
 	struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)
 		return -rte_errno;
 	}
 
-	s = rte_stack_create(name, mp->size, mp->socket_id, 0);
+	s = rte_stack_create(name, mp->size, mp->socket_id, flags);
 	if (s == NULL)
 		return -rte_errno;
 
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp)
 }
 
 static int
+stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, 0);
+}
+
+static int
+lf_stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, RTE_STACK_F_LF);
+}
+
+static int
 stack_enqueue(struct rte_mempool *mp, void * const *obj_table,
 	      unsigned int n)
 {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = {
 	.get_count = stack_get_count
 };
 
+static struct rte_mempool_ops ops_lf_stack = {
+	.name = "lf_stack",
+	.alloc = lf_stack_alloc,
+	.free = stack_free,
+	.enqueue = stack_enqueue,
+	.dequeue = stack_dequeue,
+	.get_count = stack_get_count
+};
+
 MEMPOOL_REGISTER_OPS(ops_stack);
+MEMPOOL_REGISTER_OPS(ops_lf_stack);
Signature · 1 line
2.13.6
author
#74 lore raw

v6 cover Add stack library and new mempool handler

In reply to Gage Eads

This patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.

The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.

The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.

This patchset is the successor to a patchset containing only the new mempool handler[2].

[1] http://mails.dpdk.org/archives/dev/2019-March/125751.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html

---
v6:
- Add load-acquire fence to the lock-free push function
- Correct generic implementation's pop_elems 128b CAS success and failure memorder to match those in the C11 implementation.

v5:
- Add comment to explain padding in *_get_memsize() functions
- Prefix internal functions with '__'
- Use RTE_ASSERT for performance critical run-time checks
- Don't use __atomic_load in the C11 pop_elems function, and put an acquire thread fence at the start of the 2nd do-while loop
- Change pop_elems 128b CAS success memorder to RELEASE and failure memorder to RELAXED
- Change compile-time assertion to run for all 64-bit architectures
- Reorganize the code with standard and lock-free .c and .h files

v4:
- Fix 32-bit build error in test_stack.c by using %zu format specifier for size_t
- Rebase onto master

v3:
- Rebase patchset onto master (test/test/ -> app/test/)
- Fix rte_stack_std_push() segfault introduced in v2

v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset

Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler

 MAINTAINERS                                     |   9 +-
 app/test/Makefile                               |   3 +
 app/test/meson.build                            |   7 +
 app/test/test_stack.c                           | 423 ++++++++++++++++++++++++
 app/test/test_stack_perf.c                      | 356 ++++++++++++++++++++
 config/common_base                              |   5 +
 doc/api/doxy-api-index.md                       |   1 +
 doc/api/doxy-api.conf.in                        |   1 +
 doc/guides/prog_guide/env_abstraction_layer.rst |  10 +
 doc/guides/prog_guide/index.rst                 |   1 +
 doc/guides/prog_guide/stack_lib.rst             |  83 +++++
 doc/guides/rel_notes/release_19_05.rst          |  13 +
 drivers/mempool/stack/Makefile                  |   3 +-
 drivers/mempool/stack/meson.build               |   6 +-
 drivers/mempool/stack/rte_mempool_stack.c       | 115 +++----
 lib/Makefile                                    |   2 +
 lib/librte_stack/Makefile                       |  29 ++
 lib/librte_stack/meson.build                    |  12 +
 lib/librte_stack/rte_stack.c                    | 196 +++++++++++
 lib/librte_stack/rte_stack.h                    | 259 +++++++++++++++
 lib/librte_stack/rte_stack_lf.c                 |  31 ++
 lib/librte_stack/rte_stack_lf.h                 | 106 ++++++
 lib/librte_stack/rte_stack_lf_c11.h             | 175 ++++++++++
 lib/librte_stack/rte_stack_lf_generic.h         | 164 +++++++++
 lib/librte_stack/rte_stack_pvt.h                |  34 ++
 lib/librte_stack/rte_stack_std.c                |  26 ++
 lib/librte_stack/rte_stack_std.h                | 119 +++++++
 lib/librte_stack/rte_stack_version.map          |   9 +
 lib/meson.build                                 |   2 +-
 mk/rte.app.mk                                   |   1 +
 30 files changed, 2129 insertions(+), 72 deletions(-)
 create mode 100644 app/test/test_stack.c
 create mode 100644 app/test/test_stack_perf.c
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_lf.c
 create mode 100644 lib/librte_stack/rte_stack_lf.h
 create mode 100644 lib/librte_stack/rte_stack_lf_c11.h
 create mode 100644 lib/librte_stack/rte_stack_lf_generic.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_std.c
 create mode 100644 lib/librte_stack/rte_stack_std.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
Signature · 1 line
2.13.6
author
#75 lore raw

PATCH v6 1/8 stack: introduce rte stack library

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

---
 MAINTAINERS                            |   6 +
 config/common_base                     |   5 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/index.rst        |   1 +
 doc/guides/prog_guide/stack_lib.rst    |  28 +++++
 doc/guides/rel_notes/release_19_05.rst |   5 +
 lib/Makefile                           |   2 +
 lib/librte_stack/Makefile              |  25 ++++
 lib/librte_stack/meson.build           |   8 ++
 lib/librte_stack/rte_stack.c           | 182 +++++++++++++++++++++++++++++
 lib/librte_stack/rte_stack.h           | 207 +++++++++++++++++++++++++++++++++
 lib/librte_stack/rte_stack_pvt.h       |  34 ++++++
 lib/librte_stack/rte_stack_std.c       |  26 +++++
 lib/librte_stack/rte_stack_std.h       | 119 +++++++++++++++++++
 lib/librte_stack/rte_stack_version.map |   9 ++
 lib/meson.build                        |   2 +-
 mk/rte.app.mk                          |   1 +
 18 files changed, 661 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_std.c
 create mode 100644 lib/librte_stack/rte_stack_std.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
index e9ff2b4c2..09fd99dbf 100644
@@ -416,6 +416,12 @@ F: drivers/raw/skeleton_rawdev/
 F: app/test/test_rawdev.c
 F: doc/guides/prog_guide/rawdev.rst
 
+Stack API - EXPERIMENTAL
+M: Gage Eads <[email protected]>
+M: Olivier Matz <[email protected]>
+F: lib/librte_stack/
+F: doc/guides/prog_guide/stack_lib.rst
+
 
 Memory Pool Drivers
 -------------------
config/common_base +5 -0
index 6292bc4af..fc8dba69d 100644
@@ -994,3 +994,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
 # Compile the eventdev application
 #
 CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Compile librte_stack
+#
+CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
index aacc66bd8..de1e215dd 100644
@@ -125,6 +125,7 @@ The public API headers are grouped by topics:
   [mbuf]               (@ref rte_mbuf.h),
   [mbuf pool ops]      (@ref rte_mbuf_pool_ops.h),
   [ring]               (@ref rte_ring.h),
+  [stack]              (@ref rte_stack.h),
   [tailq]              (@ref rte_tailq.h),
   [bitmap]             (@ref rte_bitmap.h)
 
doc/api/doxy-api.conf.in +1 -0
index a365e669b..7722fc3e9 100644
@@ -55,6 +55,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/librte_ring \
                           @TOPDIR@/lib/librte_sched \
                           @TOPDIR@/lib/librte_security \
+                          @TOPDIR@/lib/librte_stack \
                           @TOPDIR@/lib/librte_table \
                           @TOPDIR@/lib/librte_telemetry \
                           @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
index 6726b1e8d..f4f60862f 100644
@@ -55,6 +55,7 @@ Programmer's Guide
     metrics_lib
     bpf_lib
     ipsec_lib
+    stack_lib
     source_org
     dev_kit_build_system
     dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +28 -0
new file mode 100644
index 000000000..25a8cc38a
@@ -0,0 +1,28 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Stack Library
+=============
+
+DPDK's stack library provides an API for configuration and use of a bounded
+stack of pointers.
+
+The stack library provides the following basic operations:
+
+*  Create a uniquely named stack of a user-specified size and using a
+   user-specified socket.
+
+*  Push and pop a burst of one or more stack objects (pointers). These function
+   are multi-threading safe.
+
+*  Free a previously created stack.
+
+*  Lookup a pointer to a stack by its name.
+
+*  Query a stack's current depth and number of free entries.
+
+Implementation
+~~~~~~~~~~~~~~
+
+The stack consists of a contiguous array of pointers, a current index, and a
+spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
index bdad1ddbe..ebfbe36e5 100644
@@ -121,6 +121,11 @@ New Features
   Improved testpmd application performance on ARM platform. For ``macswap``
   forwarding mode, NEON intrinsics were used to do swap to save CPU cycles.
 
+* **Added Stack API.**
+
+  Added a new stack API for configuration and use of a bounded stack of
+  pointers. The API provides MT-safe push and pop operations that can operate
+  on one or more pointers per operation.
 
 Removed Items
 -------------
lib/Makefile +2 -0
index a358f1c19..9f90e80ad 100644
@@ -109,6 +109,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec
 DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security
 DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry
 DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev
+DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack
+DEPDIRS-librte_stack := librte_eal
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +25 -0
new file mode 100644
index 000000000..6db540073
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_stack.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+LDLIBS += -lrte_eal
+
+EXPORT_MAP := rte_stack_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \
+				   rte_stack_std.c
+
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
+					      rte_stack_std.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
new file mode 100644
index 000000000..d2e60ce9b
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+allow_experimental_apis = true
+
+version = 1
+sources = files('rte_stack.c', 'rte_stack_std.c')
+headers = files('rte_stack.h', 'rte_stack_std.h')
lib/librte_stack/rte_stack.c new file +182 -0

182 lines not on this page · raw message

lib/librte_stack/rte_stack.h new file +207 -0

207 lines not on this page · raw message

lib/librte_stack/rte_stack_pvt.h new file +34 -0
new file mode 100644
index 000000000..4a6a7bdb3
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_STACK_PVT_H_
+#define _RTE_STACK_PVT_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_log.h>
+
+extern int stack_logtype;
+
+#define STACK_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \
+		__func__, ##args)
+
+#define STACK_LOG_ERR(fmt, args...) \
+	STACK_LOG(ERR, fmt, ## args)
+
+#define STACK_LOG_WARN(fmt, args...) \
+	STACK_LOG(WARNING, fmt, ## args)
+
+#define STACK_LOG_INFO(fmt, args...) \
+	STACK_LOG(INFO, fmt, ## args)
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_std.c new file +26 -0
new file mode 100644
index 000000000..0a310d7c6
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include "rte_stack.h"
+
+void
+rte_stack_std_init(struct rte_stack *s)
+{
+	rte_spinlock_init(&s->stack_std.lock);
+}
+
+ssize_t
+rte_stack_std_get_memsize(unsigned int count)
+{
+	ssize_t sz = sizeof(struct rte_stack);
+
+	sz += RTE_CACHE_LINE_ROUNDUP(count * sizeof(void *));
+
+	/* Add padding to avoid false sharing conflicts caused by
+	 * next-line hardware prefetchers.
+	 */
+	sz += 2 * RTE_CACHE_LINE_SIZE;
+
+	return sz;
+}
lib/librte_stack/rte_stack_std.h new file +119 -0

119 lines not on this page · raw message

lib/librte_stack/rte_stack_version.map new file +9 -0
new file mode 100644
index 000000000..6662679c3
@@ -0,0 +1,9 @@
+EXPERIMENTAL {
+	global:
+
+	rte_stack_create;
+	rte_stack_free;
+	rte_stack_lookup;
+
+	local: *;
+};
lib/meson.build +1 -1
index 99957ba7d..90115477f 100644
@@ -22,7 +22,7 @@ libraries = [
 	'gro', 'gso', 'ip_frag', 'jobstats',
 	'kni', 'latencystats', 'lpm', 'member',
 	'power', 'pdump', 'rawdev',
-	'reorder', 'sched', 'security', 'vhost',
+	'reorder', 'sched', 'security', 'stack', 'vhost',
 	#ipsec lib depends on crypto and security
 	'ipsec',
 	# add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
index 262132fc6..7e033e78c 100644
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -lrte_security
 _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV)         += -lrte_rawdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER)          += -lrte_timer
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL)        += -lrte_mempool
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
Signature · 1 line
2.13.6
Reviewed-by on v6 1/8
#76 lore raw

Re: [PATCH v6 1/8] stack: introduce rte stack library

Quoted 7 lines

Subject: [PATCH v6 1/8] stack: introduce rte stack library

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

Quoted 8 lines

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

---

<snip>

#77 lore raw

Re: [PATCH v6 1/8] stack: introduce rte stack library

02/04/2019 13:14, Honnappa Nagarahalli:

Quoted 12 lines
Quoted 12 lines

Subject: [PATCH v6 1/8] stack: introduce rte stack library

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

Quoted 4 lines
Quoted 3 lines

---

It does not compile for Arm:

lib/librte_stack/rte_stack_std.h:
In function '__rte_stack_std_pop':
lib/librte_stack/rte_stack_std.h:68:6: error:
implicit declaration of function 'unlikely'

author
#78 lore raw

Re: [PATCH v6 1/8] stack: introduce rte stack library

Quoted 16 lines

02/04/2019 13:14, Honnappa Nagarahalli:

Quoted 15 lines
Quoted 15 lines

Subject: [PATCH v6 1/8] stack: introduce rte stack library

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

---

Quoted 8 lines
Quoted 1 line

It does not compile for Arm:

lib/librte_stack/rte_stack_std.h:
In function '__rte_stack_std_pop':
lib/librte_stack/rte_stack_std.h:68:6: error:
implicit declaration of function 'unlikely'

Missing rte_branch_prediction.h include -- I'll fix and resubmit. Thanks for checking the non-x86 builds.

I can hold off resubmission until the 128-bit CAS patch is merged, so this series is properly tested in the automated build + test pipeline, if you'd prefer.

Thanks,
Gage

#79 lore raw

Re: [PATCH v6 1/8] stack: introduce rte stack library

03/04/2019 19:13, Eads, Gage:

Quoted 20 lines
Quoted 20 lines

02/04/2019 13:14, Honnappa Nagarahalli:

Quoted 12 lines
Quoted 12 lines

Subject: [PATCH v6 1/8] stack: introduce rte stack library

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

Quoted 4 lines
Quoted 3 lines

---

It does not compile for Arm:

Quoted 8 lines
Quoted 4 lines

lib/librte_stack/rte_stack_std.h:
In function '__rte_stack_std_pop':
lib/librte_stack/rte_stack_std.h:68:6: error:
implicit declaration of function 'unlikely'

Missing rte_branch_prediction.h include -- I'll fix and resubmit. Thanks for checking the non-x86 builds.

I can hold off resubmission until the 128-bit CAS patch is merged, so this series is properly tested in the automated build + test pipeline, if you'd prefer.

Yes, would be the best.
But we need to merge everything tomorrow at last.

author
#80 lore raw

PATCH v6 2/8 mempool/stack: convert mempool to use rte stack

In reply to Gage Eads

The new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.

---
 MAINTAINERS                               |  2 +-
 drivers/mempool/stack/Makefile            |  3 +-
 drivers/mempool/stack/meson.build         |  6 +-
 drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++----------------------
 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
index 09fd99dbf..13fe49e2b 100644
@@ -293,7 +293,6 @@ M: Andrew Rybchenko <[email protected]>
 F: lib/librte_mempool/
 F: drivers/mempool/Makefile
 F: drivers/mempool/ring/
-F: drivers/mempool/stack/
 F: doc/guides/prog_guide/mempool_lib.rst
 F: app/test/test_mempool*
 F: app/test/test_func_reentrancy.c
@@ -421,6 +420,7 @@ M: Gage Eads <[email protected]>
 M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
+F: drivers/mempool/stack/
 
 
 Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
index 0444aedad..1681a62bc 100644
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # Headers
 CFLAGS += -I$(RTE_SDK)/lib/librte_mempool
-LDLIBS += -lrte_eal -lrte_mempool -lrte_ring
+LDLIBS += -lrte_eal -lrte_mempool -lrte_stack
 
 EXPORT_MAP := rte_mempool_stack_version.map
 
drivers/mempool/stack/meson.build +5 -1
index b75a3bb56..03e369a41 100644
@@ -1,4 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
+# Copyright(c) 2017-2019 Intel Corporation
+
+allow_experimental_apis = true
 
 sources = files('rte_mempool_stack.c')
+
+deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68

133 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#81 lore raw

PATCH v6 3/8 test/stack: add stack test

In reply to Gage Eads

stack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.

---
 MAINTAINERS           |   1 +
 app/test/Makefile     |   2 +
 app/test/meson.build  |   3 +
 app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 416 insertions(+)
 create mode 100644 app/test/test_stack.c
MAINTAINERS +1 -0
index 13fe49e2b..2842f07ab 100644
@@ -421,6 +421,7 @@ M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
 F: drivers/mempool/stack/
+F: test/test/*stack*
 
 
 Memory Pool Drivers
app/test/Makefile +2 -0
index d6aa28bad..e5bde81af 100644
@@ -90,6 +90,8 @@ endif
 
 SRCS-y += test_rwlock.c
 
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
app/test/meson.build +3 -0
index c5e65fe66..56ea13f53 100644
@@ -95,6 +95,7 @@ test_sources = files('commands.c',
 	'test_sched.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
+	'test_stack.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -133,6 +134,7 @@ test_deps = ['acl',
 	'port',
 	'reorder',
 	'ring',
+	'stack',
 	'timer'
 ]
 
@@ -174,6 +176,7 @@ fast_parallel_test_names = [
         'rwlock_autotest',
         'sched_autotest',
         'spinlock_autotest',
+        'stack_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
app/test/test_stack.c new file +410 -0

410 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#82 lore raw

PATCH v6 4/8 test/stack: add stack perf test

In reply to Gage Eads

stack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects

It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores

---
 app/test/Makefile          |   1 +
 app/test/meson.build       |   2 +
 app/test/test_stack_perf.c | 343 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 346 insertions(+)
 create mode 100644 app/test/test_stack_perf.c
app/test/Makefile +1 -0
index e5bde81af..b28bed2d4 100644
@@ -91,6 +91,7 @@ endif
 SRCS-y += test_rwlock.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
app/test/meson.build +2 -0
index 56ea13f53..02eb788a4 100644
@@ -96,6 +96,7 @@ test_sources = files('commands.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
 	'test_stack.c',
+	'test_stack_perf.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -241,6 +242,7 @@ perf_test_names = [
         'distributor_perf_autotest',
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
+        'stack_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
app/test/test_stack_perf.c new file +343 -0

343 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#83 lore raw

PATCH v6 5/8 stack: add lock-free stack implementation

In reply to Gage Eads

This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.

The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.

The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.

The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.

This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.

The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.

---
 doc/guides/prog_guide/stack_lib.rst     |  61 +++++++++++-
 doc/guides/rel_notes/release_19_05.rst  |   3 +
 lib/librte_stack/Makefile               |   7 +-
 lib/librte_stack/meson.build            |   7 +-
 lib/librte_stack/rte_stack.c            |  28 ++++--
 lib/librte_stack/rte_stack.h            |  62 +++++++++++-
 lib/librte_stack/rte_stack_lf.c         |  31 ++++++
 lib/librte_stack/rte_stack_lf.h         | 102 ++++++++++++++++++++
 lib/librte_stack/rte_stack_lf_generic.h | 164 ++++++++++++++++++++++++++++++++
 9 files changed, 446 insertions(+), 19 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_lf.c
 create mode 100644 lib/librte_stack/rte_stack_lf.h
 create mode 100644 lib/librte_stack/rte_stack_lf_generic.h
doc/guides/prog_guide/stack_lib.rst +58 -3
index 25a8cc38a..8fe8804e3 100644
@@ -10,7 +10,8 @@ stack of pointers.
 The stack library provides the following basic operations:
 
 *  Create a uniquely named stack of a user-specified size and using a
-   user-specified socket.
+   user-specified socket, with either standard (lock-based) or lock-free
+   behavior.
 
 *  Push and pop a burst of one or more stack objects (pointers). These function
    are multi-threading safe.
@@ -24,5 +25,59 @@ The stack library provides the following basic operations:
 Implementation
 ~~~~~~~~~~~~~~
 
-The stack consists of a contiguous array of pointers, a current index, and a
-spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
+The library supports two types of stacks: standard (lock-based) and lock-free.
+Both types use the same set of interfaces, but their implementations differ.
+
+Lock-based Stack
+----------------
+
+The lock-based stack consists of a contiguous array of pointers, a current
+index, and a spinlock. Accesses to the stack are made multi-thread safe by the
+spinlock.
+
+Lock-free Stack
+------------------
+
+The lock-free stack consists of a linked list of elements, each containing a
+data pointer and a next pointer, and an atomic stack depth counter. The
+lock-free property means that multiple threads can push and pop simultaneously,
+and one thread being preempted/delayed in a push or pop operation will not
+impede the forward progress of any other thread.
+
+The lock-free push operation enqueues a linked list of pointers by pointing the
+list's tail to the current stack head, and using a CAS to swing the stack head
+pointer to the head of the list. The operation retries if it is unsuccessful
+(i.e. the list changed between reading the head and modifying it), else it
+adjusts the stack length and returns.
+
+The lock-free pop operation first reserves one or more list elements by
+adjusting the stack length, to ensure the dequeue operation will succeed
+without blocking. It then dequeues pointers by walking the list -- starting
+from the head -- then swinging the head pointer (using a CAS as well). While
+walking the list, the data pointers are recorded in an object table.
+
+The linked list elements themselves are maintained in a lock-free LIFO, and are
+allocated before stack pushes and freed after stack pops. Since the stack has a
+fixed maximum depth, these elements do not need to be dynamically created.
+
+The lock-free behavior is selected by passing the *RTE_STACK_F_LF* flag to
+rte_stack_create().
+
+Preventing the ABA Problem
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To prevent the ABA problem, this algorithm stack uses a 128-bit
+compare-and-swap instruction to atomically update both the stack top pointer
+and a modification counter. The ABA problem can occur without a modification
+counter if, for example:
+
+1. Thread A reads head pointer X and stores the pointed-to list element.
+2. Other threads modify the list such that the head pointer is once again X,
+   but its pointed-to data is different than what thread A read.
+3. Thread A changes the head pointer with a compare-and-swap and succeeds.
+
+In this case thread A would not detect that the list had changed, and would
+both pop stale data and incorrect change the head pointer. By adding a
+modification counter that is updated on every push and pop as part of the
+compare-and-swap, the algorithm can detect when the list changes even if the
+head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
index ebfbe36e5..3b115b5f6 100644
@@ -127,6 +127,9 @@ New Features
   pointers. The API provides MT-safe push and pop operations that can operate
   on one or more pointers per operation.
 
+  The library supports two stack implementations: standard (lock-based) and lock-free.
+  The lock-free implementation is currently limited to x86-64 platforms.
+
 Removed Items
 -------------
 
lib/librte_stack/Makefile +5 -2
index 6db540073..311edd997 100644
@@ -16,10 +16,13 @@ LIBABIVER := 1
 
 # all source are stored in SRCS-y
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \
-				   rte_stack_std.c
+				   rte_stack_std.c \
+				   rte_stack_lf.c
 
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
-					      rte_stack_std.h
+					      rte_stack_std.h \
+					      rte_stack_lf.h \
+					      rte_stack_lf_generic.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +5 -2
index d2e60ce9b..7a09a5d66 100644
@@ -4,5 +4,8 @@
 allow_experimental_apis = true
 
 version = 1
-sources = files('rte_stack.c', 'rte_stack_std.c')
-headers = files('rte_stack.h', 'rte_stack_std.h')
+sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c')
+headers = files('rte_stack.h',
+		'rte_stack_std.h',
+		'rte_stack_lf.h',
+		'rte_stack_lf_generic.h')
lib/librte_stack/rte_stack.c +21 -7
index 610014b6c..1a4d9bd1e 100644
@@ -25,18 +25,25 @@ static struct rte_tailq_elem rte_stack_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_stack_tailq)
 
+
 static void
-rte_stack_init(struct rte_stack *s)
+rte_stack_init(struct rte_stack *s, unsigned int count, uint32_t flags)
 {
 	memset(s, 0, sizeof(*s));
 
-	rte_stack_std_init(s);
+	if (flags & RTE_STACK_F_LF)
+		rte_stack_lf_init(s, count);
+	else
+		rte_stack_std_init(s);
 }
 
 static ssize_t
-rte_stack_get_memsize(unsigned int count)
+rte_stack_get_memsize(unsigned int count, uint32_t flags)
 {
-	return rte_stack_std_get_memsize(count);
+	if (flags & RTE_STACK_F_LF)
+		return rte_stack_lf_get_memsize(count);
+	else
+		return rte_stack_std_get_memsize(count);
 }
 
 struct rte_stack *
@@ -51,9 +58,16 @@ rte_stack_create(const char *name, unsigned int count, int socket_id,
 	unsigned int sz;
 	int ret;
 
-	RTE_SET_USED(flags);
+#ifdef RTE_ARCH_64
+	RTE_BUILD_BUG_ON(sizeof(struct rte_stack_lf_head) != 16);
+#else
+	if (flags & RTE_STACK_F_LF) {
+		STACK_LOG_ERR("Lock-free stack is not supported on your platform\n");
+		return NULL;
+	}
+#endif
 
-	sz = rte_stack_get_memsize(count);
+	sz = rte_stack_get_memsize(count, flags);
 
 	ret = snprintf(mz_name, sizeof(mz_name), "%s%s",
 		       RTE_STACK_MZ_PREFIX, name);
@@ -82,7 +96,7 @@ rte_stack_create(const char *name, unsigned int count, int socket_id,
 
 	s = mz->addr;
 
-	rte_stack_init(s);
+	rte_stack_init(s, count, flags);
 
 	/* Store the name for later lookups */
 	ret = snprintf(s->name, sizeof(s->name), "%s", name);
lib/librte_stack/rte_stack.h +57 -5

101 lines not on this page · raw message

lib/librte_stack/rte_stack_lf.c new file +31 -0
new file mode 100644
index 000000000..0adcc263e
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include "rte_stack.h"
+
+void
+rte_stack_lf_init(struct rte_stack *s, unsigned int count)
+{
+	struct rte_stack_lf_elem *elems = s->stack_lf.elems;
+	unsigned int i;
+
+	for (i = 0; i < count; i++)
+		__rte_stack_lf_push_elems(&s->stack_lf.free,
+					  &elems[i], &elems[i], 1);
+}
+
+ssize_t
+rte_stack_lf_get_memsize(unsigned int count)
+{
+	ssize_t sz = sizeof(struct rte_stack);
+
+	sz += RTE_CACHE_LINE_ROUNDUP(count * sizeof(struct rte_stack_lf_elem));
+
+	/* Add padding to avoid false sharing conflicts caused by
+	 * next-line hardware prefetchers.
+	 */
+	sz += 2 * RTE_CACHE_LINE_SIZE;
+
+	return sz;
+}
lib/librte_stack/rte_stack_lf.h new file +102 -0

102 lines not on this page · raw message

lib/librte_stack/rte_stack_lf_generic.h new file +164 -0

164 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#84 lore raw

PATCH v6 6/8 stack: add C11 atomic implementation

In reply to Gage Eads

This commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.

---
 lib/librte_stack/Makefile           |   3 +-
 lib/librte_stack/meson.build        |   3 +-
 lib/librte_stack/rte_stack_lf.h     |   4 +
 lib/librte_stack/rte_stack_lf_c11.h | 175 ++++++++++++++++++++++++++++++++++++
 4 files changed, 183 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_lf_c11.h
lib/librte_stack/Makefile +2 -1
index 311edd997..8d18ce520 100644
@@ -23,6 +23,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \
 SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
 					      rte_stack_std.h \
 					      rte_stack_lf.h \
-					      rte_stack_lf_generic.h
+					      rte_stack_lf_generic.h \
+					      rte_stack_lf_c11.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
index 7a09a5d66..46fce0c20 100644
@@ -8,4 +8,5 @@ sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c')
 headers = files('rte_stack.h',
 		'rte_stack_std.h',
 		'rte_stack_lf.h',
-		'rte_stack_lf_generic.h')
+		'rte_stack_lf_generic.h',
+		'rte_stack_lf_c11.h')
lib/librte_stack/rte_stack_lf.h +4 -0
index bfd680133..518889a05 100644
@@ -5,7 +5,11 @@
 #ifndef _RTE_STACK_LF_H_
 #define _RTE_STACK_LF_H_
 
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "rte_stack_lf_c11.h"
+#else
 #include "rte_stack_lf_generic.h"
+#endif
 
 /**
  * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_lf_c11.h new file +175 -0

175 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v6 6/8
#85 lore raw

Re: [PATCH v6 6/8] stack: add C11 atomic implementation

Quoted 9 lines

Subject: [PATCH v6 6/8] stack: add C11 atomic implementation

This commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.

---

<snip>

author
#86 lore raw

PATCH v6 7/8 test/stack: add lock-free stack tests

In reply to Gage Eads

This commit adds lock-free stack variants of stack_autotest (stack_lf_autotest) and stack_perf_autotest (stack_lf_perf_autotest), which differ only in that the lock-free versions pass the RTE_STACK_F_LF flag to all rte_stack_create() calls.

---
 app/test/meson.build       |  2 ++
 app/test/test_stack.c      | 41 +++++++++++++++++++++++++++--------------
 app/test/test_stack_perf.c | 17 +++++++++++++++--
 3 files changed, 44 insertions(+), 16 deletions(-)
app/test/meson.build +2 -0
index 02eb788a4..867cc5863 100644
@@ -178,6 +178,7 @@ fast_parallel_test_names = [
         'sched_autotest',
         'spinlock_autotest',
         'stack_autotest',
+        'stack_nb_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
@@ -243,6 +244,7 @@ perf_test_names = [
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
         'stack_perf_autotest',
+        'stack_nb_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
app/test/test_stack.c +27 -14

112 lines not on this page · raw message

app/test/test_stack_perf.c +15 -2
index 484370d30..e09d5384c 100644
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s)
 }
 
 static int
-test_stack_perf(void)
+__test_stack_perf(uint32_t flags)
 {
 	struct lcore_pair cores;
 	struct rte_stack *s;
 
 	rte_atomic32_init(&lcore_barrier);
 
-	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), 0);
+	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), flags);
 	if (s == NULL) {
 		printf("[%s():%u] failed to create a stack\n",
 		       __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)
 	return 0;
 }
 
+static int
+test_stack_perf(void)
+{
+	return __test_stack_perf(0);
+}
+
+static int
+test_lf_stack_perf(void)
+{
+	return __test_stack_perf(RTE_STACK_F_LF);
+}
+
 REGISTER_TEST_COMMAND(stack_perf_autotest, test_stack_perf);
+REGISTER_TEST_COMMAND(stack_lf_perf_autotest, test_lf_stack_perf);
Signature · 1 line
2.13.6
author
#87 lore raw

PATCH v6 8/8 mempool/stack: add lock-free stack mempool handler

In reply to Gage Eads

This commit adds support for lock-free (linked list based) stack mempool handler.

In mempool_perf_autotest the lock-based stack outperforms the lock-free handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a standard (lock-based) stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the lock-free stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.

*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the lock-free stack's.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
 doc/guides/rel_notes/release_19_05.rst          |  5 +++++
 drivers/mempool/stack/rte_mempool_stack.c       | 26 +++++++++++++++++++++++--
 3 files changed, 39 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
index c1346363b..1a4391898 100644
@@ -563,6 +563,16 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
+  Alternatively, applications can use the lock-free stack mempool handler. When
+  considering this handler, note that:
+
+  - It is currently limited to the x86_64 platform, because it uses an
+    instruction (16-byte compare-and-swap) that is not yet available on other
+    platforms.
+  - It has worse average-case performance than the non-preemptive rte_ring, but
+    software caching (e.g. the mempool cache) can mitigate this by reducing the
+    number of stack accesses.
+
 + rte_timer
 
   Running  ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
index 3b115b5f6..f873984ad 100644
@@ -130,6 +130,11 @@ New Features
   The library supports two stack implementations: standard (lock-based) and lock-free.
   The lock-free implementation is currently limited to x86-64 platforms.
 
+* **Added Lock-Free Stack Mempool Handler.**
+
+  Added a new lock-free stack handler, which uses the newly added stack
+  library.
+
 Removed Items
 -------------
 
drivers/mempool/stack/rte_mempool_stack.c +24 -2
index 25ccdb9af..7e85c8d6b 100644
@@ -7,7 +7,7 @@
 #include <rte_stack.h>
 
 static int
-stack_alloc(struct rte_mempool *mp)
+__stack_alloc(struct rte_mempool *mp, uint32_t flags)
 {
 	char name[RTE_STACK_NAMESIZE];
 	struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)
 		return -rte_errno;
 	}
 
-	s = rte_stack_create(name, mp->size, mp->socket_id, 0);
+	s = rte_stack_create(name, mp->size, mp->socket_id, flags);
 	if (s == NULL)
 		return -rte_errno;
 
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp)
 }
 
 static int
+stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, 0);
+}
+
+static int
+lf_stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, RTE_STACK_F_LF);
+}
+
+static int
 stack_enqueue(struct rte_mempool *mp, void * const *obj_table,
 	      unsigned int n)
 {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = {
 	.get_count = stack_get_count
 };
 
+static struct rte_mempool_ops ops_lf_stack = {
+	.name = "lf_stack",
+	.alloc = lf_stack_alloc,
+	.free = stack_free,
+	.enqueue = stack_enqueue,
+	.dequeue = stack_dequeue,
+	.get_count = stack_get_count
+};
+
 MEMPOOL_REGISTER_OPS(ops_stack);
+MEMPOOL_REGISTER_OPS(ops_lf_stack);
Signature · 1 line
2.13.6
#88 lore raw

Re: [PATCH v6 0/8] Add stack library and new mempool handler

In reply to Gage Eads

01/04/2019 23:14, Gage Eads:

Quoted 2 lines

Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.

I'm waiting for an update of the 128-bit compare-and-swap. It is blocking the integration of this patch.

author
#89 lore raw

Re: [PATCH v6 0/8] Add stack library and new mempool handler

Quoted 7 lines

01/04/2019 23:14, Gage Eads:

Quoted 2 lines

Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.

I'm waiting for an update of the 128-bit compare-and-swap. It is blocking the integration of this patch.

Sorry for that; I misunderstood your earlier comment on that patch. I'll address it and re-submit.

Thanks,
Gage

author
#90 lore raw

v7 cover Add stack library and new mempool handler

In reply to Gage Eads

This patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.

The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.

The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.

This patchset is the successor to a patchset containing only the new mempool handler[2].

[1] http://mails.dpdk.org/archives/dev/2019-April/129014.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html

---
v7:
- Add rte_branch_prediction.h include to rte_stack_std.h for unlikely()
- Add rte_compat.h include to rte_stack.h for __rte_experimental

v6:
- Add load-acquire fence to the lock-free push function
- Correct generic implementation's pop_elems 128b CAS success and failure memorder to match those in the C11 implementation.

v5:
- Add comment to explain padding in *_get_memsize() functions
- Prefix internal functions with '__'
- Use RTE_ASSERT for performance critical run-time checks
- Don't use __atomic_load in the C11 pop_elems function, and put an acquire thread fence at the start of the 2nd do-while loop
- Change pop_elems 128b CAS success memorder to RELEASE and failure memorder to RELAXED
- Change compile-time assertion to run for all 64-bit architectures
- Reorganize the code with standard and lock-free .c and .h files

v4:
- Fix 32-bit build error in test_stack.c by using %zu format specifier for size_t
- Rebase onto master

v3:
- Rebase patchset onto master (test/test/ -> app/test/)
- Fix rte_stack_std_push() segfault introduced in v2

v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset

Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler

 MAINTAINERS                                     |   9 +-
 app/test/Makefile                               |   3 +
 app/test/meson.build                            |   7 +
 app/test/test_stack.c                           | 423 ++++++++++++++++++++++++
 app/test/test_stack_perf.c                      | 356 ++++++++++++++++++++
 config/common_base                              |   5 +
 doc/api/doxy-api-index.md                       |   1 +
 doc/api/doxy-api.conf.in                        |   1 +
 doc/guides/prog_guide/env_abstraction_layer.rst |  10 +
 doc/guides/prog_guide/index.rst                 |   1 +
 doc/guides/prog_guide/stack_lib.rst             |  83 +++++
 doc/guides/rel_notes/release_19_05.rst          |  13 +
 drivers/mempool/stack/Makefile                  |   3 +-
 drivers/mempool/stack/meson.build               |   6 +-
 drivers/mempool/stack/rte_mempool_stack.c       | 115 +++----
 lib/Makefile                                    |   2 +
 lib/librte_stack/Makefile                       |  29 ++
 lib/librte_stack/meson.build                    |  12 +
 lib/librte_stack/rte_stack.c                    | 196 +++++++++++
 lib/librte_stack/rte_stack.h                    | 260 +++++++++++++++
 lib/librte_stack/rte_stack_lf.c                 |  31 ++
 lib/librte_stack/rte_stack_lf.h                 | 106 ++++++
 lib/librte_stack/rte_stack_lf_c11.h             | 175 ++++++++++
 lib/librte_stack/rte_stack_lf_generic.h         | 164 +++++++++
 lib/librte_stack/rte_stack_pvt.h                |  34 ++
 lib/librte_stack/rte_stack_std.c                |  26 ++
 lib/librte_stack/rte_stack_std.h                | 121 +++++++
 lib/librte_stack/rte_stack_version.map          |   9 +
 lib/meson.build                                 |   2 +-
 mk/rte.app.mk                                   |   1 +
 30 files changed, 2132 insertions(+), 72 deletions(-)
 create mode 100644 app/test/test_stack.c
 create mode 100644 app/test/test_stack_perf.c
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_lf.c
 create mode 100644 lib/librte_stack/rte_stack_lf.h
 create mode 100644 lib/librte_stack/rte_stack_lf_c11.h
 create mode 100644 lib/librte_stack/rte_stack_lf_generic.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_std.c
 create mode 100644 lib/librte_stack/rte_stack_std.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
Signature · 1 line
2.13.6
author
#91 lore raw

PATCH v7 1/8 stack: introduce rte stack library

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

---
 MAINTAINERS                            |   6 +
 config/common_base                     |   5 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/index.rst        |   1 +
 doc/guides/prog_guide/stack_lib.rst    |  28 +++++
 doc/guides/rel_notes/release_19_05.rst |   5 +
 lib/Makefile                           |   2 +
 lib/librte_stack/Makefile              |  25 ++++
 lib/librte_stack/meson.build           |   8 ++
 lib/librte_stack/rte_stack.c           | 182 +++++++++++++++++++++++++++++
 lib/librte_stack/rte_stack.h           | 208 +++++++++++++++++++++++++++++++++
 lib/librte_stack/rte_stack_pvt.h       |  34 ++++++
 lib/librte_stack/rte_stack_std.c       |  26 +++++
 lib/librte_stack/rte_stack_std.h       | 121 +++++++++++++++++++
 lib/librte_stack/rte_stack_version.map |   9 ++
 lib/meson.build                        |   2 +-
 mk/rte.app.mk                          |   1 +
 18 files changed, 664 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_std.c
 create mode 100644 lib/librte_stack/rte_stack_std.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
index 71ac8cd4b..f30fc4aa6 100644
@@ -426,6 +426,12 @@ F: drivers/raw/skeleton_rawdev/
 F: app/test/test_rawdev.c
 F: doc/guides/prog_guide/rawdev.rst
 
+Stack API - EXPERIMENTAL
+M: Gage Eads <[email protected]>
+M: Olivier Matz <[email protected]>
+F: lib/librte_stack/
+F: doc/guides/prog_guide/stack_lib.rst
+
 
 Memory Pool Drivers
 -------------------
config/common_base +5 -0
index 6292bc4af..fc8dba69d 100644
@@ -994,3 +994,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
 # Compile the eventdev application
 #
 CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Compile librte_stack
+#
+CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
index aacc66bd8..de1e215dd 100644
@@ -125,6 +125,7 @@ The public API headers are grouped by topics:
   [mbuf]               (@ref rte_mbuf.h),
   [mbuf pool ops]      (@ref rte_mbuf_pool_ops.h),
   [ring]               (@ref rte_ring.h),
+  [stack]              (@ref rte_stack.h),
   [tailq]              (@ref rte_tailq.h),
   [bitmap]             (@ref rte_bitmap.h)
 
doc/api/doxy-api.conf.in +1 -0
index a365e669b..7722fc3e9 100644
@@ -55,6 +55,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/librte_ring \
                           @TOPDIR@/lib/librte_sched \
                           @TOPDIR@/lib/librte_security \
+                          @TOPDIR@/lib/librte_stack \
                           @TOPDIR@/lib/librte_table \
                           @TOPDIR@/lib/librte_telemetry \
                           @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
index 6726b1e8d..f4f60862f 100644
@@ -55,6 +55,7 @@ Programmer's Guide
     metrics_lib
     bpf_lib
     ipsec_lib
+    stack_lib
     source_org
     dev_kit_build_system
     dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +28 -0
new file mode 100644
index 000000000..25a8cc38a
@@ -0,0 +1,28 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Stack Library
+=============
+
+DPDK's stack library provides an API for configuration and use of a bounded
+stack of pointers.
+
+The stack library provides the following basic operations:
+
+*  Create a uniquely named stack of a user-specified size and using a
+   user-specified socket.
+
+*  Push and pop a burst of one or more stack objects (pointers). These function
+   are multi-threading safe.
+
+*  Free a previously created stack.
+
+*  Lookup a pointer to a stack by its name.
+
+*  Query a stack's current depth and number of free entries.
+
+Implementation
+~~~~~~~~~~~~~~
+
+The stack consists of a contiguous array of pointers, a current index, and a
+spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
index bdad1ddbe..ebfbe36e5 100644
@@ -121,6 +121,11 @@ New Features
   Improved testpmd application performance on ARM platform. For ``macswap``
   forwarding mode, NEON intrinsics were used to do swap to save CPU cycles.
 
+* **Added Stack API.**
+
+  Added a new stack API for configuration and use of a bounded stack of
+  pointers. The API provides MT-safe push and pop operations that can operate
+  on one or more pointers per operation.
 
 Removed Items
 -------------
lib/Makefile +2 -0
index a358f1c19..9f90e80ad 100644
@@ -109,6 +109,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec
 DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security
 DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry
 DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev
+DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack
+DEPDIRS-librte_stack := librte_eal
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +25 -0
new file mode 100644
index 000000000..6db540073
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_stack.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+LDLIBS += -lrte_eal
+
+EXPORT_MAP := rte_stack_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \
+				   rte_stack_std.c
+
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
+					      rte_stack_std.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
new file mode 100644
index 000000000..d2e60ce9b
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+allow_experimental_apis = true
+
+version = 1
+sources = files('rte_stack.c', 'rte_stack_std.c')
+headers = files('rte_stack.h', 'rte_stack_std.h')
lib/librte_stack/rte_stack.c new file +182 -0

182 lines not on this page · raw message

lib/librte_stack/rte_stack.h new file +208 -0

208 lines not on this page · raw message

lib/librte_stack/rte_stack_pvt.h new file +34 -0
new file mode 100644
index 000000000..4a6a7bdb3
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_STACK_PVT_H_
+#define _RTE_STACK_PVT_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_log.h>
+
+extern int stack_logtype;
+
+#define STACK_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \
+		__func__, ##args)
+
+#define STACK_LOG_ERR(fmt, args...) \
+	STACK_LOG(ERR, fmt, ## args)
+
+#define STACK_LOG_WARN(fmt, args...) \
+	STACK_LOG(WARNING, fmt, ## args)
+
+#define STACK_LOG_INFO(fmt, args...) \
+	STACK_LOG(INFO, fmt, ## args)
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_std.c new file +26 -0
new file mode 100644
index 000000000..0a310d7c6
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include "rte_stack.h"
+
+void
+rte_stack_std_init(struct rte_stack *s)
+{
+	rte_spinlock_init(&s->stack_std.lock);
+}
+
+ssize_t
+rte_stack_std_get_memsize(unsigned int count)
+{
+	ssize_t sz = sizeof(struct rte_stack);
+
+	sz += RTE_CACHE_LINE_ROUNDUP(count * sizeof(void *));
+
+	/* Add padding to avoid false sharing conflicts caused by
+	 * next-line hardware prefetchers.
+	 */
+	sz += 2 * RTE_CACHE_LINE_SIZE;
+
+	return sz;
+}
lib/librte_stack/rte_stack_std.h new file +121 -0

121 lines not on this page · raw message

lib/librte_stack/rte_stack_version.map new file +9 -0
new file mode 100644
index 000000000..6662679c3
@@ -0,0 +1,9 @@
+EXPERIMENTAL {
+	global:
+
+	rte_stack_create;
+	rte_stack_free;
+	rte_stack_lookup;
+
+	local: *;
+};
lib/meson.build +1 -1
index c3289f885..595314d7d 100644
@@ -22,7 +22,7 @@ libraries = [
 	'gro', 'gso', 'ip_frag', 'jobstats',
 	'kni', 'latencystats', 'lpm', 'member',
 	'power', 'pdump', 'rawdev',
-	'reorder', 'sched', 'security', 'vhost',
+	'reorder', 'sched', 'security', 'stack', 'vhost',
 	#ipsec lib depends on crypto and security
 	'ipsec',
 	# add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
index 262132fc6..7e033e78c 100644
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -lrte_security
 _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV)         += -lrte_rawdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER)          += -lrte_timer
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL)        += -lrte_mempool
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
Signature · 1 line
2.13.6
author
#92 lore raw

PATCH v7 2/8 mempool/stack: convert mempool to use rte stack

In reply to Gage Eads

The new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.

---
 MAINTAINERS                               |  2 +-
 drivers/mempool/stack/Makefile            |  3 +-
 drivers/mempool/stack/meson.build         |  6 +-
 drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++----------------------
 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
index f30fc4aa6..e09e7d93f 100644
@@ -303,7 +303,6 @@ M: Andrew Rybchenko <[email protected]>
 F: lib/librte_mempool/
 F: drivers/mempool/Makefile
 F: drivers/mempool/ring/
-F: drivers/mempool/stack/
 F: doc/guides/prog_guide/mempool_lib.rst
 F: app/test/test_mempool*
 F: app/test/test_func_reentrancy.c
@@ -431,6 +430,7 @@ M: Gage Eads <[email protected]>
 M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
+F: drivers/mempool/stack/
 
 
 Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
index 0444aedad..1681a62bc 100644
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # Headers
 CFLAGS += -I$(RTE_SDK)/lib/librte_mempool
-LDLIBS += -lrte_eal -lrte_mempool -lrte_ring
+LDLIBS += -lrte_eal -lrte_mempool -lrte_stack
 
 EXPORT_MAP := rte_mempool_stack_version.map
 
drivers/mempool/stack/meson.build +5 -1
index b75a3bb56..03e369a41 100644
@@ -1,4 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
+# Copyright(c) 2017-2019 Intel Corporation
+
+allow_experimental_apis = true
 
 sources = files('rte_mempool_stack.c')
+
+deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68

133 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#93 lore raw

PATCH v7 3/8 test/stack: add stack test

In reply to Gage Eads

stack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.

---
 MAINTAINERS           |   1 +
 app/test/Makefile     |   2 +
 app/test/meson.build  |   3 +
 app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 416 insertions(+)
 create mode 100644 app/test/test_stack.c
MAINTAINERS +1 -0
index e09e7d93f..e4e6d1b15 100644
@@ -431,6 +431,7 @@ M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
 F: drivers/mempool/stack/
+F: test/test/*stack*
 
 
 Memory Pool Drivers
app/test/Makefile +2 -0
index d6aa28bad..e5bde81af 100644
@@ -90,6 +90,8 @@ endif
 
 SRCS-y += test_rwlock.c
 
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
app/test/meson.build +3 -0
index c5e65fe66..56ea13f53 100644
@@ -95,6 +95,7 @@ test_sources = files('commands.c',
 	'test_sched.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
+	'test_stack.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -133,6 +134,7 @@ test_deps = ['acl',
 	'port',
 	'reorder',
 	'ring',
+	'stack',
 	'timer'
 ]
 
@@ -174,6 +176,7 @@ fast_parallel_test_names = [
         'rwlock_autotest',
         'sched_autotest',
         'spinlock_autotest',
+        'stack_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
app/test/test_stack.c new file +410 -0

410 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#94 lore raw

PATCH v7 4/8 test/stack: add stack perf test

In reply to Gage Eads

stack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects

It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores

---
 app/test/Makefile          |   1 +
 app/test/meson.build       |   2 +
 app/test/test_stack_perf.c | 343 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 346 insertions(+)
 create mode 100644 app/test/test_stack_perf.c
app/test/Makefile +1 -0
index e5bde81af..b28bed2d4 100644
@@ -91,6 +91,7 @@ endif
 SRCS-y += test_rwlock.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
app/test/meson.build +2 -0
index 56ea13f53..02eb788a4 100644
@@ -96,6 +96,7 @@ test_sources = files('commands.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
 	'test_stack.c',
+	'test_stack_perf.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -241,6 +242,7 @@ perf_test_names = [
         'distributor_perf_autotest',
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
+        'stack_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
app/test/test_stack_perf.c new file +343 -0

343 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#95 lore raw

PATCH v7 5/8 stack: add lock-free stack implementation

In reply to Gage Eads

This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.

The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.

The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.

The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.

This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.

The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.

---
 doc/guides/prog_guide/stack_lib.rst     |  61 +++++++++++-
 doc/guides/rel_notes/release_19_05.rst  |   3 +
 lib/librte_stack/Makefile               |   7 +-
 lib/librte_stack/meson.build            |   7 +-
 lib/librte_stack/rte_stack.c            |  28 ++++--
 lib/librte_stack/rte_stack.h            |  62 +++++++++++-
 lib/librte_stack/rte_stack_lf.c         |  31 ++++++
 lib/librte_stack/rte_stack_lf.h         | 102 ++++++++++++++++++++
 lib/librte_stack/rte_stack_lf_generic.h | 164 ++++++++++++++++++++++++++++++++
 9 files changed, 446 insertions(+), 19 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_lf.c
 create mode 100644 lib/librte_stack/rte_stack_lf.h
 create mode 100644 lib/librte_stack/rte_stack_lf_generic.h
doc/guides/prog_guide/stack_lib.rst +58 -3
index 25a8cc38a..8fe8804e3 100644
@@ -10,7 +10,8 @@ stack of pointers.
 The stack library provides the following basic operations:
 
 *  Create a uniquely named stack of a user-specified size and using a
-   user-specified socket.
+   user-specified socket, with either standard (lock-based) or lock-free
+   behavior.
 
 *  Push and pop a burst of one or more stack objects (pointers). These function
    are multi-threading safe.
@@ -24,5 +25,59 @@ The stack library provides the following basic operations:
 Implementation
 ~~~~~~~~~~~~~~
 
-The stack consists of a contiguous array of pointers, a current index, and a
-spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
+The library supports two types of stacks: standard (lock-based) and lock-free.
+Both types use the same set of interfaces, but their implementations differ.
+
+Lock-based Stack
+----------------
+
+The lock-based stack consists of a contiguous array of pointers, a current
+index, and a spinlock. Accesses to the stack are made multi-thread safe by the
+spinlock.
+
+Lock-free Stack
+------------------
+
+The lock-free stack consists of a linked list of elements, each containing a
+data pointer and a next pointer, and an atomic stack depth counter. The
+lock-free property means that multiple threads can push and pop simultaneously,
+and one thread being preempted/delayed in a push or pop operation will not
+impede the forward progress of any other thread.
+
+The lock-free push operation enqueues a linked list of pointers by pointing the
+list's tail to the current stack head, and using a CAS to swing the stack head
+pointer to the head of the list. The operation retries if it is unsuccessful
+(i.e. the list changed between reading the head and modifying it), else it
+adjusts the stack length and returns.
+
+The lock-free pop operation first reserves one or more list elements by
+adjusting the stack length, to ensure the dequeue operation will succeed
+without blocking. It then dequeues pointers by walking the list -- starting
+from the head -- then swinging the head pointer (using a CAS as well). While
+walking the list, the data pointers are recorded in an object table.
+
+The linked list elements themselves are maintained in a lock-free LIFO, and are
+allocated before stack pushes and freed after stack pops. Since the stack has a
+fixed maximum depth, these elements do not need to be dynamically created.
+
+The lock-free behavior is selected by passing the *RTE_STACK_F_LF* flag to
+rte_stack_create().
+
+Preventing the ABA Problem
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To prevent the ABA problem, this algorithm stack uses a 128-bit
+compare-and-swap instruction to atomically update both the stack top pointer
+and a modification counter. The ABA problem can occur without a modification
+counter if, for example:
+
+1. Thread A reads head pointer X and stores the pointed-to list element.
+2. Other threads modify the list such that the head pointer is once again X,
+   but its pointed-to data is different than what thread A read.
+3. Thread A changes the head pointer with a compare-and-swap and succeeds.
+
+In this case thread A would not detect that the list had changed, and would
+both pop stale data and incorrect change the head pointer. By adding a
+modification counter that is updated on every push and pop as part of the
+compare-and-swap, the algorithm can detect when the list changes even if the
+head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
index ebfbe36e5..3b115b5f6 100644
@@ -127,6 +127,9 @@ New Features
   pointers. The API provides MT-safe push and pop operations that can operate
   on one or more pointers per operation.
 
+  The library supports two stack implementations: standard (lock-based) and lock-free.
+  The lock-free implementation is currently limited to x86-64 platforms.
+
 Removed Items
 -------------
 
lib/librte_stack/Makefile +5 -2
index 6db540073..311edd997 100644
@@ -16,10 +16,13 @@ LIBABIVER := 1
 
 # all source are stored in SRCS-y
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \
-				   rte_stack_std.c
+				   rte_stack_std.c \
+				   rte_stack_lf.c
 
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
-					      rte_stack_std.h
+					      rte_stack_std.h \
+					      rte_stack_lf.h \
+					      rte_stack_lf_generic.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +5 -2
index d2e60ce9b..7a09a5d66 100644
@@ -4,5 +4,8 @@
 allow_experimental_apis = true
 
 version = 1
-sources = files('rte_stack.c', 'rte_stack_std.c')
-headers = files('rte_stack.h', 'rte_stack_std.h')
+sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c')
+headers = files('rte_stack.h',
+		'rte_stack_std.h',
+		'rte_stack_lf.h',
+		'rte_stack_lf_generic.h')
lib/librte_stack/rte_stack.c +21 -7
index 610014b6c..1a4d9bd1e 100644
@@ -25,18 +25,25 @@ static struct rte_tailq_elem rte_stack_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_stack_tailq)
 
+
 static void
-rte_stack_init(struct rte_stack *s)
+rte_stack_init(struct rte_stack *s, unsigned int count, uint32_t flags)
 {
 	memset(s, 0, sizeof(*s));
 
-	rte_stack_std_init(s);
+	if (flags & RTE_STACK_F_LF)
+		rte_stack_lf_init(s, count);
+	else
+		rte_stack_std_init(s);
 }
 
 static ssize_t
-rte_stack_get_memsize(unsigned int count)
+rte_stack_get_memsize(unsigned int count, uint32_t flags)
 {
-	return rte_stack_std_get_memsize(count);
+	if (flags & RTE_STACK_F_LF)
+		return rte_stack_lf_get_memsize(count);
+	else
+		return rte_stack_std_get_memsize(count);
 }
 
 struct rte_stack *
@@ -51,9 +58,16 @@ rte_stack_create(const char *name, unsigned int count, int socket_id,
 	unsigned int sz;
 	int ret;
 
-	RTE_SET_USED(flags);
+#ifdef RTE_ARCH_64
+	RTE_BUILD_BUG_ON(sizeof(struct rte_stack_lf_head) != 16);
+#else
+	if (flags & RTE_STACK_F_LF) {
+		STACK_LOG_ERR("Lock-free stack is not supported on your platform\n");
+		return NULL;
+	}
+#endif
 
-	sz = rte_stack_get_memsize(count);
+	sz = rte_stack_get_memsize(count, flags);
 
 	ret = snprintf(mz_name, sizeof(mz_name), "%s%s",
 		       RTE_STACK_MZ_PREFIX, name);
@@ -82,7 +96,7 @@ rte_stack_create(const char *name, unsigned int count, int socket_id,
 
 	s = mz->addr;
 
-	rte_stack_init(s);
+	rte_stack_init(s, count, flags);
 
 	/* Store the name for later lookups */
 	ret = snprintf(s->name, sizeof(s->name), "%s", name);
lib/librte_stack/rte_stack.h +57 -5

101 lines not on this page · raw message

lib/librte_stack/rte_stack_lf.c new file +31 -0
new file mode 100644
index 000000000..0adcc263e
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include "rte_stack.h"
+
+void
+rte_stack_lf_init(struct rte_stack *s, unsigned int count)
+{
+	struct rte_stack_lf_elem *elems = s->stack_lf.elems;
+	unsigned int i;
+
+	for (i = 0; i < count; i++)
+		__rte_stack_lf_push_elems(&s->stack_lf.free,
+					  &elems[i], &elems[i], 1);
+}
+
+ssize_t
+rte_stack_lf_get_memsize(unsigned int count)
+{
+	ssize_t sz = sizeof(struct rte_stack);
+
+	sz += RTE_CACHE_LINE_ROUNDUP(count * sizeof(struct rte_stack_lf_elem));
+
+	/* Add padding to avoid false sharing conflicts caused by
+	 * next-line hardware prefetchers.
+	 */
+	sz += 2 * RTE_CACHE_LINE_SIZE;
+
+	return sz;
+}
lib/librte_stack/rte_stack_lf.h new file +102 -0

102 lines not on this page · raw message

lib/librte_stack/rte_stack_lf_generic.h new file +164 -0

164 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#96 lore raw

PATCH v7 6/8 stack: add C11 atomic implementation

In reply to Gage Eads

This commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.

---
 lib/librte_stack/Makefile           |   3 +-
 lib/librte_stack/meson.build        |   3 +-
 lib/librte_stack/rte_stack_lf.h     |   4 +
 lib/librte_stack/rte_stack_lf_c11.h | 175 ++++++++++++++++++++++++++++++++++++
 4 files changed, 183 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_lf_c11.h
lib/librte_stack/Makefile +2 -1
index 311edd997..8d18ce520 100644
@@ -23,6 +23,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \
 SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
 					      rte_stack_std.h \
 					      rte_stack_lf.h \
-					      rte_stack_lf_generic.h
+					      rte_stack_lf_generic.h \
+					      rte_stack_lf_c11.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
index 7a09a5d66..46fce0c20 100644
@@ -8,4 +8,5 @@ sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c')
 headers = files('rte_stack.h',
 		'rte_stack_std.h',
 		'rte_stack_lf.h',
-		'rte_stack_lf_generic.h')
+		'rte_stack_lf_generic.h',
+		'rte_stack_lf_c11.h')
lib/librte_stack/rte_stack_lf.h +4 -0
index bfd680133..518889a05 100644
@@ -5,7 +5,11 @@
 #ifndef _RTE_STACK_LF_H_
 #define _RTE_STACK_LF_H_
 
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "rte_stack_lf_c11.h"
+#else
 #include "rte_stack_lf_generic.h"
+#endif
 
 /**
  * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_lf_c11.h new file +175 -0

175 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#97 lore raw

PATCH v7 7/8 test/stack: add lock-free stack tests

In reply to Gage Eads

This commit adds lock-free stack variants of stack_autotest (stack_lf_autotest) and stack_perf_autotest (stack_lf_perf_autotest), which differ only in that the lock-free versions pass the RTE_STACK_F_LF flag to all rte_stack_create() calls.

---
 app/test/meson.build       |  2 ++
 app/test/test_stack.c      | 41 +++++++++++++++++++++++++++--------------
 app/test/test_stack_perf.c | 17 +++++++++++++++--
 3 files changed, 44 insertions(+), 16 deletions(-)
app/test/meson.build +2 -0
index 02eb788a4..867cc5863 100644
@@ -178,6 +178,7 @@ fast_parallel_test_names = [
         'sched_autotest',
         'spinlock_autotest',
         'stack_autotest',
+        'stack_nb_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
@@ -243,6 +244,7 @@ perf_test_names = [
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
         'stack_perf_autotest',
+        'stack_nb_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
app/test/test_stack.c +27 -14

112 lines not on this page · raw message

app/test/test_stack_perf.c +15 -2
index 484370d30..e09d5384c 100644
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s)
 }
 
 static int
-test_stack_perf(void)
+__test_stack_perf(uint32_t flags)
 {
 	struct lcore_pair cores;
 	struct rte_stack *s;
 
 	rte_atomic32_init(&lcore_barrier);
 
-	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), 0);
+	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), flags);
 	if (s == NULL) {
 		printf("[%s():%u] failed to create a stack\n",
 		       __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)
 	return 0;
 }
 
+static int
+test_stack_perf(void)
+{
+	return __test_stack_perf(0);
+}
+
+static int
+test_lf_stack_perf(void)
+{
+	return __test_stack_perf(RTE_STACK_F_LF);
+}
+
 REGISTER_TEST_COMMAND(stack_perf_autotest, test_stack_perf);
+REGISTER_TEST_COMMAND(stack_lf_perf_autotest, test_lf_stack_perf);
Signature · 1 line
2.13.6
author
#98 lore raw

PATCH v7 8/8 mempool/stack: add lock-free stack mempool handler

In reply to Gage Eads

This commit adds support for lock-free (linked list based) stack mempool handler.

In mempool_perf_autotest the lock-based stack outperforms the lock-free handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a standard (lock-based) stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the lock-free stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.

*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the lock-free stack's.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
 doc/guides/rel_notes/release_19_05.rst          |  5 +++++
 drivers/mempool/stack/rte_mempool_stack.c       | 26 +++++++++++++++++++++++--
 3 files changed, 39 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
index 6a04c3c33..fa8afdb3a 100644
@@ -581,6 +581,16 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
+  Alternatively, applications can use the lock-free stack mempool handler. When
+  considering this handler, note that:
+
+  - It is currently limited to the x86_64 platform, because it uses an
+    instruction (16-byte compare-and-swap) that is not yet available on other
+    platforms.
+  - It has worse average-case performance than the non-preemptive rte_ring, but
+    software caching (e.g. the mempool cache) can mitigate this by reducing the
+    number of stack accesses.
+
 + rte_timer
 
   Running  ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
index 3b115b5f6..f873984ad 100644
@@ -130,6 +130,11 @@ New Features
   The library supports two stack implementations: standard (lock-based) and lock-free.
   The lock-free implementation is currently limited to x86-64 platforms.
 
+* **Added Lock-Free Stack Mempool Handler.**
+
+  Added a new lock-free stack handler, which uses the newly added stack
+  library.
+
 Removed Items
 -------------
 
drivers/mempool/stack/rte_mempool_stack.c +24 -2
index 25ccdb9af..7e85c8d6b 100644
@@ -7,7 +7,7 @@
 #include <rte_stack.h>
 
 static int
-stack_alloc(struct rte_mempool *mp)
+__stack_alloc(struct rte_mempool *mp, uint32_t flags)
 {
 	char name[RTE_STACK_NAMESIZE];
 	struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)
 		return -rte_errno;
 	}
 
-	s = rte_stack_create(name, mp->size, mp->socket_id, 0);
+	s = rte_stack_create(name, mp->size, mp->socket_id, flags);
 	if (s == NULL)
 		return -rte_errno;
 
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp)
 }
 
 static int
+stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, 0);
+}
+
+static int
+lf_stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, RTE_STACK_F_LF);
+}
+
+static int
 stack_enqueue(struct rte_mempool *mp, void * const *obj_table,
 	      unsigned int n)
 {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = {
 	.get_count = stack_get_count
 };
 
+static struct rte_mempool_ops ops_lf_stack = {
+	.name = "lf_stack",
+	.alloc = lf_stack_alloc,
+	.free = stack_free,
+	.enqueue = stack_enqueue,
+	.dequeue = stack_dequeue,
+	.get_count = stack_get_count
+};
+
 MEMPOOL_REGISTER_OPS(ops_stack);
+MEMPOOL_REGISTER_OPS(ops_lf_stack);
Signature · 1 line
2.13.6
#99 lore raw

Re: [PATCH v7 0/8] Add stack library and new mempool handler

In reply to Gage Eads

03/04/2019 22:09, Gage Eads:

Quoted 3 lines

v7:
- Add rte_branch_prediction.h include to rte_stack_std.h for unlikely()
- Add rte_compat.h include to rte_stack.h for __rte_experimental

There is another error when compiling for Arm:

lib/librte_stack/rte_stack.h:76:2: error:
implicit declaration of function 'RTE_ASSERT'

author
#100 lore raw

Re: [PATCH v7 0/8] Add stack library and new mempool handler

Quoted 14 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, April 3, 2019 3:40 PM
To: Eads, Gage <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v7 0/8] Add stack library and new mempool handler

03/04/2019 22:09, Gage Eads:

Quoted 8 lines
Quoted 3 lines

v7:
- Add rte_branch_prediction.h include to rte_stack_std.h for unlikely()
- Add rte_compat.h include to rte_stack.h for __rte_experimental

There is another error when compiling for Arm:

lib/librte_stack/rte_stack.h:76:2: error:
implicit declaration of function 'RTE_ASSERT'

Fix incoming. Looking through the rest of that header, I don't see any other externally defined macros/data structures/functions/etc. that would need another include.

Also note that v7's ci/Performance-Testing build fail (http://mails.dpdk.org/archives/test-report/2019-April/079261.html) is because dpdklab applied the patch on commit 3c45889189924067, which is older than the 128-bit CAS patch.

author
#101 lore raw

v8 cover Add stack library and new mempool handler

In reply to Gage Eads

This patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.

The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.

The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.

This patchset is the successor to a patchset containing only the new mempool handler[2].

[1] http://mails.dpdk.org/archives/dev/2019-April/129014.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html

---
v8:
- Add rte_debug.h include to rte_stack.h for RTE_ASSERT()

v7:
- Add rte_branch_prediction.h include to rte_stack_std.h for unlikely()
- Add rte_compat.h include to rte_stack.h for __rte_experimental

v6:
- Add load-acquire fence to the lock-free push function
- Correct generic implementation's pop_elems 128b CAS success and failure memorder to match those in the C11 implementation.

v5:
- Add comment to explain padding in *_get_memsize() functions
- Prefix internal functions with '__'
- Use RTE_ASSERT for performance critical run-time checks
- Don't use __atomic_load in the C11 pop_elems function, and put an acquire thread fence at the start of the 2nd do-while loop
- Change pop_elems 128b CAS success memorder to RELEASE and failure memorder to RELAXED
- Change compile-time assertion to run for all 64-bit architectures
- Reorganize the code with standard and lock-free .c and .h files

v4:
- Fix 32-bit build error in test_stack.c by using %zu format specifier for size_t
- Rebase onto master

v3:
- Rebase patchset onto master (test/test/ -> app/test/)
- Fix rte_stack_std_push() segfault introduced in v2

v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset

Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler

 MAINTAINERS                                     |   9 +-
 app/test/Makefile                               |   3 +
 app/test/meson.build                            |   7 +
 app/test/test_stack.c                           | 423 ++++++++++++++++++++++++
 app/test/test_stack_perf.c                      | 356 ++++++++++++++++++++
 config/common_base                              |   5 +
 doc/api/doxy-api-index.md                       |   1 +
 doc/api/doxy-api.conf.in                        |   1 +
 doc/guides/prog_guide/env_abstraction_layer.rst |  10 +
 doc/guides/prog_guide/index.rst                 |   1 +
 doc/guides/prog_guide/stack_lib.rst             |  83 +++++
 doc/guides/rel_notes/release_19_05.rst          |  13 +
 drivers/mempool/stack/Makefile                  |   3 +-
 drivers/mempool/stack/meson.build               |   6 +-
 drivers/mempool/stack/rte_mempool_stack.c       | 115 +++----
 lib/Makefile                                    |   2 +
 lib/librte_stack/Makefile                       |  29 ++
 lib/librte_stack/meson.build                    |  12 +
 lib/librte_stack/rte_stack.c                    | 196 +++++++++++
 lib/librte_stack/rte_stack.h                    | 261 +++++++++++++++
 lib/librte_stack/rte_stack_lf.c                 |  31 ++
 lib/librte_stack/rte_stack_lf.h                 | 106 ++++++
 lib/librte_stack/rte_stack_lf_c11.h             | 175 ++++++++++
 lib/librte_stack/rte_stack_lf_generic.h         | 164 +++++++++
 lib/librte_stack/rte_stack_pvt.h                |  34 ++
 lib/librte_stack/rte_stack_std.c                |  26 ++
 lib/librte_stack/rte_stack_std.h                | 121 +++++++
 lib/librte_stack/rte_stack_version.map          |   9 +
 lib/meson.build                                 |   2 +-
 mk/rte.app.mk                                   |   1 +
 30 files changed, 2133 insertions(+), 72 deletions(-)
 create mode 100644 app/test/test_stack.c
 create mode 100644 app/test/test_stack_perf.c
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_lf.c
 create mode 100644 lib/librte_stack/rte_stack_lf.h
 create mode 100644 lib/librte_stack/rte_stack_lf_c11.h
 create mode 100644 lib/librte_stack/rte_stack_lf_generic.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_std.c
 create mode 100644 lib/librte_stack/rte_stack_std.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
Signature · 1 line
2.13.6
author
#102 lore raw

PATCH v8 1/8 stack: introduce rte stack library

The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.

The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.

---
 MAINTAINERS                            |   6 +
 config/common_base                     |   5 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/prog_guide/index.rst        |   1 +
 doc/guides/prog_guide/stack_lib.rst    |  28 +++++
 doc/guides/rel_notes/release_19_05.rst |   5 +
 lib/Makefile                           |   2 +
 lib/librte_stack/Makefile              |  25 ++++
 lib/librte_stack/meson.build           |   8 ++
 lib/librte_stack/rte_stack.c           | 182 ++++++++++++++++++++++++++++
 lib/librte_stack/rte_stack.h           | 209 +++++++++++++++++++++++++++++++++
 lib/librte_stack/rte_stack_pvt.h       |  34 ++++++
 lib/librte_stack/rte_stack_std.c       |  26 ++++
 lib/librte_stack/rte_stack_std.h       | 121 +++++++++++++++++++
 lib/librte_stack/rte_stack_version.map |   9 ++
 lib/meson.build                        |   2 +-
 mk/rte.app.mk                          |   1 +
 18 files changed, 665 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_std.c
 create mode 100644 lib/librte_stack/rte_stack_std.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
index 71ac8cd4b..f30fc4aa6 100644
@@ -426,6 +426,12 @@ F: drivers/raw/skeleton_rawdev/
 F: app/test/test_rawdev.c
 F: doc/guides/prog_guide/rawdev.rst
 
+Stack API - EXPERIMENTAL
+M: Gage Eads <[email protected]>
+M: Olivier Matz <[email protected]>
+F: lib/librte_stack/
+F: doc/guides/prog_guide/stack_lib.rst
+
 
 Memory Pool Drivers
 -------------------
config/common_base +5 -0
index 6292bc4af..fc8dba69d 100644
@@ -994,3 +994,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
 # Compile the eventdev application
 #
 CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Compile librte_stack
+#
+CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
index aacc66bd8..de1e215dd 100644
@@ -125,6 +125,7 @@ The public API headers are grouped by topics:
   [mbuf]               (@ref rte_mbuf.h),
   [mbuf pool ops]      (@ref rte_mbuf_pool_ops.h),
   [ring]               (@ref rte_ring.h),
+  [stack]              (@ref rte_stack.h),
   [tailq]              (@ref rte_tailq.h),
   [bitmap]             (@ref rte_bitmap.h)
 
doc/api/doxy-api.conf.in +1 -0
index a365e669b..7722fc3e9 100644
@@ -55,6 +55,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/librte_ring \
                           @TOPDIR@/lib/librte_sched \
                           @TOPDIR@/lib/librte_security \
+                          @TOPDIR@/lib/librte_stack \
                           @TOPDIR@/lib/librte_table \
                           @TOPDIR@/lib/librte_telemetry \
                           @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
index 6726b1e8d..f4f60862f 100644
@@ -55,6 +55,7 @@ Programmer's Guide
     metrics_lib
     bpf_lib
     ipsec_lib
+    stack_lib
     source_org
     dev_kit_build_system
     dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +28 -0
new file mode 100644
index 000000000..25a8cc38a
@@ -0,0 +1,28 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Stack Library
+=============
+
+DPDK's stack library provides an API for configuration and use of a bounded
+stack of pointers.
+
+The stack library provides the following basic operations:
+
+*  Create a uniquely named stack of a user-specified size and using a
+   user-specified socket.
+
+*  Push and pop a burst of one or more stack objects (pointers). These function
+   are multi-threading safe.
+
+*  Free a previously created stack.
+
+*  Lookup a pointer to a stack by its name.
+
+*  Query a stack's current depth and number of free entries.
+
+Implementation
+~~~~~~~~~~~~~~
+
+The stack consists of a contiguous array of pointers, a current index, and a
+spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
index bdad1ddbe..ebfbe36e5 100644
@@ -121,6 +121,11 @@ New Features
   Improved testpmd application performance on ARM platform. For ``macswap``
   forwarding mode, NEON intrinsics were used to do swap to save CPU cycles.
 
+* **Added Stack API.**
+
+  Added a new stack API for configuration and use of a bounded stack of
+  pointers. The API provides MT-safe push and pop operations that can operate
+  on one or more pointers per operation.
 
 Removed Items
 -------------
lib/Makefile +2 -0
index a358f1c19..9f90e80ad 100644
@@ -109,6 +109,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec
 DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security
 DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry
 DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev
+DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack
+DEPDIRS-librte_stack := librte_eal
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +25 -0
new file mode 100644
index 000000000..6db540073
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_stack.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+LDLIBS += -lrte_eal
+
+EXPORT_MAP := rte_stack_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \
+				   rte_stack_std.c
+
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
+					      rte_stack_std.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
new file mode 100644
index 000000000..d2e60ce9b
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+allow_experimental_apis = true
+
+version = 1
+sources = files('rte_stack.c', 'rte_stack_std.c')
+headers = files('rte_stack.h', 'rte_stack_std.h')
lib/librte_stack/rte_stack.c new file +182 -0

182 lines not on this page · raw message

lib/librte_stack/rte_stack.h new file +209 -0

209 lines not on this page · raw message

lib/librte_stack/rte_stack_pvt.h new file +34 -0
new file mode 100644
index 000000000..4a6a7bdb3
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _RTE_STACK_PVT_H_
+#define _RTE_STACK_PVT_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_log.h>
+
+extern int stack_logtype;
+
+#define STACK_LOG(level, fmt, args...) \
+	rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \
+		__func__, ##args)
+
+#define STACK_LOG_ERR(fmt, args...) \
+	STACK_LOG(ERR, fmt, ## args)
+
+#define STACK_LOG_WARN(fmt, args...) \
+	STACK_LOG(WARNING, fmt, ## args)
+
+#define STACK_LOG_INFO(fmt, args...) \
+	STACK_LOG(INFO, fmt, ## args)
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_std.c new file +26 -0
new file mode 100644
index 000000000..0a310d7c6
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include "rte_stack.h"
+
+void
+rte_stack_std_init(struct rte_stack *s)
+{
+	rte_spinlock_init(&s->stack_std.lock);
+}
+
+ssize_t
+rte_stack_std_get_memsize(unsigned int count)
+{
+	ssize_t sz = sizeof(struct rte_stack);
+
+	sz += RTE_CACHE_LINE_ROUNDUP(count * sizeof(void *));
+
+	/* Add padding to avoid false sharing conflicts caused by
+	 * next-line hardware prefetchers.
+	 */
+	sz += 2 * RTE_CACHE_LINE_SIZE;
+
+	return sz;
+}
lib/librte_stack/rte_stack_std.h new file +121 -0

121 lines not on this page · raw message

lib/librte_stack/rte_stack_version.map new file +9 -0
new file mode 100644
index 000000000..6662679c3
@@ -0,0 +1,9 @@
+EXPERIMENTAL {
+	global:
+
+	rte_stack_create;
+	rte_stack_free;
+	rte_stack_lookup;
+
+	local: *;
+};
lib/meson.build +1 -1
index c3289f885..595314d7d 100644
@@ -22,7 +22,7 @@ libraries = [
 	'gro', 'gso', 'ip_frag', 'jobstats',
 	'kni', 'latencystats', 'lpm', 'member',
 	'power', 'pdump', 'rawdev',
-	'reorder', 'sched', 'security', 'vhost',
+	'reorder', 'sched', 'security', 'stack', 'vhost',
 	#ipsec lib depends on crypto and security
 	'ipsec',
 	# add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
index 262132fc6..7e033e78c 100644
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY)       += -lrte_security
 _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV)    += -lrte_compressdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV)       += -lrte_eventdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV)         += -lrte_rawdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER)          += -lrte_timer
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL)        += -lrte_mempool
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
Signature · 1 line
2.13.6
author
#103 lore raw

PATCH v8 2/8 mempool/stack: convert mempool to use rte stack

In reply to Gage Eads

The new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.

---
 MAINTAINERS                               |  2 +-
 drivers/mempool/stack/Makefile            |  3 +-
 drivers/mempool/stack/meson.build         |  6 +-
 drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++----------------------
 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
index f30fc4aa6..e09e7d93f 100644
@@ -303,7 +303,6 @@ M: Andrew Rybchenko <[email protected]>
 F: lib/librte_mempool/
 F: drivers/mempool/Makefile
 F: drivers/mempool/ring/
-F: drivers/mempool/stack/
 F: doc/guides/prog_guide/mempool_lib.rst
 F: app/test/test_mempool*
 F: app/test/test_func_reentrancy.c
@@ -431,6 +430,7 @@ M: Gage Eads <[email protected]>
 M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
+F: drivers/mempool/stack/
 
 
 Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
index 0444aedad..1681a62bc 100644
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # Headers
 CFLAGS += -I$(RTE_SDK)/lib/librte_mempool
-LDLIBS += -lrte_eal -lrte_mempool -lrte_ring
+LDLIBS += -lrte_eal -lrte_mempool -lrte_stack
 
 EXPORT_MAP := rte_mempool_stack_version.map
 
drivers/mempool/stack/meson.build +5 -1
index b75a3bb56..03e369a41 100644
@@ -1,4 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
+# Copyright(c) 2017-2019 Intel Corporation
+
+allow_experimental_apis = true
 
 sources = files('rte_mempool_stack.c')
+
+deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68

133 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#104 lore raw

PATCH v8 3/8 test/stack: add stack test

In reply to Gage Eads

stack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.

---
 MAINTAINERS           |   1 +
 app/test/Makefile     |   2 +
 app/test/meson.build  |   3 +
 app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 416 insertions(+)
 create mode 100644 app/test/test_stack.c
MAINTAINERS +1 -0
index e09e7d93f..e4e6d1b15 100644
@@ -431,6 +431,7 @@ M: Olivier Matz <[email protected]>
 F: lib/librte_stack/
 F: doc/guides/prog_guide/stack_lib.rst
 F: drivers/mempool/stack/
+F: test/test/*stack*
 
 
 Memory Pool Drivers
app/test/Makefile +2 -0
index d6aa28bad..e5bde81af 100644
@@ -90,6 +90,8 @@ endif
 
 SRCS-y += test_rwlock.c
 
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
app/test/meson.build +3 -0
index c5e65fe66..56ea13f53 100644
@@ -95,6 +95,7 @@ test_sources = files('commands.c',
 	'test_sched.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
+	'test_stack.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -133,6 +134,7 @@ test_deps = ['acl',
 	'port',
 	'reorder',
 	'ring',
+	'stack',
 	'timer'
 ]
 
@@ -174,6 +176,7 @@ fast_parallel_test_names = [
         'rwlock_autotest',
         'sched_autotest',
         'spinlock_autotest',
+        'stack_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
app/test/test_stack.c new file +410 -0

410 lines not on this page · raw message

Signature · 1 line
2.13.6
#105 lore raw

Re: [PATCH v8 3/8] test/stack: add stack test

03/04/2019 22:50, Gage Eads:

Quoted 5 lines

stack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.

Quoted 7 lines
---
 MAINTAINERS           |   1 +
 app/test/Makefile     |   2 +
 app/test/meson.build  |   3 +
 app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 416 insertions(+)
 create mode 100644 app/test/test_stack.c

Another error with Arm:

app/test/test_stack.c:275:2: error: unknown type name 'rte_atomic64_t'

I think you should install an Arm toolchain and run test-meson-builds.sh

author
#106 lore raw

Re: [PATCH v8 3/8] test/stack: add stack test

Quoted 28 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, April 3, 2019 5:41 PM
To: Eads, Gage <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v8 3/8] test/stack: add stack test

03/04/2019 22:50, Gage Eads:

Quoted 2 lines

stack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available

lcores.

Quoted 11 lines
---
 MAINTAINERS           |   1 +
 app/test/Makefile     |   2 +
 app/test/meson.build  |   3 +
 app/test/test_stack.c | 410
++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 416 insertions(+)
 create mode 100644 app/test/test_stack.c
Quoted 6 lines

Another error with Arm:

app/test/test_stack.c:275:2: error: unknown type name 'rte_atomic64_t'

I think you should install an Arm toolchain and run test-meson-builds.sh

I should've done that a while ago -- it was pretty painless on Ubuntu. Looks like rte_atomic.h is the last missing header.

author
#107 lore raw

PATCH v8 4/8 test/stack: add stack perf test

In reply to Gage Eads

stack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects

It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores

---
 app/test/Makefile          |   1 +
 app/test/meson.build       |   2 +
 app/test/test_stack_perf.c | 343 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 346 insertions(+)
 create mode 100644 app/test/test_stack_perf.c
app/test/Makefile +1 -0
index e5bde81af..b28bed2d4 100644
@@ -91,6 +91,7 @@ endif
 SRCS-y += test_rwlock.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c
+SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c
 
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c
 SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
app/test/meson.build +2 -0
index 56ea13f53..02eb788a4 100644
@@ -96,6 +96,7 @@ test_sources = files('commands.c',
 	'test_service_cores.c',
 	'test_spinlock.c',
 	'test_stack.c',
+	'test_stack_perf.c',
 	'test_string_fns.c',
 	'test_table.c',
 	'test_table_acl.c',
@@ -241,6 +242,7 @@ perf_test_names = [
         'distributor_perf_autotest',
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
+        'stack_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
app/test/test_stack_perf.c new file +343 -0

343 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#108 lore raw

PATCH v8 5/8 stack: add lock-free stack implementation

In reply to Gage Eads

This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.

The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.

The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.

The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.

This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.

The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.

---
 doc/guides/prog_guide/stack_lib.rst     |  61 +++++++++++-
 doc/guides/rel_notes/release_19_05.rst  |   3 +
 lib/librte_stack/Makefile               |   7 +-
 lib/librte_stack/meson.build            |   7 +-
 lib/librte_stack/rte_stack.c            |  28 ++++--
 lib/librte_stack/rte_stack.h            |  62 +++++++++++-
 lib/librte_stack/rte_stack_lf.c         |  31 ++++++
 lib/librte_stack/rte_stack_lf.h         | 102 ++++++++++++++++++++
 lib/librte_stack/rte_stack_lf_generic.h | 164 ++++++++++++++++++++++++++++++++
 9 files changed, 446 insertions(+), 19 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_lf.c
 create mode 100644 lib/librte_stack/rte_stack_lf.h
 create mode 100644 lib/librte_stack/rte_stack_lf_generic.h
doc/guides/prog_guide/stack_lib.rst +58 -3
index 25a8cc38a..8fe8804e3 100644
@@ -10,7 +10,8 @@ stack of pointers.
 The stack library provides the following basic operations:
 
 *  Create a uniquely named stack of a user-specified size and using a
-   user-specified socket.
+   user-specified socket, with either standard (lock-based) or lock-free
+   behavior.
 
 *  Push and pop a burst of one or more stack objects (pointers). These function
    are multi-threading safe.
@@ -24,5 +25,59 @@ The stack library provides the following basic operations:
 Implementation
 ~~~~~~~~~~~~~~
 
-The stack consists of a contiguous array of pointers, a current index, and a
-spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
+The library supports two types of stacks: standard (lock-based) and lock-free.
+Both types use the same set of interfaces, but their implementations differ.
+
+Lock-based Stack
+----------------
+
+The lock-based stack consists of a contiguous array of pointers, a current
+index, and a spinlock. Accesses to the stack are made multi-thread safe by the
+spinlock.
+
+Lock-free Stack
+------------------
+
+The lock-free stack consists of a linked list of elements, each containing a
+data pointer and a next pointer, and an atomic stack depth counter. The
+lock-free property means that multiple threads can push and pop simultaneously,
+and one thread being preempted/delayed in a push or pop operation will not
+impede the forward progress of any other thread.
+
+The lock-free push operation enqueues a linked list of pointers by pointing the
+list's tail to the current stack head, and using a CAS to swing the stack head
+pointer to the head of the list. The operation retries if it is unsuccessful
+(i.e. the list changed between reading the head and modifying it), else it
+adjusts the stack length and returns.
+
+The lock-free pop operation first reserves one or more list elements by
+adjusting the stack length, to ensure the dequeue operation will succeed
+without blocking. It then dequeues pointers by walking the list -- starting
+from the head -- then swinging the head pointer (using a CAS as well). While
+walking the list, the data pointers are recorded in an object table.
+
+The linked list elements themselves are maintained in a lock-free LIFO, and are
+allocated before stack pushes and freed after stack pops. Since the stack has a
+fixed maximum depth, these elements do not need to be dynamically created.
+
+The lock-free behavior is selected by passing the *RTE_STACK_F_LF* flag to
+rte_stack_create().
+
+Preventing the ABA Problem
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To prevent the ABA problem, this algorithm stack uses a 128-bit
+compare-and-swap instruction to atomically update both the stack top pointer
+and a modification counter. The ABA problem can occur without a modification
+counter if, for example:
+
+1. Thread A reads head pointer X and stores the pointed-to list element.
+2. Other threads modify the list such that the head pointer is once again X,
+   but its pointed-to data is different than what thread A read.
+3. Thread A changes the head pointer with a compare-and-swap and succeeds.
+
+In this case thread A would not detect that the list had changed, and would
+both pop stale data and incorrect change the head pointer. By adding a
+modification counter that is updated on every push and pop as part of the
+compare-and-swap, the algorithm can detect when the list changes even if the
+head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
index ebfbe36e5..3b115b5f6 100644
@@ -127,6 +127,9 @@ New Features
   pointers. The API provides MT-safe push and pop operations that can operate
   on one or more pointers per operation.
 
+  The library supports two stack implementations: standard (lock-based) and lock-free.
+  The lock-free implementation is currently limited to x86-64 platforms.
+
 Removed Items
 -------------
 
lib/librte_stack/Makefile +5 -2
index 6db540073..311edd997 100644
@@ -16,10 +16,13 @@ LIBABIVER := 1
 
 # all source are stored in SRCS-y
 SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \
-				   rte_stack_std.c
+				   rte_stack_std.c \
+				   rte_stack_lf.c
 
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
-					      rte_stack_std.h
+					      rte_stack_std.h \
+					      rte_stack_lf.h \
+					      rte_stack_lf_generic.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +5 -2
index d2e60ce9b..7a09a5d66 100644
@@ -4,5 +4,8 @@
 allow_experimental_apis = true
 
 version = 1
-sources = files('rte_stack.c', 'rte_stack_std.c')
-headers = files('rte_stack.h', 'rte_stack_std.h')
+sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c')
+headers = files('rte_stack.h',
+		'rte_stack_std.h',
+		'rte_stack_lf.h',
+		'rte_stack_lf_generic.h')
lib/librte_stack/rte_stack.c +21 -7
index 610014b6c..1a4d9bd1e 100644
@@ -25,18 +25,25 @@ static struct rte_tailq_elem rte_stack_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_stack_tailq)
 
+
 static void
-rte_stack_init(struct rte_stack *s)
+rte_stack_init(struct rte_stack *s, unsigned int count, uint32_t flags)
 {
 	memset(s, 0, sizeof(*s));
 
-	rte_stack_std_init(s);
+	if (flags & RTE_STACK_F_LF)
+		rte_stack_lf_init(s, count);
+	else
+		rte_stack_std_init(s);
 }
 
 static ssize_t
-rte_stack_get_memsize(unsigned int count)
+rte_stack_get_memsize(unsigned int count, uint32_t flags)
 {
-	return rte_stack_std_get_memsize(count);
+	if (flags & RTE_STACK_F_LF)
+		return rte_stack_lf_get_memsize(count);
+	else
+		return rte_stack_std_get_memsize(count);
 }
 
 struct rte_stack *
@@ -51,9 +58,16 @@ rte_stack_create(const char *name, unsigned int count, int socket_id,
 	unsigned int sz;
 	int ret;
 
-	RTE_SET_USED(flags);
+#ifdef RTE_ARCH_64
+	RTE_BUILD_BUG_ON(sizeof(struct rte_stack_lf_head) != 16);
+#else
+	if (flags & RTE_STACK_F_LF) {
+		STACK_LOG_ERR("Lock-free stack is not supported on your platform\n");
+		return NULL;
+	}
+#endif
 
-	sz = rte_stack_get_memsize(count);
+	sz = rte_stack_get_memsize(count, flags);
 
 	ret = snprintf(mz_name, sizeof(mz_name), "%s%s",
 		       RTE_STACK_MZ_PREFIX, name);
@@ -82,7 +96,7 @@ rte_stack_create(const char *name, unsigned int count, int socket_id,
 
 	s = mz->addr;
 
-	rte_stack_init(s);
+	rte_stack_init(s, count, flags);
 
 	/* Store the name for later lookups */
 	ret = snprintf(s->name, sizeof(s->name), "%s", name);
lib/librte_stack/rte_stack.h +57 -5

101 lines not on this page · raw message

lib/librte_stack/rte_stack_lf.c new file +31 -0
new file mode 100644
index 000000000..0adcc263e
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#include "rte_stack.h"
+
+void
+rte_stack_lf_init(struct rte_stack *s, unsigned int count)
+{
+	struct rte_stack_lf_elem *elems = s->stack_lf.elems;
+	unsigned int i;
+
+	for (i = 0; i < count; i++)
+		__rte_stack_lf_push_elems(&s->stack_lf.free,
+					  &elems[i], &elems[i], 1);
+}
+
+ssize_t
+rte_stack_lf_get_memsize(unsigned int count)
+{
+	ssize_t sz = sizeof(struct rte_stack);
+
+	sz += RTE_CACHE_LINE_ROUNDUP(count * sizeof(struct rte_stack_lf_elem));
+
+	/* Add padding to avoid false sharing conflicts caused by
+	 * next-line hardware prefetchers.
+	 */
+	sz += 2 * RTE_CACHE_LINE_SIZE;
+
+	return sz;
+}
lib/librte_stack/rte_stack_lf.h new file +102 -0

102 lines not on this page · raw message

lib/librte_stack/rte_stack_lf_generic.h new file +164 -0

164 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#109 lore raw

PATCH v8 6/8 stack: add C11 atomic implementation

In reply to Gage Eads

This commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.

---
 lib/librte_stack/Makefile           |   3 +-
 lib/librte_stack/meson.build        |   3 +-
 lib/librte_stack/rte_stack_lf.h     |   4 +
 lib/librte_stack/rte_stack_lf_c11.h | 175 ++++++++++++++++++++++++++++++++++++
 4 files changed, 183 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_stack/rte_stack_lf_c11.h
lib/librte_stack/Makefile +2 -1
index 311edd997..8d18ce520 100644
@@ -23,6 +23,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \
 SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \
 					      rte_stack_std.h \
 					      rte_stack_lf.h \
-					      rte_stack_lf_generic.h
+					      rte_stack_lf_generic.h \
+					      rte_stack_lf_c11.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
index 7a09a5d66..46fce0c20 100644
@@ -8,4 +8,5 @@ sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c')
 headers = files('rte_stack.h',
 		'rte_stack_std.h',
 		'rte_stack_lf.h',
-		'rte_stack_lf_generic.h')
+		'rte_stack_lf_generic.h',
+		'rte_stack_lf_c11.h')
lib/librte_stack/rte_stack_lf.h +4 -0
index bfd680133..518889a05 100644
@@ -5,7 +5,11 @@
 #ifndef _RTE_STACK_LF_H_
 #define _RTE_STACK_LF_H_
 
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "rte_stack_lf_c11.h"
+#else
 #include "rte_stack_lf_generic.h"
+#endif
 
 /**
  * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_lf_c11.h new file +175 -0

175 lines not on this page · raw message

Signature · 1 line
2.13.6
author
#110 lore raw

PATCH v8 7/8 test/stack: add lock-free stack tests

In reply to Gage Eads

This commit adds lock-free stack variants of stack_autotest (stack_lf_autotest) and stack_perf_autotest (stack_lf_perf_autotest), which differ only in that the lock-free versions pass the RTE_STACK_F_LF flag to all rte_stack_create() calls.

---
 app/test/meson.build       |  2 ++
 app/test/test_stack.c      | 41 +++++++++++++++++++++++++++--------------
 app/test/test_stack_perf.c | 17 +++++++++++++++--
 3 files changed, 44 insertions(+), 16 deletions(-)
app/test/meson.build +2 -0
index 02eb788a4..867cc5863 100644
@@ -178,6 +178,7 @@ fast_parallel_test_names = [
         'sched_autotest',
         'spinlock_autotest',
         'stack_autotest',
+        'stack_nb_autotest',
         'string_autotest',
         'table_autotest',
         'tailq_autotest',
@@ -243,6 +244,7 @@ perf_test_names = [
         'ring_pmd_perf_autotest',
         'pmd_perf_autotest',
         'stack_perf_autotest',
+        'stack_nb_perf_autotest',
 ]
 
 # All test cases in driver_test_names list are non-parallel
app/test/test_stack.c +27 -14

112 lines not on this page · raw message

app/test/test_stack_perf.c +15 -2
index 484370d30..e09d5384c 100644
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s)
 }
 
 static int
-test_stack_perf(void)
+__test_stack_perf(uint32_t flags)
 {
 	struct lcore_pair cores;
 	struct rte_stack *s;
 
 	rte_atomic32_init(&lcore_barrier);
 
-	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), 0);
+	s = rte_stack_create(STACK_NAME, STACK_SIZE, rte_socket_id(), flags);
 	if (s == NULL) {
 		printf("[%s():%u] failed to create a stack\n",
 		       __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)
 	return 0;
 }
 
+static int
+test_stack_perf(void)
+{
+	return __test_stack_perf(0);
+}
+
+static int
+test_lf_stack_perf(void)
+{
+	return __test_stack_perf(RTE_STACK_F_LF);
+}
+
 REGISTER_TEST_COMMAND(stack_perf_autotest, test_stack_perf);
+REGISTER_TEST_COMMAND(stack_lf_perf_autotest, test_lf_stack_perf);
Signature · 1 line
2.13.6
author
#111 lore raw

PATCH v8 8/8 mempool/stack: add lock-free stack mempool handler

In reply to Gage Eads

This commit adds support for lock-free (linked list based) stack mempool handler.

In mempool_perf_autotest the lock-based stack outperforms the lock-free handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a standard (lock-based) stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the lock-free stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.

*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the lock-free stack's.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
 doc/guides/rel_notes/release_19_05.rst          |  5 +++++
 drivers/mempool/stack/rte_mempool_stack.c       | 26 +++++++++++++++++++++++--
 3 files changed, 39 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
index 6a04c3c33..fa8afdb3a 100644
@@ -581,6 +581,16 @@ Known Issues
 
   5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR.
 
+  Alternatively, applications can use the lock-free stack mempool handler. When
+  considering this handler, note that:
+
+  - It is currently limited to the x86_64 platform, because it uses an
+    instruction (16-byte compare-and-swap) that is not yet available on other
+    platforms.
+  - It has worse average-case performance than the non-preemptive rte_ring, but
+    software caching (e.g. the mempool cache) can mitigate this by reducing the
+    number of stack accesses.
+
 + rte_timer
 
   Running  ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
index 3b115b5f6..f873984ad 100644
@@ -130,6 +130,11 @@ New Features
   The library supports two stack implementations: standard (lock-based) and lock-free.
   The lock-free implementation is currently limited to x86-64 platforms.
 
+* **Added Lock-Free Stack Mempool Handler.**
+
+  Added a new lock-free stack handler, which uses the newly added stack
+  library.
+
 Removed Items
 -------------
 
drivers/mempool/stack/rte_mempool_stack.c +24 -2
index 25ccdb9af..7e85c8d6b 100644
@@ -7,7 +7,7 @@
 #include <rte_stack.h>
 
 static int
-stack_alloc(struct rte_mempool *mp)
+__stack_alloc(struct rte_mempool *mp, uint32_t flags)
 {
 	char name[RTE_STACK_NAMESIZE];
 	struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)
 		return -rte_errno;
 	}
 
-	s = rte_stack_create(name, mp->size, mp->socket_id, 0);
+	s = rte_stack_create(name, mp->size, mp->socket_id, flags);
 	if (s == NULL)
 		return -rte_errno;
 
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp)
 }
 
 static int
+stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, 0);
+}
+
+static int
+lf_stack_alloc(struct rte_mempool *mp)
+{
+	return __stack_alloc(mp, RTE_STACK_F_LF);
+}
+
+static int
 stack_enqueue(struct rte_mempool *mp, void * const *obj_table,
 	      unsigned int n)
 {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = {
 	.get_count = stack_get_count
 };
 
+static struct rte_mempool_ops ops_lf_stack = {
+	.name = "lf_stack",
+	.alloc = lf_stack_alloc,
+	.free = stack_free,
+	.enqueue = stack_enqueue,
+	.dequeue = stack_dequeue,
+	.get_count = stack_get_count
+};
+
 MEMPOOL_REGISTER_OPS(ops_stack);
+MEMPOOL_REGISTER_OPS(ops_lf_stack);
Signature · 1 line
2.13.6
author
#112 lore raw

v9 cover Add stack library and new mempool handler

In reply to Gage Eads

This patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.

The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.

The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.

Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.

This patchset is the successor to a patchset containing only the new mempool handler[2].

[1] http://mails.dpdk.org/archives/dev/2019-April/129014.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html

---
v9:
- Add rte_atomic.h includes to rte_stack.h, test_stack.c, and test_stack_perf.c to fix ARM builds

v8:
- Add rte_debug.h include to rte_stack.h for RTE_ASSERT()

v7:
- Add rte_branch_prediction.h include to rte_stack_std.h for unlikely()
- Add rte_compat.h include to rte_stack.h for __rte_experimental

v6:
- Add load-acquire fence to the lock-free push function
- Correct generic implementation's pop_elems 128b CAS success and failure memorder to match those in the C11 implementation.

v5:
- Add comment to explain padding in *_get_memsize() functions
- Prefix internal functions with '__'
- Use RTE_ASSERT for performance critical run-time checks
- Don't use __atomic_load in the C11 pop_elems function, and put an acquire thread fence at the start of the 2nd do-while loop
- Change pop_elems 128b CAS success memorder to RELEASE and failure memorder to RELAXED
- Change compile-time assertion to run for all 64-bit architectures
- Reorganize the code with standard and lock-free .c and .h files

v4:
- Fix 32-bit build error in test_stack.c by using %zu format specifier for size_t
- Rebase onto master

v3:
- Rebase patchset onto master (test/test/ -> app/test/)
- Fix rte_stack_std_push() segfault introduced in v2

v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset

Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler

 MAINTAINERS                                     |   9 +-
 app/test/Makefile                               |   3 +
 app/test/meson.build                            |   7 +
 app/test/test_stack.c                           | 424 ++++++++++++++++++++++++
 app/test/test_stack_perf.c                      | 358 ++++++++++++++++++++
 config/common_base                              |   5 +
 doc/api/doxy-api-index.md                       |   1 +
 doc/api/doxy-api.conf.in                        |   1 +
 doc/guides/prog_guide/env_abstraction_layer.rst |  10 +
 doc/guides/prog_guide/index.rst                 |   1 +
 doc/guides/prog_guide/stack_lib.rst             |  83 +++++
 doc/guides/rel_notes/release_19_05.rst          |  13 +
 drivers/mempool/stack/Makefile                  |   3 +-
 drivers/mempool/stack/meson.build               |   6 +-
 drivers/mempool/stack/rte_mempool_stack.c       | 115 +++----
 lib/Makefile                                    |   2 +
 lib/librte_stack/Makefile                       |  29 ++
 lib/librte_stack/meson.build                    |  12 +
 lib/librte_stack/rte_stack.c                    | 196 +++++++++++
 lib/librte_stack/rte_stack.h                    | 262 +++++++++++++++
 lib/librte_stack/rte_stack_lf.c                 |  31 ++
 lib/librte_stack/rte_stack_lf.h                 | 106 ++++++
 lib/librte_stack/rte_stack_lf_c11.h             | 175 ++++++++++
 lib/librte_stack/rte_stack_lf_generic.h         | 164 +++++++++
 lib/librte_stack/rte_stack_pvt.h                |  34 ++
 lib/librte_stack/rte_stack_std.c                |  26 ++
 lib/librte_stack/rte_stack_std.h                | 121 +++++++
 lib/librte_stack/rte_stack_version.map          |   9 +
 lib/meson.build                                 |   2 +-
 mk/rte.app.mk                                   |   1 +
 30 files changed, 2137 insertions(+), 72 deletions(-)
 create mode 100644 app/test/test_stack.c
 create mode 100644 app/test/test_stack_perf.c
 create mode 100644 doc/guides/prog_guide/stack_lib.rst
 create mode 100644 lib/librte_stack/Makefile
 create mode 100644 lib/librte_stack/meson.build
 create mode 100644 lib/librte_stack/rte_stack.c
 create mode 100644 lib/librte_stack/rte_stack.h
 create mode 100644 lib/librte_stack/rte_stack_lf.c
 create mode 100644 lib/librte_stack/rte_stack_lf.h
 create mode 100644 lib/librte_stack/rte_stack_lf_c11.h
 create mode 100644 lib/librte_stack/rte_stack_lf_generic.h
 create mode 100644 lib/librte_stack/rte_stack_pvt.h
 create mode 100644 lib/librte_stack/rte_stack_std.c
 create mode 100644 lib/librte_stack/rte_stack_std.h
 create mode 100644 lib/librte_stack/rte_stack_version.map
Signature · 1 line
2.13.6