Subject: [PATCH ...] Add stack library and new mempool handler
Subsystems (2)
cover Subject: [PATCH ...] Add stack library and new mempool handler
This patchset introduces a stack library, supporting both lock-based and non-blocking stacks, and a non-blocking stack mempool handler.
The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.
The non-blocking stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.
Note that the non-blocking algorithm relies on a 128-bit compare-and-swap, so it is currently limited to the x86_64 platform.
This patchset is the successor to a patchset containing only the new mempool handler[1].
[1] http://mails.dpdk.org/archives/dev/2019-January/123555.html
Gage Eads (7):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add non-blocking stack implementation test/stack: add non-blocking stack tests mempool/stack: add non-blocking stack mempool handler
MAINTAINERS | 9 +- config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/env_abstraction_layer.rst | 5 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 66 ++++ doc/guides/rel_notes/release_19_05.rst | 13 + drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 115 +++---- lib/Makefile | 2 + lib/librte_stack/Makefile | 25 ++ lib/librte_stack/meson.build | 10 + lib/librte_stack/rte_stack.c | 220 +++++++++++++ lib/librte_stack/rte_stack.h | 406 +++++++++++++++++++++++ lib/librte_stack/rte_stack_c11_mem.h | 173 ++++++++++ lib/librte_stack/rte_stack_generic.h | 157 +++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++ lib/librte_stack/rte_stack_version.map | 9 + lib/meson.build | 2 +- mk/rte.app.mk | 1 + test/test/Makefile | 3 + test/test/meson.build | 7 + test/test/test_stack.c | 407 ++++++++++++++++++++++++ test/test/test_stack_perf.c | 356 +++++++++++++++++++++ 26 files changed, 1965 insertions(+), 72 deletions(-) create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_c11_mem.h create mode 100644 lib/librte_stack/rte_stack_generic.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_version.map create mode 100644 test/test/test_stack.c create mode 100644 test/test/test_stack_perf.c
Signature · 1 line
2.13.6
PATCH 1/7 stack: introduce rte stack library
The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.
The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.
--- MAINTAINERS | 6 + config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 26 ++++ doc/guides/rel_notes/release_19_05.rst | 5 + lib/Makefile | 2 + lib/librte_stack/Makefile | 23 +++ lib/librte_stack/meson.build | 8 + lib/librte_stack/rte_stack.c | 194 +++++++++++++++++++++++ lib/librte_stack/rte_stack.h | 277 +++++++++++++++++++++++++++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++++ lib/librte_stack/rte_stack_version.map | 9 ++ lib/meson.build | 2 +- mk/rte.app.mk | 1 + 16 files changed, 594 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
@@ -407,6 +407,12 @@ F: drivers/raw/skeleton_rawdev/ F: test/test/test_rawdev.c F: doc/guides/prog_guide/rawdev.rst +Stack API - EXPERIMENTAL +M: Gage Eads <[email protected]> +M: Olivier Matz <[email protected]> +F: lib/librte_stack/ +F: doc/guides/prog_guide/stack_lib.rst + Memory Pool Drivers -------------------
config/common_base +5 -0
@@ -980,3 +980,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y # Compile the eventdev application # CONFIG_RTE_APP_EVENTDEV=y + +# +# Compile librte_stack +# +CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
@@ -124,6 +124,7 @@ The public API headers are grouped by topics: [mbuf] (@ref rte_mbuf.h), [mbuf pool ops] (@ref rte_mbuf_pool_ops.h), [ring] (@ref rte_ring.h), + [stack] (@ref rte_stack.h), [tailq] (@ref rte_tailq.h), [bitmap] (@ref rte_bitmap.h)
doc/api/doxy-api.conf.in +1 -0
@@ -56,6 +56,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/librte_ring \ @TOPDIR@/lib/librte_sched \ @TOPDIR@/lib/librte_security \ + @TOPDIR@/lib/librte_stack \ @TOPDIR@/lib/librte_table \ @TOPDIR@/lib/librte_telemetry \ @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
@@ -55,6 +55,7 @@ Programmer's Guide metrics_lib bpf_lib ipsec_lib + stack_lib source_org dev_kit_build_system dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +26 -0
@@ -0,0 +1,26 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +Stack Library +============= + +DPDK's stack library provides an API for configuration and use of a bounded stack of +pointers. + +The stack library provides the following basic operations: + +* Create a uniquely named stack of a user-specified size and using a user-specified socket. + +* Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe. + +* Free a previously created stack. + +* Lookup a pointer to a stack by its name. + +* Query a stack's current depth and number of free entries. + +Implementation +~~~~~~~~~~~~~~ + +The stack consists of a contiguous array of pointers, a current index, and a +spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -65,6 +65,11 @@ New Features process. * Added support for Rx packet types list in a secondary process. +* **Added Stack API.** + + Added a new stack API for configuration and use of a bounded stack of + pointers. The API provides MT-safe push and pop operations that can operate + on one or more pointers per operation. Removed Items -------------
lib/Makefile +2 -0
@@ -111,6 +111,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev +DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack +DEPDIRS-librte_stack := librte_eal ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +23 -0
@@ -0,0 +1,23 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_stack.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +CFLAGS += -DALLOW_EXPERIMENTAL_API +LDLIBS += -lrte_eal + +EXPORT_MAP := rte_stack_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c + +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h + +include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
@@ -0,0 +1,8 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +allow_experimental_apis = true + +version = 1 +sources = files('rte_stack.c') +headers = files('rte_stack.h')
lib/librte_stack/rte_stack.c new file +194 -0
194 lines not on this page · raw message
lib/librte_stack/rte_stack.h new file +277 -0
277 lines not on this page · raw message
lib/librte_stack/rte_stack_pvt.h new file +34 -0
@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#ifndef _RTE_STACK_PVT_H_ +#define _RTE_STACK_PVT_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <rte_log.h> + +extern int stack_logtype; + +#define STACK_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \ + __func__, ##args) + +#define STACK_LOG_ERR(fmt, args...) \ + STACK_LOG(ERR, fmt, ## args) + +#define STACK_LOG_WARN(fmt, args...) \ + STACK_LOG(WARNING, fmt, ## args) + +#define STACK_LOG_INFO(fmt, args...) \ + STACK_LOG(INFO, fmt, ## args) + + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_version.map new file +9 -0
@@ -0,0 +1,9 @@ +EXPERIMENTAL { + global: + + rte_stack_create; + rte_stack_free; + rte_stack_lookup; + + local: *; +};
lib/meson.build +1 -1
@@ -21,7 +21,7 @@ libraries = [ 'compat', # just a header, used for versioning 'gro', 'gso', 'ip_frag', 'jobstats', 'kni', 'latencystats', 'lpm', 'member', 'power', 'pdump', 'rawdev', - 'reorder', 'sched', 'security', 'vhost', + 'reorder', 'sched', 'security', 'stack', 'vhost', #ipsec lib depends on crypto and security 'ipsec', # add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY) += -lrte_security _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += -lrte_compressdev _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += -lrte_eventdev _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV) += -lrte_rawdev +_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER) += -lrte_timer _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL) += -lrte_mempool _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
Signature · 1 line
2.13.6
Re: [PATCH 1/7] stack: introduce rte stack library
Hi Gage,
Please find few comments below.
Quoted 11 lines · Gage Eads
The
rte_stacklibrary provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.The library's interface is modeled after another DPDK data structure,
rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler torte_stack.Signed-off-by Gage Eads <[email protected]>
[...]
Quoted 13 lines
doc/guides/prog_guide/stack_lib.rst new file +10
-0@@ -0,0 +1,26 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +Stack Library +============= + +DPDK's stack library provides an API for configuration and use of a bounded stack of +pointers. + +The stack library provides the following basic operations:
doc/guides/prog_guide/stack_lib.rst line 20
Quoted 10 lines
doc/guides/prog_guide/stack_lib.rst new file +10
-0+ +* Create a uniquely named stack of a user-specified size and using a user-specified socket. + +* Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe. + +* Free a previously created stack. + +* Lookup a pointer to a stack by its name. + +* Query a stack's current depth and number of free entries.
It seems the 80-cols limitation also applies to documentation:
https://mails.dpdk.org/archives/dev/2019-February/124917.html
[...]
Quoted 61 lines
lib/librte_stack/rte_stack.h new file +58
-0@@ -0,0 +1,277 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +/** +* @file rte_stack.h +* @b EXPERIMENTAL: this API may change without prior notice +* +* RTE Stack. +* librte_stack provides an API for configuration and use of a bounded stack of +* pointers. Push and pop operations are MT-safe, allowing concurrent access, +* and the interface supports pushing and popping multiple pointers at a time. +*/ + +#ifndef _RTE_STACK_H_ +#define _RTE_STACK_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <rte_errno.h> +#include <rte_memzone.h> +#include <rte_spinlock.h> + +#define RTE_TAILQ_STACK_NAME "RTE_STACK" +#define RTE_STACK_MZ_PREFIX "STK_" +/**< The maximum length of a stack name. */ +#define RTE_STACK_NAMESIZE (RTE_MEMZONE_NAMESIZE - \ + sizeof(RTE_STACK_MZ_PREFIX) + 1) + +/* Structure containing the LIFO, its current length, and a lock for mutual + * exclusion. + */ +struct rte_lifo { + rte_spinlock_t lock; /**< LIFO lock */ + uint32_t len; /**< LIFO len */ + void *objs[]; /**< LIFO pointer table */ +}; + +/* The RTE stack structure contains the LIFO structure itself, plus metadata + * such as its name and memzone pointer. + */ +struct rte_stack { + /** Name of the stack. */ + char name[RTE_STACK_NAMESIZE] __rte_cache_aligned; + /** Memzone containing the rte_stack structure */ + const struct rte_memzone *memzone; + uint32_t capacity; /**< Usable size of the stack */ + uint32_t flags; /**< Flags supplied at creation */ + struct rte_lifo lifo; /**< LIFO structure */ +} __rte_cache_aligned; + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * @internal Push several objects on the stack (MT-safe)
lib/librte_stack/rte_stack.h line 68
Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10
-0+* +* @param s +* A pointer to the stack structure. +* @param obj_table +* A pointer to a table of void * pointers (objects). +* @param n +* The number of objects to push on the stack from the obj_table. +* @return +* Actual number of objects pushed (either 0 or *n*). +*/
Minor: a dot is missing at the end of the title. There are few in this patch, and maybe in next ones.
[...]
Quoted 6 lines
lib/librte_stack/rte_stack.h new file +6
-0+/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Return the number of used entries in a stack. +*
lib/librte_stack/rte_stack.h line 194
Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10
-0+* @param s +* A pointer to the stack structure. +* @return +* The number of used entries in the stack. +*/ +static __rte_always_inline unsigned int __rte_experimental +rte_stack_count (struct rte_stack *s) +{ +return (unsigned int )s->lifo.len; +}
The argument can be const.
Quoted 7 lines
lib/librte_stack/rte_stack.h new file +7
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Return the number of free entries in a stack. +*
lib/librte_stack/rte_stack.h line 211
Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10
-0+* @param s +* A pointer to the stack structure. +* @return +* The number of free entries in the stack. +*/ +static __rte_always_inline unsigned int __rte_experimental +rte_stack_free_count (struct rte_stack *s) +{ +return s->capacity -rte_stack_count (s); +}
Same here.
Re: [PATCH 1/7] stack: introduce rte stack library
Quoted 50 lines
-----Original Message-----
From: Olivier Matz [mailto:[email protected]]
Sent: Monday, February 25, 2019 4:43 AM
To: Eads, Gage <[email protected]>
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 1/7] stack: introduce rte stack libraryHi Gage,
Please find few comments below.
Quoted 11 lines · Gage Eads
The
rte_stacklibrary provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.The library's interface is modeled after another DPDK data structure,
rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler torte_stack.Signed-off-by Gage Eads <[email protected]>[...]
Quoted 5 lines
doc/guides/prog_guide/stack_lib.rst new file +2
-0@@ -0,0 +1,26 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation.doc/guides/prog_guide/stack_lib.rst line 12
Quoted 10 lines
doc/guides/prog_guide/stack_lib.rst new file +10
-0+ +Stack Library +============= + +DPDK's stack library provides an API for configuration and use of a +bounded stack of pointers. + +The stack library provides the following basic operations: + +* Create a uniquely named stack of a user-specified size and using a user-specified socket.
doc/guides/prog_guide/stack_lib.rst line 14
Quoted 2 lines
doc/guides/prog_guide/stack_lib.rst new file +2
-0+ +* Push and pop a burst of one or more stack objects (pointers). Thesefunction are multi-threading safe.
Quoted 10 lines
doc/guides/prog_guide/stack_lib.rst line 20
Quoted 6 lines
doc/guides/prog_guide/stack_lib.rst new file +6
-0+ +* Free a previously created stack. + +* Lookup a pointer to a stack by its name. + +* Query a stack's current depth and number of free entries.It seems the 80-cols limitation also applies to documentation:
https://mails.dpdk.org/archives/dev/2019-February/124917.html
Sure, will fix in v2.
Quoted 69 lines
[...]
Quoted 67 lines
lib/librte_stack/rte_stack.h new file +64
-0@@ -0,0 +1,277 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation */ + +/** +* @file rte_stack.h +* @b EXPERIMENTAL: this API may change without prior notice +* +* RTE Stack. +* librte_stack provides an API for configuration and use of a +bounded stack of +* pointers. Push and pop operations are MT-safe, allowing concurrent +access, +* and the interface supports pushing and popping multiple pointers at a time. +*/ + +#ifndef _RTE_STACK_H_ +#define _RTE_STACK_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <rte_errno.h> +#include <rte_memzone.h> +#include <rte_spinlock.h> + +#define RTE_TAILQ_STACK_NAME "RTE_STACK" +#define RTE_STACK_MZ_PREFIX "STK_" +/**< The maximum length of a stack name. */ #define +RTE_STACK_NAMESIZE (RTE_MEMZONE_NAMESIZE - \ + sizeof(RTE_STACK_MZ_PREFIX) + 1) + +/* Structure containing the LIFO, its current length, and a lock for +mutual + * exclusion. + */ +struct rte_lifo { + rte_spinlock_t lock; /**< LIFO lock */ + uint32_t len; /**< LIFO len */ + void *objs[]; /**< LIFO pointer table */ }; + +/* The RTE stack structure contains the LIFO structure itself, plus +metadata + * such as its name and memzone pointer. + */ +struct rte_stack { + /** Name of the stack. */ + char name[RTE_STACK_NAMESIZE] __rte_cache_aligned; + /** Memzone containing the rte_stack structure */ + const struct rte_memzone *memzone; + uint32_t capacity; /**< Usable size of the stack */ + uint32_t flags; /**< Flags supplied at creation */ + struct rte_lifo lifo; /**< LIFO structure */ } __rte_cache_aligned; + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * @internal Push several objects on the stack (MT-safe) + * + * @param s + * A pointer to the stack structure. + * @param obj_table + * A pointer to a table of void * pointers (objects).
Quoted 9 lines
lib/librte_stack/rte_stack.h line 68
Quoted 5 lines
lib/librte_stack/rte_stack.h new file +5
-0+* @param n +* The number of objects to push on the stack from the obj_table. +* @return +* Actual number of objects pushed (either 0 or *n*). +*/ Minor: a dot is missing at the end of the title. There are few in this patch, and maybe in next ones.
Will fix.
Quoted 29 lines
[...]
Quoted 5 lines
lib/librte_stack/rte_stack.h new file +5
-0+/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Return the number of used entries in a stack. lib/librte_stack/rte_stack.h line 194
Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10
-0+* +* @param s +* A pointer to the stack structure. +* @return +* The number of used entries in the stack. +*/ +static __rte_always_inline unsigned int __rte_experimental +rte_stack_count (struct rte_stack *s) { +return (unsigned int )s->lifo.len; +}The argument can be const.
Quoted 9 lines
lib/librte_stack/rte_stack.h new file +9
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Return the number of free entries in a stack. +* +* @param s +* A pointer to the stack structure.
Quoted 8 lines
lib/librte_stack/rte_stack.h
Quoted 6 lines
+ * @return
+ * The number of free entries in the stack.
+ */lib/librte_stack/rte_stack.h new file +3
-0+static __rte_always_inline unsigned int __rte_experimental +rte_stack_free_count (struct rte_stack *s) { +return s->capacity -rte_stack_count (s); }Same here.
Unfortunately the const keyword causes a discarded-qualifiers warning in the non-blocking implementation, due to the call to rte_atomic64_read().
Thanks,
Gage
PATCH 2/7 mempool/stack: convert mempool to use rte stack
In reply to Gage EadsThe new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.
--- MAINTAINERS | 2 +- drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++---------------------- 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
@@ -284,7 +284,6 @@ M: Andrew Rybchenko <[email protected]> F: lib/librte_mempool/ F: drivers/mempool/Makefile F: drivers/mempool/ring/ -F: drivers/mempool/stack/ F: doc/guides/prog_guide/mempool_lib.rst F: test/test/test_mempool* F: test/test/test_func_reentrancy.c
@@ -412,6 +411,7 @@ M: Gage Eads <[email protected]> M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst +F: drivers/mempool/stack/ Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a CFLAGS += -O3 CFLAGS += $(WERROR_FLAGS) +CFLAGS += -DALLOW_EXPERIMENTAL_API # Headers CFLAGS += -I$(RTE_SDK)/lib/librte_mempool -LDLIBS += -lrte_eal -lrte_mempool -lrte_ring +LDLIBS += -lrte_eal -lrte_mempool -lrte_stack EXPORT_MAP := rte_mempool_stack_version.map
drivers/mempool/stack/meson.build +5 -1
@@ -1,4 +1,8 @@ # SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation +# Copyright(c) 2017-2019 Intel Corporation + +allow_experimental_apis = true sources = files('rte_mempool_stack.c') + +deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68
133 lines not on this page · raw message
Signature · 1 line
2.13.6
Re: [PATCH 2/7] mempool/stack: convert mempool to use rte stack
Quoted 5 lines · Gage Eads
The new
rte_stacklibrary is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.Signed-off-by Gage Eads <[email protected]>
PATCH 3/7 test/stack: add stack test
In reply to Gage Eadsstack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.
--- MAINTAINERS | 1 + test/test/Makefile | 2 + test/test/meson.build | 3 + test/test/test_stack.c | 394 +++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 400 insertions(+) create mode 100644 test/test/test_stack.c
MAINTAINERS +1 -0
@@ -412,6 +412,7 @@ M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst F: drivers/mempool/stack/ +F: test/test/*stack* Memory Pool Drivers
test/test/Makefile +2 -0
@@ -89,6 +89,8 @@ endif SRCS-y += test_rwlock.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c + SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
test/test/meson.build +3 -0
@@ -95,6 +95,7 @@ test_sources = files('commands.c', 'test_sched.c', 'test_service_cores.c', 'test_spinlock.c', + 'test_stack.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -132,6 +133,7 @@ test_deps = ['acl', 'port', 'reorder', 'ring', + 'stack', 'timer' ]
@@ -173,6 +175,7 @@ fast_parallel_test_names = [ 'rwlock_autotest', 'sched_autotest', 'spinlock_autotest', + 'stack_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
test/test/test_stack.c new file +394 -0
394 lines not on this page · raw message
Signature · 1 line
2.13.6
Re: [PATCH 3/7] test/stack: add stack test
Quoted 4 lines · Gage Eads
stack_autotestperforms positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.Signed-off-by Gage Eads <[email protected]>
[...]
Quoted 14 lines
test/test/test_stack.c new file +11
-0@@ -0,0 +1,394 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#include <string.h> + +#include <rte_lcore.h> +#include <rte_malloc.h> +#include <rte_random.h> +#include <rte_stack.h> +
test/test/test_stack.c line 21
Quoted 10 lines
test/test/test_stack.c new file +10
-0+#include "test.h" + +#define STACK_SIZE 4096 +#define MAX_BULK 32 + +static int +test_stack_push_pop (struct rte_stack *s,void **obj_table,unsigned int bulk_sz) +{ +void *popped_objs[STACK_SIZE ]; +unsigned int i, ret;
Here, a dynamic sized table is used. In test_stack_basic() below, it uses a heap-based allocation for the same purpose. I think it would be more consistent to have the same method for both. I suggest to allocate in heap to avoid a stack overflow if STACK_SIZE is increased in the future.
[...]
Quoted 68 lines
test/test/test_stack.c new file +68
-0+static int +test_stack_basic (void ) +{ +struct rte_stack *s =NULL ; +void **obj_table =NULL ; +int i, ret =-1 ; + + obj_table =rte_calloc (NULL ,STACK_SIZE ,sizeof (void *),0 ); +if (obj_table ==NULL ) { +printf ("[%s():%u] failed to calloc %lu bytes\n" , + __func__, __LINE__,STACK_SIZE *sizeof (void *)); + goto fail_test; + } + +for (i =0 ; i <STACK_SIZE ; i++) + obj_table[i] = (void *)(uintptr_t )i; + + s =rte_stack_create (__func__,STACK_SIZE ,rte_socket_id (),0 ); +if (s ==NULL ) { +printf ("[%s():%u] failed to create a stack\n" , + __func__, __LINE__); + goto fail_test; + } + +if (rte_stack_lookup (__func__) != s) { +printf ("[%s():%u] failed to lookup a stack\n" , + __func__, __LINE__); + goto fail_test; + } + +if (rte_stack_count (s) !=0 ) { +printf ("[%s():%u] stack count: %u (expected 0)\n" , + __func__, __LINE__,rte_stack_count (s)); + goto fail_test; + } + +if (rte_stack_free_count (s) !=STACK_SIZE ) { +printf ("[%s():%u] stack free count: %u (expected %u)\n" , + __func__, __LINE__,rte_stack_count (s),STACK_SIZE ); + goto fail_test; + } + + ret =test_stack_push_pop (s, obj_table,1 ); +if (ret) { +printf ("[%s():%u] Single object push/pop failed\n" , + __func__, __LINE__); + goto fail_test; + } + + ret =test_stack_push_pop (s, obj_table,MAX_BULK ); +if (ret) { +printf ("[%s():%u] Bulk object push/pop failed\n" , + __func__, __LINE__); + goto fail_test; + } + + ret =rte_stack_push (s, obj_table,2 *STACK_SIZE ); +if (ret !=0 ) { +printf ("[%s():%u] Excess objects push succeeded\n" , + __func__, __LINE__); + goto fail_test; + } + + ret =rte_stack_pop (s, obj_table,1 ); +if (ret !=0 ) { +printf ("[%s():%u] Empty stack pop succeeded\n" , + __func__, __LINE__); + goto fail_test;
test/test/test_stack.c line 160
Quoted 10 lines
test/test/test_stack.c new file +10
-0+ } + + ret =0 ; + +fail_test: +rte_stack_free (s); + +if (obj_table !=NULL ) +rte_free (obj_table); +
The if can be removed.
Quoted 4 lines
test/test/test_stack.c new file +4
-0+static int +test_stack_name_length (void ) +{ +char name[RTE_STACK_NAMESIZE +1 ];
test/test/test_stack.c line 201
Quoted 10 lines
test/test/test_stack.c new file +10
-0+struct rte_stack *s; + +memset (name,'s' ,sizeof (name)); + + s =rte_stack_create (name,STACK_SIZE ,rte_socket_id (),0 ); +if (s !=NULL ) { +printf ("[%s():%u] Failed to prevent long name\n" , + __func__, __LINE__); +return -1 ; + }
Here, "name" is not a valid string (no \0 at the end). It does not hurt because the length check is properly done in the lib, but we could imagine that the wrong name is logged by the library on error, which would trigger a crash here. So I suggest to pass a valid string instead.
Re: [PATCH 3/7] test/stack: add stack test
Quoted 41 lines
-----Original Message-----
From: Olivier Matz [mailto:[email protected]]
Sent: Monday, February 25, 2019 4:59 AM
To: Eads, Gage <[email protected]>
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 3/7] test/stack: add stack testQuoted 2 lines · Gage Eads
stack_autotestperforms positive and negative testing of the stack API, and exercises the push and pop datapath functions with all availablelcores.
Quoted 2 lines
Signed-off-by Gage Eads <[email protected]>[...]
Quoted 20 lines
test/test/test_stack.c new file +17
-0@@ -0,0 +1,394 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation */ + +#include <string.h> + +#include <rte_lcore.h> +#include <rte_malloc.h> +#include <rte_random.h> +#include <rte_stack.h> + +#include "test.h" + +#define STACK_SIZE 4096 +#define MAX_BULK 32 + +static int +test_stack_push_pop(struct rte_stack *s, void **obj_table, unsigned
Quoted 9 lines
test/test/test_stack.c line 21
Quoted 3 lines
test/test/test_stack.c new file +3
-0+int bulk_sz) { +void *popped_objs[STACK_SIZE ]; +unsigned int i, ret;Here, a dynamic sized table is used. In
test_stack_basic()below, it uses a heap- based allocation for the same purpose. I think it would be more consistent to have the same method for both. I suggest to allocate in heap to avoid a stack overflow ifSTACK_SIZEis increased in the future.
Sure, I'll make popped_objs dynamically allocated.
Quoted 74 lines
[...]
Quoted 72 lines
test/test/test_stack.c new file +72
-0+static int +test_stack_basic (void ) +{ +struct rte_stack *s =NULL ; +void **obj_table =NULL ; +int i, ret =-1 ; + + obj_table =rte_calloc (NULL ,STACK_SIZE ,sizeof (void *),0 ); +if (obj_table ==NULL ) { +printf ("[%s():%u] failed to calloc %lu bytes\n" , + __func__, __LINE__,STACK_SIZE *sizeof (void *)); + goto fail_test; + } + +for (i =0 ; i <STACK_SIZE ; i++) + obj_table[i] = (void *)(uintptr_t )i; + + s =rte_stack_create (__func__,STACK_SIZE ,rte_socket_id (),0 ); +if (s ==NULL ) { +printf ("[%s():%u] failed to create a stack\n" , + __func__, __LINE__); + goto fail_test; + } + +if (rte_stack_lookup (__func__) != s) { +printf ("[%s():%u] failed to lookup a stack\n" , + __func__, __LINE__); + goto fail_test; + } + +if (rte_stack_count (s) !=0 ) { +printf ("[%s():%u] stack count: %u (expected 0)\n" , + __func__, __LINE__,rte_stack_count (s)); + goto fail_test; + } + +if (rte_stack_free_count (s) !=STACK_SIZE ) { +printf ("[%s():%u] stack free count: %u (expected %u)\n" , + __func__, __LINE__,rte_stack_count (s),STACK_SIZE ); + goto fail_test; + } + + ret =test_stack_push_pop (s, obj_table,1 ); +if (ret) { +printf ("[%s():%u] Single object push/pop failed\n" , + __func__, __LINE__); + goto fail_test; + } + + ret =test_stack_push_pop (s, obj_table,MAX_BULK ); +if (ret) { +printf ("[%s():%u] Bulk object push/pop failed\n" , + __func__, __LINE__); + goto fail_test; + } + + ret =rte_stack_push (s, obj_table,2 *STACK_SIZE ); +if (ret !=0 ) { +printf ("[%s():%u] Excess objects push succeeded\n" , + __func__, __LINE__); + goto fail_test; + } + + ret =rte_stack_pop (s, obj_table,1 ); +if (ret !=0 ) { +printf ("[%s():%u] Empty stack pop succeeded\n" , + __func__, __LINE__); + goto fail_test; + } + + ret =0 ; +
Quoted 8 lines
test/test/test_stack.c line 160
Quoted 6 lines
test/test/test_stack.c new file +6
-0+fail_test: +rte_stack_free (s); + +if (obj_table !=NULL ) +rte_free (obj_table); +The if can be removed.
Ah, I didn't know rte_free() checks for NULL. Will remove.
Quoted 12 lines
Quoted 12 lines
test/test/test_stack.c new file +11
-0+static int +test_stack_name_length (void ) +{ +char name[RTE_STACK_NAMESIZE +1 ]; +struct rte_stack *s; + +memset (name,'s' ,sizeof (name)); + + s =rte_stack_create (name,STACK_SIZE ,rte_socket_id (),0 ); +if (s !=NULL ) { + printf("[%s():%u] Failed to prevent long name\n",
Quoted 8 lines
test/test/test_stack.c line 174
Quoted 3 lines
test/test/test_stack.c new file +3
-0+ __func__, __LINE__); +return -1 ; + }Here, "name" is not a valid string (no \0 at the end). It does not hurt because the length check is properly done in the lib, but we could imagine that the wrong name is logged by the library on error, which would trigger a crash here. So I suggest to pass a valid string instead.
Good catch. Will fix.
Thanks,
Gage
PATCH 4/7 test/stack: add stack perf test
In reply to Gage Eadsstack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects
It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores
--- test/test/Makefile | 1 + test/test/meson.build | 2 + test/test/test_stack_perf.c | 343 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 346 insertions(+) create mode 100644 test/test/test_stack_perf.c
test/test/Makefile +1 -0
@@ -90,6 +90,7 @@ endif SRCS-y += test_rwlock.c SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
test/test/meson.build +2 -0
@@ -96,6 +96,7 @@ test_sources = files('commands.c', 'test_service_cores.c', 'test_spinlock.c', 'test_stack.c', + 'test_stack_perf.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -240,6 +241,7 @@ perf_test_names = [ 'distributor_perf_autotest', 'ring_pmd_perf_autotest', 'pmd_perf_autotest', + 'stack_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
test/test/test_stack_perf.c new file +343 -0
343 lines not on this page · raw message
Signature · 1 line
2.13.6
Re: [PATCH 4/7] test/stack: add stack perf test
Quoted 5 lines · Gage Eads
stack_perf_autotesttests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects
Quoted 8 lines
It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcoresSigned-off-by Gage Eads <[email protected]>
PATCH 5/7 stack: add non-blocking stack implementation
In reply to Gage EadsThis commit adds support for a non-blocking (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, STACK_F_NB.
The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.
The non-blocking push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.
The non-blocking pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.
This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.
The linked list elements themselves are maintained in a non-blocking LIFO, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.
--- doc/guides/prog_guide/stack_lib.rst | 46 ++++++++- doc/guides/rel_notes/release_19_05.rst | 3 + lib/librte_stack/Makefile | 4 +- lib/librte_stack/meson.build | 4 +- lib/librte_stack/rte_stack.c | 42 ++++++-- lib/librte_stack/rte_stack.h | 139 +++++++++++++++++++++++++- lib/librte_stack/rte_stack_c11_mem.h | 173 +++++++++++++++++++++++++++++++++ lib/librte_stack/rte_stack_generic.h | 157 ++++++++++++++++++++++++++++++ 8 files changed, 550 insertions(+), 18 deletions(-) create mode 100644 lib/librte_stack/rte_stack_c11_mem.h create mode 100644 lib/librte_stack/rte_stack_generic.h
doc/guides/prog_guide/stack_lib.rst +43 -3
@@ -9,7 +9,7 @@ pointers. The stack library provides the following basic operations: -* Create a uniquely named stack of a user-specified size and using a user-specified socket. +* Create a uniquely named stack of a user-specified size and using a user-specified socket, with either lock-based or non-blocking behavior. * Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe.
@@ -22,5 +22,45 @@ The stack library provides the following basic operations: Implementation ~~~~~~~~~~~~~~ -The stack consists of a contiguous array of pointers, a current index, and a -spinlock. Accesses to the stack are made multi-thread safe by the spinlock. +The library supports two types of stacks: lock-based and non-blocking. +Both types use the same set of interfaces, but their implementations differ. + +Lock-based Stack +---------------- + +The lock-based stack consists of a contiguous array of pointers, a current index, and a spinlock. +Accesses to the stack are made multi-thread safe by the spinlock. + +The lock-based stack consists of a contiguous array of pointers, a current +index, and a spinlock. Accesses to the stack are made multi-thread safe by the +spinlock. + +Non-blocking Stack +------------------ + +The non-blocking stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter. +The non-blocking property means that multiple threads can push and pop simultaneously, and one thread being preempted/delayed in a push or pop operation will not impede the forward progress of any other thread. + +The non-blocking push operation enqueues a linked list of pointers by pointing the list's tail to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. +The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns. + +The non-blocking pop operation first reserves one or more list elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. +It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). +While walking the list, the data pointers are recorded in an object table. + +The linked list elements themselves are maintained in a non-blocking LIFO, and are allocated before stack pushes and freed after stack pops. +Since the stack has a fixed maximum depth, these elements do not need to be dynamically created. + +The non-blocking behavior is selected by passing the *STACK_F_NB* flag to rte_stack_create(). + +Preventing the ABA Problem +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To prevent the ABA problem, this algorithm stack uses a 128-bit compare-and-swap instruction to atomically update both the stack top pointer and a modification counter. The ABA problem can occur without a modification counter if, for example: + +1. Thread A reads head pointer X and stores the pointed-to list element. +2. Other threads modify the list such that the head pointer is once again X, but its pointed-to data is different than what thread A read. +3. Thread A changes the head pointer with a compare-and-swap and succeeds. + +In this case thread A would not detect that the list had changed, and would both pop stale data and incorrect change the head pointer. +By adding a modification counter that is updated on every push and pop as part of the compare-and-swap, the algorithm can detect when the list changes even if the head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
@@ -71,6 +71,9 @@ New Features pointers. The API provides MT-safe push and pop operations that can operate on one or more pointers per operation. + The library supports two stack implementations: lock-based and non-blocking. + The non-blocking implementation is currently limited to x86-64 platforms. + Removed Items -------------
lib/librte_stack/Makefile +3 -1
@@ -18,6 +18,8 @@ LIBABIVER := 1 SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c # install includes -SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h +SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ + rte_stack_generic.h \ + rte_stack_c11_mem.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +3 -1
@@ -5,4 +5,6 @@ allow_experimental_apis = true version = 1 sources = files('rte_stack.c') -headers = files('rte_stack.h') +headers = files('rte_stack.h', + 'rte_stack_c11_mem.h', + 'rte_stack_generic.h')
lib/librte_stack/rte_stack.c +34 -8
77 lines not on this page · raw message
lib/librte_stack/rte_stack.h +134 -5
178 lines not on this page · raw message
lib/librte_stack/rte_stack_c11_mem.h new file +173 -0
173 lines not on this page · raw message
lib/librte_stack/rte_stack_generic.h new file +157 -0
157 lines not on this page · raw message
Signature · 1 line
2.13.6
Re: [PATCH 5/7] stack: add non-blocking stack implementation
Quoted 29 lines · Gage Eads
This commit adds support for a non-blocking (linked list based) stack to the stack API. This behavior is selected through a new
rte_stack_create()flag,STACK_F_NB.The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.
The non-blocking push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.
The non-blocking pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.
This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.
The linked list elements themselves are maintained in a non-blocking LIFO, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.
Signed-off-by Gage Eads <[email protected]>
[...]
doc/guides/prog_guide/stack_lib.rst pointers. · line 15
Quoted 13 lines
doc/guides/prog_guide/stack_lib.rst +1
-1@@ -9,7 +9,7 @@ pointers. The stack library provides the following basic operations: -* Create a uniquely named stack of a user-specified size and using a user-specified socket. +* Create a uniquely named stack of a user-specified size and using a user-specified socket, with either lock-based or non-blocking behavior. * Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe.
Same comment about 80-cols than in the first patch.
[...]
Quoted 5 lines
lib/librte_stack/rte_stack.c +0
-0@@ -26,27 +26,46 @@ static struct rte_tailq_elem rte_stack_tailq = {EAL_REGISTER_TAILQ (rte_stack_tailq)
lib/librte_stack/rte_stack.c static struct rte_tailq_elem rte_stack_tailq = { · line 37
Quoted 10 lines
lib/librte_stack/rte_stack.c +9
-0static void +nb_lifo_init (struct rte_stack *s,unsigned int count) +{ +struct rte_nb_lifo_elem *elems; +unsigned int i; + + elems = (struct rte_nb_lifo_elem *)&s[1 ]; +for (i =0 ; i < count; i++) +__nb_lifo_push (&s->nb_lifo.free, &elems[i], &elems[i],1 ); +}
Would it be possible to add:
struct rte_nb_lifo {
/** LIFO list of elements */
struct rte_nb_lifo_list used __rte_cache_aligned;
/** LIFO list of free elements */
struct rte_nb_lifo_list free __rte_cache_aligned;
+ struct rte_nb_lifo_elem elems[];
};I think it is more consistent with the non-blocking structure.
[...]
Quoted 23 lines
lib/librte_stack/rte_stack.h +17
-0@@ -29,6 +29,33 @@ extern "C" {#define RTE_STACK_NAMESIZE (RTE_MEMZONE_NAMESIZE - \ sizeof(RTE_STACK_MZ_PREFIX) + 1) +struct rte_nb_lifo_elem { +void *data;/**< Data pointer */ +struct rte_nb_lifo_elem *next;/**< Next pointer */ +}; + +struct rte_nb_lifo_head { +struct rte_nb_lifo_elem *top;/**< Stack top */ +uint64_t cnt;/**< Modification counter for avoiding ABA problem */ +}; + +struct rte_nb_lifo_list { +/** List head */ + struct rte_nb_lifo_head head __rte_aligned(16); +/** List len */ +rte_atomic64_t len; +}; +
lib/librte_stack/rte_stack.h extern "C" { · line 58
Quoted 10 lines
lib/librte_stack/rte_stack.h +10
-0+/* Structure containing two non-blocking LIFO lists: the stack itself and a +* list of free linked-list elements. +*/ +struct rte_nb_lifo { +/** LIFO list of elements */ + struct rte_nb_lifo_list used __rte_cache_aligned; +/** LIFO list of free elements */ + struct rte_nb_lifo_list free __rte_cache_aligned; +}; +
The names "rte_nb_lifo*" bothers me a bit. I think a more usual name format is "rte_<module_name>_<struct_name>".
What would you think about names like this?
rte_nb_lifo -> rte_stack_nb
rte_nb_lifo_elem -> rte_stack_nb_elem
rte_nb_lifo_head -> rte_stack_nb_head
rte_nb_lifo_list -> rte_stack_nb_list
rte_lifo -> rte_stack_std
I even wonder if "nonblock", "noblk", or "lockless" shouldn't be used in place of "nb" (which is also a common abbreviation for number). This can also applies to the STACK_F_NB flag name.
[...]
Quoted 10 lines
lib/librte_stack/rte_stack.h +2
-1/* Structure containing the LIFO, its current length, and a lock for mutual * exclusion. */ @@ -48,10 +75,69 @@ struct rte_stack {const struct rte_memzone *memzone;uint32_t capacity;/**< Usable size of the stack */ uint32_t flags;/**< Flags supplied at creation */ -struct rte_lifo lifo;/**< LIFO structure */ +RTE_STD_C11 + union {
lib/librte_stack/rte_stack.h struct rte_stack { · line 89
Quoted 10 lines
lib/librte_stack/rte_stack.h +7
-0+struct rte_nb_lifo nb_lifo;/**< Non-blocking LIFO structure */ +struct rte_lifo lifo;/**< LIFO structure */ + }; } __rte_cache_aligned;/** +* The stack uses non-blocking push and pop functions. This flag is only +* supported on x86_64 platforms, currently. +*/ +#define STACK_F_NB 0x0001
What about adding the RTE_ prefix?
Quoted 17 lines
lib/librte_stack/rte_stack.h +17
-0+static __rte_always_inline unsigned int __rte_experimental +rte_nb_lifo_push (struct rte_stack *s,void *const *obj_table,unsigned int n) +{ +struct rte_nb_lifo_elem *tmp, *first, *last =NULL ; +unsigned int i; + +if (unlikely (n ==0 )) +return 0 ; + +/* Pop n free elements */ + first =__nb_lifo_pop (&s->nb_lifo.free, n,NULL ,NULL ); +if (unlikely (first ==NULL )) +return 0 ; + +/* Construct the list elements */ + tmp = first; +for (i =0 ; i < n; i++) {
lib/librte_stack/rte_stack.h struct rte_stack { · line 138
Quoted 10 lines
lib/librte_stack/rte_stack.h +10
-0+ tmp->data = obj_table[n - i -1 ]; + last = tmp; + tmp = tmp->next; + } + +/* Push them to the used list */ +__nb_lifo_push (&s->nb_lifo.used, first, last, n); + +return n; +}
Here, I didn't get why "last" is not retrieved through __nb_lifo_pop(), like it's done in rte_nb_lifo_pop(). Is there a reason for that?
[...]
lib/librte_stack/rte_stack_c11_mem.h
Quoted 2 lines
lib/librte_stack/rte_stack_c11_mem.h new file +0
-0
For the c11 memory model, please consider having an additional reviewer ;)
[...]
Quoted 99 lines
lib/librte_stack/rte_stack_generic.h new file +96
-0@@ -0,0 +1,157 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#ifndef _NB_LIFO_GENERIC_H_ +#define _NB_LIFO_GENERIC_H_ + +#include <rte_branch_prediction.h> +#include <rte_prefetch.h> + +static __rte_always_inline unsigned int +rte_nb_lifo_len (struct rte_stack *s) +{ +/* nb_lifo_push() and nb_lifo_pop() do not update the list's contents +* and nb_lifo->len atomically, which can cause the list to appear +* shorter than it actually is if this function is called while other +* threads are modifying the list. +* +* However, given the inherently approximate nature of the get_count +* callback -- even if the list and its size were updated atomically, +* the size could change between when get_count executes and when the +* value is returned to the caller -- this is acceptable. +* +* The nb_lifo->len updates are placed such that the list may appear to +* have fewer elements than it does, but will never appear to have more +* elements. If the mempool is near-empty to the point that this is a +* concern, the user should consider increasing the mempool size. +*/ +return (unsigned int )rte_atomic64_read (&s->nb_lifo.used.len); +} + +static __rte_always_inline void +__nb_lifo_push (struct rte_nb_lifo_list *lifo, +struct rte_nb_lifo_elem *first, +struct rte_nb_lifo_elem *last, +unsigned int num) +{ +#ifndef RTE_ARCH_X86_64 +RTE_SET_USED (first); +RTE_SET_USED (last); +RTE_SET_USED (lifo); +RTE_SET_USED (num); +#else +struct rte_nb_lifo_head old_head; +int success; + + old_head = lifo->head; + +do { +struct rte_nb_lifo_head new_head; + +/* Swing the top pointer to the first element in the list and +* make the last element point to the old top. +*/ + new_head.top = first; + new_head.cnt = old_head.cnt +1 ; + + last->next = old_head.top; + +/* Ensure the list entry writes are visible before pushing them +* to the stack. +*/ +rte_wmb (); + +/* old_head is updated on failure */ + success =rte_atomic128_cmpxchg ((rte_int128_t *)&lifo->head, + (rte_int128_t *)&old_head, + (rte_int128_t *)&new_head, +1 , __ATOMIC_RELEASE, + __ATOMIC_RELAXED); + }while (success ==0 ); + +rte_atomic64_add (&lifo->len, num); +#endif +} + +static __rte_always_inline struct rte_nb_lifo_elem * +__nb_lifo_pop (struct rte_nb_lifo_list *lifo, +unsigned int num, +void **obj_table, +struct rte_nb_lifo_elem **last) +{ +#ifndef RTE_ARCH_X86_64 +RTE_SET_USED (obj_table); +RTE_SET_USED (last); +RTE_SET_USED (lifo); +RTE_SET_USED (num); + +return NULL ; +#else +struct rte_nb_lifo_head old_head; +int success; + +/* Reserve num elements, if available */ +while (1 ) { +uint64_t len =rte_atomic64_read (&lifo->len);
lib/librte_stack/rte_stack_generic.h line 106
Quoted 10 lines
lib/librte_stack/rte_stack_generic.h new file +10
-0+ +/* Does the list contain enough elements? */ +if (unlikely (len < num)) +return NULL ; + +if (rte_atomic64_cmpset ((volatile uint64_t *)&lifo->len, + len, len - num)) +break ; + } +
Here, accessing the length with a compare and set costs probably more than a standard atomic sub fonction. I understand that was done for the reason described above:
The nb_lifo->len updates are placed such that the list may appear to have fewer elements than it does, but will never appear to have more elements.
Another strategy could be to use a rte_atomic64_sub() after the effective pop and change rte_nb_lifo_len() to bound the result to [0:size].
A message that is not in the archive (sent off-list or not yet received): 2EC44CCD3517A842B44C82651A5557A14AF13386@fmsmsx118.amr.corp.intel.com
PATCH 5/7 stack: add non-blocking stack implementation
Quoted 17 lines
-----Original Message-----
From: Olivier Matz [mailto:[email protected]]
Sent: Monday, February 25, 2019 5:28 AM
To: Eads, Gage <[email protected]>
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 5/7] stack: add non-blocking stack implementationQuoted 4 lines · Gage Eads
This commit adds support for a non-blocking (linked list based) stack to the stack API. This behavior is selected through a new
rte_stack_create()flag,STACK_F_NB.
Quoted 8 lines
Quoted 4 lines
The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.
The non-blocking push operation enqueues a linked list of pointers
by pointing the tail of the list to the current stack head, and
Quoted 3 lines
using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length
and returns.
Quoted 71 lines
Quoted 7 lines
The non-blocking pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in
an object table.
Quoted 11 lines
This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.
The linked list elements themselves are maintained in a non-blocking LIFO, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.
Signed-off-by Gage Eads <[email protected]>[...]
Quoted 11 lines
doc/guides/prog_guide/stack_lib.rst +0
-1@@ -9,7 +9,7 @@ pointers. The stack library provides the following basic operations: -* Create a uniquely named stack of a user-specified size and usinga user-
specified socket.
doc/guides/prog_guide/stack_lib.rst
Quoted 2 lines
doc/guides/prog_guide/stack_lib.rst +2
-0+* Create a uniquely named stack of a user-specified size and using +a user-specified socket, with either lock-based or non-blocking behavior.
Quoted 3 lines
* Push and pop a burst of one or more stack objects (pointers).
Thesefunction are multi-threading safe.
Quoted 1 line
Same comment about 80-cols than in the first patch.
[...]
lib/librte_stack/rte_stack.c static struct rte_tailq_elem rte_stack_tailq = { · line 35
Quoted 13 lines
lib/librte_stack/rte_stack.c +7
-0@@ -26,27 +26,46 @@ static struct rte_tailq_elem rte_stack_tailq = {EAL_REGISTER_TAILQ (rte_stack_tailq) static void +nb_lifo_init (struct rte_stack *s,unsigned int count) { +struct rte_nb_lifo_elem *elems; +unsigned int i; + + elems = (struct rte_nb_lifo_elem *)&s[1 ]; +for (i =0 ; i < count; i++) +__nb_lifo_push (&s->nb_lifo.free, &elems[i], &elems[i],1 ); }Would it be possible to add:
struct rte_nb_lifo { /** LIFO list of elements */ struct rte_nb_lifo_list used __rte_cache_aligned; /** LIFO list of free elements */ struct rte_nb_lifo_list free __rte_cache_aligned; + struct rte_nb_lifo_elem elems[]; };
Quoted 2 lines
I think it is more consistent with the non-blocking structure.
Will do.
Quoted 46 lines
[...]
Quoted 23 lines
lib/librte_stack/rte_stack.h +17
-0@@ -29,6 +29,33 @@ extern "C" {#define RTE_STACK_NAMESIZE (RTE_MEMZONE_NAMESIZE - \ sizeof(RTE_STACK_MZ_PREFIX) + 1) +struct rte_nb_lifo_elem { +void *data;/**< Data pointer */ +struct rte_nb_lifo_elem *next;/**< Next pointer */ +}; + +struct rte_nb_lifo_head { +struct rte_nb_lifo_elem *top;/**< Stack top */ +uint64_t cnt;/**< Modification counter for avoiding ABA problem +*/ }; + +struct rte_nb_lifo_list { +/** List head */ + struct rte_nb_lifo_head head __rte_aligned(16); +/** List len */ +rte_atomic64_t len; +}; +lib/librte_stack/rte_stack.h extern "C" { · line 58
Quoted 10 lines
lib/librte_stack/rte_stack.h +10
-0+/* Structure containing two non-blocking LIFO lists: the stack +itself and a +* list of free linked-list elements. +*/ +struct rte_nb_lifo { +/** LIFO list of elements */ + struct rte_nb_lifo_list used __rte_cache_aligned; +/** LIFO list of free elements */ + struct rte_nb_lifo_list free __rte_cache_aligned; }; +The names "rte_nb_lifo*" bothers me a bit. I think a more usual name format is "rte_<module_name>_<struct_name>".
What would you think about names like this?
rte_nb_lifo->rte_stack_nb
rte_nb_lifo_elem->rte_stack_nb_elem
rte_nb_lifo_head->rte_stack_nb_head
rte_nb_lifo_list->rte_stack_nb_list
rte_lifo->rte_stack_std
Quoted 4 lines
I even wonder if "nonblock", "noblk", or "lockless" shouldn't be used in place of "nb" (which is also a common abbreviation for number). This can also applies to the
STACK_F_NBflag name.
How about std and lf (lock-free)?
Quoted 17 lines
[...]
Quoted 15 lines
/* Structure containing the LIFO, its current length, and a lock for mutual
* exclusion.
*/lib/librte_stack/rte_stack.h +5
-1@@ -48,10 +75,69 @@ struct rte_stack {const struct rte_memzone *memzone;uint32_t capacity;/**< Usable size of the stack */ uint32_t flags;/**< Flags supplied at creation */ -struct rte_lifo lifo;/**< LIFO structure */ +RTE_STD_C11 + union { +struct rte_nb_lifo nb_lifo;/**< Non-blocking LIFO structure */ +struct rte_lifo lifo;/**< LIFO structure */ + }; } __rte_cache_aligned;
Quoted 8 lines
lib/librte_stack/rte_stack.h struct rte_stack { · line 89
Quoted 6 lines
lib/librte_stack/rte_stack.h +5
-0/** +* The stack uses non-blocking push and pop functions. This flag is +only +* supported on x86_64 platforms, currently. +*/ +#define STACK_F_NB 0x0001What about adding the
RTE_prefix?
I'm fine with either, but there's precedent for flag macros named <module_name>_*. E.g. RING_F_*, MEMPOOL_F_*, ETH_*, and SERVICE_F_*.
Quoted 23 lines
Quoted 23 lines
lib/librte_stack/rte_stack.h +22
-0+static __rte_always_inline unsigned int __rte_experimental +rte_nb_lifo_push (struct rte_stack *s,void *const *obj_table, +unsigned int n) { +struct rte_nb_lifo_elem *tmp, *first, *last =NULL ; +unsigned int i; + +if (unlikely (n ==0 )) +return 0 ; + +/* Pop n free elements */ + first =__nb_lifo_pop (&s->nb_lifo.free, n,NULL ,NULL ); +if (unlikely (first ==NULL )) +return 0 ; + +/* Construct the list elements */ + tmp = first; +for (i =0 ; i < n; i++) { + tmp->data = obj_table[n - i -1 ]; + last = tmp; + tmp = tmp->next; + } +
Quoted 9 lines
lib/librte_stack/rte_stack.h struct rte_stack { · line 138
Quoted 5 lines
lib/librte_stack/rte_stack.h +5
-0+/* Push them to the used list */ +__nb_lifo_push (&s->nb_lifo.used, first, last, n); + +return n; +}Here, I didn't get why "last" is not retrieved through
__nb_lifo_pop(), like it's done inrte_nb_lifo_pop(). Is there a reason for that?
Just an simple oversight -- that works, and I'll change it for v2.
Quoted 7 lines
[...]
lib/librte_stack/rte_stack_c11_mem.h
Quoted 2 lines
lib/librte_stack/rte_stack_c11_mem.h new file +0
-0For the c11 memory model, please consider having an additional reviewer ;)
No problem, and I'll break out the C11 implementation into a separate patch in case that makes reviewing it easier.
Quoted 112 lines
[...]
Quoted 12 lines
lib/librte_stack/rte_stack_generic.h new file +9
-0@@ -0,0 +1,157 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation */ + +#ifndef _NB_LIFO_GENERIC_H_ +#define _NB_LIFO_GENERIC_H_ + +#include <rte_branch_prediction.h> +#include <rte_prefetch.h> +lib/librte_stack/rte_stack_generic.h line 19
Quoted 10 lines
lib/librte_stack/rte_stack_generic.h new file +10
-0+static __rte_always_inline unsigned int rte_nb_lifo_len(struct +rte_stack *s) { +/* nb_lifo_push() and nb_lifo_pop() do not update the list's contents +* and nb_lifo->len atomically, which can cause the list to appear +* shorter than it actually is if this function is called while other +* threads are modifying the list. +* +* However, given the inherently approximate nature of the get_count +* callback -- even if the list and its size were updated atomically, +* the size could change between when get_count executes and when the
Quoted 71 lines
lib/librte_stack/rte_stack_generic.h new file +71
-0+* value is returned to the caller -- this is acceptable. +* +* The nb_lifo->len updates are placed such that the list may appear to +* have fewer elements than it does, but will never appear to have more +* elements. If the mempool is near-empty to the point that this is a +* concern, the user should consider increasing the mempool size. +*/ +return (unsigned int )rte_atomic64_read (&s->nb_lifo.used.len); +} + +static __rte_always_inline void +__nb_lifo_push (struct rte_nb_lifo_list *lifo, +struct rte_nb_lifo_elem *first, +struct rte_nb_lifo_elem *last, +unsigned int num) +{ +#ifndef RTE_ARCH_X86_64 +RTE_SET_USED (first); +RTE_SET_USED (last); +RTE_SET_USED (lifo); +RTE_SET_USED (num); +#else +struct rte_nb_lifo_head old_head; +int success; + + old_head = lifo->head; + +do { +struct rte_nb_lifo_head new_head; + +/* Swing the top pointer to the first element in the list and +* make the last element point to the old top. +*/ + new_head.top = first; + new_head.cnt = old_head.cnt +1 ; + + last->next = old_head.top; + +/* Ensure the list entry writes are visible before pushing them +* to the stack. +*/ +rte_wmb (); + +/* old_head is updated on failure */ + success =rte_atomic128_cmpxchg ((rte_int128_t *)&lifo->head, + (rte_int128_t *)&old_head, + (rte_int128_t *)&new_head, +1 , __ATOMIC_RELEASE, + __ATOMIC_RELAXED); + }while (success ==0 ); + +rte_atomic64_add (&lifo->len, num);#endif } + +static __rte_always_inline struct rte_nb_lifo_elem * +__nb_lifo_pop (struct rte_nb_lifo_list *lifo, +unsigned int num, +void **obj_table, +struct rte_nb_lifo_elem **last) {#ifndef RTE_ARCH_X86_64 +RTE_SET_USED (obj_table); +RTE_SET_USED (last); +RTE_SET_USED (lifo); +RTE_SET_USED (num); + +return NULL ; +#else +struct rte_nb_lifo_head old_head; +int success; + +/* Reserve num elements, if available */ +while (1 ) { +uint64_t len =rte_atomic64_read (&lifo->len);lib/librte_stack/rte_stack_generic.h line 106
Quoted 10 lines
lib/librte_stack/rte_stack_generic.h new file +10
-0+ +/* Does the list contain enough elements? */ +if (unlikely (len < num)) +return NULL ; + +if (rte_atomic64_cmpset ((volatile uint64_t *)&lifo->len, + len, len - num)) +break ; + } +Here, accessing the length with a compare and set costs probably more than a standard atomic sub fonction. I understand that was done for the reason described above:
Quoted 6 lines
The
nb_lifo->lenupdates are placed such that the list may appear to have fewer elements than it does, but will never appear to have more elements.Another strategy could be to use a
rte_atomic64_sub()after the effective pop and changerte_nb_lifo_len()to bound the result to [0:size].
It serves a second purpose: if the CAS succeeds, the subsequent do-while loop is guaranteed to (eventually) succeed because we've effectively reserved num elements. Otherwise there's the chance that the list runs empty after popping fewer than num elements.
If folks are interested in this patchset, please also consider reviewing the 128-bit CAS patch here: http://mails.dpdk.org/archives/dev/2019-February/125059.html
Thanks!
Gage
Re: FW: [PATCH 5/7] stack: add non-blocking stack implementation
01/03/2019 21:53, Eads, Gage:
Quoted 8 lines
From: Olivier Matz [mailto:[email protected]]
Quoted 4 lines · Gage Eads
Quoted 1 line
+#define
STACK_F_NB0x0001What about adding the
RTE_prefix?I'm fine with either, but there's precedent for flag macros named <module_name>_*. E.g. RING_F_*, MEMPOOL_F_*, ETH_*, and SERVICE_F_*.
They should be fixed.
Every public symbols should be prefixed to avoid namespace conflict. At first, we should rename them and keep the old name as an alias. Later, non-prefixed names should be removed after a deprecation notice.
Re: FW: [PATCH 5/7] stack: add non-blocking stack implementation
Quoted 19 lines
-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Friday, March 1, 2019 3:13 PM
To: Eads, Gage <[email protected]>
Cc: Olivier Matz <[email protected]>; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: FW: [PATCH 5/7] stack: add non-blocking stack implementation01/03/2019 21:53, Eads, Gage:
Quoted 6 lines
From: Olivier Matz [mailto:[email protected]]
Quoted 4 lines · Gage Eads
Quoted 1 line
+#define
STACK_F_NB0x0001What about adding the
RTE_prefix?
Quoted 9 lines
Quoted 2 lines
I'm fine with either, but there's precedent for flag macros named <module_name>_*. E.g. RING_F_*, MEMPOOL_F_*, ETH_*, and
SERVICE_F_*.
They should be fixed.
Every public symbols should be prefixed to avoid namespace conflict. At first, we should rename them and keep the old name as an alias. Later, non-prefixed names should be removed after a deprecation notice.
Ok, will fix.
Thanks,
Gage
PATCH 6/7 test/stack: add non-blocking stack tests
In reply to Gage EadsThis commit adds non-blocking stack variants of stack_autotest (stack_nb_autotest) and stack_perf_autotest (stack_nb_perf_autotest), which differ only in that the non-blocking versions pass the STACK_F_NB flag to all rte_stack_create() calls.
--- test/test/meson.build | 2 ++ test/test/test_stack.c | 41 +++++++++++++++++++++++++++-------------- test/test/test_stack_perf.c | 17 +++++++++++++++-- 3 files changed, 44 insertions(+), 16 deletions(-)
test/test/meson.build +2 -0
@@ -177,6 +177,7 @@ fast_parallel_test_names = [ 'sched_autotest', 'spinlock_autotest', 'stack_autotest', + 'stack_nb_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
@@ -242,6 +243,7 @@ perf_test_names = [ 'ring_pmd_perf_autotest', 'pmd_perf_autotest', 'stack_perf_autotest', + 'stack_nb_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
test/test/test_stack.c +27 -14
112 lines not on this page · raw message
test/test/test_stack_perf.c +15 -2
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s) }static int -test_stack_perf (void ) +__test_stack_perf (uint32_t flags) {struct lcore_pair cores;struct rte_stack *s;rte_atomic32_init (&lcore_barrier); - s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (),0 ); + s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (), flags);if (s ==NULL ) {printf ("[%s():%u] failed to create a stack\n" , __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)return 0 ; } +static int +test_stack_perf (void ) +{ +return __test_stack_perf (0 ); +} + +static int +test_nb_stack_perf (void ) +{ +return __test_stack_perf (STACK_F_NB ); +} +REGISTER_TEST_COMMAND (stack_perf_autotest, test_stack_perf); +REGISTER_TEST_COMMAND (stack_nb_perf_autotest, test_nb_stack_perf);
Signature · 1 line
2.13.6
Re: [PATCH 6/7] test/stack: add non-blocking stack tests
Quoted 6 lines · Gage Eads
This commit adds non-blocking stack variants of
stack_autotest(stack_nb_autotest) andstack_perf_autotest(stack_nb_perf_autotest), which differ only in that the non-blocking versions pass theSTACK_F_NBflag to allrte_stack_create()calls.Signed-off-by Gage Eads <[email protected]>
PATCH 7/7 mempool/stack: add non-blocking stack mempool handler
In reply to Gage EadsThis commit adds support for non-blocking (linked list based) stack mempool handler.
In mempool_perf_autotest the lock-based stack outperforms the non-blocking handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a lock-based stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the non-blocking stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.
*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the non-blocking stack's.
--- doc/guides/prog_guide/env_abstraction_layer.rst | 5 +++++ doc/guides/rel_notes/release_19_05.rst | 5 +++++ drivers/mempool/stack/rte_mempool_stack.c | 26 +++++++++++++++++++++++-- 3 files changed, 34 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +5 -0
@@ -541,6 +541,11 @@ Known Issues 5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR. + Alternatively, applications can use the non-blocking stack mempool handler. When considering this handler, note that: + + - it is currently limited to the x86_64 platform, because it uses an instruction (16-byte compare-and-swap) that is not yet available on other platforms. + - it has worse average-case performance than the non-preemptive rte_ring, but software caching (e.g. the mempool cache) can mitigate this by reducing the number of stack accesses. + + rte_timer Running ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -74,6 +74,11 @@ New Features The library supports two stack implementations: lock-based and non-blocking. The non-blocking implementation is currently limited to x86-64 platforms. +* **Added Non-blocking Stack Mempool Handler.** + + Added a new non-blocking stack handler, which uses the newly added stack + library. + Removed Items -------------
drivers/mempool/stack/rte_mempool_stack.c +24 -2
@@ -7,7 +7,7 @@#include <rte_stack.h> static int -stack_alloc (struct rte_mempool *mp) +__stack_alloc (struct rte_mempool *mp,uint32_t flags) {char name[RTE_STACK_NAMESIZE ];struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)return -rte_errno; } - s =rte_stack_create (name, mp->size, mp->socket_id,0 ); + s =rte_stack_create (name, mp->size, mp->socket_id, flags);if (s ==NULL )return -rte_errno;
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp) }static int +stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,0 ); +} + +static int +nb_stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,STACK_F_NB ); +} + +static int stack_enqueue (struct rte_mempool *mp,void *const *obj_table,unsigned int n) {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = { .get_count = stack_get_count }; +static struct rte_mempool_ops ops_nb_stack = { + .name ="nb_stack" , + .alloc = nb_stack_alloc, + .free = stack_free, + .enqueue = stack_enqueue, + .dequeue = stack_dequeue, + .get_count = stack_get_count +}; +MEMPOOL_REGISTER_OPS (ops_stack); +MEMPOOL_REGISTER_OPS (ops_nb_stack);
Signature · 1 line
2.13.6
Re: [PATCH 7/7] mempool/stack: add non-blocking stack mempool handler
Quoted 12 lines · Gage Eads
This commit adds support for non-blocking (linked list based) stack mempool handler.
In
mempool_perf_autotestthe lock-based stack outperforms the non-blocking handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a lock-based stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the non-blocking stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.
Quoted 5 lines
*Test setup:
x86_64build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack'srate_persecwas 0.6x-3.5x the non-blocking stack's.Signed-off-by Gage Eads <[email protected]>
v2 cover Add stack library and new mempool handler
In reply to Gage EadsThis patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.
The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.
The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.
Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.
This patchset is the successor to a patchset containing only the new mempool handler[2].
[1] http://mails.dpdk.org/archives/dev/2019-March/125751.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html
---
v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset
Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler
MAINTAINERS | 9 +- config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/env_abstraction_layer.rst | 10 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 83 +++++ doc/guides/rel_notes/release_19_05.rst | 13 + drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 115 +++---- lib/Makefile | 2 + lib/librte_stack/Makefile | 25 ++ lib/librte_stack/meson.build | 10 + lib/librte_stack/rte_stack.c | 219 ++++++++++++ lib/librte_stack/rte_stack.h | 395 ++++++++++++++++++++++ lib/librte_stack/rte_stack_c11_mem.h | 175 ++++++++++ lib/librte_stack/rte_stack_generic.h | 151 +++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++ lib/librte_stack/rte_stack_version.map | 9 + lib/meson.build | 2 +- mk/rte.app.mk | 1 + test/test/Makefile | 3 + test/test/meson.build | 7 + test/test/test_stack.c | 423 ++++++++++++++++++++++++ test/test/test_stack_perf.c | 356 ++++++++++++++++++++ 26 files changed, 1987 insertions(+), 72 deletions(-) create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_c11_mem.h create mode 100644 lib/librte_stack/rte_stack_generic.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_version.map create mode 100644 test/test/test_stack.c create mode 100644 test/test/test_stack_perf.c
Signature · 1 line
2.13.6
PATCH v2 1/8 stack: introduce rte stack library
The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.
The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.
--- MAINTAINERS | 6 + config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 28 ++++ doc/guides/rel_notes/release_19_05.rst | 5 + lib/Makefile | 2 + lib/librte_stack/Makefile | 23 +++ lib/librte_stack/meson.build | 8 + lib/librte_stack/rte_stack.c | 194 +++++++++++++++++++++++ lib/librte_stack/rte_stack.h | 274 +++++++++++++++++++++++++++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++++ lib/librte_stack/rte_stack_version.map | 9 ++ lib/meson.build | 2 +- mk/rte.app.mk | 1 + 16 files changed, 593 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
@@ -407,6 +407,12 @@ F: drivers/raw/skeleton_rawdev/ F: test/test/test_rawdev.c F: doc/guides/prog_guide/rawdev.rst +Stack API - EXPERIMENTAL +M: Gage Eads <[email protected]> +M: Olivier Matz <[email protected]> +F: lib/librte_stack/ +F: doc/guides/prog_guide/stack_lib.rst + Memory Pool Drivers -------------------
config/common_base +5 -0
@@ -980,3 +980,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y # Compile the eventdev application # CONFIG_RTE_APP_EVENTDEV=y + +# +# Compile librte_stack +# +CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
@@ -124,6 +124,7 @@ The public API headers are grouped by topics: [mbuf] (@ref rte_mbuf.h), [mbuf pool ops] (@ref rte_mbuf_pool_ops.h), [ring] (@ref rte_ring.h), + [stack] (@ref rte_stack.h), [tailq] (@ref rte_tailq.h), [bitmap] (@ref rte_bitmap.h)
doc/api/doxy-api.conf.in +1 -0
@@ -56,6 +56,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/librte_ring \ @TOPDIR@/lib/librte_sched \ @TOPDIR@/lib/librte_security \ + @TOPDIR@/lib/librte_stack \ @TOPDIR@/lib/librte_table \ @TOPDIR@/lib/librte_telemetry \ @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
@@ -55,6 +55,7 @@ Programmer's Guide metrics_lib bpf_lib ipsec_lib + stack_lib source_org dev_kit_build_system dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +28 -0
@@ -0,0 +1,28 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +Stack Library +============= + +DPDK's stack library provides an API for configuration and use of a bounded +stack of pointers. + +The stack library provides the following basic operations: + +* Create a uniquely named stack of a user-specified size and using a + user-specified socket. + +* Push and pop a burst of one or more stack objects (pointers). These function + are multi-threading safe. + +* Free a previously created stack. + +* Lookup a pointer to a stack by its name. + +* Query a stack's current depth and number of free entries. + +Implementation +~~~~~~~~~~~~~~ + +The stack consists of a contiguous array of pointers, a current index, and a +spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -65,6 +65,11 @@ New Features process. * Added support for Rx packet types list in a secondary process. +* **Added Stack API.** + + Added a new stack API for configuration and use of a bounded stack of + pointers. The API provides MT-safe push and pop operations that can operate + on one or more pointers per operation. Removed Items -------------
lib/Makefile +2 -0
@@ -111,6 +111,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev +DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack +DEPDIRS-librte_stack := librte_eal ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +23 -0
@@ -0,0 +1,23 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_stack.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +CFLAGS += -DALLOW_EXPERIMENTAL_API +LDLIBS += -lrte_eal + +EXPORT_MAP := rte_stack_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c + +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h + +include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
@@ -0,0 +1,8 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +allow_experimental_apis = true + +version = 1 +sources = files('rte_stack.c') +headers = files('rte_stack.h')
lib/librte_stack/rte_stack.c new file +194 -0
194 lines not on this page · raw message
lib/librte_stack/rte_stack.h new file +274 -0
274 lines not on this page · raw message
lib/librte_stack/rte_stack_pvt.h new file +34 -0
@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#ifndef _RTE_STACK_PVT_H_ +#define _RTE_STACK_PVT_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <rte_log.h> + +extern int stack_logtype; + +#define STACK_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \ + __func__, ##args) + +#define STACK_LOG_ERR(fmt, args...) \ + STACK_LOG(ERR, fmt, ## args) + +#define STACK_LOG_WARN(fmt, args...) \ + STACK_LOG(WARNING, fmt, ## args) + +#define STACK_LOG_INFO(fmt, args...) \ + STACK_LOG(INFO, fmt, ## args) + + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_version.map new file +9 -0
@@ -0,0 +1,9 @@ +EXPERIMENTAL { + global: + + rte_stack_create; + rte_stack_free; + rte_stack_lookup; + + local: *; +};
lib/meson.build +1 -1
@@ -21,7 +21,7 @@ libraries = [ 'compat', # just a header, used for versioning 'gro', 'gso', 'ip_frag', 'jobstats', 'kni', 'latencystats', 'lpm', 'member', 'power', 'pdump', 'rawdev', - 'reorder', 'sched', 'security', 'vhost', + 'reorder', 'sched', 'security', 'stack', 'vhost', #ipsec lib depends on crypto and security 'ipsec', # add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY) += -lrte_security _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += -lrte_compressdev _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += -lrte_eventdev _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV) += -lrte_rawdev +_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER) += -lrte_timer _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL) += -lrte_mempool _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
Signature · 1 line
2.13.6
PATCH v2 2/8 mempool/stack: convert mempool to use rte stack
In reply to Gage EadsThe new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.
--- MAINTAINERS | 2 +- drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++---------------------- 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
@@ -284,7 +284,6 @@ M: Andrew Rybchenko <[email protected]> F: lib/librte_mempool/ F: drivers/mempool/Makefile F: drivers/mempool/ring/ -F: drivers/mempool/stack/ F: doc/guides/prog_guide/mempool_lib.rst F: test/test/test_mempool* F: test/test/test_func_reentrancy.c
@@ -412,6 +411,7 @@ M: Gage Eads <[email protected]> M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst +F: drivers/mempool/stack/ Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a CFLAGS += -O3 CFLAGS += $(WERROR_FLAGS) +CFLAGS += -DALLOW_EXPERIMENTAL_API # Headers CFLAGS += -I$(RTE_SDK)/lib/librte_mempool -LDLIBS += -lrte_eal -lrte_mempool -lrte_ring +LDLIBS += -lrte_eal -lrte_mempool -lrte_stack EXPORT_MAP := rte_mempool_stack_version.map
drivers/mempool/stack/meson.build +5 -1
@@ -1,4 +1,8 @@ # SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation +# Copyright(c) 2017-2019 Intel Corporation + +allow_experimental_apis = true sources = files('rte_mempool_stack.c') + +deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68
133 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v2 3/8 test/stack: add stack test
In reply to Gage Eadsstack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.
--- MAINTAINERS | 1 + test/test/Makefile | 2 + test/test/meson.build | 3 + test/test/test_stack.c | 410 +++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 416 insertions(+) create mode 100644 test/test/test_stack.c
MAINTAINERS +1 -0
@@ -412,6 +412,7 @@ M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst F: drivers/mempool/stack/ +F: test/test/*stack* Memory Pool Drivers
test/test/Makefile +2 -0
@@ -89,6 +89,8 @@ endif SRCS-y += test_rwlock.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c + SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
test/test/meson.build +3 -0
@@ -95,6 +95,7 @@ test_sources = files('commands.c', 'test_sched.c', 'test_service_cores.c', 'test_spinlock.c', + 'test_stack.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -132,6 +133,7 @@ test_deps = ['acl', 'port', 'reorder', 'ring', + 'stack', 'timer' ]
@@ -173,6 +175,7 @@ fast_parallel_test_names = [ 'rwlock_autotest', 'sched_autotest', 'spinlock_autotest', + 'stack_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
test/test/test_stack.c new file +410 -0
410 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v2 4/8 test/stack: add stack perf test
In reply to Gage Eadsstack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects
It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores
--- test/test/Makefile | 1 + test/test/meson.build | 2 + test/test/test_stack_perf.c | 343 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 346 insertions(+) create mode 100644 test/test/test_stack_perf.c
test/test/Makefile +1 -0
@@ -90,6 +90,7 @@ endif SRCS-y += test_rwlock.c SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
test/test/meson.build +2 -0
@@ -96,6 +96,7 @@ test_sources = files('commands.c', 'test_service_cores.c', 'test_spinlock.c', 'test_stack.c', + 'test_stack_perf.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -240,6 +241,7 @@ perf_test_names = [ 'distributor_perf_autotest', 'ring_pmd_perf_autotest', 'pmd_perf_autotest', + 'stack_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
test/test/test_stack_perf.c new file +343 -0
343 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v2 5/8 stack: add lock-free stack implementation
In reply to Gage EadsThis commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.
The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.
The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.
The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.
This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.
The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.
--- doc/guides/prog_guide/stack_lib.rst | 61 ++++++++++++- doc/guides/rel_notes/release_19_05.rst | 3 + lib/librte_stack/Makefile | 3 +- lib/librte_stack/meson.build | 3 +- lib/librte_stack/rte_stack.c | 41 +++++++-- lib/librte_stack/rte_stack.h | 127 +++++++++++++++++++++++++-- lib/librte_stack/rte_stack_generic.h | 151 +++++++++++++++++++++++++++++++++ 7 files changed, 371 insertions(+), 18 deletions(-) create mode 100644 lib/librte_stack/rte_stack_generic.h
doc/guides/prog_guide/stack_lib.rst +58 -3
@@ -10,7 +10,8 @@ stack of pointers. The stack library provides the following basic operations: * Create a uniquely named stack of a user-specified size and using a - user-specified socket. + user-specified socket, with either standard (lock-based) or lock-free + behavior. * Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe.
@@ -24,5 +25,59 @@ The stack library provides the following basic operations: Implementation ~~~~~~~~~~~~~~ -The stack consists of a contiguous array of pointers, a current index, and a -spinlock. Accesses to the stack are made multi-thread safe by the spinlock. +The library supports two types of stacks: standard (lock-based) and lock-free. +Both types use the same set of interfaces, but their implementations differ. + +Lock-based Stack +---------------- + +The lock-based stack consists of a contiguous array of pointers, a current +index, and a spinlock. Accesses to the stack are made multi-thread safe by the +spinlock. + +Lock-free Stack +------------------ + +The lock-free stack consists of a linked list of elements, each containing a +data pointer and a next pointer, and an atomic stack depth counter. The +lock-free property means that multiple threads can push and pop simultaneously, +and one thread being preempted/delayed in a push or pop operation will not +impede the forward progress of any other thread. + +The lock-free push operation enqueues a linked list of pointers by pointing the +list's tail to the current stack head, and using a CAS to swing the stack head +pointer to the head of the list. The operation retries if it is unsuccessful +(i.e. the list changed between reading the head and modifying it), else it +adjusts the stack length and returns. + +The lock-free pop operation first reserves one or more list elements by +adjusting the stack length, to ensure the dequeue operation will succeed +without blocking. It then dequeues pointers by walking the list -- starting +from the head -- then swinging the head pointer (using a CAS as well). While +walking the list, the data pointers are recorded in an object table. + +The linked list elements themselves are maintained in a lock-free LIFO, and are +allocated before stack pushes and freed after stack pops. Since the stack has a +fixed maximum depth, these elements do not need to be dynamically created. + +The lock-free behavior is selected by passing the *RTE_STACK_F_LF* flag to +rte_stack_create(). + +Preventing the ABA Problem +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To prevent the ABA problem, this algorithm stack uses a 128-bit +compare-and-swap instruction to atomically update both the stack top pointer +and a modification counter. The ABA problem can occur without a modification +counter if, for example: + +1. Thread A reads head pointer X and stores the pointed-to list element. +2. Other threads modify the list such that the head pointer is once again X, + but its pointed-to data is different than what thread A read. +3. Thread A changes the head pointer with a compare-and-swap and succeeds. + +In this case thread A would not detect that the list had changed, and would +both pop stale data and incorrect change the head pointer. By adding a +modification counter that is updated on every push and pop as part of the +compare-and-swap, the algorithm can detect when the list changes even if the +head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
@@ -71,6 +71,9 @@ New Features pointers. The API provides MT-safe push and pop operations that can operate on one or more pointers per operation. + The library supports two stack implementations: standard (lock-based) and lock-free. + The lock-free implementation is currently limited to x86-64 platforms. + Removed Items -------------
lib/librte_stack/Makefile +2 -1
@@ -18,6 +18,7 @@ LIBABIVER := 1 SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c # install includes -SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h +SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ + rte_stack_generic.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
@@ -5,4 +5,5 @@ allow_experimental_apis = true version = 1 sources = files('rte_stack.c') -headers = files('rte_stack.h') +headers = files('rte_stack.h', + 'rte_stack_generic.h')
lib/librte_stack/rte_stack.c +33 -8
76 lines not on this page · raw message
lib/librte_stack/rte_stack.h +122 -5
166 lines not on this page · raw message
lib/librte_stack/rte_stack_generic.h new file +151 -0
151 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v2 6/8 stack: add C11 atomic implementation
In reply to Gage EadsThis commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.
--- lib/librte_stack/Makefile | 3 +- lib/librte_stack/meson.build | 3 +- lib/librte_stack/rte_stack.h | 4 + lib/librte_stack/rte_stack_c11_mem.h | 175 +++++++++++++++++++++++++++++++++++ 4 files changed, 183 insertions(+), 2 deletions(-) create mode 100644 lib/librte_stack/rte_stack_c11_mem.h
lib/librte_stack/Makefile +2 -1
@@ -19,6 +19,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ - rte_stack_generic.h + rte_stack_generic.h \ + rte_stack_c11_mem.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
@@ -6,4 +6,5 @@ allow_experimental_apis = true version = 1 sources = files('rte_stack.c') headers = files('rte_stack.h', - 'rte_stack_generic.h') + 'rte_stack_generic.h', + 'rte_stack_c11_mem.h')
lib/librte_stack/rte_stack.h +4 -0
@@ -91,7 +91,11 @@ struct rte_stack {*/ #define RTE_STACK_F_LF 0x0001 +#ifdef RTE_USE_C11_MEM_MODEL +#include "rte_stack_c11_mem.h" +#else #include "rte_stack_generic.h" +#endif /** * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_c11_mem.h new file +175 -0
175 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v2 7/8 test/stack: add lock-free stack tests
In reply to Gage EadsThis commit adds lock-free stack variants of stack_autotest (stack_lf_autotest) and stack_perf_autotest (stack_lf_perf_autotest), which differ only in that the lock-free versions pass the RTE_STACK_F_LF flag to all rte_stack_create() calls.
--- test/test/meson.build | 2 ++ test/test/test_stack.c | 41 +++++++++++++++++++++++++++-------------- test/test/test_stack_perf.c | 17 +++++++++++++++-- 3 files changed, 44 insertions(+), 16 deletions(-)
test/test/meson.build +2 -0
@@ -177,6 +177,7 @@ fast_parallel_test_names = [ 'sched_autotest', 'spinlock_autotest', 'stack_autotest', + 'stack_nb_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
@@ -242,6 +243,7 @@ perf_test_names = [ 'ring_pmd_perf_autotest', 'pmd_perf_autotest', 'stack_perf_autotest', + 'stack_nb_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
test/test/test_stack.c +27 -14
112 lines not on this page · raw message
test/test/test_stack_perf.c +15 -2
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s) }static int -test_stack_perf (void ) +__test_stack_perf (uint32_t flags) {struct lcore_pair cores;struct rte_stack *s;rte_atomic32_init (&lcore_barrier); - s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (),0 ); + s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (), flags);if (s ==NULL ) {printf ("[%s():%u] failed to create a stack\n" , __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)return 0 ; } +static int +test_stack_perf (void ) +{ +return __test_stack_perf (0 ); +} + +static int +test_lf_stack_perf (void ) +{ +return __test_stack_perf (RTE_STACK_F_LF ); +} +REGISTER_TEST_COMMAND (stack_perf_autotest, test_stack_perf); +REGISTER_TEST_COMMAND (stack_lf_perf_autotest, test_lf_stack_perf);
Signature · 1 line
2.13.6
PATCH v2 8/8 mempool/stack: add lock-free stack mempool handler
In reply to Gage EadsThis commit adds support for lock-free (linked list based) stack mempool handler.
In mempool_perf_autotest the lock-based stack outperforms the lock-free handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a standard (lock-based) stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the lock-free stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.
*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the lock-free stack's.
--- doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++ doc/guides/rel_notes/release_19_05.rst | 5 +++++ drivers/mempool/stack/rte_mempool_stack.c | 26 +++++++++++++++++++++++-- 3 files changed, 39 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
@@ -541,6 +541,16 @@ Known Issues 5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR. + Alternatively, applications can use the lock-free stack mempool handler. When + considering this handler, note that: + + - It is currently limited to the x86_64 platform, because it uses an + instruction (16-byte compare-and-swap) that is not yet available on other + platforms. + - It has worse average-case performance than the non-preemptive rte_ring, but + software caching (e.g. the mempool cache) can mitigate this by reducing the + number of stack accesses. + + rte_timer Running ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -74,6 +74,11 @@ New Features The library supports two stack implementations: standard (lock-based) and lock-free. The lock-free implementation is currently limited to x86-64 platforms. +* **Added Lock-Free Stack Mempool Handler.** + + Added a new lock-free stack handler, which uses the newly added stack + library. + Removed Items -------------
drivers/mempool/stack/rte_mempool_stack.c +24 -2
@@ -7,7 +7,7 @@#include <rte_stack.h> static int -stack_alloc (struct rte_mempool *mp) +__stack_alloc (struct rte_mempool *mp,uint32_t flags) {char name[RTE_STACK_NAMESIZE ];struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)return -rte_errno; } - s =rte_stack_create (name, mp->size, mp->socket_id,0 ); + s =rte_stack_create (name, mp->size, mp->socket_id, flags);if (s ==NULL )return -rte_errno;
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp) }static int +stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,0 ); +} + +static int +lf_stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,RTE_STACK_F_LF ); +} + +static int stack_enqueue (struct rte_mempool *mp,void *const *obj_table,unsigned int n) {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = { .get_count = stack_get_count }; +static struct rte_mempool_ops ops_lf_stack = { + .name ="lf_stack" , + .alloc = lf_stack_alloc, + .free = stack_free, + .enqueue = stack_enqueue, + .dequeue = stack_dequeue, + .get_count = stack_get_count +}; +MEMPOOL_REGISTER_OPS (ops_stack); +MEMPOOL_REGISTER_OPS (ops_lf_stack);
Signature · 1 line
2.13.6
v3 cover Add stack library and new mempool handler
In reply to Gage EadsThis patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.
The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.
The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.
Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.
This patchset is the successor to a patchset containing only the new mempool handler[2].
[1] http://mails.dpdk.org/archives/dev/2019-March/125751.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html
---
v3:
- Rebase patchset onto master (test/test/ -> app/test/)
- Fix rte_stack_std_push() segfault introduced in v2
v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset
Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler
MAINTAINERS | 9 +- app/test/Makefile | 3 + app/test/meson.build | 7 + app/test/test_stack.c | 423 ++++++++++++++++++++++++ app/test/test_stack_perf.c | 356 ++++++++++++++++++++ config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/env_abstraction_layer.rst | 10 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 83 +++++ doc/guides/rel_notes/release_19_05.rst | 13 + drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 115 +++---- lib/Makefile | 2 + lib/librte_stack/Makefile | 25 ++ lib/librte_stack/meson.build | 10 + lib/librte_stack/rte_stack.c | 219 ++++++++++++ lib/librte_stack/rte_stack.h | 395 ++++++++++++++++++++++ lib/librte_stack/rte_stack_c11_mem.h | 175 ++++++++++ lib/librte_stack/rte_stack_generic.h | 151 +++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++ lib/librte_stack/rte_stack_version.map | 9 + lib/meson.build | 2 +- mk/rte.app.mk | 1 + 26 files changed, 1987 insertions(+), 72 deletions(-) create mode 100644 app/test/test_stack.c create mode 100644 app/test/test_stack_perf.c create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_c11_mem.h create mode 100644 lib/librte_stack/rte_stack_generic.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_version.map
Signature · 1 line
2.13.6
PATCH v3 1/8 stack: introduce rte stack library
The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.
The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.
--- MAINTAINERS | 6 + config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 28 ++++ doc/guides/rel_notes/release_19_05.rst | 5 + lib/Makefile | 2 + lib/librte_stack/Makefile | 23 +++ lib/librte_stack/meson.build | 8 + lib/librte_stack/rte_stack.c | 194 +++++++++++++++++++++++ lib/librte_stack/rte_stack.h | 274 +++++++++++++++++++++++++++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++++ lib/librte_stack/rte_stack_version.map | 9 ++ lib/meson.build | 2 +- mk/rte.app.mk | 1 + 16 files changed, 593 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
@@ -405,6 +405,12 @@ F: drivers/raw/skeleton_rawdev/ F: app/test/test_rawdev.c F: doc/guides/prog_guide/rawdev.rst +Stack API - EXPERIMENTAL +M: Gage Eads <[email protected]> +M: Olivier Matz <[email protected]> +F: lib/librte_stack/ +F: doc/guides/prog_guide/stack_lib.rst + Memory Pool Drivers -------------------
config/common_base +5 -0
@@ -980,3 +980,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y # Compile the eventdev application # CONFIG_RTE_APP_EVENTDEV=y + +# +# Compile librte_stack +# +CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
@@ -124,6 +124,7 @@ The public API headers are grouped by topics: [mbuf] (@ref rte_mbuf.h), [mbuf pool ops] (@ref rte_mbuf_pool_ops.h), [ring] (@ref rte_ring.h), + [stack] (@ref rte_stack.h), [tailq] (@ref rte_tailq.h), [bitmap] (@ref rte_bitmap.h)
doc/api/doxy-api.conf.in +1 -0
@@ -55,6 +55,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/librte_ring \ @TOPDIR@/lib/librte_sched \ @TOPDIR@/lib/librte_security \ + @TOPDIR@/lib/librte_stack \ @TOPDIR@/lib/librte_table \ @TOPDIR@/lib/librte_telemetry \ @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
@@ -55,6 +55,7 @@ Programmer's Guide metrics_lib bpf_lib ipsec_lib + stack_lib source_org dev_kit_build_system dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +28 -0
@@ -0,0 +1,28 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +Stack Library +============= + +DPDK's stack library provides an API for configuration and use of a bounded +stack of pointers. + +The stack library provides the following basic operations: + +* Create a uniquely named stack of a user-specified size and using a + user-specified socket. + +* Push and pop a burst of one or more stack objects (pointers). These function + are multi-threading safe. + +* Free a previously created stack. + +* Lookup a pointer to a stack by its name. + +* Query a stack's current depth and number of free entries. + +Implementation +~~~~~~~~~~~~~~ + +The stack consists of a contiguous array of pointers, a current index, and a +spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -77,6 +77,11 @@ New Features which includes the directory name, lib name, filenames, makefile, docs, macros, functions, structs and any other strings in the code. +* **Added Stack API.** + + Added a new stack API for configuration and use of a bounded stack of + pointers. The API provides MT-safe push and pop operations that can operate + on one or more pointers per operation. Removed Items -------------
lib/Makefile +2 -0
@@ -109,6 +109,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev +DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack +DEPDIRS-librte_stack := librte_eal ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +23 -0
@@ -0,0 +1,23 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_stack.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +CFLAGS += -DALLOW_EXPERIMENTAL_API +LDLIBS += -lrte_eal + +EXPORT_MAP := rte_stack_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c + +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h + +include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
@@ -0,0 +1,8 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +allow_experimental_apis = true + +version = 1 +sources = files('rte_stack.c') +headers = files('rte_stack.h')
lib/librte_stack/rte_stack.c new file +194 -0
194 lines not on this page · raw message
lib/librte_stack/rte_stack.h new file +274 -0
274 lines not on this page · raw message
lib/librte_stack/rte_stack_pvt.h new file +34 -0
@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#ifndef _RTE_STACK_PVT_H_ +#define _RTE_STACK_PVT_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <rte_log.h> + +extern int stack_logtype; + +#define STACK_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \ + __func__, ##args) + +#define STACK_LOG_ERR(fmt, args...) \ + STACK_LOG(ERR, fmt, ## args) + +#define STACK_LOG_WARN(fmt, args...) \ + STACK_LOG(WARNING, fmt, ## args) + +#define STACK_LOG_INFO(fmt, args...) \ + STACK_LOG(INFO, fmt, ## args) + + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_version.map new file +9 -0
@@ -0,0 +1,9 @@ +EXPERIMENTAL { + global: + + rte_stack_create; + rte_stack_free; + rte_stack_lookup; + + local: *; +};
lib/meson.build +1 -1
@@ -22,7 +22,7 @@ libraries = [ 'gro', 'gso', 'ip_frag', 'jobstats', 'kni', 'latencystats', 'lpm', 'member', 'power', 'pdump', 'rawdev', - 'reorder', 'sched', 'security', 'vhost', + 'reorder', 'sched', 'security', 'stack', 'vhost', #ipsec lib depends on crypto and security 'ipsec', # add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY) += -lrte_security _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += -lrte_compressdev _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += -lrte_eventdev _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV) += -lrte_rawdev +_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER) += -lrte_timer _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL) += -lrte_mempool _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
Signature · 1 line
2.13.6
Re: [PATCH v3 1/8] stack: introduce rte stack library
Quoted 11 lines · Gage Eads
The
rte_stacklibrary provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.The library's interface is modeled after another DPDK data structure,
rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler torte_stack.Signed-off-by Gage Eads <[email protected]>
Re: [PATCH v3 1/8] stack: introduce rte stack library
In reply to Gage EadsHi Gage,
Apologies for the late comments.
Quoted 262 lines
-----Original Message-----
From: Gage Eads <[email protected]>
Sent: Wednesday, March 6, 2019 8:46 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Gavin Hu (Arm Technology China) <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>; [email protected]
Subject: [PATCH v3 1/8] stack: introduce rte stack libraryThe
rte_stacklibrary provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.The library's interface is modeled after another DPDK data structure,
rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler torte_stack.Signed-off-by Gage Eads <[email protected]>--- MAINTAINERS | 6 + config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 28 ++++ doc/guides/rel_notes/release_19_05.rst | 5 + lib/Makefile | 2 + lib/librte_stack/Makefile | 23 +++ lib/librte_stack/meson.build | 8 + lib/librte_stack/rte_stack.c | 194 +++++++++++++++++++++++ lib/librte_stack/rte_stack.h | 274 +++++++++++++++++++++++++++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++++ lib/librte_stack/rte_stack_version.map | 9 ++ lib/meson.build | 2 +- mk/rte.app.mk | 1 + 16 files changed, 593 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644
lib/librte_stack/meson.buildcreate mode 100644lib/librte_stack/rte_stack.ccreate mode 100644lib/librte_stack/rte_stack.hcreate mode 100644lib/librte_stack/rte_stack_pvt.hcreate mode 100644lib/librte_stack/rte_stack_version.mapMAINTAINERS +6
-0@@ -405,6 +405,12 @@ F: drivers/raw/skeleton_rawdev/ F: app/test/test_rawdev.c F: doc/guides/prog_guide/rawdev.rst +Stack API - EXPERIMENTAL +M: Gage Eads <[email protected]> +M: Olivier Matz <[email protected]> +F: lib/librte_stack/ +F: doc/guides/prog_guide/stack_lib.rst + Memory Pool Drivers -------------------config/common_base index renamed from config/common_base +0
-0
0b09a9348..1b45dea6c100644config/common_base +0
-0@@ -980,3 +980,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y # Compile theeventdev application #
CONFIG_RTE_APP_EVENTDEV=yconfig/common_base +5
-0+ +# +# Compile librte_stack +# +CONFIG_RTE_LIBRTE_STACK=ydoc/api/doxy-api-index.md index renamed from doc/api/doxy-api-index.md +0
-0
d95ad566c..0df8848c0100644doc/api/doxy-api-index.md +1
-0@@ -124,6 +124,7 @@ The public API headers are grouped by topics: [mbuf] (@ref rte_mbuf.h), [mbuf pool ops] (@ref rte_mbuf_pool_ops.h), [ring] (@ref rte_ring.h), + [stack] (@ref rte_stack.h), [tailq] (@ref rte_tailq.h), [bitmap] (@ref rte_bitmap.h)doc/api/doxy-api.conf.in index renamed from doc/api/doxy-api.conf.in +0
-0
a365e669b..7722fc3e9100644doc/api/doxy-api.conf.in +1
-0@@ -55,6 +55,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api- index.md \ @TOPDIR@/lib/librte_ring \ @TOPDIR@/lib/librte_sched \ @TOPDIR@/lib/librte_security \ + @TOPDIR@/lib/librte_stack \ @TOPDIR@/lib/librte_table \ @TOPDIR@/lib/librte_telemetry \@TOPDIR@/lib/librte_timer \ diff --git
a/doc/guides/prog_guide/index.rstb/doc/guides/prog_guide/index.rstindex6726b1e8d..f4f60862f100644doc/guides/prog_guide/index.rst +1
-0@@ -55,6 +55,7 @@ Programmer's Guide metrics_lib bpf_lib ipsec_lib + stack_lib source_org dev_kit_build_system dev_kit_root_make_helpdoc/guides/prog_guide/stack_lib.rst new file +15
-0@@ -0,0 +1,28 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +Stack Library +============= + +DPDK's stack library provides an API for configuration and use of a +bounded stack of pointers. + +The stack library provides the following basic operations: + +* Create a uniquely named stack of a user-specified size and using a + user-specified socket. + +* Push and pop a burst of one or more stack objects (pointers). Thesefunction
doc/guides/prog_guide/stack_lib.rst new file +13
-0+ are multi-threading safe. + +* Free a previously created stack. + +* Lookup a pointer to a stack by its name. + +* Query a stack's current depth and number of free entries. + +Implementation +~~~~~~~~~~~~~~ + +The stack consists of a contiguous array of pointers, a current index, +and a spinlock. Accesses to the stack are made multi-thread safe by thespinlock.
doc/guides/rel_notes/release_19_05.rst +5
-0@@ -77,6 +77,11 @@ New Features which includes the directory name, lib name, filenames, makefile, docs, macros, functions, structs and any other strings in the code. +* **Added Stack API.** + + Added a new stack API for configuration and use of a bounded stack of + pointers. The API provides MT-safe push and pop operations that can + operate on one or more pointers per operation. Removed Items -------------lib/Makefile +0
-0@@ -109,6 +109,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsecDEPDIRS-librte_ipsec :=
librte_eallibrte_mbuflibrte_cryptodevlibrte_security
DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) +=librte_telemetryDEPDIRS-librte_telemetry:=librte_eallibrte_metricslibrte_ethdevlib/Makefile +2
-0+DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack DEPDIRS-librte_stack := +librte_eal ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni diff --gita/lib/librte_stack/Makefile b/lib/librte_stack/Makefile new file mode 100644 index 000000000..e956b6535
lib/librte_stack/Makefile new file +22
-0@@ -0,0 +1,23 @@ +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2019 Intel +Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_stack.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 CFLAGS += +-DALLOW_EXPERIMENTAL_API LDLIBS += -lrte_eal + +EXPORT_MAP := rte_stack_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c + +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h + +include $(RTE_SDK)/mk/rte.lib.mklib/librte_stack/meson.build new renamed from lib/librte_stack/meson.build +0
-0file mode 100644 index 000000000..99f43710e
lib/librte_stack/meson.build new file +8
-0@@ -0,0 +1,8 @@ +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2019 Intel +Corporation + +allow_experimental_apis = true + +version = 1 +sources = files('rte_stack.c') +headers = files('rte_stack.h')lib/librte_stack/rte_stack.c new file renamed from lib/librte_stack/rte_stack.c +0
-0mode 100644 index 000000000..96dffdf44
lib/librte_stack/rte_stack.c new file +37
-0@@ -0,0 +1,194 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#include <string.h> + +#include <rte_atomic.h> +#include <rte_eal.h> +#include <rte_eal_memconfig.h> +#include <rte_errno.h> +#include <rte_malloc.h> +#include <rte_memzone.h> +#include <rte_rwlock.h> +#include <rte_tailq.h> + +#include "rte_stack.h" +#include "rte_stack_pvt.h" + +int stack_logtype; + +TAILQ_HEAD (rte_stack_list, rte_tailq_entry); + +static struct rte_tailq_elem rte_stack_tailq = { + .name =RTE_TAILQ_STACK_NAME , +}; +EAL_REGISTER_TAILQ (rte_stack_tailq) + +static void +rte_stack_std_init (struct rte_stack *s) { +rte_spinlock_init (&s->stack_std.lock); +} + +static void +rte_stack_init (struct rte_stack *s) +{ +memset (s,0 ,sizeof (*s)); +
lib/librte_stack/rte_stack.c line 47
Quoted 10 lines
lib/librte_stack/rte_stack.c new file +10
-0+rte_stack_std_init (s); +} + +static ssize_t +rte_stack_get_memsize (unsigned int count) { +ssize_t sz =sizeof (struct rte_stack ); + +/* Add padding to avoid false sharing conflicts */ + sz +=RTE_CACHE_LINE_ROUNDUP (count *sizeof (void *)) + +2 *RTE_CACHE_LINE_SIZE ;
I did not understand how the false sharing is caused and how this padding is solving the issue. Verbose comments would help.
Quoted 26 lines
lib/librte_stack/rte_stack.c new file +26
-0+ +return sz; +} + +struct rte_stack * +rte_stack_create (const char *name,unsigned int count,int socket_id, +uint32_t flags) +{ +char mz_name[RTE_MEMZONE_NAMESIZE ]; +struct rte_stack_list *stack_list; +const struct rte_memzone *mz; +struct rte_tailq_entry *te; +struct rte_stack *s; +unsigned int sz; +int ret; + +RTE_SET_USED (flags); + + sz =rte_stack_get_memsize (count); + + ret =snprintf (mz_name,sizeof (mz_name),"%s%s" , +RTE_STACK_MZ_PREFIX , name); +if (ret <0 || ret >= (int )sizeof (mz_name)) { + rte_errno =ENAMETOOLONG ; +return NULL ; + }
lib/librte_stack/rte_stack.c line 85
Quoted 10 lines
lib/librte_stack/rte_stack.c new file +10
-0+ + te =rte_zmalloc ("STACK_TAILQ_ENTRY" ,sizeof (*te),0 ); +if (te ==NULL ) { +STACK_LOG_ERR ("Cannot reserve memory for tailq\n" ); + rte_errno =ENOMEM ; +return NULL ; + } + +rte_rwlock_write_lock (RTE_EAL_TAILQ_RWLOCK ); +
I think there is a need to check if a stack with the same name exists already.
Quoted 38 lines
lib/librte_stack/rte_stack.c new file +38
-0+ mz =rte_memzone_reserve_aligned (mz_name, sz, socket_id, + 0, __alignof__(*s)); +if (mz ==NULL ) { +STACK_LOG_ERR ("Cannot reserve stack memzone!\n" ); +rte_rwlock_write_unlock (RTE_EAL_TAILQ_RWLOCK ); +rte_free (te); +return NULL ; + } + + s = mz->addr; + +rte_stack_init (s); + +/* Store the name for later lookups */ + ret =snprintf (s->name,sizeof (s->name),"%s" , name); +if (ret <0 || ret >= (int )sizeof (s->name)) { +rte_rwlock_write_unlock (RTE_EAL_TAILQ_RWLOCK ); + + rte_errno =ENAMETOOLONG ; +rte_free (te); +rte_memzone_free (mz); +return NULL ; + } + + s->memzone = mz; + s->capacity = count; + s->flags = flags; + + te->data = s; + + stack_list =RTE_TAILQ_CAST (rte_stack_tailq.head, rte_stack_list); + +TAILQ_INSERT_TAIL (stack_list, te, next); + +rte_rwlock_write_unlock (RTE_EAL_TAILQ_RWLOCK ); + +return s; +}
lib/librte_stack/rte_stack.c line 133
Quoted 10 lines
lib/librte_stack/rte_stack.c new file +10
-0+ +void +rte_stack_free (struct rte_stack *s) +{ +struct rte_stack_list *stack_list; +struct rte_tailq_entry *te; + +if (s ==NULL ) +return ; +
Adding a check to make sure the length of the stack is 0 would help catch issues?
Quoted 86 lines
lib/librte_stack/rte_stack.c new file +60
-0+ stack_list =RTE_TAILQ_CAST (rte_stack_tailq.head, rte_stack_list); +rte_rwlock_write_lock (RTE_EAL_TAILQ_RWLOCK ); + +/* find out tailq entry */ + TAILQ_FOREACH(te, stack_list, next) { +if (te->data == s) +break ; + } + +if (te ==NULL ) { +rte_rwlock_write_unlock (RTE_EAL_TAILQ_RWLOCK ); +return ; + } + +TAILQ_REMOVE (stack_list, te, next); + +rte_rwlock_write_unlock (RTE_EAL_TAILQ_RWLOCK ); + +rte_free (te); + +rte_memzone_free (s->memzone); +} + +struct rte_stack * +rte_stack_lookup (const char *name) +{ +struct rte_stack_list *stack_list; +struct rte_tailq_entry *te; +struct rte_stack *r =NULL ; + +if (name ==NULL ) { + rte_errno =EINVAL ; +return NULL ; + } + + stack_list =RTE_TAILQ_CAST (rte_stack_tailq.head, rte_stack_list); + +rte_rwlock_read_lock (RTE_EAL_TAILQ_RWLOCK ); + + TAILQ_FOREACH(te, stack_list, next) { + r = (struct rte_stack *) te->data; +if (strncmp (name, r->name,RTE_STACK_NAMESIZE ) ==0 ) +break ; + } + +rte_rwlock_read_unlock (RTE_EAL_TAILQ_RWLOCK ); + +if (te ==NULL ) { + rte_errno =ENOENT ; +return NULL ; + } + +return r; +} + +RTE_INIT (librte_stack_init_log) +{ + stack_logtype =rte_log_register ("lib.stack" ); +if (stack_logtype >=0 ) +rte_log_set_level (stack_logtype,RTE_LOG_NOTICE ); }lib/librte_stack/rte_stack.h new file renamed from lib/librte_stack/rte_stack.h +0
-0mode 100644 index 000000000..7a633deb5
lib/librte_stack/rte_stack.h new file +15
-0@@ -0,0 +1,274 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +/** +* @file rte_stack.h +* @b EXPERIMENTAL: this API may change without prior notice +* +* RTE Stack +* +* librte_stack provides an API for configuration and use of a bounded +stack of +* pointers. Push and pop operations are MT-safe, allowing concurrent +access, +* and the interface supports pushing and popping multiple pointers at a time.
+ */lib/librte_stack/rte_stack.h new file +4
-0+ +#ifndef _RTE_STACK_H_ +#define _RTE_STACK_H_ +
lib/librte_stack/rte_stack.h line 28
Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10
-0+#ifdef __cplusplus +extern "C" { +#endif + +#include <rte_errno.h> +#include <rte_memzone.h> +#include <rte_spinlock.h> + +#define RTE_TAILQ_STACK_NAME "RTE_STACK" +#define RTE_STACK_MZ_PREFIX "STK_"
Nit, "STACK_" would be easier to debug
Quoted 30 lines
lib/librte_stack/rte_stack.h new file +30
-0+/** The maximum length of a stack name. */ #define RTE_STACK_NAMESIZE +(RTE_MEMZONE_NAMESIZE - \ +sizeof (RTE_STACK_MZ_PREFIX ) +1 ) + +/* Structure containing the LIFO, its current length, and a lock for +mutual +* exclusion. + */ +struct rte_stack_std { +rte_spinlock_t lock;/**< LIFO lock */ +uint32_t len;/**< LIFO len */ +void *objs[];/**< LIFO pointer table */ }; + +/* The RTE stack structure contains the LIFO structure itself, plus +metadata +* such as its name and memzone pointer. +*/ +struct rte_stack { +/** Name of the stack. */ + char name[RTE_STACK_NAMESIZE] __rte_cache_aligned; +/** Memzone containing the rte_stack structure. */ +const struct rte_memzone *memzone; +uint32_t capacity;/**< Usable size of the stack. */ +uint32_t flags;/**< Flags supplied at creation. */ +struct rte_stack_std stack_std;/**< LIFO structure. */ } +__rte_cache_aligned; + +/** +* @internal Push several objects on the stack (MT-safe). +*
lib/librte_stack/rte_stack.h line 67
Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10
-0+* @param s +* A pointer to the stack structure. +* @param obj_table +* A pointer to a table of void * pointers (objects). +* @param n +* The number of objects to push on the stack from the obj_table. +* @return +* Actual number of objects pushed (either 0 or *n*). +*/ +static __rte_always_inline unsigned int __rte_experimental
This is an internal function. Is '__rte_experimental' tag required?
lib/librte_stack/rte_stack.h
Quoted 2 lines
lib/librte_stack/rte_stack.h new file +2
-0+rte_stack_std_push(struct rte_stack *s, void * const *obj_table, +unsigned int n) {
Since this is an internal function, does it make sense to add '__' to the beginning of the function name (similar to what is done in rte_ring?).
Quoted 46 lines
lib/librte_stack/rte_stack.h new file +46
-0+struct rte_stack_std *stack = &s->stack_std; +unsigned int index; +void **cache_objs; + +rte_spinlock_lock (&stack->lock); + cache_objs = &stack->objs[stack->len]; + +/* Is there sufficient space in the stack? */ +if ((stack->len + n) > s->capacity) { +rte_spinlock_unlock (&stack->lock); +return 0 ; + } + +/* Add elements back into the cache */ +for (index =0 ; index < n; ++index, obj_table++) + cache_objs[index] = *obj_table; + + stack->len += n; + +rte_spinlock_unlock (&stack->lock); +return n; +} + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Push several objects on the stack (MT-safe). +* +* @param s +* A pointer to the stack structure. +* @param obj_table +* A pointer to a table of void * pointers (objects). +* @param n +* The number of objects to push on the stack from the obj_table. +* @return +* Actual number of objects pushed (either 0 or *n*). +*/ +static __rte_always_inline unsigned int __rte_experimental +rte_stack_push (struct rte_stack *s,void *const *obj_table,unsigned +int n) { +return rte_stack_std_push (s, obj_table, n); } + +/** +* @internal Pop several objects from the stack (MT-safe). +*
lib/librte_stack/rte_stack.h line 126
Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10
-0+* @param s +* A pointer to the stack structure. +* @param obj_table +* A pointer to a table of void * pointers (objects). +* @param n +* The number of objects to pull from the stack. +* @return +* Actual number of objects popped (either 0 or *n*). +*/ +static __rte_always_inline unsigned int __rte_experimental
This is an internal function. Is '__rte_experimental' tag required?
Quoted 34 lines
lib/librte_stack/rte_stack.h new file +34
-0+rte_stack_std_pop(struct rte_stack *s, void **obj_table, unsigned int +n) { + struct rte_stack_std *stack = &s->stack_std; +unsigned int index, len; +void **cache_objs; + +rte_spinlock_lock (&stack->lock); + +if (unlikely (n > stack->len)) { +rte_spinlock_unlock (&stack->lock); +return 0 ; + } + + cache_objs = stack->objs; + +for (index =0 , len = stack->len -1 ; index < n; + ++index, len--, obj_table++) + *obj_table = cache_objs[len]; + + stack->len -= n; +rte_spinlock_unlock (&stack->lock); + +return n; +} + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Pop several objects from the stack (MT-safe). +* +* @param s +* A pointer to the stack structure. +* @param obj_table
lib/librte_stack/rte_stack.h line 171
Quoted 10 lines
lib/librte_stack/rte_stack.h new file +10
-0+* A pointer to a table of void * pointers (objects). +* @param n +* The number of objects to pull from the stack. +* @return +* Actual number of objects popped (either 0 or *n*). +*/ +static __rte_always_inline unsigned int __rte_experimental +rte_stack_pop (struct rte_stack *s,void **obj_table,unsigned int n) { +if (unlikely (n ==0 || obj_table ==NULL )) +return 0 ;
's == NULL' can be added as well. Similar check is missing in 'rte_stack_push'. Since these are data-path APIs, RTE_ASSERT would be better.
Quoted 57 lines
lib/librte_stack/rte_stack.h new file +57
-0+ +return rte_stack_std_pop (s, obj_table, n); } + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Return the number of used entries in a stack. +* +* @param s +* A pointer to the stack structure. +* @return +* The number of used entries in the stack. +*/ +static __rte_always_inline unsigned int __rte_experimental +rte_stack_count (struct rte_stack *s) { +return (unsigned int )s->stack_std.len; } + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Return the number of free entries in a stack. +* +* @param s +* A pointer to the stack structure. +* @return +* The number of free entries in the stack. +*/ +static __rte_always_inline unsigned int __rte_experimental +rte_stack_free_count (struct rte_stack *s) { +return s->capacity -rte_stack_count (s); } + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Create a new stack named *name* in memory. +* +* This function uses ``memzone_reserve()`` to allocate memory for a +stack of +* size *count*. The behavior of the stack is controlled by the *flags*. +* +* @param name +* The name of the stack. +* @param count +* The size of the stack. +* @param socket_id +* The *socket_id* argument is the socket identifier in case of +* NUMA. The value can be *SOCKET_ID_ANY* if there is no NUMA +* constraint for the reserved zone. +* @param flags +* Reserved for future use. +* @return +* On success, the pointer to the new allocated stack. NULL on error with +* rte_errno set appropriately. Possible errno values include: +* - ENOSPC - the maximum number of memzones has already been
Quoted 2 lines
allocated
+ * - EEXIST - a stack with the same name already exists
This is not implemented currently
Quoted 124 lines
+ * - ENOMEM - insufficient memory to create the stack
+ * - ENAMETOOLONG - name size exceedsRTE_STACK_NAMESIZE
+ */lib/librte_stack/rte_stack.h new file +38
-0+struct rte_stack *__rte_experimental +rte_stack_create (const char *name,unsigned int count,int socket_id, +uint32_t flags); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Free all memory used by the stack. +* +* @param s +* Stack to free +*/ +void __rte_experimental +rte_stack_free (struct rte_stack *s); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Lookup a stack by its name. +* +* @param name +* The name of the stack. +* @return +* The pointer to the stack matching the name, or NULL if not found, +* with rte_errno set appropriately. Possible rte_errno values include: +* - ENOENT - Stack with name *name* not found. +* - EINVAL - *name* pointer is NULL. +*/ +struct rte_stack * __rte_experimental +rte_stack_lookup (const char *name); + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_STACK_H_ */ lib/librte_stack/rte_stack_pvt.h new file +34
-0@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#ifndef _RTE_STACK_PVT_H_ +#define _RTE_STACK_PVT_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <rte_log.h> + +extern int stack_logtype; + +#define STACK_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \ + __func__, ##args) + +#define STACK_LOG_ERR(fmt, args...) \ + STACK_LOG(ERR, fmt, ## args) + +#define STACK_LOG_WARN(fmt, args...) \ + STACK_LOG(WARNING, fmt, ## args) + +#define STACK_LOG_INFO(fmt, args...) \ + STACK_LOG(INFO, fmt, ## args) + + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_STACK_PVT_H_ */ lib/librte_stack/rte_stack_version.map new file +9
-0@@ -0,0 +1,9 @@ +EXPERIMENTAL { + global: + + rte_stack_create; + rte_stack_free; + rte_stack_lookup; + + local: *; +};lib/meson.build index 99957ba7d..90115477f renamed from lib/meson.build +0
-0100644
lib/meson.build +1
-1@@ -22,7 +22,7 @@ libraries = [ 'gro', 'gso', 'ip_frag', 'jobstats', 'kni', 'latencystats', 'lpm', 'member', 'power', 'pdump', 'rawdev', - 'reorder', 'sched', 'security', 'vhost', + 'reorder', 'sched', 'security', 'stack', 'vhost', #ipsec lib depends on crypto and security 'ipsec', # add pkt framework libs which use other libs from above diff --git
a/mk/rte.app.mkb/mk/rte.app.mkindex3c40f9df2..8decfb851100644mk/rte.app.mk +0
-0@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY) += -
lrte_security_LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += -lrte_compressdev _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += -lrte_eventdev _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV) += -lrte_rawdev +_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER) += -lrte_timer _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL) += -lrte_mempool _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring--
2.13.6
Re: [PATCH v3 1/8] stack: introduce rte stack library
@Thomas: I expect I can address Honnappa's feedback within a day or two. Since today is the 19.05 merge deadline, what do you think about these options for merging?
1. Merge V4 now and address these comments during RC1.
2. Delay merge until RC2, with all comments addressed.
In terms of risk, Honnappa identified an incorrect memory ordering argument (patch 6/8), but that doesn't affect the one platform (x86-64) that can (currently) use this library. His other comments address readability, error-checking, and performance, but aren't critical. Beyond that, this patchset is isolated from the rest of DPDK. So, I think the risk to the project is very low.
(Also, note that I accidentally left off Olivier's Reviewed-by tag in V4's patches 1, 3, 5, and 6 -- I'll address that as well)
Quoted 11 lines
-----Original Message-----
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Thursday, March 28, 2019 6:27 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v3 1/8] stack: introduce rte stack library
Quoted 3 lines
Hi Gage,
Apologies for the late comments.
No problem, I appreciate the feedback.
[snip]
Quoted 9 lines
lib/librte_stack/rte_stack.c line 49
Quoted 7 lines
lib/librte_stack/rte_stack.c new file +7
-0+static ssize_t +rte_stack_get_memsize (unsigned int count) { +ssize_t sz =sizeof (struct rte_stack ); + +/* Add padding to avoid false sharing conflicts */ + sz +=RTE_CACHE_LINE_ROUNDUP (count *sizeof (void *)) + +2 *RTE_CACHE_LINE_SIZE ;I did not understand how the false sharing is caused and how this padding is solving the issue. Verbose comments would help.
The additional padding (beyond the CACHE_LINE_ROUNDUP) is to prevent false sharing caused by adjacent/next-line hardware prefetchers. I'll address this.
[snip]
Quoted 25 lines
Quoted 25 lines
lib/librte_stack/rte_stack.c new file +25
-0+struct rte_stack * +rte_stack_create (const char *name,unsigned int count,int socket_id, +uint32_t flags) +{ +char mz_name[RTE_MEMZONE_NAMESIZE ]; +struct rte_stack_list *stack_list; +const struct rte_memzone *mz; +struct rte_tailq_entry *te; +struct rte_stack *s; +unsigned int sz; +int ret; + +RTE_SET_USED (flags); + + sz =rte_stack_get_memsize (count); + + ret =snprintf (mz_name,sizeof (mz_name),"%s%s" , +RTE_STACK_MZ_PREFIX , name); +if (ret <0 || ret >= (int )sizeof (mz_name)) { + rte_errno =ENAMETOOLONG ; +return NULL ; + } + + te =rte_zmalloc ("STACK_TAILQ_ENTRY" ,sizeof (*te),0 ); +if (te ==NULL ) {
Quoted 8 lines
lib/librte_stack/rte_stack.c line 85
Quoted 7 lines
lib/librte_stack/rte_stack.c new file +7
-0+STACK_LOG_ERR ("Cannot reserve memory for tailq\n" ); + rte_errno =ENOMEM ; +return NULL ; + } + +rte_rwlock_write_lock (RTE_EAL_TAILQ_RWLOCK ); +I think there is a need to check if a stack with the same name exists already.
rte_memzone_reserve_aligned() does just that. This behavior is tested in the function test_stack_name_reuse(), added in commit " test/stack: add stack test".
Quoted 8 lines
Quoted 8 lines
lib/librte_stack/rte_stack.c new file +8
-0+ mz =rte_memzone_reserve_aligned (mz_name, sz, socket_id, + 0, __alignof__(*s)); +if (mz ==NULL ) { +STACK_LOG_ERR ("Cannot reserve stack memzone!\n" ); +rte_rwlock_write_unlock (RTE_EAL_TAILQ_RWLOCK ); +rte_free (te); +return NULL ; + }
[snip]
Quoted 11 lines
lib/librte_stack/rte_stack.c line 133
Quoted 9 lines
lib/librte_stack/rte_stack.c new file +9
-0+void +rte_stack_free (struct rte_stack *s) +{ +struct rte_stack_list *stack_list; +struct rte_tailq_entry *te; + +if (s ==NULL ) +return ; +Adding a check to make sure the length of the stack is 0 would help catch issues?
My preference is to leave that check to the user, for any apps that want to/can safely free non-empty stacks.
[snip]
Quoted 3 lines
lib/librte_stack/rte_stack.h line 28
Quoted 2 lines
lib/librte_stack/rte_stack.h new file +2
-0+#define RTE_TAILQ_STACK_NAME "RTE_STACK" +#define RTE_STACK_MZ_PREFIX "STK_"Nit, "
STACK_" would be easier to debug
Since RTE_MEMZONE_NAMESIZE (32) doesn't give us a lot of space, I kept the prefix short. Adding 2 more characters *probably* won't make a difference...but I'd prefer the shortened name.
Quoted 5 lines
Quoted 1 line
+/** The maximum length of a stack name. */ #define
RTE_STACK_NAMESIZEQuoted 3 lines
lib/librte_stack/rte_stack.h new file +3
-0+(RTE_MEMZONE_NAMESIZE - \ +sizeof (RTE_STACK_MZ_PREFIX ) +1 ) +
[snip]
Quoted 6 lines
Quoted 6 lines
+/**
+ * @internal Push several objects on the stack (MT-safe).
+ *
+ * @param s
+ * A pointer to the stack structure.
+ * @paramobj_table
Quoted 8 lines
Quoted 7 lines
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to push on the stack from theobj_table.
+ * @return
+ * Actual number of objects pushed (either 0 or *n*).
+ */
+static__rte_always_inlineunsigned int__rte_experimentalThis is an internal function. Is '
__rte_experimental' tag required?
I don't think so, but I erred on the side of caution. I don't think the tag causes any problems.
Quoted 5 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+rte_stack_std_push(struct rte_stack *s, void * const *obj_table, +unsigned int n) {Since this is an internal function, does it make sense to add '__' to the beginning of the function name (similar to what is done in
rte_ring?).
Makes sense. I'll address this.
[snip]
Quoted 6 lines
Quoted 6 lines
+/**
+ * @internal Pop several objects from the stack (MT-safe).
+ *
+ * @param s
+ * A pointer to the stack structure.
+ * @paramobj_table
Quoted 8 lines
Quoted 7 lines
+ * A pointer to a table of void * pointers (objects).
+ * @param n
+ * The number of objects to pull from the stack.
+ * @return
+ * Actual number of objects popped (either 0 or *n*).
+ */
+static__rte_always_inlineunsigned int__rte_experimentalThis is an internal function. Is '
__rte_experimental' tag required?
(see above)
[snip]
Quoted 7 lines
lib/librte_stack/rte_stack.h line 171
Quoted 4 lines
lib/librte_stack/rte_stack.h new file +4
-0+static __rte_always_inline unsigned int __rte_experimental +rte_stack_pop (struct rte_stack *s,void **obj_table,unsigned int n) { +if (unlikely (n ==0 || obj_table ==NULL )) +return 0 ;'s == NULL' can be added as well. Similar check is missing in '
rte_stack_push'. Since these are data-path APIs,RTE_ASSERTwould be better.
Good point. I'll add RTE_ASSERT for obj_table and s. That won't work for "n == 0" -- the pop code assumes n > 0, so we can't allow that check to be compiled out.
[snip]
Quoted 19 lines
Quoted 19 lines
lib/librte_stack/rte_stack.h new file +19
-0+/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Create a new stack named *name* in memory. +* +* This function uses ``memzone_reserve()`` to allocate memory for a +stack of +* size *count*. The behavior of the stack is controlled by the *flags*. +* +* @param name +* The name of the stack. +* @param count +* The size of the stack. +* @param socket_id +* The *socket_id* argument is the socket identifier in case of +* NUMA. The value can be *SOCKET_ID_ANY* if there is no NUMA +* constraint for the reserved zone. +* @param flags
Quoted 8 lines
Quoted 7 lines
+ * Reserved for future use.
+ * @return
+ * On success, the pointer to the new allocated stack. NULL on error with
+ *rte_errnoset appropriately. Possible errno values include:
+ * - ENOSPC - the maximum number of memzones has already been
allocated
+ * - EEXIST - a stack with the same name already existsThis is not implemented currently
It is -- see above.
Thanks,
Gage
Re: [PATCH v3 1/8] stack: introduce rte stack library
29/03/2019 20:23, Eads, Gage:
Quoted 3 lines
@Thomas: I expect I can address Honnappa's feedback within a day or two. Since today is the 19.05 merge deadline, what do you think about these options for merging?
1. Merge V4 now and address these comments during RC1.
2. Delay merge until RC2, with all comments addressed.
I plan to release RC1 on Wednesday,
allowing last revision to be sent on Tuesday.
If it does not impact the rest of DPDK, the RC2 is also an option to consider.
Re: [PATCH v3 1/8] stack: introduce rte stack library
In reply to Eads, GageQuoted 11 lines
Quoted 10 lines
lib/librte_stack/rte_stack.c line 49
Quoted 8 lines
lib/librte_stack/rte_stack.c new file +7
-0+static ssize_t +rte_stack_get_memsize (unsigned int count) { +ssize_t sz =sizeof (struct rte_stack ); + +/* Add padding to avoid false sharing conflicts */ + sz +=RTE_CACHE_LINE_ROUNDUP (count *sizeof (void *)) + +2 *RTE_CACHE_LINE_SIZE ;I did not understand how the false sharing is caused and how this padding is solving the issue. Verbose comments would help.
Quoted 4 lines
The additional padding (beyond the
CACHE_LINE_ROUNDUP) is to prevent false sharing caused by adjacent/next-line hardware prefetchers. I'll address this.
Is it not a generic problem? Or is it specific to this library?
Re: [PATCH v3 1/8] stack: introduce rte stack library
Quoted 22 lines
-----Original Message-----
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Monday, April 1, 2019 12:41 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v3 1/8] stack: introduce rte stack libraryQuoted 11 lines
Quoted 10 lines
lib/librte_stack/rte_stack.c line 49
Quoted 8 lines
lib/librte_stack/rte_stack.c new file +7
-0+static ssize_t +rte_stack_get_memsize (unsigned int count) { +ssize_t sz =sizeof (struct rte_stack ); + +/* Add padding to avoid false sharing conflicts */ + sz +=RTE_CACHE_LINE_ROUNDUP (count *sizeof (void *)) + +2 *RTE_CACHE_LINE_SIZE ;I did not understand how the false sharing is caused and how this padding is solving the issue. Verbose comments would help.
Quoted 5 lines
Quoted 4 lines
The additional padding (beyond the
CACHE_LINE_ROUNDUP) is to prevent false sharing caused by adjacent/next-line hardware prefetchers. I'll address this.Is it not a generic problem? Or is it specific to this library?
This is not limited to this library, but it only affects systems with (enabled) next-line prefetchers, for example Intel products with an L2 adjacent cache line prefetcher[1]. For those systems, additional padding can potentially improve performance. As I understand it, this was the reason behind the 128B alignment added to rte_ring a couple years ago[2].
[1] https://software.intel.com/en-us/articles/disclosure-of-hw-prefetcher-control-on-some-intel-processors
[2] http://mails.dpdk.org/archives/dev/2017-February/058613.html
PATCH v3 2/8 mempool/stack: convert mempool to use rte stack
In reply to Gage EadsThe new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.
--- MAINTAINERS | 2 +- drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++---------------------- 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
@@ -282,7 +282,6 @@ M: Andrew Rybchenko <[email protected]> F: lib/librte_mempool/ F: drivers/mempool/Makefile F: drivers/mempool/ring/ -F: drivers/mempool/stack/ F: doc/guides/prog_guide/mempool_lib.rst F: app/test/test_mempool* F: app/test/test_func_reentrancy.c
@@ -410,6 +409,7 @@ M: Gage Eads <[email protected]> M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst +F: drivers/mempool/stack/ Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a CFLAGS += -O3 CFLAGS += $(WERROR_FLAGS) +CFLAGS += -DALLOW_EXPERIMENTAL_API # Headers CFLAGS += -I$(RTE_SDK)/lib/librte_mempool -LDLIBS += -lrte_eal -lrte_mempool -lrte_ring +LDLIBS += -lrte_eal -lrte_mempool -lrte_stack EXPORT_MAP := rte_mempool_stack_version.map
drivers/mempool/stack/meson.build +5 -1
@@ -1,4 +1,8 @@ # SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation +# Copyright(c) 2017-2019 Intel Corporation + +allow_experimental_apis = true sources = files('rte_mempool_stack.c') + +deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68
133 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v3 3/8 test/stack: add stack test
In reply to Gage Eadsstack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.
--- MAINTAINERS | 1 + app/test/Makefile | 2 + app/test/meson.build | 3 + app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 416 insertions(+) create mode 100644 app/test/test_stack.c
MAINTAINERS +1 -0
@@ -410,6 +410,7 @@ M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst F: drivers/mempool/stack/ +F: test/test/*stack* Memory Pool Drivers
app/test/Makefile +2 -0
@@ -89,6 +89,8 @@ endif SRCS-y += test_rwlock.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c + SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
app/test/meson.build +3 -0
@@ -95,6 +95,7 @@ test_sources = files('commands.c', 'test_sched.c', 'test_service_cores.c', 'test_spinlock.c', + 'test_stack.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -132,6 +133,7 @@ test_deps = ['acl', 'port', 'reorder', 'ring', + 'stack', 'timer' ]
@@ -173,6 +175,7 @@ fast_parallel_test_names = [ 'rwlock_autotest', 'sched_autotest', 'spinlock_autotest', + 'stack_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
app/test/test_stack.c new file +410 -0
410 lines not on this page · raw message
Signature · 1 line
2.13.6
Re: [PATCH v3 3/8] test/stack: add stack test
Quoted 4 lines · Gage Eads
stack_autotestperforms positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.Signed-off-by Gage Eads <[email protected]>
PATCH v3 4/8 test/stack: add stack perf test
In reply to Gage Eadsstack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects
It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores
--- app/test/Makefile | 1 + app/test/meson.build | 2 + app/test/test_stack_perf.c | 343 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 346 insertions(+) create mode 100644 app/test/test_stack_perf.c
app/test/Makefile +1 -0
@@ -90,6 +90,7 @@ endif SRCS-y += test_rwlock.c SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
app/test/meson.build +2 -0
@@ -96,6 +96,7 @@ test_sources = files('commands.c', 'test_service_cores.c', 'test_spinlock.c', 'test_stack.c', + 'test_stack_perf.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -240,6 +241,7 @@ perf_test_names = [ 'distributor_perf_autotest', 'ring_pmd_perf_autotest', 'pmd_perf_autotest', + 'stack_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
app/test/test_stack_perf.c new file +343 -0
343 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v3 5/8 stack: add lock-free stack implementation
In reply to Gage EadsThis commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.
The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.
The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.
The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.
This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.
The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.
--- doc/guides/prog_guide/stack_lib.rst | 61 ++++++++++++- doc/guides/rel_notes/release_19_05.rst | 3 + lib/librte_stack/Makefile | 3 +- lib/librte_stack/meson.build | 3 +- lib/librte_stack/rte_stack.c | 41 +++++++-- lib/librte_stack/rte_stack.h | 127 +++++++++++++++++++++++++-- lib/librte_stack/rte_stack_generic.h | 151 +++++++++++++++++++++++++++++++++ 7 files changed, 371 insertions(+), 18 deletions(-) create mode 100644 lib/librte_stack/rte_stack_generic.h
doc/guides/prog_guide/stack_lib.rst +58 -3
@@ -10,7 +10,8 @@ stack of pointers. The stack library provides the following basic operations: * Create a uniquely named stack of a user-specified size and using a - user-specified socket. + user-specified socket, with either standard (lock-based) or lock-free + behavior. * Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe.
@@ -24,5 +25,59 @@ The stack library provides the following basic operations: Implementation ~~~~~~~~~~~~~~ -The stack consists of a contiguous array of pointers, a current index, and a -spinlock. Accesses to the stack are made multi-thread safe by the spinlock. +The library supports two types of stacks: standard (lock-based) and lock-free. +Both types use the same set of interfaces, but their implementations differ. + +Lock-based Stack +---------------- + +The lock-based stack consists of a contiguous array of pointers, a current +index, and a spinlock. Accesses to the stack are made multi-thread safe by the +spinlock. + +Lock-free Stack +------------------ + +The lock-free stack consists of a linked list of elements, each containing a +data pointer and a next pointer, and an atomic stack depth counter. The +lock-free property means that multiple threads can push and pop simultaneously, +and one thread being preempted/delayed in a push or pop operation will not +impede the forward progress of any other thread. + +The lock-free push operation enqueues a linked list of pointers by pointing the +list's tail to the current stack head, and using a CAS to swing the stack head +pointer to the head of the list. The operation retries if it is unsuccessful +(i.e. the list changed between reading the head and modifying it), else it +adjusts the stack length and returns. + +The lock-free pop operation first reserves one or more list elements by +adjusting the stack length, to ensure the dequeue operation will succeed +without blocking. It then dequeues pointers by walking the list -- starting +from the head -- then swinging the head pointer (using a CAS as well). While +walking the list, the data pointers are recorded in an object table. + +The linked list elements themselves are maintained in a lock-free LIFO, and are +allocated before stack pushes and freed after stack pops. Since the stack has a +fixed maximum depth, these elements do not need to be dynamically created. + +The lock-free behavior is selected by passing the *RTE_STACK_F_LF* flag to +rte_stack_create(). + +Preventing the ABA Problem +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To prevent the ABA problem, this algorithm stack uses a 128-bit +compare-and-swap instruction to atomically update both the stack top pointer +and a modification counter. The ABA problem can occur without a modification +counter if, for example: + +1. Thread A reads head pointer X and stores the pointed-to list element. +2. Other threads modify the list such that the head pointer is once again X, + but its pointed-to data is different than what thread A read. +3. Thread A changes the head pointer with a compare-and-swap and succeeds. + +In this case thread A would not detect that the list had changed, and would +both pop stale data and incorrect change the head pointer. By adding a +modification counter that is updated on every push and pop as part of the +compare-and-swap, the algorithm can detect when the list changes even if the +head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
@@ -83,6 +83,9 @@ New Features pointers. The API provides MT-safe push and pop operations that can operate on one or more pointers per operation. + The library supports two stack implementations: standard (lock-based) and lock-free. + The lock-free implementation is currently limited to x86-64 platforms. + Removed Items -------------
lib/librte_stack/Makefile +2 -1
@@ -18,6 +18,7 @@ LIBABIVER := 1 SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c # install includes -SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h +SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ + rte_stack_generic.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
@@ -5,4 +5,5 @@ allow_experimental_apis = true version = 1 sources = files('rte_stack.c') -headers = files('rte_stack.h') +headers = files('rte_stack.h', + 'rte_stack_generic.h')
lib/librte_stack/rte_stack.c +33 -8
76 lines not on this page · raw message
lib/librte_stack/rte_stack.h +122 -5
166 lines not on this page · raw message
lib/librte_stack/rte_stack_generic.h new file +151 -0
151 lines not on this page · raw message
Signature · 1 line
2.13.6
Re: [PATCH v3 5/8] stack: add lock-free stack implementation
Quoted 23 lines · Gage Eads
This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new
rte_stack_create()flag,RTE_STACK_F_LF.The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.
The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.
The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.
This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.
Quoted 6 lines
The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.
Signed-off-by Gage Eads <[email protected]>
Re: [PATCH v3 5/8] stack: add lock-free stack implementation
In reply to Gage Eads<snip>
Quoted 4 lines
lib/librte_stack/rte_stack.c index renamed from lib/librte_stack/rte_stack.c +0
-0
96dffdf44..8f0361ea1100644lib/librte_stack/rte_stack.c +0
-0
<snip>
lib/librte_stack/rte_stack.c rte_stack_create(const char *name, unsigned int · line 86
Quoted 8 lines
lib/librte_stack/rte_stack.c +2
-1@@ -63,9 +81,16 @@ rte_stack_create(const char *name, unsigned int count, int socket_id, unsigned int sz;int ret; -RTE_SET_USED (flags); +#ifdef RTE_ARCH_X86_64 +RTE_BUILD_BUG_ON (sizeof (struct rte_stack_lf_head ) !=16 );
This check should be independent of the platform.
#else
Quoted 27 lines
lib/librte_stack/rte_stack.c +7
-2+if (flags &RTE_STACK_F_LF ) { +STACK_LOG_ERR ("Lock-free stack is not supported on your platform\n"); +return NULL ; + } +#endif - sz =rte_stack_get_memsize (count); + sz =rte_stack_get_memsize (count, flags); ret =snprintf (mz_name,sizeof (mz_name),"%s%s" ,RTE_STACK_MZ_PREFIX , name);@@ -94,7 +119,7 @@ rte_stack_create(const char *name, unsigned int count, int socket_id, s = mz->addr; -rte_stack_init (s); +rte_stack_init (s, count, flags);/* Store the name for later lookups */ ret =
snprintf(s->name, sizeof(s->name), "%s", name); diff --gita/lib/librte_stack/rte_stack.hb/lib/librte_stack/rte_stack.hindex7a633deb5..b484313bb100644lib/librte_stack/rte_stack.h +0
-0@@ -30,6 +30,35 @@ extern "C" {
<snip>
lib/librte_stack/rte_stack.h struct rte_stack { · line 108
Quoted 13 lines
lib/librte_stack/rte_stack.h +13
-0+/** +* @internal Push several objects on the lock-free stack (MT-safe). +* +* @param s +* A pointer to the stack structure. +* @param obj_table +* A pointer to a table of void * pointers (objects). +* @param n +* The number of objects to push on the stack from the obj_table. +* @return +* Actual number of objects enqueued. +*/ +static __rte_always_inline unsigned int __rte_experimental
This is an internal function. Is '__rte_experimental' tag required? (applies to other instances in this patch)
Quoted 23 lines
lib/librte_stack/rte_stack.h +23
-0+rte_stack_lf_push(struct rte_stack *s, void * const *obj_table, +unsigned int n) { + struct rte_stack_lf_elem *tmp, *first, *last = NULL; +unsigned int i; + +if (unlikely (n ==0 )) +return 0 ; + +/* Pop n free elements */ + first =__rte_stack_lf_pop (&s->stack_lf.free, n,NULL , &last); +if (unlikely (first ==NULL )) +return 0 ; + +/* Construct the list elements */ +for (tmp = first, i =0 ; i < n; i++, tmp = tmp->next) + tmp->data = obj_table[n - i -1 ]; + +/* Push them to the used list */ +__rte_stack_lf_push (&s->stack_lf.used, first, last, n); + +return n; +} +
<snip>
lib/librte_stack/rte_stack_generic.h
Quoted 20 lines
/**
lib/librte_stack/rte_stack.h +4
-1@@ -225,7 +339,10 @@ rte_stack_free_count(struct rte_stack *s)* NUMA. The value can be *SOCKET_ID_ANY* if there is no NUMA * constraint for the reserved zone. * @param flags -* Reserved for future use. +* An OR of the following: +* - RTE_STACK_F_LF: If this flag is set, the stack uses lock-free +* variants of the push and pop functions. Otherwise, it achieves +* thread-safety using a lock. * @return * On success, the pointer to the new allocated stack. NULL on error with * rte_errno set appropriately. Possible errno values include: lib/librte_stack/rte_stack_generic.h new file +0
-0
The name "...stack_generic.h" is confusing. It is implementing LF algorithm.
IMO, the code should be re-organized differently.rte_stack.h, rte_stack.c - Contain the APIs (calling std or LF based on the flag) and top level structure definition rte_stack_std.c, rte_stack_std.h - Contain the standard implementationrte_stack_lf.c, rte_stack_lf.h - Contain the LF implementationrte_stack_lf_c11.h - Contain the LF C11 implementation
Quoted 11 lines
lib/librte_stack/rte_stack_generic.h new file +10
-0@@ -0,0 +1,151 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#ifndef _RTE_STACK_GENERIC_H_ +#define _RTE_STACK_GENERIC_H_ + +#include <rte_branch_prediction.h> +#include <rte_prefetch.h> +
<snip>
Quoted 64 lines
lib/librte_stack/rte_stack_generic.h new file +46
-0+ +static __rte_always_inline struct rte_stack_lf_elem * +__rte_stack_lf_pop (struct rte_stack_lf_list *list, +unsigned int num, +void **obj_table, +struct rte_stack_lf_elem **last) +{ +#ifndef RTE_ARCH_X86_64 + RTE_SET_USED(obj_table); + RTE_SET_USED(last); + RTE_SET_USED(list); + RTE_SET_USED(num); + + return NULL; +#else + struct rte_stack_lf_head old_head; + int success; + + /* Reserve num elements, if available */ + while (1) { + uint64_t len = rte_atomic64_read(&list->len); + + /* Does the list contain enough elements? */ + if (unlikely(len < num)) + return NULL; + + if (rte_atomic64_cmpset((volatile uint64_t *)&list->len, + len, len - num)) + break; + } + + old_head = list->head; + + /* Pop num elements */ + do { + struct rte_stack_lf_head new_head; + struct rte_stack_lf_elem *tmp; + unsigned int i; + + rte_prefetch0(old_head.top); + + tmp = old_head.top; + + /* Traverse the list to find the new head. A next pointer will + * either point to another element or NULL; if a thread + * encounters a pointer that has already been popped, theCAS
lib/librte_stack/rte_stack_generic.h new file +17
-0+* will fail. +*/ +for (i =0 ; i < num && tmp !=NULL ; i++) { +rte_prefetch0 (tmp->next); +if (obj_table) + obj_table[i] = tmp->data; +if (last) + *last = tmp; + tmp = tmp->next; + } + +/* If NULL was encountered, the list was modified while +* traversing it. Retry. +*/ +if (i != num) +continue ; +
lib/librte_stack/rte_stack_generic.h line 144
Quoted 10 lines
lib/librte_stack/rte_stack_generic.h new file +10
-0+ new_head.top = tmp; + new_head.cnt = old_head.cnt +1 ; + +/* old_head is updated on failure */ + success =rte_atomic128_cmp_exchange ( + (rte_int128_t *)&list->head, + (rte_int128_t *)&old_head, + (rte_int128_t *)&new_head, +1 , __ATOMIC_ACQUIRE, + __ATOMIC_ACQUIRE);
Just wondering if 'rte_atomic128_cmp_exchange' for x86 should have compiler barriers based on the memory order passed? C++11 memory model is getting mixed with barrier based model. I think this is something that needs to be discussed at a wider level.
Quoted 9 lines
lib/librte_stack/rte_stack_generic.h new file +7
-1+ } while (success == 0); + +return old_head.top; +#endif +} + +#endif /* _RTE_STACK_GENERIC_H_ */ --2.13.6
Re: [PATCH v3 5/8] stack: add lock-free stack implementation
Quoted 28 lines
-----Original Message-----
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Thursday, March 28, 2019 6:27 PM
To: Eads, Gage <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v3 5/8] stack: add lock-free stack implementation<snip>
Quoted 5 lines
(quoted diff) +0
-0
b/lib/librte_stack/rte_stack.cindex96dffdf44..8f0361ea1100644lib/librte_stack/rte_stack.c +0
-0<snip>
Quoted 5 lines
lib/librte_stack/rte_stack.c +0
-0@@ -63,9 +81,16 @@ rte_stack_create(const char *name, unsigned int count, int socket_id, unsigned int sz;int ret;
Quoted 4 lines
lib/librte_stack/rte_stack.c rte_stack_create(const char *name, unsigned int count, int socket_id, · line 85
Quoted 3 lines
lib/librte_stack/rte_stack.c +2
-1-RTE_SET_USED (flags); +#ifdef RTE_ARCH_X86_64 +RTE_BUILD_BUG_ON (sizeof (struct rte_stack_lf_head ) !=16 );This check should be independent of the platform.
Good catch. Will change the ifdef to RTE_ARCH_64.
[snip]
Quoted 7 lines
Quoted 7 lines
lib/librte_stack/rte_stack.c +7
-0+/** +* @internal Push several objects on the lock-free stack (MT-safe). +* +* @param s +* A pointer to the stack structure. +* @param obj_table +* A pointer to a table of void * pointers (objects).
Quoted 8 lines
lib/librte_stack/rte_stack.c rte_stack_create(const char *name, unsigned int
Quoted 6 lines
lib/librte_stack/rte_stack.c +6
-0+* @param n +* The number of objects to push on the stack from the obj_table. +* @return +* Actual number of objects enqueued. +*/ +static __rte_always_inline unsigned int __rte_experimentalThis is an internal function. Is '
__rte_experimental' tag required? (applies to other instances in this patch)
(Addressed in comments to patch 1/8)
[snip]
Quoted 18 lines
Quoted 18 lines
/**
lib/librte_stack/rte_stack.c +4
-1@@ -225,7 +339,10 @@ rte_stack_free_count(struct rte_stack *s)* NUMA. The value can be *SOCKET_ID_ANY* if there is no NUMA * constraint for the reserved zone. * @param flags -* Reserved for future use. +* An OR of the following: +* - RTE_STACK_F_LF: If this flag is set, the stack uses lock-free +* variants of the push and pop functions. Otherwise, it achieves +* thread-safety using a lock. * @return * On success, the pointer to the new allocated stack. NULL on error with * rte_errno set appropriately. Possible errno values include: lib/librte_stack/rte_stack_generic.h new file +0
-0
Quoted 9 lines
lib/librte_stack/rte_stack_generic.h
Quoted 2 lines
lib/librte_stack/rte_stack_generic.h new file +0
-0The name "
...stack_generic.h" is confusing. It is implementing LF algorithm. IMO, the code should be re-organized differently.rte_stack.h,rte_stack.c- Contain the APIs (calling std or LF based on the flag) and top level structure definitionrte_stack_std.c,rte_stack_std.h- Contain the standard implementationrte_stack_lf.c,rte_stack_lf.h- Contain the LF implementationrte_stack_lf_c11.h- Contain the LF C11 implementation
'generic' here refers to the "generic API for atomic operations" (generic/rte_atomic.h:12), but I see how that can be misleading.
Yeah, I like this proposal, but with one tweak: use three lock-free header files: rte_stack_lf.h (common inline lock-free functions like rte_stack_lf_pop()), rte_stack_lf_c11.h (C11 implementation), rte_stack_lf_generic.h (generic atomic implementation). Since the name is *_lf_generic.h, it should be clear that it implements the lock-free functions, and this naming matches rte ring's (easier to pick up for those already used to the ring organization).
[snip]
Quoted 11 lines
lib/librte_stack/rte_stack_generic.h line 144
Quoted 7 lines
lib/librte_stack/rte_stack_generic.h new file +7
-0+/* old_head is updated on failure */ + success =rte_atomic128_cmp_exchange ( + (rte_int128_t *)&list->head, + (rte_int128_t *)&old_head, + (rte_int128_t *)&new_head, +1 , __ATOMIC_ACQUIRE, + __ATOMIC_ACQUIRE);Just wondering if '
rte_atomic128_cmp_exchange' for x86 should have compiler barriers based on the memory order passed?
C++11 memory model is getting mixed with barrier based model. I think this is something that needs to be discussed at a wider level.
The x86 implementation uses a compiler barrier (i.e. the inline assembly clobber list contains "memory") regardless of the memory order, so we're (conservatively) adhering to the C11 model, which guarantees ordering at both the compiler and processor levels. Whether/how we relax the x86 implementation (e.g. no compiler barrier if ordering == relaxed) is an interesting question.
Thanks,
Gage
PATCH v3 6/8 stack: add C11 atomic implementation
In reply to Gage EadsThis commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.
--- lib/librte_stack/Makefile | 3 +- lib/librte_stack/meson.build | 3 +- lib/librte_stack/rte_stack.h | 4 + lib/librte_stack/rte_stack_c11_mem.h | 175 +++++++++++++++++++++++++++++++++++ 4 files changed, 183 insertions(+), 2 deletions(-) create mode 100644 lib/librte_stack/rte_stack_c11_mem.h
lib/librte_stack/Makefile +2 -1
@@ -19,6 +19,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ - rte_stack_generic.h + rte_stack_generic.h \ + rte_stack_c11_mem.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
@@ -6,4 +6,5 @@ allow_experimental_apis = true version = 1 sources = files('rte_stack.c') headers = files('rte_stack.h', - 'rte_stack_generic.h') + 'rte_stack_generic.h', + 'rte_stack_c11_mem.h')
lib/librte_stack/rte_stack.h +4 -0
@@ -91,7 +91,11 @@ struct rte_stack {*/ #define RTE_STACK_F_LF 0x0001 +#ifdef RTE_USE_C11_MEM_MODEL +#include "rte_stack_c11_mem.h" +#else #include "rte_stack_generic.h" +#endif /** * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_c11_mem.h new file +175 -0
175 lines not on this page · raw message
Signature · 1 line
2.13.6
Re: [PATCH v3 6/8] stack: add C11 atomic implementation
Quoted 5 lines · Gage Eads
This commit adds an implementation of the lock-free stack push, pop, and length functions that use
__atomicbuiltins, for systems that benefit from the finer-grained memory ordering control.Signed-off-by Gage Eads <[email protected]>
Re: [PATCH v3 6/8] stack: add C11 atomic implementation
In reply to Gage EadsQuoted 102 lines
This commit adds an implementation of the lock-free stack push, pop, and length functions that use
__atomicbuiltins, for systems that benefit from the finer-grained memory ordering control.Signed-off-by Gage Eads <[email protected]>--- lib/librte_stack/Makefile | 3 +- lib/librte_stack/meson.build | 3 +- lib/librte_stack/rte_stack.h | 4 + lib/librte_stack/rte_stack_c11_mem.h | 175 +++++++++++++++++++++++++++++++++++ 4 files changed, 183 insertions(+), 2 deletions(-) create mode 100644
lib/librte_stack/rte_stack_c11_mem.hlib/librte_stack/Makefile index renamed from lib/librte_stack/Makefile +0
-0
3ecddf033..94a7c1476100644lib/librte_stack/Makefile +2
-1@@ -19,6 +19,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ - rte_stack_generic.h + rte_stack_generic.h \ + rte_stack_c11_mem.h include $(RTE_SDK)/mk/rte.lib.mklib/librte_stack/meson.build index renamed from lib/librte_stack/meson.build +0
-0
99d7f9ec5..7e2d1dbb8100644lib/librte_stack/meson.build +2
-1@@ -6,4 +6,5 @@ allow_experimental_apis = true version = 1 sources = files('rte_stack.c') headers = files('rte_stack.h', - 'rte_stack_generic.h') + 'rte_stack_generic.h', + 'rte_stack_c11_mem.h')lib/librte_stack/rte_stack.h index renamed from lib/librte_stack/rte_stack.h +0
-0
b484313bb..de16f8fff100644lib/librte_stack/rte_stack.h +4
-0@@ -91,7 +91,11 @@ struct rte_stack {*/ #define RTE_STACK_F_LF 0x0001 +#ifdef RTE_USE_C11_MEM_MODEL +#include "rte_stack_c11_mem.h" +#else #include "rte_stack_generic.h" +#endif /** * @internal Push several objects on the lock-free stack (MT-safe). lib/librte_stack/rte_stack_c11_mem.h new file +13
-0@@ -0,0 +1,175 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#ifndef _RTE_STACK_C11_MEM_H_ +#define _RTE_STACK_C11_MEM_H_ + +#include <rte_branch_prediction.h> +#include <rte_prefetch.h> + +static __rte_always_inline unsigned int rte_stack_lf_len(struct +rte_stack *s) { +/* stack_lf_push() and stack_lf_pop() do not update the list's contents
lib/librte_stack/rte_stack_c11_mem.h new file +5
-0+* and stack_lf->len atomically, which can cause the list to appear +* shorter than it actually is if this function is called while other +* threads are modifying the list. +* +* However, given the inherently approximate nature of the
get_countlib/librte_stack/rte_stack_c11_mem.h new file +2
-0+* callback -- even if the list and its size were updated atomically, +* the size could change between when get_count executes and when the
lib/librte_stack/rte_stack_c11_mem.h new file +3
-0+* value is returned to the caller -- this is acceptable. +* +* The stack_lf->len updates are placed such that the list may appear to
lib/librte_stack/rte_stack_c11_mem.h new file +1
-0+* have fewer elements than it does, but will never appear to have more
lib/librte_stack/rte_stack_c11_mem.h new file +13
-0+* elements. If the mempool is near-empty to the point that this is a +* concern, the user should consider increasing the mempool size. +*/ +return (unsigned int )__atomic_load_n (&s->stack_lf.used.len.cnt, + __ATOMIC_RELAXED); +} + +static __rte_always_inline void +__rte_stack_lf_push (struct rte_stack_lf_list *list, +struct rte_stack_lf_elem *first, +struct rte_stack_lf_elem *last, +unsigned int num) +{
lib/librte_stack/rte_stack_c11_mem.h line 48
Quoted 10 lines
lib/librte_stack/rte_stack_c11_mem.h new file +10
-0+#ifndef RTE_ARCH_X86_64 +RTE_SET_USED (first); +RTE_SET_USED (last); +RTE_SET_USED (list); +RTE_SET_USED (num); +#else +struct rte_stack_lf_head old_head; +int success; + + old_head = list->head;
This can be a torn read (same as you have mentioned in __rte_stack_lf_pop). I suggest we use acquire thread fence here as well (please see the comments in __rte_stack_lf_pop).
lib/librte_stack/rte_stack_c11_mem.h line 52
Quoted 4 lines
lib/librte_stack/rte_stack_c11_mem.h new file +4
-0+ + do { + struct rte_stack_lf_head new_head; +
We can add __atomic_thread_fence(__ATOMIC_ACQUIRE) here (please see the comments in __rte_stack_lf_pop).
Quoted 10 lines
lib/librte_stack/rte_stack_c11_mem.h new file +9
-0+/* Swing the top pointer to the first element in the list and +* make the last element point to the old top. +*/ + new_head.top = first; + new_head.cnt = old_head.cnt +1 ; + + last->next = old_head.top; + +/* Use the release memmodel to ensure the writes to the LF LIFO
lib/librte_stack/rte_stack_c11_mem.h line 69
Quoted 8 lines
lib/librte_stack/rte_stack_c11_mem.h new file +8
-0+* elements are visible before the head pointer write. +*/ + success =rte_atomic128_cmp_exchange ( + (rte_int128_t *)&list->head, + (rte_int128_t *)&old_head, + (rte_int128_t *)&new_head, +1 , __ATOMIC_RELEASE, + __ATOMIC_RELAXED);
Success memory order can be RELAXED as the store to list->len.cnt is RELEASE.
Quoted 19 lines
lib/librte_stack/rte_stack_c11_mem.h new file +6
-0+ }while (success ==0 ); + +/* Ensure the stack modifications are not reordered with respect +* to the LIFO len update. +*/ +__atomic_add_fetch (&list->len.cnt, num, __ATOMIC_RELEASE);#endif }
lib/librte_stack/rte_stack_c11_mem.h new file +12
-0+ +static __rte_always_inline struct rte_stack_lf_elem * +__rte_stack_lf_pop (struct rte_stack_lf_list *list, +unsigned int num, +void **obj_table, +struct rte_stack_lf_elem **last) +{ +#ifndef RTE_ARCH_X86_64 +RTE_SET_USED (obj_table); +RTE_SET_USED (last); +RTE_SET_USED (list); +RTE_SET_USED (num);
lib/librte_stack/rte_stack_c11_mem.h line 99
Quoted 10 lines
lib/librte_stack/rte_stack_c11_mem.h new file +10
-0+ +return NULL ; +#else +struct rte_stack_lf_head old_head; +int success; + +/* Reserve num elements, if available */ +while (1 ) { +uint64_t len =__atomic_load_n (&list->len.cnt, + __ATOMIC_ACQUIRE);
This can be outside the loop.
Quoted 19 lines
lib/librte_stack/rte_stack_c11_mem.h new file +14
-0+ +/* Does the list contain enough elements? */ +if (unlikely (len < num)) +return NULL ; + +if (__atomic_compare_exchange_n (&list->len.cnt, + &len, len - num, +0 , __ATOMIC_RELAXED, + __ATOMIC_RELAXED)) +break ; + } + +#ifndef RTE_ARCH_X86_64 +/* Use the acquire memmodel to ensure the reads to the LF LIFO elements
lib/librte_stack/rte_stack_c11_mem.h new file +3
-0+* are properly ordered with respect to the head pointer read. +* +* Note that for aarch64, GCC's implementation of __atomic_load_16 in
lib/librte_stack/rte_stack_c11_mem.h line 118
Quoted 2 lines
lib/librte_stack/rte_stack_c11_mem.h new file +2
-0+* libatomic uses locks, and so this function should be replaced by +* a new function (e.g. "rte_atomic128_load()").
aarch64 does not have 'pure' atomic 128b load instructions. They have to be implemented using load/store exclusives.
lib/librte_stack/rte_stack_c11_mem.h line 122
Quoted 4 lines
lib/librte_stack/rte_stack_c11_mem.h new file +4
-0+*/ +__atomic_load ((volatile __int128 *)&list->head, + &old_head, + __ATOMIC_ACQUIRE);
Since, we know of x86/aarch64 (power?) that cannot implement pure atomic 128b loads, should we just use relaxed reads and assume the possibility of torn reads for all architectures? Then, we can use acquire fence to prevent the reordering (see below)
lib/librte_stack/rte_stack_c11_mem.h
Quoted 2 lines
lib/librte_stack/rte_stack_c11_mem.h new file +2
-0+#else +/* x86-64 does not require an atomic load here; if a torn read occurs,
IMO, we should not make architecture specific distinctions as this algorithm is based on C11 memory model.
lib/librte_stack/rte_stack_c11_mem.h line 135
Quoted 11 lines
lib/librte_stack/rte_stack_c11_mem.h new file +11
-0+* the CAS will fail and set old_head to the correct/latest value. +*/ + old_head = list->head; +#endif + +/* Pop num elements */ + do { + struct rte_stack_lf_head new_head; + struct rte_stack_lf_elem *tmp; + unsigned int i; +
We can add __atomic_thread_fence(__ATOMIC_ACQUIRE) here.
Quoted 24 lines
lib/librte_stack/rte_stack_c11_mem.h new file +7
-0+rte_prefetch0 (old_head.top); + + tmp = old_head.top; + +/* Traverse the list to find the new head. A next pointer will +* either point to another element or NULL; if a thread +* encounters a pointer that has already been popped, the CAS
lib/librte_stack/rte_stack_c11_mem.h new file +16
-0+* will fail. +*/ +for (i =0 ; i < num && tmp !=NULL ; i++) { +rte_prefetch0 (tmp->next); +if (obj_table) + obj_table[i] = tmp->data; +if (last) + *last = tmp; + tmp = tmp->next; + } + +/* If NULL was encountered, the list was modified while +* traversing it. Retry. +*/ +if (i != num) +continue ;
lib/librte_stack/rte_stack_c11_mem.h line 168
Quoted 10 lines
lib/librte_stack/rte_stack_c11_mem.h new file +10
-0+ + new_head.top = tmp; + new_head.cnt = old_head.cnt +1 ; + + success =rte_atomic128_cmp_exchange ( + (rte_int128_t *)&list->head, + (rte_int128_t *)&old_head, + (rte_int128_t *)&new_head, +1 , __ATOMIC_ACQUIRE, + __ATOMIC_ACQUIRE);
The success order should be __ATOMIC_RELEASE as the write to list->len.cnt is relaxed. The failure order can be __ATOMIC_RELAXED if the thread fence is added.
Quoted 9 lines
lib/librte_stack/rte_stack_c11_mem.h new file +7
-1+ } while (success == 0); + +return old_head.top; +#endif +} + +#endif /* _RTE_STACK_C11_MEM_H_ */ --2.13.6
Re: [PATCH v3 6/8] stack: add C11 atomic implementation
[snip]
Quoted 11 lines
Quoted 11 lines
lib/librte_stack/rte_stack_c11_mem.h new file +11
-0+static __rte_always_inline void +__rte_stack_lf_push (struct rte_stack_lf_list *list, +struct rte_stack_lf_elem *first, +struct rte_stack_lf_elem *last, +unsigned int num) +{ +#ifndef RTE_ARCH_X86_64 +RTE_SET_USED (first); +RTE_SET_USED (last); +RTE_SET_USED (list); +RTE_SET_USED (num);
Quoted 8 lines
lib/librte_stack/rte_stack_c11_mem.h line 48
Quoted 5 lines
lib/librte_stack/rte_stack_c11_mem.h new file +5
-0+#else +struct rte_stack_lf_head old_head; +int success; + + old_head = list->head;This can be a torn read (same as you have mentioned in
__rte_stack_lf_pop). I suggest we use acquire thread fence here as well (please see the comments in__rte_stack_lf_pop).
Agreed. I'll add the acquire fence.
Quoted 6 lines
lib/librte_stack/rte_stack_c11_mem.h line 52
Quoted 4 lines
lib/librte_stack/rte_stack_c11_mem.h new file +4
-0+ + do { + struct rte_stack_lf_head new_head; +We can add
__atomic_thread_fence(__ATOMIC_ACQUIRE)here (please see the comments in__rte_stack_lf_pop).
Will add the fence here.
Quoted 12 lines
Quoted 12 lines
lib/librte_stack/rte_stack_c11_mem.h new file +9
-0+/* Swing the top pointer to the first element in the list and +* make the last element point to the old top. +*/ + new_head.top = first; + new_head.cnt = old_head.cnt +1 ; + + last->next = old_head.top; + +/* Use the release memmodel to ensure the writes to the LF LIFO
(quoted diff) +2
-0+ * elements are visible before the head pointer write. + */
Quoted 8 lines
lib/librte_stack/rte_stack_c11_mem.h line 69
Quoted 6 lines
lib/librte_stack/rte_stack_c11_mem.h new file +6
-0+ success =rte_atomic128_cmp_exchange ( + (rte_int128_t *)&list->head, + (rte_int128_t *)&old_head, + (rte_int128_t *)&new_head, +1 , __ATOMIC_RELEASE, + __ATOMIC_RELAXED);Success memory order can be RELAXED as the store to
list->len.cntis RELEASE.
The RELEASE success order here ensures that the store to 'last->next' is visible before the head update. The RELEASE in the list->len.cnt store only guarantees that the preceding stores are visible before list->len.cnt's store, but doesn't guarantee any ordering between the 'last->next' store and the head update, so we can't rely on that.
Quoted 14 lines
lib/librte_stack/rte_stack_c11_mem.h
Quoted 13 lines
lib/librte_stack/rte_stack_c11_mem.h new file +6
-0+ }while (success ==0 ); + +/* Ensure the stack modifications are not reordered with respect +* to the LIFO len update. +*/ +__atomic_add_fetch (&list->len.cnt, num, __ATOMIC_RELEASE);#endif }
lib/librte_stack/rte_stack_c11_mem.h new file +6
-0+ +static __rte_always_inline struct rte_stack_lf_elem * +__rte_stack_lf_pop (struct rte_stack_lf_list *list, +unsigned int num, +void **obj_table, + struct rte_stack_lf_elem **last) { #ifndef
RTE_ARCH_X86_64
Quoted 15 lines
Quoted 4 lines
lib/librte_stack/rte_stack_c11_mem.h new file +4
-0+RTE_SET_USED (obj_table); +RTE_SET_USED (last); +RTE_SET_USED (list); +RTE_SET_USED (num);lib/librte_stack/rte_stack_c11_mem.h line 99
Quoted 10 lines
lib/librte_stack/rte_stack_c11_mem.h new file +10
-0+ +return NULL ; +#else +struct rte_stack_lf_head old_head; +int success; + +/* Reserve num elements, if available */ +while (1 ) { +uint64_t len =__atomic_load_n (&list->len.cnt, + __ATOMIC_ACQUIRE);This can be outside the loop.
Good idea. I'll move this out of the loop and change the atomic_compare_exchange_n's failure memory order to ACQUIRE.
Quoted 24 lines
Quoted 19 lines
lib/librte_stack/rte_stack_c11_mem.h new file +14
-0+ +/* Does the list contain enough elements? */ +if (unlikely (len < num)) +return NULL ; + +if (__atomic_compare_exchange_n (&list->len.cnt, + &len, len - num, +0 , __ATOMIC_RELAXED, + __ATOMIC_RELAXED)) +break ; + } + +#ifndef RTE_ARCH_X86_64 +/* Use the acquire memmodel to ensure the reads to the LF LIFO elements
lib/librte_stack/rte_stack_c11_mem.h new file +3
-0+* are properly ordered with respect to the head pointer read. +* +* Note that for aarch64, GCC's implementation of __atomic_load_16 in
lib/librte_stack/rte_stack_c11_mem.h line 118
Quoted 2 lines
lib/librte_stack/rte_stack_c11_mem.h new file +2
-0+* libatomic uses locks, and so this function should be replaced by +* a new function (e.g. "rte_atomic128_load()"). aarch64 does not have 'pure' atomic 128b load instructions. They have to be implemented using load/store exclusives.
Quoted 8 lines
lib/librte_stack/rte_stack_c11_mem.h line 122
Quoted 4 lines
lib/librte_stack/rte_stack_c11_mem.h new file +4
-0+*/ +__atomic_load ((volatile __int128 *)&list->head, + &old_head, + __ATOMIC_ACQUIRE);Since, we know of x86/aarch64 (power?) that cannot implement pure atomic 128b loads, should we just use relaxed reads and assume the possibility of torn reads for all architectures? Then, we can use acquire fence to prevent the reordering (see below)
That's a cleaner solution. I'll implement that, dropping the architecture distinction.
Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+#else + /* x86-64 does not require an atomic load here; if a torn read +occurs,IMO, we should not make architecture specific distinctions as this algorithm is based on C11 memory model.
Quoted 12 lines
lib/librte_stack/rte_stack_c11_mem.h line 135
Quoted 11 lines
lib/librte_stack/rte_stack_c11_mem.h new file +11
-0+* the CAS will fail and set old_head to the correct/latest value. +*/ + old_head = list->head; +#endif + +/* Pop num elements */ + do { + struct rte_stack_lf_head new_head; + struct rte_stack_lf_elem *tmp; + unsigned int i; +We can add
__atomic_thread_fence(__ATOMIC_ACQUIRE)here.
Will do.
Quoted 29 lines
Quoted 29 lines
lib/librte_stack/rte_stack_c11_mem.h new file +7
-0+rte_prefetch0 (old_head.top); + + tmp = old_head.top; + +/* Traverse the list to find the new head. A next pointer will +* either point to another element or NULL; if a thread +* encounters a pointer that has already been popped, the CAS
lib/librte_stack/rte_stack_c11_mem.h new file +21
-0+* will fail. +*/ +for (i =0 ; i < num && tmp !=NULL ; i++) { +rte_prefetch0 (tmp->next); +if (obj_table) + obj_table[i] = tmp->data; +if (last) + *last = tmp; + tmp = tmp->next; + } + +/* If NULL was encountered, the list was modified while +* traversing it. Retry. +*/ +if (i != num) +continue ; + + new_head.top = tmp; + new_head.cnt = old_head.cnt +1 ; + + success = rte_atomic128_cmp_exchange(
Quoted 8 lines
lib/librte_stack/rte_stack_c11_mem.h line 168
Quoted 5 lines
lib/librte_stack/rte_stack_c11_mem.h new file +5
-0+ (rte_int128_t *)&list->head, + (rte_int128_t *)&old_head, + (rte_int128_t *)&new_head, +1 , __ATOMIC_ACQUIRE, + __ATOMIC_ACQUIRE);The success order should be
__ATOMIC_RELEASEas the write tolist->len.cntis relaxed.
The failure order can be__ATOMIC_RELAXEDif the thread fence is added.
Agreed on both counts. Will address.
Quoted 9 lines
Quoted 9 lines
lib/librte_stack/rte_stack_c11_mem.h new file +7
-1+ } while (success == 0); + +return old_head.top; +#endif +} + +#endif /* _RTE_STACK_C11_MEM_H_ */ --2.13.6
Re: [PATCH v3 6/8] stack: add C11 atomic implementation
Quoted 36 lines
-----Original Message-----
From: Eads, Gage
Sent: Friday, March 29, 2019 2:25 PM
To: Honnappa Nagarahalli <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>; Thomas Monjalon <[email protected]>
Subject: RE: [PATCH v3 6/8] stack: add C11 atomic implementation[snip]
Quoted 11 lines
Quoted 11 lines
lib/librte_stack/rte_stack_c11_mem.h new file +11
-0+static __rte_always_inline void +__rte_stack_lf_push (struct rte_stack_lf_list *list, +struct rte_stack_lf_elem *first, +struct rte_stack_lf_elem *last, +unsigned int num) +{ +#ifndef RTE_ARCH_X86_64 +RTE_SET_USED (first); +RTE_SET_USED (last); +RTE_SET_USED (list); +RTE_SET_USED (num);Quoted 8 lines
lib/librte_stack/rte_stack_c11_mem.h line 48
Quoted 5 lines
lib/librte_stack/rte_stack_c11_mem.h new file +5
-0+#else +struct rte_stack_lf_head old_head; +int success; + + old_head = list->head;This can be a torn read (same as you have mentioned in
__rte_stack_lf_pop). I suggest we use acquire thread fence here as well (please see the comments in__rte_stack_lf_pop).
Quoted 2 lines
Agreed. I'll add the acquire fence.
On second thought, an acquire fence isn't necessary. The acquire fence in __rte_stack_lf_pop() ensures the list->head is ordered before the list element reads. That isn't necessary here; we need to ensure that the last->next write occurs (and is observed) before the list->head write, which the CAS's RELEASE success memorder accomplishes.
If a torn read occurs, the CAS will fail and will atomically re-load &old_head.
Re: [PATCH v3 6/8] stack: add C11 atomic implementation
Quoted 27 lines
Quoted 24 lines
Subject: RE: [PATCH v3 6/8] stack: add C11 atomic implementation
[snip]
Quoted 11 lines
Quoted 11 lines
lib/librte_stack/rte_stack_c11_mem.h new file +11
-0+static __rte_always_inline void +__rte_stack_lf_push (struct rte_stack_lf_list *list, +struct rte_stack_lf_elem *first, +struct rte_stack_lf_elem *last, +unsigned int num) +{ +#ifndef RTE_ARCH_X86_64 +RTE_SET_USED (first); +RTE_SET_USED (last); +RTE_SET_USED (list); +RTE_SET_USED (num);Quoted 8 lines
lib/librte_stack/rte_stack_c11_mem.h line 48
Quoted 5 lines
lib/librte_stack/rte_stack_c11_mem.h new file +5
-0+#else +struct rte_stack_lf_head old_head; +int success; + + old_head = list->head;This can be a torn read (same as you have mentioned in
__rte_stack_lf_pop). I suggest we use acquire thread fence here as well (please see the comments in__rte_stack_lf_pop).Quoted 2 lines
Agreed. I'll add the acquire fence.
Quoted 7 lines
On second thought, an acquire fence isn't necessary. The acquire fence in
__rte_stack_lf_pop()ensures thelist->headis ordered before the list element reads. That isn't necessary here; we need to ensure that thelast->nextwrite occurs (and is observed) before thelist->headwrite, which the CAS's RELEASE success memorder accomplishes.If a torn read occurs, the CAS will fail and will atomically re-load
&old_head.
Following is my understanding:
The general guideline is there should be a load-acquire for every store-release. In both xxx_lf_pop and xxx_lf_push, the head is store-released, hence the load of the head should be load-acquire.
Quoted 1 line
From the code (for ex: in function
_xxx_lf_push), you can notice that there is dependency from 'old_headtonew_headtolist->head(in compare_exchange)'. When such a dependency exists, if the memory orderings have to be avoided, one needs to use__ATOMIC_CONSUME. Currently, the compilers will use a stronger memory order (which is__ATOMIC_ACQUIRE) as__ATOMIC_CONSUMEis not well defined. Please refer to [1] and [2] for more info.
IMO, since, for 128b, we do not have a pure load-acquire, I suggest we use thread_fence with acquire semantics. It is a heavier barrier, but I think it is a safer code which will adhere to C11 memory model.
[1] https://preshing.com/20140709/the-purpose-of-memory_order_consume-in-cpp11/
[2] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0750r1.html
Re: [PATCH v3 6/8] stack: add C11 atomic implementation
Quoted 62 lines
-----Original Message-----
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Monday, April 1, 2019 2:07 PM
To: Eads, Gage <[email protected]>; '[email protected]' <[email protected]>
Cc: '[email protected]' <[email protected]>; '[email protected]' <[email protected]>; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH v3 6/8] stack: add C11 atomic implementationQuoted 27 lines
Quoted 24 lines
Subject: RE: [PATCH v3 6/8] stack: add C11 atomic implementation
[snip]
Quoted 11 lines
Quoted 11 lines
lib/librte_stack/rte_stack_c11_mem.h new file +11
-0+static __rte_always_inline void __rte_stack_lf_push(struct +rte_stack_lf_list *list, +struct rte_stack_lf_elem *first, +struct rte_stack_lf_elem *last, +unsigned int num) +{ +#ifndef RTE_ARCH_X86_64 +RTE_SET_USED (first); +RTE_SET_USED (last); +RTE_SET_USED (list); +RTE_SET_USED (num);Quoted 8 lines
lib/librte_stack/rte_stack_c11_mem.h line 48
Quoted 5 lines
lib/librte_stack/rte_stack_c11_mem.h new file +5
-0+#else +struct rte_stack_lf_head old_head; +int success; + + old_head = list->head;This can be a torn read (same as you have mentioned in
__rte_stack_lf_pop). I suggest we use acquire thread fence here as well (please see the comments in__rte_stack_lf_pop).Quoted 2 lines
Agreed. I'll add the acquire fence.
Quoted 8 lines
On second thought, an acquire fence isn't necessary. The acquire fence in
__rte_stack_lf_pop()ensures thelist->headis ordered before the list element reads. That isn't necessary here; we need to ensure that thelast->nextwrite occurs (and is observed) before thelist->headwrite, which the CAS's RELEASE success memorder accomplishes.If a torn read occurs, the CAS will fail and will atomically re-load
&old_head.Following is my understanding:
The general guideline is there should be a load-acquire for every store- release. In bothxxx_lf_popandxxx_lf_push, the head is store-released, hence the load of the head should be load-acquire.
From the code (for ex: in function_xxx_lf_push), you can notice that there is dependency from 'old_headtonew_headtolist->head(in compare_exchange)'. When such a dependency exists, if the memory orderings have to be avoided, one needs to use__ATOMIC_CONSUME. Currently, the compilers will use a stronger memory order (which is__ATOMIC_ACQUIRE) as__ATOMIC_CONSUMEis not well defined. Please refer to [1] and [2] for more info.
Quoted 8 lines
IMO, since, for 128b, we do not have a pure load-acquire, I suggest we use
thread_fencewith acquire semantics. It is a heavier barrier, but I think it is a safer code which will adhere to C11 memory model.[1] https://preshing.com/20140709/the-purpose-of-
memory_order_consume-in-cpp11/
[2] http://www.open-
std.org/jtc1/sc22/wg21/docs/papers/2018/p0750r1.html
Thanks for those two links, they're good resources.
I agree with your understanding. I admit I'm not fully convinced the synchronized-with relationship is needed between pop's list->head store and push's list->head load (or between push's list->head store and its list->head load), but it's better to err on the side of caution to ensure it's functionally correct...at least until I can manage to convince you :).
I'll send out a V6 with the acquire thread fence.
Thanks,
Gage
PATCH v3 7/8 test/stack: add lock-free stack tests
In reply to Gage EadsThis commit adds lock-free stack variants of stack_autotest (stack_lf_autotest) and stack_perf_autotest (stack_lf_perf_autotest), which differ only in that the lock-free versions pass the RTE_STACK_F_LF flag to all rte_stack_create() calls.
--- app/test/meson.build | 2 ++ app/test/test_stack.c | 41 +++++++++++++++++++++++++++-------------- app/test/test_stack_perf.c | 17 +++++++++++++++-- 3 files changed, 44 insertions(+), 16 deletions(-)
app/test/meson.build +2 -0
@@ -177,6 +177,7 @@ fast_parallel_test_names = [ 'sched_autotest', 'spinlock_autotest', 'stack_autotest', + 'stack_nb_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
@@ -242,6 +243,7 @@ perf_test_names = [ 'ring_pmd_perf_autotest', 'pmd_perf_autotest', 'stack_perf_autotest', + 'stack_nb_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
app/test/test_stack.c +27 -14
112 lines not on this page · raw message
app/test/test_stack_perf.c +15 -2
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s) }static int -test_stack_perf (void ) +__test_stack_perf (uint32_t flags) {struct lcore_pair cores;struct rte_stack *s;rte_atomic32_init (&lcore_barrier); - s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (),0 ); + s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (), flags);if (s ==NULL ) {printf ("[%s():%u] failed to create a stack\n" , __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)return 0 ; } +static int +test_stack_perf (void ) +{ +return __test_stack_perf (0 ); +} + +static int +test_lf_stack_perf (void ) +{ +return __test_stack_perf (RTE_STACK_F_LF ); +} +REGISTER_TEST_COMMAND (stack_perf_autotest, test_stack_perf); +REGISTER_TEST_COMMAND (stack_lf_perf_autotest, test_lf_stack_perf);
Signature · 1 line
2.13.6
PATCH v3 8/8 mempool/stack: add lock-free stack mempool handler
In reply to Gage EadsThis commit adds support for lock-free (linked list based) stack mempool handler.
In mempool_perf_autotest the lock-based stack outperforms the lock-free handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a standard (lock-based) stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the lock-free stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.
*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the lock-free stack's.
--- doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++ doc/guides/rel_notes/release_19_05.rst | 5 +++++ drivers/mempool/stack/rte_mempool_stack.c | 26 +++++++++++++++++++++++-- 3 files changed, 39 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
@@ -541,6 +541,16 @@ Known Issues 5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR. + Alternatively, applications can use the lock-free stack mempool handler. When + considering this handler, note that: + + - It is currently limited to the x86_64 platform, because it uses an + instruction (16-byte compare-and-swap) that is not yet available on other + platforms. + - It has worse average-case performance than the non-preemptive rte_ring, but + software caching (e.g. the mempool cache) can mitigate this by reducing the + number of stack accesses. + + rte_timer Running ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -86,6 +86,11 @@ New Features The library supports two stack implementations: standard (lock-based) and lock-free. The lock-free implementation is currently limited to x86-64 platforms. +* **Added Lock-Free Stack Mempool Handler.** + + Added a new lock-free stack handler, which uses the newly added stack + library. + Removed Items -------------
drivers/mempool/stack/rte_mempool_stack.c +24 -2
@@ -7,7 +7,7 @@#include <rte_stack.h> static int -stack_alloc (struct rte_mempool *mp) +__stack_alloc (struct rte_mempool *mp,uint32_t flags) {char name[RTE_STACK_NAMESIZE ];struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)return -rte_errno; } - s =rte_stack_create (name, mp->size, mp->socket_id,0 ); + s =rte_stack_create (name, mp->size, mp->socket_id, flags);if (s ==NULL )return -rte_errno;
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp) }static int +stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,0 ); +} + +static int +lf_stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,RTE_STACK_F_LF ); +} + +static int stack_enqueue (struct rte_mempool *mp,void *const *obj_table,unsigned int n) {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = { .get_count = stack_get_count }; +static struct rte_mempool_ops ops_lf_stack = { + .name ="lf_stack" , + .alloc = lf_stack_alloc, + .free = stack_free, + .enqueue = stack_enqueue, + .dequeue = stack_dequeue, + .get_count = stack_get_count +}; +MEMPOOL_REGISTER_OPS (ops_stack); +MEMPOOL_REGISTER_OPS (ops_lf_stack);
Signature · 1 line
2.13.6
v4 cover Add stack library and new mempool handler
In reply to Gage EadsThis patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.
The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.
The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.
Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.
This patchset is the successor to a patchset containing only the new mempool handler[2].
[1] http://mails.dpdk.org/archives/dev/2019-March/125751.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html
---
v4:
- Fix test_stack.c 32-bit build by using %zu format specifier for size_t
- Rebase onto master
v3:
- Rebase patchset onto master (test/test/ -> app/test/)
- Fix rte_stack_std_push() segfault introduced in v2
v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset
Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler
MAINTAINERS | 9 +- app/test/Makefile | 3 + app/test/meson.build | 7 + app/test/test_stack.c | 423 ++++++++++++++++++++++++ app/test/test_stack_perf.c | 356 ++++++++++++++++++++ config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/env_abstraction_layer.rst | 10 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 83 +++++ doc/guides/rel_notes/release_19_05.rst | 13 + drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 115 +++---- lib/Makefile | 2 + lib/librte_stack/Makefile | 25 ++ lib/librte_stack/meson.build | 10 + lib/librte_stack/rte_stack.c | 219 ++++++++++++ lib/librte_stack/rte_stack.h | 395 ++++++++++++++++++++++ lib/librte_stack/rte_stack_c11_mem.h | 175 ++++++++++ lib/librte_stack/rte_stack_generic.h | 151 +++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++ lib/librte_stack/rte_stack_version.map | 9 + lib/meson.build | 2 +- mk/rte.app.mk | 1 + 26 files changed, 1987 insertions(+), 72 deletions(-) create mode 100644 app/test/test_stack.c create mode 100644 app/test/test_stack_perf.c create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_c11_mem.h create mode 100644 lib/librte_stack/rte_stack_generic.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_version.map
Signature · 1 line
2.13.6
PATCH v4 1/8 stack: introduce rte stack library
The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.
The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.
--- MAINTAINERS | 6 + config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 28 ++++ doc/guides/rel_notes/release_19_05.rst | 5 + lib/Makefile | 2 + lib/librte_stack/Makefile | 23 +++ lib/librte_stack/meson.build | 8 + lib/librte_stack/rte_stack.c | 194 +++++++++++++++++++++++ lib/librte_stack/rte_stack.h | 274 +++++++++++++++++++++++++++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++++ lib/librte_stack/rte_stack_version.map | 9 ++ lib/meson.build | 2 +- mk/rte.app.mk | 1 + 16 files changed, 593 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
@@ -416,6 +416,12 @@ F: drivers/raw/skeleton_rawdev/ F: app/test/test_rawdev.c F: doc/guides/prog_guide/rawdev.rst +Stack API - EXPERIMENTAL +M: Gage Eads <[email protected]> +M: Olivier Matz <[email protected]> +F: lib/librte_stack/ +F: doc/guides/prog_guide/stack_lib.rst + Memory Pool Drivers -------------------
config/common_base +5 -0
@@ -994,3 +994,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y # Compile the eventdev application # CONFIG_RTE_APP_EVENTDEV=y + +# +# Compile librte_stack +# +CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
@@ -125,6 +125,7 @@ The public API headers are grouped by topics: [mbuf] (@ref rte_mbuf.h), [mbuf pool ops] (@ref rte_mbuf_pool_ops.h), [ring] (@ref rte_ring.h), + [stack] (@ref rte_stack.h), [tailq] (@ref rte_tailq.h), [bitmap] (@ref rte_bitmap.h)
doc/api/doxy-api.conf.in +1 -0
@@ -55,6 +55,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/librte_ring \ @TOPDIR@/lib/librte_sched \ @TOPDIR@/lib/librte_security \ + @TOPDIR@/lib/librte_stack \ @TOPDIR@/lib/librte_table \ @TOPDIR@/lib/librte_telemetry \ @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
@@ -55,6 +55,7 @@ Programmer's Guide metrics_lib bpf_lib ipsec_lib + stack_lib source_org dev_kit_build_system dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +28 -0
@@ -0,0 +1,28 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +Stack Library +============= + +DPDK's stack library provides an API for configuration and use of a bounded +stack of pointers. + +The stack library provides the following basic operations: + +* Create a uniquely named stack of a user-specified size and using a + user-specified socket. + +* Push and pop a burst of one or more stack objects (pointers). These function + are multi-threading safe. + +* Free a previously created stack. + +* Lookup a pointer to a stack by its name. + +* Query a stack's current depth and number of free entries. + +Implementation +~~~~~~~~~~~~~~ + +The stack consists of a contiguous array of pointers, a current index, and a +spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -105,6 +105,11 @@ New Features Improved testpmd application performance on ARM platform. For ``macswap`` forwarding mode, NEON intrinsics were used to do swap to save CPU cycles. +* **Added Stack API.** + + Added a new stack API for configuration and use of a bounded stack of + pointers. The API provides MT-safe push and pop operations that can operate + on one or more pointers per operation. Removed Items -------------
lib/Makefile +2 -0
@@ -109,6 +109,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev +DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack +DEPDIRS-librte_stack := librte_eal ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +23 -0
@@ -0,0 +1,23 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_stack.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +CFLAGS += -DALLOW_EXPERIMENTAL_API +LDLIBS += -lrte_eal + +EXPORT_MAP := rte_stack_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c + +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h + +include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
@@ -0,0 +1,8 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +allow_experimental_apis = true + +version = 1 +sources = files('rte_stack.c') +headers = files('rte_stack.h')
lib/librte_stack/rte_stack.c new file +194 -0
194 lines not on this page · raw message
lib/librte_stack/rte_stack.h new file +274 -0
274 lines not on this page · raw message
lib/librte_stack/rte_stack_pvt.h new file +34 -0
@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#ifndef _RTE_STACK_PVT_H_ +#define _RTE_STACK_PVT_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <rte_log.h> + +extern int stack_logtype; + +#define STACK_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \ + __func__, ##args) + +#define STACK_LOG_ERR(fmt, args...) \ + STACK_LOG(ERR, fmt, ## args) + +#define STACK_LOG_WARN(fmt, args...) \ + STACK_LOG(WARNING, fmt, ## args) + +#define STACK_LOG_INFO(fmt, args...) \ + STACK_LOG(INFO, fmt, ## args) + + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_version.map new file +9 -0
@@ -0,0 +1,9 @@ +EXPERIMENTAL { + global: + + rte_stack_create; + rte_stack_free; + rte_stack_lookup; + + local: *; +};
lib/meson.build +1 -1
@@ -22,7 +22,7 @@ libraries = [ 'gro', 'gso', 'ip_frag', 'jobstats', 'kni', 'latencystats', 'lpm', 'member', 'power', 'pdump', 'rawdev', - 'reorder', 'sched', 'security', 'vhost', + 'reorder', 'sched', 'security', 'stack', 'vhost', #ipsec lib depends on crypto and security 'ipsec', # add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY) += -lrte_security _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += -lrte_compressdev _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += -lrte_eventdev _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV) += -lrte_rawdev +_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER) += -lrte_timer _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL) += -lrte_mempool _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
Signature · 1 line
2.13.6
PATCH v4 2/8 mempool/stack: convert mempool to use rte stack
In reply to Gage EadsThe new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.
--- MAINTAINERS | 2 +- drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++---------------------- 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
@@ -293,7 +293,6 @@ M: Andrew Rybchenko <[email protected]> F: lib/librte_mempool/ F: drivers/mempool/Makefile F: drivers/mempool/ring/ -F: drivers/mempool/stack/ F: doc/guides/prog_guide/mempool_lib.rst F: app/test/test_mempool* F: app/test/test_func_reentrancy.c
@@ -421,6 +420,7 @@ M: Gage Eads <[email protected]> M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst +F: drivers/mempool/stack/ Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a CFLAGS += -O3 CFLAGS += $(WERROR_FLAGS) +CFLAGS += -DALLOW_EXPERIMENTAL_API # Headers CFLAGS += -I$(RTE_SDK)/lib/librte_mempool -LDLIBS += -lrte_eal -lrte_mempool -lrte_ring +LDLIBS += -lrte_eal -lrte_mempool -lrte_stack EXPORT_MAP := rte_mempool_stack_version.map
drivers/mempool/stack/meson.build +5 -1
@@ -1,4 +1,8 @@ # SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation +# Copyright(c) 2017-2019 Intel Corporation + +allow_experimental_apis = true sources = files('rte_mempool_stack.c') + +deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68
133 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v4 3/8 test/stack: add stack test
In reply to Gage Eadsstack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.
--- MAINTAINERS | 1 + app/test/Makefile | 2 + app/test/meson.build | 3 + app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 416 insertions(+) create mode 100644 app/test/test_stack.c
MAINTAINERS +1 -0
@@ -421,6 +421,7 @@ M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst F: drivers/mempool/stack/ +F: test/test/*stack* Memory Pool Drivers
app/test/Makefile +2 -0
@@ -90,6 +90,8 @@ endif SRCS-y += test_rwlock.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c + SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
app/test/meson.build +3 -0
@@ -95,6 +95,7 @@ test_sources = files('commands.c', 'test_sched.c', 'test_service_cores.c', 'test_spinlock.c', + 'test_stack.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -133,6 +134,7 @@ test_deps = ['acl', 'port', 'reorder', 'ring', + 'stack', 'timer' ]
@@ -174,6 +176,7 @@ fast_parallel_test_names = [ 'rwlock_autotest', 'sched_autotest', 'spinlock_autotest', + 'stack_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
app/test/test_stack.c new file +410 -0
410 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v4 4/8 test/stack: add stack perf test
In reply to Gage Eadsstack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects
It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores
--- app/test/Makefile | 1 + app/test/meson.build | 2 + app/test/test_stack_perf.c | 343 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 346 insertions(+) create mode 100644 app/test/test_stack_perf.c
app/test/Makefile +1 -0
@@ -91,6 +91,7 @@ endif SRCS-y += test_rwlock.c SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
app/test/meson.build +2 -0
@@ -96,6 +96,7 @@ test_sources = files('commands.c', 'test_service_cores.c', 'test_spinlock.c', 'test_stack.c', + 'test_stack_perf.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -241,6 +242,7 @@ perf_test_names = [ 'distributor_perf_autotest', 'ring_pmd_perf_autotest', 'pmd_perf_autotest', + 'stack_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
app/test/test_stack_perf.c new file +343 -0
343 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v4 5/8 stack: add lock-free stack implementation
In reply to Gage EadsThis commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.
The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.
The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.
The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.
This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.
The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.
--- doc/guides/prog_guide/stack_lib.rst | 61 ++++++++++++- doc/guides/rel_notes/release_19_05.rst | 3 + lib/librte_stack/Makefile | 3 +- lib/librte_stack/meson.build | 3 +- lib/librte_stack/rte_stack.c | 41 +++++++-- lib/librte_stack/rte_stack.h | 127 +++++++++++++++++++++++++-- lib/librte_stack/rte_stack_generic.h | 151 +++++++++++++++++++++++++++++++++ 7 files changed, 371 insertions(+), 18 deletions(-) create mode 100644 lib/librte_stack/rte_stack_generic.h
doc/guides/prog_guide/stack_lib.rst +58 -3
@@ -10,7 +10,8 @@ stack of pointers. The stack library provides the following basic operations: * Create a uniquely named stack of a user-specified size and using a - user-specified socket. + user-specified socket, with either standard (lock-based) or lock-free + behavior. * Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe.
@@ -24,5 +25,59 @@ The stack library provides the following basic operations: Implementation ~~~~~~~~~~~~~~ -The stack consists of a contiguous array of pointers, a current index, and a -spinlock. Accesses to the stack are made multi-thread safe by the spinlock. +The library supports two types of stacks: standard (lock-based) and lock-free. +Both types use the same set of interfaces, but their implementations differ. + +Lock-based Stack +---------------- + +The lock-based stack consists of a contiguous array of pointers, a current +index, and a spinlock. Accesses to the stack are made multi-thread safe by the +spinlock. + +Lock-free Stack +------------------ + +The lock-free stack consists of a linked list of elements, each containing a +data pointer and a next pointer, and an atomic stack depth counter. The +lock-free property means that multiple threads can push and pop simultaneously, +and one thread being preempted/delayed in a push or pop operation will not +impede the forward progress of any other thread. + +The lock-free push operation enqueues a linked list of pointers by pointing the +list's tail to the current stack head, and using a CAS to swing the stack head +pointer to the head of the list. The operation retries if it is unsuccessful +(i.e. the list changed between reading the head and modifying it), else it +adjusts the stack length and returns. + +The lock-free pop operation first reserves one or more list elements by +adjusting the stack length, to ensure the dequeue operation will succeed +without blocking. It then dequeues pointers by walking the list -- starting +from the head -- then swinging the head pointer (using a CAS as well). While +walking the list, the data pointers are recorded in an object table. + +The linked list elements themselves are maintained in a lock-free LIFO, and are +allocated before stack pushes and freed after stack pops. Since the stack has a +fixed maximum depth, these elements do not need to be dynamically created. + +The lock-free behavior is selected by passing the *RTE_STACK_F_LF* flag to +rte_stack_create(). + +Preventing the ABA Problem +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To prevent the ABA problem, this algorithm stack uses a 128-bit +compare-and-swap instruction to atomically update both the stack top pointer +and a modification counter. The ABA problem can occur without a modification +counter if, for example: + +1. Thread A reads head pointer X and stores the pointed-to list element. +2. Other threads modify the list such that the head pointer is once again X, + but its pointed-to data is different than what thread A read. +3. Thread A changes the head pointer with a compare-and-swap and succeeds. + +In this case thread A would not detect that the list had changed, and would +both pop stale data and incorrect change the head pointer. By adding a +modification counter that is updated on every push and pop as part of the +compare-and-swap, the algorithm can detect when the list changes even if the +head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
@@ -111,6 +111,9 @@ New Features pointers. The API provides MT-safe push and pop operations that can operate on one or more pointers per operation. + The library supports two stack implementations: standard (lock-based) and lock-free. + The lock-free implementation is currently limited to x86-64 platforms. + Removed Items -------------
lib/librte_stack/Makefile +2 -1
@@ -18,6 +18,7 @@ LIBABIVER := 1 SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c # install includes -SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h +SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ + rte_stack_generic.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
@@ -5,4 +5,5 @@ allow_experimental_apis = true version = 1 sources = files('rte_stack.c') -headers = files('rte_stack.h') +headers = files('rte_stack.h', + 'rte_stack_generic.h')
lib/librte_stack/rte_stack.c +33 -8
76 lines not on this page · raw message
lib/librte_stack/rte_stack.h +122 -5
166 lines not on this page · raw message
lib/librte_stack/rte_stack_generic.h new file +151 -0
151 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v4 6/8 stack: add C11 atomic implementation
In reply to Gage EadsThis commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.
--- lib/librte_stack/Makefile | 3 +- lib/librte_stack/meson.build | 3 +- lib/librte_stack/rte_stack.h | 4 + lib/librte_stack/rte_stack_c11_mem.h | 175 +++++++++++++++++++++++++++++++++++ 4 files changed, 183 insertions(+), 2 deletions(-) create mode 100644 lib/librte_stack/rte_stack_c11_mem.h
lib/librte_stack/Makefile +2 -1
@@ -19,6 +19,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ - rte_stack_generic.h + rte_stack_generic.h \ + rte_stack_c11_mem.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
@@ -6,4 +6,5 @@ allow_experimental_apis = true version = 1 sources = files('rte_stack.c') headers = files('rte_stack.h', - 'rte_stack_generic.h') + 'rte_stack_generic.h', + 'rte_stack_c11_mem.h')
lib/librte_stack/rte_stack.h +4 -0
@@ -91,7 +91,11 @@ struct rte_stack {*/ #define RTE_STACK_F_LF 0x0001 +#ifdef RTE_USE_C11_MEM_MODEL +#include "rte_stack_c11_mem.h" +#else #include "rte_stack_generic.h" +#endif /** * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_c11_mem.h new file +175 -0
175 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v4 7/8 test/stack: add lock-free stack tests
In reply to Gage EadsThis commit adds lock-free stack variants of stack_autotest (stack_lf_autotest) and stack_perf_autotest (stack_lf_perf_autotest), which differ only in that the lock-free versions pass the RTE_STACK_F_LF flag to all rte_stack_create() calls.
--- app/test/meson.build | 2 ++ app/test/test_stack.c | 41 +++++++++++++++++++++++++++-------------- app/test/test_stack_perf.c | 17 +++++++++++++++-- 3 files changed, 44 insertions(+), 16 deletions(-)
app/test/meson.build +2 -0
@@ -178,6 +178,7 @@ fast_parallel_test_names = [ 'sched_autotest', 'spinlock_autotest', 'stack_autotest', + 'stack_nb_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
@@ -243,6 +244,7 @@ perf_test_names = [ 'ring_pmd_perf_autotest', 'pmd_perf_autotest', 'stack_perf_autotest', + 'stack_nb_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
app/test/test_stack.c +27 -14
112 lines not on this page · raw message
app/test/test_stack_perf.c +15 -2
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s) }static int -test_stack_perf (void ) +__test_stack_perf (uint32_t flags) {struct lcore_pair cores;struct rte_stack *s;rte_atomic32_init (&lcore_barrier); - s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (),0 ); + s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (), flags);if (s ==NULL ) {printf ("[%s():%u] failed to create a stack\n" , __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)return 0 ; } +static int +test_stack_perf (void ) +{ +return __test_stack_perf (0 ); +} + +static int +test_lf_stack_perf (void ) +{ +return __test_stack_perf (RTE_STACK_F_LF ); +} +REGISTER_TEST_COMMAND (stack_perf_autotest, test_stack_perf); +REGISTER_TEST_COMMAND (stack_lf_perf_autotest, test_lf_stack_perf);
Signature · 1 line
2.13.6
PATCH v4 8/8 mempool/stack: add lock-free stack mempool handler
In reply to Gage EadsThis commit adds support for lock-free (linked list based) stack mempool handler.
In mempool_perf_autotest the lock-based stack outperforms the lock-free handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a standard (lock-based) stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the lock-free stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.
*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the lock-free stack's.
--- doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++ doc/guides/rel_notes/release_19_05.rst | 5 +++++ drivers/mempool/stack/rte_mempool_stack.c | 26 +++++++++++++++++++++++-- 3 files changed, 39 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
@@ -563,6 +563,16 @@ Known Issues 5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR. + Alternatively, applications can use the lock-free stack mempool handler. When + considering this handler, note that: + + - It is currently limited to the x86_64 platform, because it uses an + instruction (16-byte compare-and-swap) that is not yet available on other + platforms. + - It has worse average-case performance than the non-preemptive rte_ring, but + software caching (e.g. the mempool cache) can mitigate this by reducing the + number of stack accesses. + + rte_timer Running ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -114,6 +114,11 @@ New Features The library supports two stack implementations: standard (lock-based) and lock-free. The lock-free implementation is currently limited to x86-64 platforms. +* **Added Lock-Free Stack Mempool Handler.** + + Added a new lock-free stack handler, which uses the newly added stack + library. + Removed Items -------------
drivers/mempool/stack/rte_mempool_stack.c +24 -2
@@ -7,7 +7,7 @@#include <rte_stack.h> static int -stack_alloc (struct rte_mempool *mp) +__stack_alloc (struct rte_mempool *mp,uint32_t flags) {char name[RTE_STACK_NAMESIZE ];struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)return -rte_errno; } - s =rte_stack_create (name, mp->size, mp->socket_id,0 ); + s =rte_stack_create (name, mp->size, mp->socket_id, flags);if (s ==NULL )return -rte_errno;
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp) }static int +stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,0 ); +} + +static int +lf_stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,RTE_STACK_F_LF ); +} + +static int stack_enqueue (struct rte_mempool *mp,void *const *obj_table,unsigned int n) {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = { .get_count = stack_get_count }; +static struct rte_mempool_ops ops_lf_stack = { + .name ="lf_stack" , + .alloc = lf_stack_alloc, + .free = stack_free, + .enqueue = stack_enqueue, + .dequeue = stack_dequeue, + .get_count = stack_get_count +}; +MEMPOOL_REGISTER_OPS (ops_stack); +MEMPOOL_REGISTER_OPS (ops_lf_stack);
Signature · 1 line
2.13.6
v5 cover Add stack library and new mempool handler
In reply to Gage EadsThis patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.
The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.
The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.
Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.
This patchset is the successor to a patchset containing only the new mempool handler[2].
[1] http://mails.dpdk.org/archives/dev/2019-March/125751.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html
---
v5:
- Add comment to explain padding in *_get_memsize() functions
- Prefix internal functions with '__'
- Use RTE_ASSERT for performance critical run-time checks
- Don't use __atomic_load in the C11 pop_elems function, and put an acquire thread fence at the start of the 2nd do-while loop
- Change pop_elems 128b CAS success memorder to RELEASE and failure memorder to RELAXED
- Change compile-time assertion to run for all 64-bit architectures
- Reorganize the code with standard and lock-free .c and .h files
v4:
- Fix 32-bit build error in test_stack.c by using %zu format specifier for size_t
- Rebase onto master
v3:
- Rebase patchset onto master (test/test/ -> app/test/)
- Fix rte_stack_std_push() segfault introduced in v2
v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset
Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler
MAINTAINERS | 9 +- app/test/Makefile | 3 + app/test/meson.build | 7 + app/test/test_stack.c | 423 ++++++++++++++++++++++++ app/test/test_stack_perf.c | 356 ++++++++++++++++++++ config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/env_abstraction_layer.rst | 10 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 83 +++++ doc/guides/rel_notes/release_19_05.rst | 13 + drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 115 +++---- lib/Makefile | 2 + lib/librte_stack/Makefile | 29 ++ lib/librte_stack/meson.build | 12 + lib/librte_stack/rte_stack.c | 196 +++++++++++ lib/librte_stack/rte_stack.h | 259 +++++++++++++++ lib/librte_stack/rte_stack_lf.c | 31 ++ lib/librte_stack/rte_stack_lf.h | 106 ++++++ lib/librte_stack/rte_stack_lf_c11.h | 169 ++++++++++ lib/librte_stack/rte_stack_lf_generic.h | 151 +++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++ lib/librte_stack/rte_stack_std.c | 26 ++ lib/librte_stack/rte_stack_std.h | 119 +++++++ lib/librte_stack/rte_stack_version.map | 9 + lib/meson.build | 2 +- mk/rte.app.mk | 1 + 30 files changed, 2110 insertions(+), 72 deletions(-) create mode 100644 app/test/test_stack.c create mode 100644 app/test/test_stack_perf.c create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_lf.c create mode 100644 lib/librte_stack/rte_stack_lf.h create mode 100644 lib/librte_stack/rte_stack_lf_c11.h create mode 100644 lib/librte_stack/rte_stack_lf_generic.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_std.c create mode 100644 lib/librte_stack/rte_stack_std.h create mode 100644 lib/librte_stack/rte_stack_version.map
Signature · 1 line
2.13.6
PATCH v5 1/8 stack: introduce rte stack library
The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.
The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.
--- MAINTAINERS | 6 + config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 28 +++++ doc/guides/rel_notes/release_19_05.rst | 5 + lib/Makefile | 2 + lib/librte_stack/Makefile | 25 ++++ lib/librte_stack/meson.build | 8 ++ lib/librte_stack/rte_stack.c | 182 +++++++++++++++++++++++++++++ lib/librte_stack/rte_stack.h | 207 +++++++++++++++++++++++++++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++++++ lib/librte_stack/rte_stack_std.c | 26 +++++ lib/librte_stack/rte_stack_std.h | 119 +++++++++++++++++++ lib/librte_stack/rte_stack_version.map | 9 ++ lib/meson.build | 2 +- mk/rte.app.mk | 1 + 18 files changed, 661 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_std.c create mode 100644 lib/librte_stack/rte_stack_std.h create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
@@ -416,6 +416,12 @@ F: drivers/raw/skeleton_rawdev/ F: app/test/test_rawdev.c F: doc/guides/prog_guide/rawdev.rst +Stack API - EXPERIMENTAL +M: Gage Eads <[email protected]> +M: Olivier Matz <[email protected]> +F: lib/librte_stack/ +F: doc/guides/prog_guide/stack_lib.rst + Memory Pool Drivers -------------------
config/common_base +5 -0
@@ -994,3 +994,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y # Compile the eventdev application # CONFIG_RTE_APP_EVENTDEV=y + +# +# Compile librte_stack +# +CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
@@ -125,6 +125,7 @@ The public API headers are grouped by topics: [mbuf] (@ref rte_mbuf.h), [mbuf pool ops] (@ref rte_mbuf_pool_ops.h), [ring] (@ref rte_ring.h), + [stack] (@ref rte_stack.h), [tailq] (@ref rte_tailq.h), [bitmap] (@ref rte_bitmap.h)
doc/api/doxy-api.conf.in +1 -0
@@ -55,6 +55,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/librte_ring \ @TOPDIR@/lib/librte_sched \ @TOPDIR@/lib/librte_security \ + @TOPDIR@/lib/librte_stack \ @TOPDIR@/lib/librte_table \ @TOPDIR@/lib/librte_telemetry \ @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
@@ -55,6 +55,7 @@ Programmer's Guide metrics_lib bpf_lib ipsec_lib + stack_lib source_org dev_kit_build_system dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +28 -0
@@ -0,0 +1,28 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +Stack Library +============= + +DPDK's stack library provides an API for configuration and use of a bounded +stack of pointers. + +The stack library provides the following basic operations: + +* Create a uniquely named stack of a user-specified size and using a + user-specified socket. + +* Push and pop a burst of one or more stack objects (pointers). These function + are multi-threading safe. + +* Free a previously created stack. + +* Lookup a pointer to a stack by its name. + +* Query a stack's current depth and number of free entries. + +Implementation +~~~~~~~~~~~~~~ + +The stack consists of a contiguous array of pointers, a current index, and a +spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -121,6 +121,11 @@ New Features Improved testpmd application performance on ARM platform. For ``macswap`` forwarding mode, NEON intrinsics were used to do swap to save CPU cycles. +* **Added Stack API.** + + Added a new stack API for configuration and use of a bounded stack of + pointers. The API provides MT-safe push and pop operations that can operate + on one or more pointers per operation. Removed Items -------------
lib/Makefile +2 -0
@@ -109,6 +109,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev +DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack +DEPDIRS-librte_stack := librte_eal ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +25 -0
@@ -0,0 +1,25 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_stack.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +CFLAGS += -DALLOW_EXPERIMENTAL_API +LDLIBS += -lrte_eal + +EXPORT_MAP := rte_stack_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \ + rte_stack_std.c + +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ + rte_stack_std.h + +include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
@@ -0,0 +1,8 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +allow_experimental_apis = true + +version = 1 +sources = files('rte_stack.c', 'rte_stack_std.c') +headers = files('rte_stack.h', 'rte_stack_std.h')
lib/librte_stack/rte_stack.c new file +182 -0
182 lines not on this page · raw message
lib/librte_stack/rte_stack.h new file +207 -0
207 lines not on this page · raw message
lib/librte_stack/rte_stack_pvt.h new file +34 -0
@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#ifndef _RTE_STACK_PVT_H_ +#define _RTE_STACK_PVT_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <rte_log.h> + +extern int stack_logtype; + +#define STACK_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \ + __func__, ##args) + +#define STACK_LOG_ERR(fmt, args...) \ + STACK_LOG(ERR, fmt, ## args) + +#define STACK_LOG_WARN(fmt, args...) \ + STACK_LOG(WARNING, fmt, ## args) + +#define STACK_LOG_INFO(fmt, args...) \ + STACK_LOG(INFO, fmt, ## args) + + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_std.c new file +26 -0
@@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#include "rte_stack.h" + +void +rte_stack_std_init (struct rte_stack *s) +{ +rte_spinlock_init (&s->stack_std.lock); +} + +ssize_t +rte_stack_std_get_memsize (unsigned int count) +{ +ssize_t sz =sizeof (struct rte_stack ); + + sz +=RTE_CACHE_LINE_ROUNDUP (count *sizeof (void *)); + +/* Add padding to avoid false sharing conflicts caused by +* next-line hardware prefetchers. +*/ + sz +=2 *RTE_CACHE_LINE_SIZE ; + +return sz; +}
lib/librte_stack/rte_stack_std.h new file +119 -0
119 lines not on this page · raw message
lib/librte_stack/rte_stack_version.map new file +9 -0
@@ -0,0 +1,9 @@ +EXPERIMENTAL { + global: + + rte_stack_create; + rte_stack_free; + rte_stack_lookup; + + local: *; +};
lib/meson.build +1 -1
@@ -22,7 +22,7 @@ libraries = [ 'gro', 'gso', 'ip_frag', 'jobstats', 'kni', 'latencystats', 'lpm', 'member', 'power', 'pdump', 'rawdev', - 'reorder', 'sched', 'security', 'vhost', + 'reorder', 'sched', 'security', 'stack', 'vhost', #ipsec lib depends on crypto and security 'ipsec', # add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY) += -lrte_security _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += -lrte_compressdev _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += -lrte_eventdev _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV) += -lrte_rawdev +_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER) += -lrte_timer _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL) += -lrte_mempool _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
Signature · 1 line
2.13.6
PATCH v5 2/8 mempool/stack: convert mempool to use rte stack
In reply to Gage EadsThe new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.
--- MAINTAINERS | 2 +- drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++---------------------- 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
@@ -293,7 +293,6 @@ M: Andrew Rybchenko <[email protected]> F: lib/librte_mempool/ F: drivers/mempool/Makefile F: drivers/mempool/ring/ -F: drivers/mempool/stack/ F: doc/guides/prog_guide/mempool_lib.rst F: app/test/test_mempool* F: app/test/test_func_reentrancy.c
@@ -421,6 +420,7 @@ M: Gage Eads <[email protected]> M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst +F: drivers/mempool/stack/ Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a CFLAGS += -O3 CFLAGS += $(WERROR_FLAGS) +CFLAGS += -DALLOW_EXPERIMENTAL_API # Headers CFLAGS += -I$(RTE_SDK)/lib/librte_mempool -LDLIBS += -lrte_eal -lrte_mempool -lrte_ring +LDLIBS += -lrte_eal -lrte_mempool -lrte_stack EXPORT_MAP := rte_mempool_stack_version.map
drivers/mempool/stack/meson.build +5 -1
@@ -1,4 +1,8 @@ # SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation +# Copyright(c) 2017-2019 Intel Corporation + +allow_experimental_apis = true sources = files('rte_mempool_stack.c') + +deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68
133 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v5 3/8 test/stack: add stack test
In reply to Gage Eadsstack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.
--- MAINTAINERS | 1 + app/test/Makefile | 2 + app/test/meson.build | 3 + app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 416 insertions(+) create mode 100644 app/test/test_stack.c
MAINTAINERS +1 -0
@@ -421,6 +421,7 @@ M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst F: drivers/mempool/stack/ +F: test/test/*stack* Memory Pool Drivers
app/test/Makefile +2 -0
@@ -90,6 +90,8 @@ endif SRCS-y += test_rwlock.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c + SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
app/test/meson.build +3 -0
@@ -95,6 +95,7 @@ test_sources = files('commands.c', 'test_sched.c', 'test_service_cores.c', 'test_spinlock.c', + 'test_stack.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -133,6 +134,7 @@ test_deps = ['acl', 'port', 'reorder', 'ring', + 'stack', 'timer' ]
@@ -174,6 +176,7 @@ fast_parallel_test_names = [ 'rwlock_autotest', 'sched_autotest', 'spinlock_autotest', + 'stack_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
app/test/test_stack.c new file +410 -0
410 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v5 4/8 test/stack: add stack perf test
In reply to Gage Eadsstack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects
It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores
--- app/test/Makefile | 1 + app/test/meson.build | 2 + app/test/test_stack_perf.c | 343 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 346 insertions(+) create mode 100644 app/test/test_stack_perf.c
app/test/Makefile +1 -0
@@ -91,6 +91,7 @@ endif SRCS-y += test_rwlock.c SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
app/test/meson.build +2 -0
@@ -96,6 +96,7 @@ test_sources = files('commands.c', 'test_service_cores.c', 'test_spinlock.c', 'test_stack.c', + 'test_stack_perf.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -241,6 +242,7 @@ perf_test_names = [ 'distributor_perf_autotest', 'ring_pmd_perf_autotest', 'pmd_perf_autotest', + 'stack_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
app/test/test_stack_perf.c new file +343 -0
343 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v5 5/8 stack: add lock-free stack implementation
In reply to Gage EadsThis commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.
The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.
The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.
The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.
This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.
The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.
--- doc/guides/prog_guide/stack_lib.rst | 61 ++++++++++++- doc/guides/rel_notes/release_19_05.rst | 3 + lib/librte_stack/Makefile | 7 +- lib/librte_stack/meson.build | 7 +- lib/librte_stack/rte_stack.c | 28 ++++-- lib/librte_stack/rte_stack.h | 62 +++++++++++-- lib/librte_stack/rte_stack_lf.c | 31 +++++++ lib/librte_stack/rte_stack_lf.h | 102 +++++++++++++++++++++ lib/librte_stack/rte_stack_lf_generic.h | 151 ++++++++++++++++++++++++++++++++ 9 files changed, 433 insertions(+), 19 deletions(-) create mode 100644 lib/librte_stack/rte_stack_lf.c create mode 100644 lib/librte_stack/rte_stack_lf.h create mode 100644 lib/librte_stack/rte_stack_lf_generic.h
doc/guides/prog_guide/stack_lib.rst +58 -3
@@ -10,7 +10,8 @@ stack of pointers. The stack library provides the following basic operations: * Create a uniquely named stack of a user-specified size and using a - user-specified socket. + user-specified socket, with either standard (lock-based) or lock-free + behavior. * Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe.
@@ -24,5 +25,59 @@ The stack library provides the following basic operations: Implementation ~~~~~~~~~~~~~~ -The stack consists of a contiguous array of pointers, a current index, and a -spinlock. Accesses to the stack are made multi-thread safe by the spinlock. +The library supports two types of stacks: standard (lock-based) and lock-free. +Both types use the same set of interfaces, but their implementations differ. + +Lock-based Stack +---------------- + +The lock-based stack consists of a contiguous array of pointers, a current +index, and a spinlock. Accesses to the stack are made multi-thread safe by the +spinlock. + +Lock-free Stack +------------------ + +The lock-free stack consists of a linked list of elements, each containing a +data pointer and a next pointer, and an atomic stack depth counter. The +lock-free property means that multiple threads can push and pop simultaneously, +and one thread being preempted/delayed in a push or pop operation will not +impede the forward progress of any other thread. + +The lock-free push operation enqueues a linked list of pointers by pointing the +list's tail to the current stack head, and using a CAS to swing the stack head +pointer to the head of the list. The operation retries if it is unsuccessful +(i.e. the list changed between reading the head and modifying it), else it +adjusts the stack length and returns. + +The lock-free pop operation first reserves one or more list elements by +adjusting the stack length, to ensure the dequeue operation will succeed +without blocking. It then dequeues pointers by walking the list -- starting +from the head -- then swinging the head pointer (using a CAS as well). While +walking the list, the data pointers are recorded in an object table. + +The linked list elements themselves are maintained in a lock-free LIFO, and are +allocated before stack pushes and freed after stack pops. Since the stack has a +fixed maximum depth, these elements do not need to be dynamically created. + +The lock-free behavior is selected by passing the *RTE_STACK_F_LF* flag to +rte_stack_create(). + +Preventing the ABA Problem +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To prevent the ABA problem, this algorithm stack uses a 128-bit +compare-and-swap instruction to atomically update both the stack top pointer +and a modification counter. The ABA problem can occur without a modification +counter if, for example: + +1. Thread A reads head pointer X and stores the pointed-to list element. +2. Other threads modify the list such that the head pointer is once again X, + but its pointed-to data is different than what thread A read. +3. Thread A changes the head pointer with a compare-and-swap and succeeds. + +In this case thread A would not detect that the list had changed, and would +both pop stale data and incorrect change the head pointer. By adding a +modification counter that is updated on every push and pop as part of the +compare-and-swap, the algorithm can detect when the list changes even if the +head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
@@ -127,6 +127,9 @@ New Features pointers. The API provides MT-safe push and pop operations that can operate on one or more pointers per operation. + The library supports two stack implementations: standard (lock-based) and lock-free. + The lock-free implementation is currently limited to x86-64 platforms. + Removed Items -------------
lib/librte_stack/Makefile +5 -2
@@ -16,10 +16,13 @@ LIBABIVER := 1 # all source are stored in SRCS-y SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \ - rte_stack_std.c + rte_stack_std.c \ + rte_stack_lf.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ - rte_stack_std.h + rte_stack_std.h \ + rte_stack_lf.h \ + rte_stack_lf_generic.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +5 -2
@@ -4,5 +4,8 @@ allow_experimental_apis = true version = 1 -sources = files('rte_stack.c', 'rte_stack_std.c') -headers = files('rte_stack.h', 'rte_stack_std.h') +sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c') +headers = files('rte_stack.h', + 'rte_stack_std.h', + 'rte_stack_lf.h', + 'rte_stack_lf_generic.h')
lib/librte_stack/rte_stack.c +21 -7
@@ -25,18 +25,25 @@ static struct rte_tailq_elem rte_stack_tailq = { };EAL_REGISTER_TAILQ (rte_stack_tailq) + static void -rte_stack_init (struct rte_stack *s) +rte_stack_init (struct rte_stack *s,unsigned int count,uint32_t flags) {memset (s,0 ,sizeof (*s)); -rte_stack_std_init (s); +if (flags &RTE_STACK_F_LF ) +rte_stack_lf_init (s, count); +else +rte_stack_std_init (s); }static ssize_t -rte_stack_get_memsize (unsigned int count) +rte_stack_get_memsize (unsigned int count,uint32_t flags) { -return rte_stack_std_get_memsize (count); +if (flags &RTE_STACK_F_LF ) +return rte_stack_lf_get_memsize (count); +else +return rte_stack_std_get_memsize (count); } struct rte_stack *
@@ -51,9 +58,16 @@ rte_stack_create(const char *name, unsigned int count, int socket_id,unsigned int sz;int ret; -RTE_SET_USED (flags); +#ifdef RTE_ARCH_64 +RTE_BUILD_BUG_ON (sizeof (struct rte_stack_lf_head ) !=16 ); +#else +if (flags &RTE_STACK_F_LF ) { +STACK_LOG_ERR ("Lock-free stack is not supported on your platform\n" ); +return NULL ; + } +#endif - sz =rte_stack_get_memsize (count); + sz =rte_stack_get_memsize (count, flags); ret =snprintf (mz_name,sizeof (mz_name),"%s%s" ,RTE_STACK_MZ_PREFIX , name);
@@ -82,7 +96,7 @@ rte_stack_create(const char *name, unsigned int count, int socket_id, s = mz->addr; -rte_stack_init (s); +rte_stack_init (s, count, flags);/* Store the name for later lookups */ ret =snprintf (s->name,sizeof (s->name),"%s" , name);
lib/librte_stack/rte_stack.h +57 -5
101 lines not on this page · raw message
lib/librte_stack/rte_stack_lf.c new file +31 -0
@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#include "rte_stack.h" + +void +rte_stack_lf_init (struct rte_stack *s,unsigned int count) +{ +struct rte_stack_lf_elem *elems = s->stack_lf.elems; +unsigned int i; + +for (i =0 ; i < count; i++) +__rte_stack_lf_push_elems (&s->stack_lf.free, + &elems[i], &elems[i],1 ); +} + +ssize_t +rte_stack_lf_get_memsize (unsigned int count) +{ +ssize_t sz =sizeof (struct rte_stack ); + + sz +=RTE_CACHE_LINE_ROUNDUP (count *sizeof (struct rte_stack_lf_elem )); + +/* Add padding to avoid false sharing conflicts caused by +* next-line hardware prefetchers. +*/ + sz +=2 *RTE_CACHE_LINE_SIZE ; + +return sz; +}
lib/librte_stack/rte_stack_lf.h new file +102 -0
102 lines not on this page · raw message
lib/librte_stack/rte_stack_lf_generic.h new file +151 -0
151 lines not on this page · raw message
Signature · 1 line
2.13.6
Re: [PATCH v5 5/8] stack: add lock-free stack implementation
Quoted 25 lines
Subject: [PATCH v5 5/8] stack: add lock-free stack implementation
This commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new
rte_stack_create()flag,RTE_STACK_F_LF.The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.
The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.
The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.
This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.
Quoted 8 lines
The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.
Signed-off-by Gage Eads <[email protected]>Reviewed-by Olivier Matz <[email protected]>---
PATCH v5 6/8 stack: add C11 atomic implementation
In reply to Gage EadsThis commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.
--- lib/librte_stack/Makefile | 3 +- lib/librte_stack/meson.build | 3 +- lib/librte_stack/rte_stack_lf.h | 4 + lib/librte_stack/rte_stack_lf_c11.h | 169 ++++++++++++++++++++++++++++++++++++ 4 files changed, 177 insertions(+), 2 deletions(-) create mode 100644 lib/librte_stack/rte_stack_lf_c11.h
lib/librte_stack/Makefile +2 -1
@@ -23,6 +23,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \ SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ rte_stack_std.h \ rte_stack_lf.h \ - rte_stack_lf_generic.h + rte_stack_lf_generic.h \ + rte_stack_lf_c11.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
@@ -8,4 +8,5 @@ sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c') headers = files('rte_stack.h', 'rte_stack_std.h', 'rte_stack_lf.h', - 'rte_stack_lf_generic.h') + 'rte_stack_lf_generic.h', + 'rte_stack_lf_c11.h')
lib/librte_stack/rte_stack_lf.h +4 -0
@@ -5,7 +5,11 @@#ifndef _RTE_STACK_LF_H_#define _RTE_STACK_LF_H_ +#ifdef RTE_USE_C11_MEM_MODEL +#include "rte_stack_lf_c11.h" +#else #include "rte_stack_lf_generic.h" +#endif /** * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_lf_c11.h new file +169 -0
169 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v5 7/8 test/stack: add lock-free stack tests
In reply to Gage EadsThis commit adds lock-free stack variants of stack_autotest (stack_lf_autotest) and stack_perf_autotest (stack_lf_perf_autotest), which differ only in that the lock-free versions pass the RTE_STACK_F_LF flag to all rte_stack_create() calls.
--- app/test/meson.build | 2 ++ app/test/test_stack.c | 41 +++++++++++++++++++++++++++-------------- app/test/test_stack_perf.c | 17 +++++++++++++++-- 3 files changed, 44 insertions(+), 16 deletions(-)
app/test/meson.build +2 -0
@@ -178,6 +178,7 @@ fast_parallel_test_names = [ 'sched_autotest', 'spinlock_autotest', 'stack_autotest', + 'stack_nb_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
@@ -243,6 +244,7 @@ perf_test_names = [ 'ring_pmd_perf_autotest', 'pmd_perf_autotest', 'stack_perf_autotest', + 'stack_nb_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
app/test/test_stack.c +27 -14
112 lines not on this page · raw message
app/test/test_stack_perf.c +15 -2
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s) }static int -test_stack_perf (void ) +__test_stack_perf (uint32_t flags) {struct lcore_pair cores;struct rte_stack *s;rte_atomic32_init (&lcore_barrier); - s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (),0 ); + s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (), flags);if (s ==NULL ) {printf ("[%s():%u] failed to create a stack\n" , __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)return 0 ; } +static int +test_stack_perf (void ) +{ +return __test_stack_perf (0 ); +} + +static int +test_lf_stack_perf (void ) +{ +return __test_stack_perf (RTE_STACK_F_LF ); +} +REGISTER_TEST_COMMAND (stack_perf_autotest, test_stack_perf); +REGISTER_TEST_COMMAND (stack_lf_perf_autotest, test_lf_stack_perf);
Signature · 1 line
2.13.6
PATCH v5 8/8 mempool/stack: add lock-free stack mempool handler
In reply to Gage EadsThis commit adds support for lock-free (linked list based) stack mempool handler.
In mempool_perf_autotest the lock-based stack outperforms the lock-free handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a standard (lock-based) stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the lock-free stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.
*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the lock-free stack's.
--- doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++ doc/guides/rel_notes/release_19_05.rst | 5 +++++ drivers/mempool/stack/rte_mempool_stack.c | 26 +++++++++++++++++++++++-- 3 files changed, 39 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
@@ -563,6 +563,16 @@ Known Issues 5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR. + Alternatively, applications can use the lock-free stack mempool handler. When + considering this handler, note that: + + - It is currently limited to the x86_64 platform, because it uses an + instruction (16-byte compare-and-swap) that is not yet available on other + platforms. + - It has worse average-case performance than the non-preemptive rte_ring, but + software caching (e.g. the mempool cache) can mitigate this by reducing the + number of stack accesses. + + rte_timer Running ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -130,6 +130,11 @@ New Features The library supports two stack implementations: standard (lock-based) and lock-free. The lock-free implementation is currently limited to x86-64 platforms. +* **Added Lock-Free Stack Mempool Handler.** + + Added a new lock-free stack handler, which uses the newly added stack + library. + Removed Items -------------
drivers/mempool/stack/rte_mempool_stack.c +24 -2
@@ -7,7 +7,7 @@#include <rte_stack.h> static int -stack_alloc (struct rte_mempool *mp) +__stack_alloc (struct rte_mempool *mp,uint32_t flags) {char name[RTE_STACK_NAMESIZE ];struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)return -rte_errno; } - s =rte_stack_create (name, mp->size, mp->socket_id,0 ); + s =rte_stack_create (name, mp->size, mp->socket_id, flags);if (s ==NULL )return -rte_errno;
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp) }static int +stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,0 ); +} + +static int +lf_stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,RTE_STACK_F_LF ); +} + +static int stack_enqueue (struct rte_mempool *mp,void *const *obj_table,unsigned int n) {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = { .get_count = stack_get_count }; +static struct rte_mempool_ops ops_lf_stack = { + .name ="lf_stack" , + .alloc = lf_stack_alloc, + .free = stack_free, + .enqueue = stack_enqueue, + .dequeue = stack_dequeue, + .get_count = stack_get_count +}; +MEMPOOL_REGISTER_OPS (ops_stack); +MEMPOOL_REGISTER_OPS (ops_lf_stack);
Signature · 1 line
2.13.6
v6 cover Add stack library and new mempool handler
In reply to Gage EadsThis patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.
The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.
The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.
Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.
This patchset is the successor to a patchset containing only the new mempool handler[2].
[1] http://mails.dpdk.org/archives/dev/2019-March/125751.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html
---
v6:
- Add load-acquire fence to the lock-free push function
- Correct generic implementation's pop_elems 128b CAS success and failure memorder to match those in the C11 implementation.
v5:
- Add comment to explain padding in *_get_memsize() functions
- Prefix internal functions with '__'
- Use RTE_ASSERT for performance critical run-time checks
- Don't use __atomic_load in the C11 pop_elems function, and put an acquire thread fence at the start of the 2nd do-while loop
- Change pop_elems 128b CAS success memorder to RELEASE and failure memorder to RELAXED
- Change compile-time assertion to run for all 64-bit architectures
- Reorganize the code with standard and lock-free .c and .h files
v4:
- Fix 32-bit build error in test_stack.c by using %zu format specifier for size_t
- Rebase onto master
v3:
- Rebase patchset onto master (test/test/ -> app/test/)
- Fix rte_stack_std_push() segfault introduced in v2
v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset
Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler
MAINTAINERS | 9 +- app/test/Makefile | 3 + app/test/meson.build | 7 + app/test/test_stack.c | 423 ++++++++++++++++++++++++ app/test/test_stack_perf.c | 356 ++++++++++++++++++++ config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/env_abstraction_layer.rst | 10 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 83 +++++ doc/guides/rel_notes/release_19_05.rst | 13 + drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 115 +++---- lib/Makefile | 2 + lib/librte_stack/Makefile | 29 ++ lib/librte_stack/meson.build | 12 + lib/librte_stack/rte_stack.c | 196 +++++++++++ lib/librte_stack/rte_stack.h | 259 +++++++++++++++ lib/librte_stack/rte_stack_lf.c | 31 ++ lib/librte_stack/rte_stack_lf.h | 106 ++++++ lib/librte_stack/rte_stack_lf_c11.h | 175 ++++++++++ lib/librte_stack/rte_stack_lf_generic.h | 164 +++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++ lib/librte_stack/rte_stack_std.c | 26 ++ lib/librte_stack/rte_stack_std.h | 119 +++++++ lib/librte_stack/rte_stack_version.map | 9 + lib/meson.build | 2 +- mk/rte.app.mk | 1 + 30 files changed, 2129 insertions(+), 72 deletions(-) create mode 100644 app/test/test_stack.c create mode 100644 app/test/test_stack_perf.c create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_lf.c create mode 100644 lib/librte_stack/rte_stack_lf.h create mode 100644 lib/librte_stack/rte_stack_lf_c11.h create mode 100644 lib/librte_stack/rte_stack_lf_generic.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_std.c create mode 100644 lib/librte_stack/rte_stack_std.h create mode 100644 lib/librte_stack/rte_stack_version.map
Signature · 1 line
2.13.6
PATCH v6 1/8 stack: introduce rte stack library
The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.
The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.
--- MAINTAINERS | 6 + config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 28 +++++ doc/guides/rel_notes/release_19_05.rst | 5 + lib/Makefile | 2 + lib/librte_stack/Makefile | 25 ++++ lib/librte_stack/meson.build | 8 ++ lib/librte_stack/rte_stack.c | 182 +++++++++++++++++++++++++++++ lib/librte_stack/rte_stack.h | 207 +++++++++++++++++++++++++++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++++++ lib/librte_stack/rte_stack_std.c | 26 +++++ lib/librte_stack/rte_stack_std.h | 119 +++++++++++++++++++ lib/librte_stack/rte_stack_version.map | 9 ++ lib/meson.build | 2 +- mk/rte.app.mk | 1 + 18 files changed, 661 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_std.c create mode 100644 lib/librte_stack/rte_stack_std.h create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
@@ -416,6 +416,12 @@ F: drivers/raw/skeleton_rawdev/ F: app/test/test_rawdev.c F: doc/guides/prog_guide/rawdev.rst +Stack API - EXPERIMENTAL +M: Gage Eads <[email protected]> +M: Olivier Matz <[email protected]> +F: lib/librte_stack/ +F: doc/guides/prog_guide/stack_lib.rst + Memory Pool Drivers -------------------
config/common_base +5 -0
@@ -994,3 +994,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y # Compile the eventdev application # CONFIG_RTE_APP_EVENTDEV=y + +# +# Compile librte_stack +# +CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
@@ -125,6 +125,7 @@ The public API headers are grouped by topics: [mbuf] (@ref rte_mbuf.h), [mbuf pool ops] (@ref rte_mbuf_pool_ops.h), [ring] (@ref rte_ring.h), + [stack] (@ref rte_stack.h), [tailq] (@ref rte_tailq.h), [bitmap] (@ref rte_bitmap.h)
doc/api/doxy-api.conf.in +1 -0
@@ -55,6 +55,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/librte_ring \ @TOPDIR@/lib/librte_sched \ @TOPDIR@/lib/librte_security \ + @TOPDIR@/lib/librte_stack \ @TOPDIR@/lib/librte_table \ @TOPDIR@/lib/librte_telemetry \ @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
@@ -55,6 +55,7 @@ Programmer's Guide metrics_lib bpf_lib ipsec_lib + stack_lib source_org dev_kit_build_system dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +28 -0
@@ -0,0 +1,28 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +Stack Library +============= + +DPDK's stack library provides an API for configuration and use of a bounded +stack of pointers. + +The stack library provides the following basic operations: + +* Create a uniquely named stack of a user-specified size and using a + user-specified socket. + +* Push and pop a burst of one or more stack objects (pointers). These function + are multi-threading safe. + +* Free a previously created stack. + +* Lookup a pointer to a stack by its name. + +* Query a stack's current depth and number of free entries. + +Implementation +~~~~~~~~~~~~~~ + +The stack consists of a contiguous array of pointers, a current index, and a +spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -121,6 +121,11 @@ New Features Improved testpmd application performance on ARM platform. For ``macswap`` forwarding mode, NEON intrinsics were used to do swap to save CPU cycles. +* **Added Stack API.** + + Added a new stack API for configuration and use of a bounded stack of + pointers. The API provides MT-safe push and pop operations that can operate + on one or more pointers per operation. Removed Items -------------
lib/Makefile +2 -0
@@ -109,6 +109,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev +DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack +DEPDIRS-librte_stack := librte_eal ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +25 -0
@@ -0,0 +1,25 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_stack.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +CFLAGS += -DALLOW_EXPERIMENTAL_API +LDLIBS += -lrte_eal + +EXPORT_MAP := rte_stack_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \ + rte_stack_std.c + +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ + rte_stack_std.h + +include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
@@ -0,0 +1,8 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +allow_experimental_apis = true + +version = 1 +sources = files('rte_stack.c', 'rte_stack_std.c') +headers = files('rte_stack.h', 'rte_stack_std.h')
lib/librte_stack/rte_stack.c new file +182 -0
182 lines not on this page · raw message
lib/librte_stack/rte_stack.h new file +207 -0
207 lines not on this page · raw message
lib/librte_stack/rte_stack_pvt.h new file +34 -0
@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#ifndef _RTE_STACK_PVT_H_ +#define _RTE_STACK_PVT_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <rte_log.h> + +extern int stack_logtype; + +#define STACK_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \ + __func__, ##args) + +#define STACK_LOG_ERR(fmt, args...) \ + STACK_LOG(ERR, fmt, ## args) + +#define STACK_LOG_WARN(fmt, args...) \ + STACK_LOG(WARNING, fmt, ## args) + +#define STACK_LOG_INFO(fmt, args...) \ + STACK_LOG(INFO, fmt, ## args) + + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_std.c new file +26 -0
@@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#include "rte_stack.h" + +void +rte_stack_std_init (struct rte_stack *s) +{ +rte_spinlock_init (&s->stack_std.lock); +} + +ssize_t +rte_stack_std_get_memsize (unsigned int count) +{ +ssize_t sz =sizeof (struct rte_stack ); + + sz +=RTE_CACHE_LINE_ROUNDUP (count *sizeof (void *)); + +/* Add padding to avoid false sharing conflicts caused by +* next-line hardware prefetchers. +*/ + sz +=2 *RTE_CACHE_LINE_SIZE ; + +return sz; +}
lib/librte_stack/rte_stack_std.h new file +119 -0
119 lines not on this page · raw message
lib/librte_stack/rte_stack_version.map new file +9 -0
@@ -0,0 +1,9 @@ +EXPERIMENTAL { + global: + + rte_stack_create; + rte_stack_free; + rte_stack_lookup; + + local: *; +};
lib/meson.build +1 -1
@@ -22,7 +22,7 @@ libraries = [ 'gro', 'gso', 'ip_frag', 'jobstats', 'kni', 'latencystats', 'lpm', 'member', 'power', 'pdump', 'rawdev', - 'reorder', 'sched', 'security', 'vhost', + 'reorder', 'sched', 'security', 'stack', 'vhost', #ipsec lib depends on crypto and security 'ipsec', # add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY) += -lrte_security _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += -lrte_compressdev _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += -lrte_eventdev _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV) += -lrte_rawdev +_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER) += -lrte_timer _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL) += -lrte_mempool _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
Signature · 1 line
2.13.6
Re: [PATCH v6 1/8] stack: introduce rte stack library
Quoted 7 lines
Subject: [PATCH v6 1/8] stack: introduce rte stack library
The
rte_stacklibrary provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.
Quoted 8 lines
The library's interface is modeled after another DPDK data structure,
rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler torte_stack.Signed-off-by Gage Eads <[email protected]>Reviewed-by Olivier Matz <[email protected]>---
<snip>
Re: [PATCH v6 1/8] stack: introduce rte stack library
02/04/2019 13:14, Honnappa Nagarahalli:
Quoted 12 lines
Quoted 12 lines
Subject: [PATCH v6 1/8] stack: introduce rte stack library
The
rte_stacklibrary provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.The library's interface is modeled after another DPDK data structure,
rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler torte_stack.
Quoted 4 lines
Quoted 3 lines
Signed-off-by Gage Eads <[email protected]>Reviewed-by Olivier Matz <[email protected]>---
Reviewed-by Honnappa Nagarahalli <[email protected]>
It does not compile for Arm:
lib/librte_stack/rte_stack_std.h:
In function '__rte_stack_std_pop':lib/librte_stack/rte_stack_std.h:68:6: error:
implicit declaration of function 'unlikely'
Re: [PATCH v6 1/8] stack: introduce rte stack library
Quoted 16 lines
02/04/2019 13:14, Honnappa Nagarahalli:
Quoted 15 lines
Quoted 15 lines
Subject: [PATCH v6 1/8] stack: introduce rte stack library
The
rte_stacklibrary provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.The library's interface is modeled after another DPDK data structure,
rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler torte_stack.Signed-off-by Gage Eads <[email protected]>Reviewed-by Olivier Matz <[email protected]>---
Quoted 8 lines
Quoted 1 line
Reviewed-by Honnappa Nagarahalli <[email protected]>It does not compile for Arm:
lib/librte_stack/rte_stack_std.h:
In function '__rte_stack_std_pop':lib/librte_stack/rte_stack_std.h:68:6: error:
implicit declaration of function 'unlikely'
Missing rte_branch_prediction.h include -- I'll fix and resubmit. Thanks for checking the non-x86 builds.
I can hold off resubmission until the 128-bit CAS patch is merged, so this series is properly tested in the automated build + test pipeline, if you'd prefer.
Thanks,
Gage
Re: [PATCH v6 1/8] stack: introduce rte stack library
03/04/2019 19:13, Eads, Gage:
Quoted 20 lines
Quoted 20 lines
02/04/2019 13:14, Honnappa Nagarahalli:
Quoted 12 lines
Quoted 12 lines
Subject: [PATCH v6 1/8] stack: introduce rte stack library
The
rte_stacklibrary provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.The library's interface is modeled after another DPDK data structure,
rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler torte_stack.Quoted 4 lines
Quoted 3 lines
Signed-off-by Gage Eads <[email protected]>Reviewed-by Olivier Matz <[email protected]>---
Reviewed-by Honnappa Nagarahalli <[email protected]>It does not compile for Arm:
Quoted 8 lines
Quoted 4 lines
lib/librte_stack/rte_stack_std.h:
In function '__rte_stack_std_pop':lib/librte_stack/rte_stack_std.h:68:6: error:
implicit declaration of function 'unlikely'Missing
rte_branch_prediction.hinclude -- I'll fix and resubmit. Thanks for checking the non-x86 builds.I can hold off resubmission until the 128-bit CAS patch is merged, so this series is properly tested in the automated build + test pipeline, if you'd prefer.
Yes, would be the best.
But we need to merge everything tomorrow at last.
PATCH v6 2/8 mempool/stack: convert mempool to use rte stack
In reply to Gage EadsThe new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.
--- MAINTAINERS | 2 +- drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++---------------------- 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
@@ -293,7 +293,6 @@ M: Andrew Rybchenko <[email protected]> F: lib/librte_mempool/ F: drivers/mempool/Makefile F: drivers/mempool/ring/ -F: drivers/mempool/stack/ F: doc/guides/prog_guide/mempool_lib.rst F: app/test/test_mempool* F: app/test/test_func_reentrancy.c
@@ -421,6 +420,7 @@ M: Gage Eads <[email protected]> M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst +F: drivers/mempool/stack/ Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a CFLAGS += -O3 CFLAGS += $(WERROR_FLAGS) +CFLAGS += -DALLOW_EXPERIMENTAL_API # Headers CFLAGS += -I$(RTE_SDK)/lib/librte_mempool -LDLIBS += -lrte_eal -lrte_mempool -lrte_ring +LDLIBS += -lrte_eal -lrte_mempool -lrte_stack EXPORT_MAP := rte_mempool_stack_version.map
drivers/mempool/stack/meson.build +5 -1
@@ -1,4 +1,8 @@ # SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation +# Copyright(c) 2017-2019 Intel Corporation + +allow_experimental_apis = true sources = files('rte_mempool_stack.c') + +deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68
133 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v6 3/8 test/stack: add stack test
In reply to Gage Eadsstack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.
--- MAINTAINERS | 1 + app/test/Makefile | 2 + app/test/meson.build | 3 + app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 416 insertions(+) create mode 100644 app/test/test_stack.c
MAINTAINERS +1 -0
@@ -421,6 +421,7 @@ M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst F: drivers/mempool/stack/ +F: test/test/*stack* Memory Pool Drivers
app/test/Makefile +2 -0
@@ -90,6 +90,8 @@ endif SRCS-y += test_rwlock.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c + SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
app/test/meson.build +3 -0
@@ -95,6 +95,7 @@ test_sources = files('commands.c', 'test_sched.c', 'test_service_cores.c', 'test_spinlock.c', + 'test_stack.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -133,6 +134,7 @@ test_deps = ['acl', 'port', 'reorder', 'ring', + 'stack', 'timer' ]
@@ -174,6 +176,7 @@ fast_parallel_test_names = [ 'rwlock_autotest', 'sched_autotest', 'spinlock_autotest', + 'stack_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
app/test/test_stack.c new file +410 -0
410 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v6 4/8 test/stack: add stack perf test
In reply to Gage Eadsstack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects
It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores
--- app/test/Makefile | 1 + app/test/meson.build | 2 + app/test/test_stack_perf.c | 343 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 346 insertions(+) create mode 100644 app/test/test_stack_perf.c
app/test/Makefile +1 -0
@@ -91,6 +91,7 @@ endif SRCS-y += test_rwlock.c SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
app/test/meson.build +2 -0
@@ -96,6 +96,7 @@ test_sources = files('commands.c', 'test_service_cores.c', 'test_spinlock.c', 'test_stack.c', + 'test_stack_perf.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -241,6 +242,7 @@ perf_test_names = [ 'distributor_perf_autotest', 'ring_pmd_perf_autotest', 'pmd_perf_autotest', + 'stack_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
app/test/test_stack_perf.c new file +343 -0
343 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v6 5/8 stack: add lock-free stack implementation
In reply to Gage EadsThis commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.
The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.
The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.
The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.
This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.
The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.
--- doc/guides/prog_guide/stack_lib.rst | 61 +++++++++++- doc/guides/rel_notes/release_19_05.rst | 3 + lib/librte_stack/Makefile | 7 +- lib/librte_stack/meson.build | 7 +- lib/librte_stack/rte_stack.c | 28 ++++-- lib/librte_stack/rte_stack.h | 62 +++++++++++- lib/librte_stack/rte_stack_lf.c | 31 ++++++ lib/librte_stack/rte_stack_lf.h | 102 ++++++++++++++++++++ lib/librte_stack/rte_stack_lf_generic.h | 164 ++++++++++++++++++++++++++++++++ 9 files changed, 446 insertions(+), 19 deletions(-) create mode 100644 lib/librte_stack/rte_stack_lf.c create mode 100644 lib/librte_stack/rte_stack_lf.h create mode 100644 lib/librte_stack/rte_stack_lf_generic.h
doc/guides/prog_guide/stack_lib.rst +58 -3
@@ -10,7 +10,8 @@ stack of pointers. The stack library provides the following basic operations: * Create a uniquely named stack of a user-specified size and using a - user-specified socket. + user-specified socket, with either standard (lock-based) or lock-free + behavior. * Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe.
@@ -24,5 +25,59 @@ The stack library provides the following basic operations: Implementation ~~~~~~~~~~~~~~ -The stack consists of a contiguous array of pointers, a current index, and a -spinlock. Accesses to the stack are made multi-thread safe by the spinlock. +The library supports two types of stacks: standard (lock-based) and lock-free. +Both types use the same set of interfaces, but their implementations differ. + +Lock-based Stack +---------------- + +The lock-based stack consists of a contiguous array of pointers, a current +index, and a spinlock. Accesses to the stack are made multi-thread safe by the +spinlock. + +Lock-free Stack +------------------ + +The lock-free stack consists of a linked list of elements, each containing a +data pointer and a next pointer, and an atomic stack depth counter. The +lock-free property means that multiple threads can push and pop simultaneously, +and one thread being preempted/delayed in a push or pop operation will not +impede the forward progress of any other thread. + +The lock-free push operation enqueues a linked list of pointers by pointing the +list's tail to the current stack head, and using a CAS to swing the stack head +pointer to the head of the list. The operation retries if it is unsuccessful +(i.e. the list changed between reading the head and modifying it), else it +adjusts the stack length and returns. + +The lock-free pop operation first reserves one or more list elements by +adjusting the stack length, to ensure the dequeue operation will succeed +without blocking. It then dequeues pointers by walking the list -- starting +from the head -- then swinging the head pointer (using a CAS as well). While +walking the list, the data pointers are recorded in an object table. + +The linked list elements themselves are maintained in a lock-free LIFO, and are +allocated before stack pushes and freed after stack pops. Since the stack has a +fixed maximum depth, these elements do not need to be dynamically created. + +The lock-free behavior is selected by passing the *RTE_STACK_F_LF* flag to +rte_stack_create(). + +Preventing the ABA Problem +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To prevent the ABA problem, this algorithm stack uses a 128-bit +compare-and-swap instruction to atomically update both the stack top pointer +and a modification counter. The ABA problem can occur without a modification +counter if, for example: + +1. Thread A reads head pointer X and stores the pointed-to list element. +2. Other threads modify the list such that the head pointer is once again X, + but its pointed-to data is different than what thread A read. +3. Thread A changes the head pointer with a compare-and-swap and succeeds. + +In this case thread A would not detect that the list had changed, and would +both pop stale data and incorrect change the head pointer. By adding a +modification counter that is updated on every push and pop as part of the +compare-and-swap, the algorithm can detect when the list changes even if the +head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
@@ -127,6 +127,9 @@ New Features pointers. The API provides MT-safe push and pop operations that can operate on one or more pointers per operation. + The library supports two stack implementations: standard (lock-based) and lock-free. + The lock-free implementation is currently limited to x86-64 platforms. + Removed Items -------------
lib/librte_stack/Makefile +5 -2
@@ -16,10 +16,13 @@ LIBABIVER := 1 # all source are stored in SRCS-y SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \ - rte_stack_std.c + rte_stack_std.c \ + rte_stack_lf.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ - rte_stack_std.h + rte_stack_std.h \ + rte_stack_lf.h \ + rte_stack_lf_generic.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +5 -2
@@ -4,5 +4,8 @@ allow_experimental_apis = true version = 1 -sources = files('rte_stack.c', 'rte_stack_std.c') -headers = files('rte_stack.h', 'rte_stack_std.h') +sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c') +headers = files('rte_stack.h', + 'rte_stack_std.h', + 'rte_stack_lf.h', + 'rte_stack_lf_generic.h')
lib/librte_stack/rte_stack.c +21 -7
@@ -25,18 +25,25 @@ static struct rte_tailq_elem rte_stack_tailq = { };EAL_REGISTER_TAILQ (rte_stack_tailq) + static void -rte_stack_init (struct rte_stack *s) +rte_stack_init (struct rte_stack *s,unsigned int count,uint32_t flags) {memset (s,0 ,sizeof (*s)); -rte_stack_std_init (s); +if (flags &RTE_STACK_F_LF ) +rte_stack_lf_init (s, count); +else +rte_stack_std_init (s); }static ssize_t -rte_stack_get_memsize (unsigned int count) +rte_stack_get_memsize (unsigned int count,uint32_t flags) { -return rte_stack_std_get_memsize (count); +if (flags &RTE_STACK_F_LF ) +return rte_stack_lf_get_memsize (count); +else +return rte_stack_std_get_memsize (count); } struct rte_stack *
@@ -51,9 +58,16 @@ rte_stack_create(const char *name, unsigned int count, int socket_id,unsigned int sz;int ret; -RTE_SET_USED (flags); +#ifdef RTE_ARCH_64 +RTE_BUILD_BUG_ON (sizeof (struct rte_stack_lf_head ) !=16 ); +#else +if (flags &RTE_STACK_F_LF ) { +STACK_LOG_ERR ("Lock-free stack is not supported on your platform\n" ); +return NULL ; + } +#endif - sz =rte_stack_get_memsize (count); + sz =rte_stack_get_memsize (count, flags); ret =snprintf (mz_name,sizeof (mz_name),"%s%s" ,RTE_STACK_MZ_PREFIX , name);
@@ -82,7 +96,7 @@ rte_stack_create(const char *name, unsigned int count, int socket_id, s = mz->addr; -rte_stack_init (s); +rte_stack_init (s, count, flags);/* Store the name for later lookups */ ret =snprintf (s->name,sizeof (s->name),"%s" , name);
lib/librte_stack/rte_stack.h +57 -5
101 lines not on this page · raw message
lib/librte_stack/rte_stack_lf.c new file +31 -0
@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#include "rte_stack.h" + +void +rte_stack_lf_init (struct rte_stack *s,unsigned int count) +{ +struct rte_stack_lf_elem *elems = s->stack_lf.elems; +unsigned int i; + +for (i =0 ; i < count; i++) +__rte_stack_lf_push_elems (&s->stack_lf.free, + &elems[i], &elems[i],1 ); +} + +ssize_t +rte_stack_lf_get_memsize (unsigned int count) +{ +ssize_t sz =sizeof (struct rte_stack ); + + sz +=RTE_CACHE_LINE_ROUNDUP (count *sizeof (struct rte_stack_lf_elem )); + +/* Add padding to avoid false sharing conflicts caused by +* next-line hardware prefetchers. +*/ + sz +=2 *RTE_CACHE_LINE_SIZE ; + +return sz; +}
lib/librte_stack/rte_stack_lf.h new file +102 -0
102 lines not on this page · raw message
lib/librte_stack/rte_stack_lf_generic.h new file +164 -0
164 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v6 6/8 stack: add C11 atomic implementation
In reply to Gage EadsThis commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.
--- lib/librte_stack/Makefile | 3 +- lib/librte_stack/meson.build | 3 +- lib/librte_stack/rte_stack_lf.h | 4 + lib/librte_stack/rte_stack_lf_c11.h | 175 ++++++++++++++++++++++++++++++++++++ 4 files changed, 183 insertions(+), 2 deletions(-) create mode 100644 lib/librte_stack/rte_stack_lf_c11.h
lib/librte_stack/Makefile +2 -1
@@ -23,6 +23,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \ SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ rte_stack_std.h \ rte_stack_lf.h \ - rte_stack_lf_generic.h + rte_stack_lf_generic.h \ + rte_stack_lf_c11.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
@@ -8,4 +8,5 @@ sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c') headers = files('rte_stack.h', 'rte_stack_std.h', 'rte_stack_lf.h', - 'rte_stack_lf_generic.h') + 'rte_stack_lf_generic.h', + 'rte_stack_lf_c11.h')
lib/librte_stack/rte_stack_lf.h +4 -0
@@ -5,7 +5,11 @@#ifndef _RTE_STACK_LF_H_#define _RTE_STACK_LF_H_ +#ifdef RTE_USE_C11_MEM_MODEL +#include "rte_stack_lf_c11.h" +#else #include "rte_stack_lf_generic.h" +#endif /** * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_lf_c11.h new file +175 -0
175 lines not on this page · raw message
Signature · 1 line
2.13.6
Re: [PATCH v6 6/8] stack: add C11 atomic implementation
Quoted 9 lines
Subject: [PATCH v6 6/8] stack: add C11 atomic implementation
This commit adds an implementation of the lock-free stack push, pop, and length functions that use
__atomicbuiltins, for systems that benefit from the finer-grained memory ordering control.Signed-off-by Gage Eads <[email protected]>Reviewed-by Olivier Matz <[email protected]>---
<snip>
PATCH v6 7/8 test/stack: add lock-free stack tests
In reply to Gage EadsThis commit adds lock-free stack variants of stack_autotest (stack_lf_autotest) and stack_perf_autotest (stack_lf_perf_autotest), which differ only in that the lock-free versions pass the RTE_STACK_F_LF flag to all rte_stack_create() calls.
--- app/test/meson.build | 2 ++ app/test/test_stack.c | 41 +++++++++++++++++++++++++++-------------- app/test/test_stack_perf.c | 17 +++++++++++++++-- 3 files changed, 44 insertions(+), 16 deletions(-)
app/test/meson.build +2 -0
@@ -178,6 +178,7 @@ fast_parallel_test_names = [ 'sched_autotest', 'spinlock_autotest', 'stack_autotest', + 'stack_nb_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
@@ -243,6 +244,7 @@ perf_test_names = [ 'ring_pmd_perf_autotest', 'pmd_perf_autotest', 'stack_perf_autotest', + 'stack_nb_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
app/test/test_stack.c +27 -14
112 lines not on this page · raw message
app/test/test_stack_perf.c +15 -2
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s) }static int -test_stack_perf (void ) +__test_stack_perf (uint32_t flags) {struct lcore_pair cores;struct rte_stack *s;rte_atomic32_init (&lcore_barrier); - s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (),0 ); + s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (), flags);if (s ==NULL ) {printf ("[%s():%u] failed to create a stack\n" , __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)return 0 ; } +static int +test_stack_perf (void ) +{ +return __test_stack_perf (0 ); +} + +static int +test_lf_stack_perf (void ) +{ +return __test_stack_perf (RTE_STACK_F_LF ); +} +REGISTER_TEST_COMMAND (stack_perf_autotest, test_stack_perf); +REGISTER_TEST_COMMAND (stack_lf_perf_autotest, test_lf_stack_perf);
Signature · 1 line
2.13.6
PATCH v6 8/8 mempool/stack: add lock-free stack mempool handler
In reply to Gage EadsThis commit adds support for lock-free (linked list based) stack mempool handler.
In mempool_perf_autotest the lock-based stack outperforms the lock-free handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a standard (lock-based) stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the lock-free stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.
*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the lock-free stack's.
--- doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++ doc/guides/rel_notes/release_19_05.rst | 5 +++++ drivers/mempool/stack/rte_mempool_stack.c | 26 +++++++++++++++++++++++-- 3 files changed, 39 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
@@ -563,6 +563,16 @@ Known Issues 5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR. + Alternatively, applications can use the lock-free stack mempool handler. When + considering this handler, note that: + + - It is currently limited to the x86_64 platform, because it uses an + instruction (16-byte compare-and-swap) that is not yet available on other + platforms. + - It has worse average-case performance than the non-preemptive rte_ring, but + software caching (e.g. the mempool cache) can mitigate this by reducing the + number of stack accesses. + + rte_timer Running ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -130,6 +130,11 @@ New Features The library supports two stack implementations: standard (lock-based) and lock-free. The lock-free implementation is currently limited to x86-64 platforms. +* **Added Lock-Free Stack Mempool Handler.** + + Added a new lock-free stack handler, which uses the newly added stack + library. + Removed Items -------------
drivers/mempool/stack/rte_mempool_stack.c +24 -2
@@ -7,7 +7,7 @@#include <rte_stack.h> static int -stack_alloc (struct rte_mempool *mp) +__stack_alloc (struct rte_mempool *mp,uint32_t flags) {char name[RTE_STACK_NAMESIZE ];struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)return -rte_errno; } - s =rte_stack_create (name, mp->size, mp->socket_id,0 ); + s =rte_stack_create (name, mp->size, mp->socket_id, flags);if (s ==NULL )return -rte_errno;
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp) }static int +stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,0 ); +} + +static int +lf_stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,RTE_STACK_F_LF ); +} + +static int stack_enqueue (struct rte_mempool *mp,void *const *obj_table,unsigned int n) {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = { .get_count = stack_get_count }; +static struct rte_mempool_ops ops_lf_stack = { + .name ="lf_stack" , + .alloc = lf_stack_alloc, + .free = stack_free, + .enqueue = stack_enqueue, + .dequeue = stack_dequeue, + .get_count = stack_get_count +}; +MEMPOOL_REGISTER_OPS (ops_stack); +MEMPOOL_REGISTER_OPS (ops_lf_stack);
Signature · 1 line
2.13.6
Re: [PATCH v6 0/8] Add stack library and new mempool handler
In reply to Gage Eads01/04/2019 23:14, Gage Eads:
Quoted 2 lines
Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the
x86_64platform.
I'm waiting for an update of the 128-bit compare-and-swap. It is blocking the integration of this patch.
Re: [PATCH v6 0/8] Add stack library and new mempool handler
Quoted 7 lines
01/04/2019 23:14, Gage Eads:
Quoted 2 lines
Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the
x86_64platform.I'm waiting for an update of the 128-bit compare-and-swap. It is blocking the integration of this patch.
Sorry for that; I misunderstood your earlier comment on that patch. I'll address it and re-submit.
Thanks,
Gage
v7 cover Add stack library and new mempool handler
In reply to Gage EadsThis patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.
The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.
The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.
Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.
This patchset is the successor to a patchset containing only the new mempool handler[2].
[1] http://mails.dpdk.org/archives/dev/2019-April/129014.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html
---
v7:
- Add rte_branch_prediction.h include to rte_stack_std.h for unlikely()
- Add rte_compat.h include to rte_stack.h for __rte_experimental
v6:
- Add load-acquire fence to the lock-free push function
- Correct generic implementation's pop_elems 128b CAS success and failure memorder to match those in the C11 implementation.
v5:
- Add comment to explain padding in *_get_memsize() functions
- Prefix internal functions with '__'
- Use RTE_ASSERT for performance critical run-time checks
- Don't use __atomic_load in the C11 pop_elems function, and put an acquire thread fence at the start of the 2nd do-while loop
- Change pop_elems 128b CAS success memorder to RELEASE and failure memorder to RELAXED
- Change compile-time assertion to run for all 64-bit architectures
- Reorganize the code with standard and lock-free .c and .h files
v4:
- Fix 32-bit build error in test_stack.c by using %zu format specifier for size_t
- Rebase onto master
v3:
- Rebase patchset onto master (test/test/ -> app/test/)
- Fix rte_stack_std_push() segfault introduced in v2
v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset
Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler
MAINTAINERS | 9 +- app/test/Makefile | 3 + app/test/meson.build | 7 + app/test/test_stack.c | 423 ++++++++++++++++++++++++ app/test/test_stack_perf.c | 356 ++++++++++++++++++++ config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/env_abstraction_layer.rst | 10 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 83 +++++ doc/guides/rel_notes/release_19_05.rst | 13 + drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 115 +++---- lib/Makefile | 2 + lib/librte_stack/Makefile | 29 ++ lib/librte_stack/meson.build | 12 + lib/librte_stack/rte_stack.c | 196 +++++++++++ lib/librte_stack/rte_stack.h | 260 +++++++++++++++ lib/librte_stack/rte_stack_lf.c | 31 ++ lib/librte_stack/rte_stack_lf.h | 106 ++++++ lib/librte_stack/rte_stack_lf_c11.h | 175 ++++++++++ lib/librte_stack/rte_stack_lf_generic.h | 164 +++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++ lib/librte_stack/rte_stack_std.c | 26 ++ lib/librte_stack/rte_stack_std.h | 121 +++++++ lib/librte_stack/rte_stack_version.map | 9 + lib/meson.build | 2 +- mk/rte.app.mk | 1 + 30 files changed, 2132 insertions(+), 72 deletions(-) create mode 100644 app/test/test_stack.c create mode 100644 app/test/test_stack_perf.c create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_lf.c create mode 100644 lib/librte_stack/rte_stack_lf.h create mode 100644 lib/librte_stack/rte_stack_lf_c11.h create mode 100644 lib/librte_stack/rte_stack_lf_generic.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_std.c create mode 100644 lib/librte_stack/rte_stack_std.h create mode 100644 lib/librte_stack/rte_stack_version.map
Signature · 1 line
2.13.6
PATCH v7 1/8 stack: introduce rte stack library
The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.
The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.
--- MAINTAINERS | 6 + config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 28 +++++ doc/guides/rel_notes/release_19_05.rst | 5 + lib/Makefile | 2 + lib/librte_stack/Makefile | 25 ++++ lib/librte_stack/meson.build | 8 ++ lib/librte_stack/rte_stack.c | 182 +++++++++++++++++++++++++++++ lib/librte_stack/rte_stack.h | 208 +++++++++++++++++++++++++++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++++++ lib/librte_stack/rte_stack_std.c | 26 +++++ lib/librte_stack/rte_stack_std.h | 121 +++++++++++++++++++ lib/librte_stack/rte_stack_version.map | 9 ++ lib/meson.build | 2 +- mk/rte.app.mk | 1 + 18 files changed, 664 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_std.c create mode 100644 lib/librte_stack/rte_stack_std.h create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
@@ -426,6 +426,12 @@ F: drivers/raw/skeleton_rawdev/ F: app/test/test_rawdev.c F: doc/guides/prog_guide/rawdev.rst +Stack API - EXPERIMENTAL +M: Gage Eads <[email protected]> +M: Olivier Matz <[email protected]> +F: lib/librte_stack/ +F: doc/guides/prog_guide/stack_lib.rst + Memory Pool Drivers -------------------
config/common_base +5 -0
@@ -994,3 +994,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y # Compile the eventdev application # CONFIG_RTE_APP_EVENTDEV=y + +# +# Compile librte_stack +# +CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
@@ -125,6 +125,7 @@ The public API headers are grouped by topics: [mbuf] (@ref rte_mbuf.h), [mbuf pool ops] (@ref rte_mbuf_pool_ops.h), [ring] (@ref rte_ring.h), + [stack] (@ref rte_stack.h), [tailq] (@ref rte_tailq.h), [bitmap] (@ref rte_bitmap.h)
doc/api/doxy-api.conf.in +1 -0
@@ -55,6 +55,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/librte_ring \ @TOPDIR@/lib/librte_sched \ @TOPDIR@/lib/librte_security \ + @TOPDIR@/lib/librte_stack \ @TOPDIR@/lib/librte_table \ @TOPDIR@/lib/librte_telemetry \ @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
@@ -55,6 +55,7 @@ Programmer's Guide metrics_lib bpf_lib ipsec_lib + stack_lib source_org dev_kit_build_system dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +28 -0
@@ -0,0 +1,28 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +Stack Library +============= + +DPDK's stack library provides an API for configuration and use of a bounded +stack of pointers. + +The stack library provides the following basic operations: + +* Create a uniquely named stack of a user-specified size and using a + user-specified socket. + +* Push and pop a burst of one or more stack objects (pointers). These function + are multi-threading safe. + +* Free a previously created stack. + +* Lookup a pointer to a stack by its name. + +* Query a stack's current depth and number of free entries. + +Implementation +~~~~~~~~~~~~~~ + +The stack consists of a contiguous array of pointers, a current index, and a +spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -121,6 +121,11 @@ New Features Improved testpmd application performance on ARM platform. For ``macswap`` forwarding mode, NEON intrinsics were used to do swap to save CPU cycles. +* **Added Stack API.** + + Added a new stack API for configuration and use of a bounded stack of + pointers. The API provides MT-safe push and pop operations that can operate + on one or more pointers per operation. Removed Items -------------
lib/Makefile +2 -0
@@ -109,6 +109,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev +DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack +DEPDIRS-librte_stack := librte_eal ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +25 -0
@@ -0,0 +1,25 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_stack.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +CFLAGS += -DALLOW_EXPERIMENTAL_API +LDLIBS += -lrte_eal + +EXPORT_MAP := rte_stack_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \ + rte_stack_std.c + +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ + rte_stack_std.h + +include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
@@ -0,0 +1,8 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +allow_experimental_apis = true + +version = 1 +sources = files('rte_stack.c', 'rte_stack_std.c') +headers = files('rte_stack.h', 'rte_stack_std.h')
lib/librte_stack/rte_stack.c new file +182 -0
182 lines not on this page · raw message
lib/librte_stack/rte_stack.h new file +208 -0
208 lines not on this page · raw message
lib/librte_stack/rte_stack_pvt.h new file +34 -0
@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#ifndef _RTE_STACK_PVT_H_ +#define _RTE_STACK_PVT_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <rte_log.h> + +extern int stack_logtype; + +#define STACK_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \ + __func__, ##args) + +#define STACK_LOG_ERR(fmt, args...) \ + STACK_LOG(ERR, fmt, ## args) + +#define STACK_LOG_WARN(fmt, args...) \ + STACK_LOG(WARNING, fmt, ## args) + +#define STACK_LOG_INFO(fmt, args...) \ + STACK_LOG(INFO, fmt, ## args) + + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_std.c new file +26 -0
@@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#include "rte_stack.h" + +void +rte_stack_std_init (struct rte_stack *s) +{ +rte_spinlock_init (&s->stack_std.lock); +} + +ssize_t +rte_stack_std_get_memsize (unsigned int count) +{ +ssize_t sz =sizeof (struct rte_stack ); + + sz +=RTE_CACHE_LINE_ROUNDUP (count *sizeof (void *)); + +/* Add padding to avoid false sharing conflicts caused by +* next-line hardware prefetchers. +*/ + sz +=2 *RTE_CACHE_LINE_SIZE ; + +return sz; +}
lib/librte_stack/rte_stack_std.h new file +121 -0
121 lines not on this page · raw message
lib/librte_stack/rte_stack_version.map new file +9 -0
@@ -0,0 +1,9 @@ +EXPERIMENTAL { + global: + + rte_stack_create; + rte_stack_free; + rte_stack_lookup; + + local: *; +};
lib/meson.build +1 -1
@@ -22,7 +22,7 @@ libraries = [ 'gro', 'gso', 'ip_frag', 'jobstats', 'kni', 'latencystats', 'lpm', 'member', 'power', 'pdump', 'rawdev', - 'reorder', 'sched', 'security', 'vhost', + 'reorder', 'sched', 'security', 'stack', 'vhost', #ipsec lib depends on crypto and security 'ipsec', # add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY) += -lrte_security _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += -lrte_compressdev _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += -lrte_eventdev _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV) += -lrte_rawdev +_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER) += -lrte_timer _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL) += -lrte_mempool _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
Signature · 1 line
2.13.6
PATCH v7 2/8 mempool/stack: convert mempool to use rte stack
In reply to Gage EadsThe new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.
--- MAINTAINERS | 2 +- drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++---------------------- 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
@@ -303,7 +303,6 @@ M: Andrew Rybchenko <[email protected]> F: lib/librte_mempool/ F: drivers/mempool/Makefile F: drivers/mempool/ring/ -F: drivers/mempool/stack/ F: doc/guides/prog_guide/mempool_lib.rst F: app/test/test_mempool* F: app/test/test_func_reentrancy.c
@@ -431,6 +430,7 @@ M: Gage Eads <[email protected]> M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst +F: drivers/mempool/stack/ Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a CFLAGS += -O3 CFLAGS += $(WERROR_FLAGS) +CFLAGS += -DALLOW_EXPERIMENTAL_API # Headers CFLAGS += -I$(RTE_SDK)/lib/librte_mempool -LDLIBS += -lrte_eal -lrte_mempool -lrte_ring +LDLIBS += -lrte_eal -lrte_mempool -lrte_stack EXPORT_MAP := rte_mempool_stack_version.map
drivers/mempool/stack/meson.build +5 -1
@@ -1,4 +1,8 @@ # SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation +# Copyright(c) 2017-2019 Intel Corporation + +allow_experimental_apis = true sources = files('rte_mempool_stack.c') + +deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68
133 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v7 3/8 test/stack: add stack test
In reply to Gage Eadsstack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.
--- MAINTAINERS | 1 + app/test/Makefile | 2 + app/test/meson.build | 3 + app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 416 insertions(+) create mode 100644 app/test/test_stack.c
MAINTAINERS +1 -0
@@ -431,6 +431,7 @@ M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst F: drivers/mempool/stack/ +F: test/test/*stack* Memory Pool Drivers
app/test/Makefile +2 -0
@@ -90,6 +90,8 @@ endif SRCS-y += test_rwlock.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c + SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
app/test/meson.build +3 -0
@@ -95,6 +95,7 @@ test_sources = files('commands.c', 'test_sched.c', 'test_service_cores.c', 'test_spinlock.c', + 'test_stack.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -133,6 +134,7 @@ test_deps = ['acl', 'port', 'reorder', 'ring', + 'stack', 'timer' ]
@@ -174,6 +176,7 @@ fast_parallel_test_names = [ 'rwlock_autotest', 'sched_autotest', 'spinlock_autotest', + 'stack_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
app/test/test_stack.c new file +410 -0
410 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v7 4/8 test/stack: add stack perf test
In reply to Gage Eadsstack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects
It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores
--- app/test/Makefile | 1 + app/test/meson.build | 2 + app/test/test_stack_perf.c | 343 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 346 insertions(+) create mode 100644 app/test/test_stack_perf.c
app/test/Makefile +1 -0
@@ -91,6 +91,7 @@ endif SRCS-y += test_rwlock.c SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
app/test/meson.build +2 -0
@@ -96,6 +96,7 @@ test_sources = files('commands.c', 'test_service_cores.c', 'test_spinlock.c', 'test_stack.c', + 'test_stack_perf.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -241,6 +242,7 @@ perf_test_names = [ 'distributor_perf_autotest', 'ring_pmd_perf_autotest', 'pmd_perf_autotest', + 'stack_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
app/test/test_stack_perf.c new file +343 -0
343 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v7 5/8 stack: add lock-free stack implementation
In reply to Gage EadsThis commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.
The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.
The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.
The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.
This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.
The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.
--- doc/guides/prog_guide/stack_lib.rst | 61 +++++++++++- doc/guides/rel_notes/release_19_05.rst | 3 + lib/librte_stack/Makefile | 7 +- lib/librte_stack/meson.build | 7 +- lib/librte_stack/rte_stack.c | 28 ++++-- lib/librte_stack/rte_stack.h | 62 +++++++++++- lib/librte_stack/rte_stack_lf.c | 31 ++++++ lib/librte_stack/rte_stack_lf.h | 102 ++++++++++++++++++++ lib/librte_stack/rte_stack_lf_generic.h | 164 ++++++++++++++++++++++++++++++++ 9 files changed, 446 insertions(+), 19 deletions(-) create mode 100644 lib/librte_stack/rte_stack_lf.c create mode 100644 lib/librte_stack/rte_stack_lf.h create mode 100644 lib/librte_stack/rte_stack_lf_generic.h
doc/guides/prog_guide/stack_lib.rst +58 -3
@@ -10,7 +10,8 @@ stack of pointers. The stack library provides the following basic operations: * Create a uniquely named stack of a user-specified size and using a - user-specified socket. + user-specified socket, with either standard (lock-based) or lock-free + behavior. * Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe.
@@ -24,5 +25,59 @@ The stack library provides the following basic operations: Implementation ~~~~~~~~~~~~~~ -The stack consists of a contiguous array of pointers, a current index, and a -spinlock. Accesses to the stack are made multi-thread safe by the spinlock. +The library supports two types of stacks: standard (lock-based) and lock-free. +Both types use the same set of interfaces, but their implementations differ. + +Lock-based Stack +---------------- + +The lock-based stack consists of a contiguous array of pointers, a current +index, and a spinlock. Accesses to the stack are made multi-thread safe by the +spinlock. + +Lock-free Stack +------------------ + +The lock-free stack consists of a linked list of elements, each containing a +data pointer and a next pointer, and an atomic stack depth counter. The +lock-free property means that multiple threads can push and pop simultaneously, +and one thread being preempted/delayed in a push or pop operation will not +impede the forward progress of any other thread. + +The lock-free push operation enqueues a linked list of pointers by pointing the +list's tail to the current stack head, and using a CAS to swing the stack head +pointer to the head of the list. The operation retries if it is unsuccessful +(i.e. the list changed between reading the head and modifying it), else it +adjusts the stack length and returns. + +The lock-free pop operation first reserves one or more list elements by +adjusting the stack length, to ensure the dequeue operation will succeed +without blocking. It then dequeues pointers by walking the list -- starting +from the head -- then swinging the head pointer (using a CAS as well). While +walking the list, the data pointers are recorded in an object table. + +The linked list elements themselves are maintained in a lock-free LIFO, and are +allocated before stack pushes and freed after stack pops. Since the stack has a +fixed maximum depth, these elements do not need to be dynamically created. + +The lock-free behavior is selected by passing the *RTE_STACK_F_LF* flag to +rte_stack_create(). + +Preventing the ABA Problem +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To prevent the ABA problem, this algorithm stack uses a 128-bit +compare-and-swap instruction to atomically update both the stack top pointer +and a modification counter. The ABA problem can occur without a modification +counter if, for example: + +1. Thread A reads head pointer X and stores the pointed-to list element. +2. Other threads modify the list such that the head pointer is once again X, + but its pointed-to data is different than what thread A read. +3. Thread A changes the head pointer with a compare-and-swap and succeeds. + +In this case thread A would not detect that the list had changed, and would +both pop stale data and incorrect change the head pointer. By adding a +modification counter that is updated on every push and pop as part of the +compare-and-swap, the algorithm can detect when the list changes even if the +head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
@@ -127,6 +127,9 @@ New Features pointers. The API provides MT-safe push and pop operations that can operate on one or more pointers per operation. + The library supports two stack implementations: standard (lock-based) and lock-free. + The lock-free implementation is currently limited to x86-64 platforms. + Removed Items -------------
lib/librte_stack/Makefile +5 -2
@@ -16,10 +16,13 @@ LIBABIVER := 1 # all source are stored in SRCS-y SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \ - rte_stack_std.c + rte_stack_std.c \ + rte_stack_lf.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ - rte_stack_std.h + rte_stack_std.h \ + rte_stack_lf.h \ + rte_stack_lf_generic.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +5 -2
@@ -4,5 +4,8 @@ allow_experimental_apis = true version = 1 -sources = files('rte_stack.c', 'rte_stack_std.c') -headers = files('rte_stack.h', 'rte_stack_std.h') +sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c') +headers = files('rte_stack.h', + 'rte_stack_std.h', + 'rte_stack_lf.h', + 'rte_stack_lf_generic.h')
lib/librte_stack/rte_stack.c +21 -7
@@ -25,18 +25,25 @@ static struct rte_tailq_elem rte_stack_tailq = { };EAL_REGISTER_TAILQ (rte_stack_tailq) + static void -rte_stack_init (struct rte_stack *s) +rte_stack_init (struct rte_stack *s,unsigned int count,uint32_t flags) {memset (s,0 ,sizeof (*s)); -rte_stack_std_init (s); +if (flags &RTE_STACK_F_LF ) +rte_stack_lf_init (s, count); +else +rte_stack_std_init (s); }static ssize_t -rte_stack_get_memsize (unsigned int count) +rte_stack_get_memsize (unsigned int count,uint32_t flags) { -return rte_stack_std_get_memsize (count); +if (flags &RTE_STACK_F_LF ) +return rte_stack_lf_get_memsize (count); +else +return rte_stack_std_get_memsize (count); } struct rte_stack *
@@ -51,9 +58,16 @@ rte_stack_create(const char *name, unsigned int count, int socket_id,unsigned int sz;int ret; -RTE_SET_USED (flags); +#ifdef RTE_ARCH_64 +RTE_BUILD_BUG_ON (sizeof (struct rte_stack_lf_head ) !=16 ); +#else +if (flags &RTE_STACK_F_LF ) { +STACK_LOG_ERR ("Lock-free stack is not supported on your platform\n" ); +return NULL ; + } +#endif - sz =rte_stack_get_memsize (count); + sz =rte_stack_get_memsize (count, flags); ret =snprintf (mz_name,sizeof (mz_name),"%s%s" ,RTE_STACK_MZ_PREFIX , name);
@@ -82,7 +96,7 @@ rte_stack_create(const char *name, unsigned int count, int socket_id, s = mz->addr; -rte_stack_init (s); +rte_stack_init (s, count, flags);/* Store the name for later lookups */ ret =snprintf (s->name,sizeof (s->name),"%s" , name);
lib/librte_stack/rte_stack.h +57 -5
101 lines not on this page · raw message
lib/librte_stack/rte_stack_lf.c new file +31 -0
@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#include "rte_stack.h" + +void +rte_stack_lf_init (struct rte_stack *s,unsigned int count) +{ +struct rte_stack_lf_elem *elems = s->stack_lf.elems; +unsigned int i; + +for (i =0 ; i < count; i++) +__rte_stack_lf_push_elems (&s->stack_lf.free, + &elems[i], &elems[i],1 ); +} + +ssize_t +rte_stack_lf_get_memsize (unsigned int count) +{ +ssize_t sz =sizeof (struct rte_stack ); + + sz +=RTE_CACHE_LINE_ROUNDUP (count *sizeof (struct rte_stack_lf_elem )); + +/* Add padding to avoid false sharing conflicts caused by +* next-line hardware prefetchers. +*/ + sz +=2 *RTE_CACHE_LINE_SIZE ; + +return sz; +}
lib/librte_stack/rte_stack_lf.h new file +102 -0
102 lines not on this page · raw message
lib/librte_stack/rte_stack_lf_generic.h new file +164 -0
164 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v7 6/8 stack: add C11 atomic implementation
In reply to Gage EadsThis commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.
--- lib/librte_stack/Makefile | 3 +- lib/librte_stack/meson.build | 3 +- lib/librte_stack/rte_stack_lf.h | 4 + lib/librte_stack/rte_stack_lf_c11.h | 175 ++++++++++++++++++++++++++++++++++++ 4 files changed, 183 insertions(+), 2 deletions(-) create mode 100644 lib/librte_stack/rte_stack_lf_c11.h
lib/librte_stack/Makefile +2 -1
@@ -23,6 +23,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \ SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ rte_stack_std.h \ rte_stack_lf.h \ - rte_stack_lf_generic.h + rte_stack_lf_generic.h \ + rte_stack_lf_c11.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
@@ -8,4 +8,5 @@ sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c') headers = files('rte_stack.h', 'rte_stack_std.h', 'rte_stack_lf.h', - 'rte_stack_lf_generic.h') + 'rte_stack_lf_generic.h', + 'rte_stack_lf_c11.h')
lib/librte_stack/rte_stack_lf.h +4 -0
@@ -5,7 +5,11 @@#ifndef _RTE_STACK_LF_H_#define _RTE_STACK_LF_H_ +#ifdef RTE_USE_C11_MEM_MODEL +#include "rte_stack_lf_c11.h" +#else #include "rte_stack_lf_generic.h" +#endif /** * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_lf_c11.h new file +175 -0
175 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v7 7/8 test/stack: add lock-free stack tests
In reply to Gage EadsThis commit adds lock-free stack variants of stack_autotest (stack_lf_autotest) and stack_perf_autotest (stack_lf_perf_autotest), which differ only in that the lock-free versions pass the RTE_STACK_F_LF flag to all rte_stack_create() calls.
--- app/test/meson.build | 2 ++ app/test/test_stack.c | 41 +++++++++++++++++++++++++++-------------- app/test/test_stack_perf.c | 17 +++++++++++++++-- 3 files changed, 44 insertions(+), 16 deletions(-)
app/test/meson.build +2 -0
@@ -178,6 +178,7 @@ fast_parallel_test_names = [ 'sched_autotest', 'spinlock_autotest', 'stack_autotest', + 'stack_nb_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
@@ -243,6 +244,7 @@ perf_test_names = [ 'ring_pmd_perf_autotest', 'pmd_perf_autotest', 'stack_perf_autotest', + 'stack_nb_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
app/test/test_stack.c +27 -14
112 lines not on this page · raw message
app/test/test_stack_perf.c +15 -2
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s) }static int -test_stack_perf (void ) +__test_stack_perf (uint32_t flags) {struct lcore_pair cores;struct rte_stack *s;rte_atomic32_init (&lcore_barrier); - s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (),0 ); + s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (), flags);if (s ==NULL ) {printf ("[%s():%u] failed to create a stack\n" , __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)return 0 ; } +static int +test_stack_perf (void ) +{ +return __test_stack_perf (0 ); +} + +static int +test_lf_stack_perf (void ) +{ +return __test_stack_perf (RTE_STACK_F_LF ); +} +REGISTER_TEST_COMMAND (stack_perf_autotest, test_stack_perf); +REGISTER_TEST_COMMAND (stack_lf_perf_autotest, test_lf_stack_perf);
Signature · 1 line
2.13.6
PATCH v7 8/8 mempool/stack: add lock-free stack mempool handler
In reply to Gage EadsThis commit adds support for lock-free (linked list based) stack mempool handler.
In mempool_perf_autotest the lock-based stack outperforms the lock-free handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a standard (lock-based) stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the lock-free stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.
*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the lock-free stack's.
--- doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++ doc/guides/rel_notes/release_19_05.rst | 5 +++++ drivers/mempool/stack/rte_mempool_stack.c | 26 +++++++++++++++++++++++-- 3 files changed, 39 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
@@ -581,6 +581,16 @@ Known Issues 5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR. + Alternatively, applications can use the lock-free stack mempool handler. When + considering this handler, note that: + + - It is currently limited to the x86_64 platform, because it uses an + instruction (16-byte compare-and-swap) that is not yet available on other + platforms. + - It has worse average-case performance than the non-preemptive rte_ring, but + software caching (e.g. the mempool cache) can mitigate this by reducing the + number of stack accesses. + + rte_timer Running ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -130,6 +130,11 @@ New Features The library supports two stack implementations: standard (lock-based) and lock-free. The lock-free implementation is currently limited to x86-64 platforms. +* **Added Lock-Free Stack Mempool Handler.** + + Added a new lock-free stack handler, which uses the newly added stack + library. + Removed Items -------------
drivers/mempool/stack/rte_mempool_stack.c +24 -2
@@ -7,7 +7,7 @@#include <rte_stack.h> static int -stack_alloc (struct rte_mempool *mp) +__stack_alloc (struct rte_mempool *mp,uint32_t flags) {char name[RTE_STACK_NAMESIZE ];struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)return -rte_errno; } - s =rte_stack_create (name, mp->size, mp->socket_id,0 ); + s =rte_stack_create (name, mp->size, mp->socket_id, flags);if (s ==NULL )return -rte_errno;
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp) }static int +stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,0 ); +} + +static int +lf_stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,RTE_STACK_F_LF ); +} + +static int stack_enqueue (struct rte_mempool *mp,void *const *obj_table,unsigned int n) {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = { .get_count = stack_get_count }; +static struct rte_mempool_ops ops_lf_stack = { + .name ="lf_stack" , + .alloc = lf_stack_alloc, + .free = stack_free, + .enqueue = stack_enqueue, + .dequeue = stack_dequeue, + .get_count = stack_get_count +}; +MEMPOOL_REGISTER_OPS (ops_stack); +MEMPOOL_REGISTER_OPS (ops_lf_stack);
Signature · 1 line
2.13.6
Re: [PATCH v7 0/8] Add stack library and new mempool handler
In reply to Gage Eads03/04/2019 22:09, Gage Eads:
Quoted 3 lines
v7:
- Addrte_branch_prediction.hinclude torte_stack_std.hforunlikely()
- Addrte_compat.hinclude torte_stack.hfor__rte_experimental
There is another error when compiling for Arm:
lib/librte_stack/rte_stack.h:76:2: error:
implicit declaration of function 'RTE_ASSERT'
Re: [PATCH v7 0/8] Add stack library and new mempool handler
Quoted 14 lines
-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, April 3, 2019 3:40 PM
To: Eads, Gage <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v7 0/8] Add stack library and new mempool handler03/04/2019 22:09, Gage Eads:
Quoted 8 lines
Quoted 3 lines
v7:
- Addrte_branch_prediction.hinclude torte_stack_std.hforunlikely()
- Addrte_compat.hinclude torte_stack.hfor__rte_experimentalThere is another error when compiling for Arm:
lib/librte_stack/rte_stack.h:76:2: error:
implicit declaration of function 'RTE_ASSERT'
Fix incoming. Looking through the rest of that header, I don't see any other externally defined macros/data structures/functions/etc. that would need another include.
Also note that v7's ci/Performance-Testing build fail (http://mails.dpdk.org/archives/test-report/2019-April/079261.html) is because dpdklab applied the patch on commit 3c45889189924067, which is older than the 128-bit CAS patch.
v8 cover Add stack library and new mempool handler
In reply to Gage EadsThis patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.
The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.
The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.
Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.
This patchset is the successor to a patchset containing only the new mempool handler[2].
[1] http://mails.dpdk.org/archives/dev/2019-April/129014.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html
---
v8:
- Add rte_debug.h include to rte_stack.h for RTE_ASSERT()
v7:
- Add rte_branch_prediction.h include to rte_stack_std.h for unlikely()
- Add rte_compat.h include to rte_stack.h for __rte_experimental
v6:
- Add load-acquire fence to the lock-free push function
- Correct generic implementation's pop_elems 128b CAS success and failure memorder to match those in the C11 implementation.
v5:
- Add comment to explain padding in *_get_memsize() functions
- Prefix internal functions with '__'
- Use RTE_ASSERT for performance critical run-time checks
- Don't use __atomic_load in the C11 pop_elems function, and put an acquire thread fence at the start of the 2nd do-while loop
- Change pop_elems 128b CAS success memorder to RELEASE and failure memorder to RELAXED
- Change compile-time assertion to run for all 64-bit architectures
- Reorganize the code with standard and lock-free .c and .h files
v4:
- Fix 32-bit build error in test_stack.c by using %zu format specifier for size_t
- Rebase onto master
v3:
- Rebase patchset onto master (test/test/ -> app/test/)
- Fix rte_stack_std_push() segfault introduced in v2
v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset
Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler
MAINTAINERS | 9 +- app/test/Makefile | 3 + app/test/meson.build | 7 + app/test/test_stack.c | 423 ++++++++++++++++++++++++ app/test/test_stack_perf.c | 356 ++++++++++++++++++++ config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/env_abstraction_layer.rst | 10 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 83 +++++ doc/guides/rel_notes/release_19_05.rst | 13 + drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 115 +++---- lib/Makefile | 2 + lib/librte_stack/Makefile | 29 ++ lib/librte_stack/meson.build | 12 + lib/librte_stack/rte_stack.c | 196 +++++++++++ lib/librte_stack/rte_stack.h | 261 +++++++++++++++ lib/librte_stack/rte_stack_lf.c | 31 ++ lib/librte_stack/rte_stack_lf.h | 106 ++++++ lib/librte_stack/rte_stack_lf_c11.h | 175 ++++++++++ lib/librte_stack/rte_stack_lf_generic.h | 164 +++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++ lib/librte_stack/rte_stack_std.c | 26 ++ lib/librte_stack/rte_stack_std.h | 121 +++++++ lib/librte_stack/rte_stack_version.map | 9 + lib/meson.build | 2 +- mk/rte.app.mk | 1 + 30 files changed, 2133 insertions(+), 72 deletions(-) create mode 100644 app/test/test_stack.c create mode 100644 app/test/test_stack_perf.c create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_lf.c create mode 100644 lib/librte_stack/rte_stack_lf.h create mode 100644 lib/librte_stack/rte_stack_lf_c11.h create mode 100644 lib/librte_stack/rte_stack_lf_generic.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_std.c create mode 100644 lib/librte_stack/rte_stack_std.h create mode 100644 lib/librte_stack/rte_stack_version.map
Signature · 1 line
2.13.6
PATCH v8 1/8 stack: introduce rte stack library
The rte_stack library provides an API for configuration and use of a bounded stack of pointers. Push and pop operations are MT-safe, allowing concurrent access, and the interface supports pushing and popping multiple pointers at a time.
The library's interface is modeled after another DPDK data structure, rte_ring, and its lock-based implementation is derived from the stack mempool handler. An upcoming commit will migrate the stack mempool handler to rte_stack.
--- MAINTAINERS | 6 + config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 28 +++++ doc/guides/rel_notes/release_19_05.rst | 5 + lib/Makefile | 2 + lib/librte_stack/Makefile | 25 ++++ lib/librte_stack/meson.build | 8 ++ lib/librte_stack/rte_stack.c | 182 ++++++++++++++++++++++++++++ lib/librte_stack/rte_stack.h | 209 +++++++++++++++++++++++++++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++++++ lib/librte_stack/rte_stack_std.c | 26 ++++ lib/librte_stack/rte_stack_std.h | 121 +++++++++++++++++++ lib/librte_stack/rte_stack_version.map | 9 ++ lib/meson.build | 2 +- mk/rte.app.mk | 1 + 18 files changed, 665 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_std.c create mode 100644 lib/librte_stack/rte_stack_std.h create mode 100644 lib/librte_stack/rte_stack_version.map
MAINTAINERS +6 -0
@@ -426,6 +426,12 @@ F: drivers/raw/skeleton_rawdev/ F: app/test/test_rawdev.c F: doc/guides/prog_guide/rawdev.rst +Stack API - EXPERIMENTAL +M: Gage Eads <[email protected]> +M: Olivier Matz <[email protected]> +F: lib/librte_stack/ +F: doc/guides/prog_guide/stack_lib.rst + Memory Pool Drivers -------------------
config/common_base +5 -0
@@ -994,3 +994,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y # Compile the eventdev application # CONFIG_RTE_APP_EVENTDEV=y + +# +# Compile librte_stack +# +CONFIG_RTE_LIBRTE_STACK=y
doc/api/doxy-api-index.md +1 -0
@@ -125,6 +125,7 @@ The public API headers are grouped by topics: [mbuf] (@ref rte_mbuf.h), [mbuf pool ops] (@ref rte_mbuf_pool_ops.h), [ring] (@ref rte_ring.h), + [stack] (@ref rte_stack.h), [tailq] (@ref rte_tailq.h), [bitmap] (@ref rte_bitmap.h)
doc/api/doxy-api.conf.in +1 -0
@@ -55,6 +55,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/librte_ring \ @TOPDIR@/lib/librte_sched \ @TOPDIR@/lib/librte_security \ + @TOPDIR@/lib/librte_stack \ @TOPDIR@/lib/librte_table \ @TOPDIR@/lib/librte_telemetry \ @TOPDIR@/lib/librte_timer \
doc/guides/prog_guide/index.rst +1 -0
@@ -55,6 +55,7 @@ Programmer's Guide metrics_lib bpf_lib ipsec_lib + stack_lib source_org dev_kit_build_system dev_kit_root_make_help
doc/guides/prog_guide/stack_lib.rst new file +28 -0
@@ -0,0 +1,28 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +Stack Library +============= + +DPDK's stack library provides an API for configuration and use of a bounded +stack of pointers. + +The stack library provides the following basic operations: + +* Create a uniquely named stack of a user-specified size and using a + user-specified socket. + +* Push and pop a burst of one or more stack objects (pointers). These function + are multi-threading safe. + +* Free a previously created stack. + +* Lookup a pointer to a stack by its name. + +* Query a stack's current depth and number of free entries. + +Implementation +~~~~~~~~~~~~~~ + +The stack consists of a contiguous array of pointers, a current index, and a +spinlock. Accesses to the stack are made multi-thread safe by the spinlock.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -121,6 +121,11 @@ New Features Improved testpmd application performance on ARM platform. For ``macswap`` forwarding mode, NEON intrinsics were used to do swap to save CPU cycles. +* **Added Stack API.** + + Added a new stack API for configuration and use of a bounded stack of + pointers. The API provides MT-safe push and pop operations that can operate + on one or more pointers per operation. Removed Items -------------
lib/Makefile +2 -0
@@ -109,6 +109,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_IPSEC) += librte_ipsec DEPDIRS-librte_ipsec := librte_eal librte_mbuf librte_cryptodev librte_security DIRS-$(CONFIG_RTE_LIBRTE_TELEMETRY) += librte_telemetry DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev +DIRS-$(CONFIG_RTE_LIBRTE_STACK) += librte_stack +DEPDIRS-librte_stack := librte_eal ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_stack/Makefile new file +25 -0
@@ -0,0 +1,25 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_stack.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +CFLAGS += -DALLOW_EXPERIMENTAL_API +LDLIBS += -lrte_eal + +EXPORT_MAP := rte_stack_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \ + rte_stack_std.c + +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ + rte_stack_std.h + +include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build new file +8 -0
@@ -0,0 +1,8 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +allow_experimental_apis = true + +version = 1 +sources = files('rte_stack.c', 'rte_stack_std.c') +headers = files('rte_stack.h', 'rte_stack_std.h')
lib/librte_stack/rte_stack.c new file +182 -0
182 lines not on this page · raw message
lib/librte_stack/rte_stack.h new file +209 -0
209 lines not on this page · raw message
lib/librte_stack/rte_stack_pvt.h new file +34 -0
@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#ifndef _RTE_STACK_PVT_H_ +#define _RTE_STACK_PVT_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <rte_log.h> + +extern int stack_logtype; + +#define STACK_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ##level, stack_logtype, "%s(): "fmt "\n", \ + __func__, ##args) + +#define STACK_LOG_ERR(fmt, args...) \ + STACK_LOG(ERR, fmt, ## args) + +#define STACK_LOG_WARN(fmt, args...) \ + STACK_LOG(WARNING, fmt, ## args) + +#define STACK_LOG_INFO(fmt, args...) \ + STACK_LOG(INFO, fmt, ## args) + + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_STACK_PVT_H_ */
lib/librte_stack/rte_stack_std.c new file +26 -0
@@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#include "rte_stack.h" + +void +rte_stack_std_init (struct rte_stack *s) +{ +rte_spinlock_init (&s->stack_std.lock); +} + +ssize_t +rte_stack_std_get_memsize (unsigned int count) +{ +ssize_t sz =sizeof (struct rte_stack ); + + sz +=RTE_CACHE_LINE_ROUNDUP (count *sizeof (void *)); + +/* Add padding to avoid false sharing conflicts caused by +* next-line hardware prefetchers. +*/ + sz +=2 *RTE_CACHE_LINE_SIZE ; + +return sz; +}
lib/librte_stack/rte_stack_std.h new file +121 -0
121 lines not on this page · raw message
lib/librte_stack/rte_stack_version.map new file +9 -0
@@ -0,0 +1,9 @@ +EXPERIMENTAL { + global: + + rte_stack_create; + rte_stack_free; + rte_stack_lookup; + + local: *; +};
lib/meson.build +1 -1
@@ -22,7 +22,7 @@ libraries = [ 'gro', 'gso', 'ip_frag', 'jobstats', 'kni', 'latencystats', 'lpm', 'member', 'power', 'pdump', 'rawdev', - 'reorder', 'sched', 'security', 'vhost', + 'reorder', 'sched', 'security', 'stack', 'vhost', #ipsec lib depends on crypto and security 'ipsec', # add pkt framework libs which use other libs from above
mk/rte.app.mk +1 -0
@@ -87,6 +87,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SECURITY) += -lrte_security _LDLIBS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += -lrte_compressdev _LDLIBS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += -lrte_eventdev _LDLIBS-$(CONFIG_RTE_LIBRTE_RAWDEV) += -lrte_rawdev +_LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_LIBRTE_TIMER) += -lrte_timer _LDLIBS-$(CONFIG_RTE_LIBRTE_MEMPOOL) += -lrte_mempool _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
Signature · 1 line
2.13.6
PATCH v8 2/8 mempool/stack: convert mempool to use rte stack
In reply to Gage EadsThe new rte_stack library is derived from the mempool handler, so this commit removes duplicated code and simplifies the handler by migrating it to this new API.
--- MAINTAINERS | 2 +- drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 93 +++++++++---------------------- 4 files changed, 33 insertions(+), 71 deletions(-)
MAINTAINERS +1 -1
@@ -303,7 +303,6 @@ M: Andrew Rybchenko <[email protected]> F: lib/librte_mempool/ F: drivers/mempool/Makefile F: drivers/mempool/ring/ -F: drivers/mempool/stack/ F: doc/guides/prog_guide/mempool_lib.rst F: app/test/test_mempool* F: app/test/test_func_reentrancy.c
@@ -431,6 +430,7 @@ M: Gage Eads <[email protected]> M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst +F: drivers/mempool/stack/ Memory Pool Drivers
drivers/mempool/stack/Makefile +2 -1
@@ -10,10 +10,11 @@ LIB = librte_mempool_stack.a CFLAGS += -O3 CFLAGS += $(WERROR_FLAGS) +CFLAGS += -DALLOW_EXPERIMENTAL_API # Headers CFLAGS += -I$(RTE_SDK)/lib/librte_mempool -LDLIBS += -lrte_eal -lrte_mempool -lrte_ring +LDLIBS += -lrte_eal -lrte_mempool -lrte_stack EXPORT_MAP := rte_mempool_stack_version.map
drivers/mempool/stack/meson.build +5 -1
@@ -1,4 +1,8 @@ # SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation +# Copyright(c) 2017-2019 Intel Corporation + +allow_experimental_apis = true sources = files('rte_mempool_stack.c') + +deps += ['stack']
drivers/mempool/stack/rte_mempool_stack.c +25 -68
133 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v8 3/8 test/stack: add stack test
In reply to Gage Eadsstack_autotest performs positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.
--- MAINTAINERS | 1 + app/test/Makefile | 2 + app/test/meson.build | 3 + app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 416 insertions(+) create mode 100644 app/test/test_stack.c
MAINTAINERS +1 -0
@@ -431,6 +431,7 @@ M: Olivier Matz <[email protected]> F: lib/librte_stack/ F: doc/guides/prog_guide/stack_lib.rst F: drivers/mempool/stack/ +F: test/test/*stack* Memory Pool Drivers
app/test/Makefile +2 -0
@@ -90,6 +90,8 @@ endif SRCS-y += test_rwlock.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c + SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_racecond.c
app/test/meson.build +3 -0
@@ -95,6 +95,7 @@ test_sources = files('commands.c', 'test_sched.c', 'test_service_cores.c', 'test_spinlock.c', + 'test_stack.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -133,6 +134,7 @@ test_deps = ['acl', 'port', 'reorder', 'ring', + 'stack', 'timer' ]
@@ -174,6 +176,7 @@ fast_parallel_test_names = [ 'rwlock_autotest', 'sched_autotest', 'spinlock_autotest', + 'stack_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
app/test/test_stack.c new file +410 -0
410 lines not on this page · raw message
Signature · 1 line
2.13.6
Re: [PATCH v8 3/8] test/stack: add stack test
03/04/2019 22:50, Gage Eads:
Quoted 5 lines
stack_autotestperforms positive and negative testing of the stack API, and exercises the push and pop datapath functions with all available lcores.Signed-off-by Gage Eads <[email protected]>Reviewed-by Olivier Matz <[email protected]>
Quoted 7 lines
--- MAINTAINERS | 1 + app/test/Makefile | 2 + app/test/meson.build | 3 + app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 416 insertions(+) create mode 100644 app/test/test_stack.c
Another error with Arm:
app/test/test_stack.c:275:2: error: unknown type name 'rte_atomic64_t'
I think you should install an Arm toolchain and run test-meson-builds.sh
Re: [PATCH v8 3/8] test/stack: add stack test
Quoted 28 lines
-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, April 3, 2019 5:41 PM
To: Eads, Gage <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [PATCH v8 3/8] test/stack: add stack test03/04/2019 22:50, Gage Eads:
Quoted 2 lines
stack_autotestperforms positive and negative testing of the stack API, and exercises the push and pop datapath functions with all availablelcores.
Quoted 11 lines
Signed-off-by Gage Eads <[email protected]>Reviewed-by Olivier Matz <[email protected]>--- MAINTAINERS | 1 + app/test/Makefile | 2 + app/test/meson.build | 3 + app/test/test_stack.c | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 416 insertions(+) create mode 100644 app/test/test_stack.c
Quoted 6 lines
Another error with Arm:
app/test/test_stack.c:275:2: error: unknown type name 'rte_atomic64_t'I think you should install an Arm toolchain and run
test-meson-builds.sh
I should've done that a while ago -- it was pretty painless on Ubuntu. Looks like rte_atomic.h is the last missing header.
PATCH v8 4/8 test/stack: add stack perf test
In reply to Gage Eadsstack_perf_autotest tests the following with one lcore:
- Cycles to attempt to pop an empty stack
- Cycles to push then pop a single object
- Cycles to push then pop a burst of 32 objects
It also tests the cycles to push then pop a burst of 8 and 32 objects with the following lcore combinations (if possible):
- Two hyperthreads
- Two physical cores
- Two physical cores on separate NUMA nodes
- All available lcores
--- app/test/Makefile | 1 + app/test/meson.build | 2 + app/test/test_stack_perf.c | 343 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 346 insertions(+) create mode 100644 app/test/test_stack_perf.c
app/test/Makefile +1 -0
@@ -91,6 +91,7 @@ endif SRCS-y += test_rwlock.c SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack.c +SRCS-$(CONFIG_RTE_LIBRTE_STACK) += test_stack_perf.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer.c SRCS-$(CONFIG_RTE_LIBRTE_TIMER) += test_timer_perf.c
app/test/meson.build +2 -0
@@ -96,6 +96,7 @@ test_sources = files('commands.c', 'test_service_cores.c', 'test_spinlock.c', 'test_stack.c', + 'test_stack_perf.c', 'test_string_fns.c', 'test_table.c', 'test_table_acl.c',
@@ -241,6 +242,7 @@ perf_test_names = [ 'distributor_perf_autotest', 'ring_pmd_perf_autotest', 'pmd_perf_autotest', + 'stack_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
app/test/test_stack_perf.c new file +343 -0
343 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v8 5/8 stack: add lock-free stack implementation
In reply to Gage EadsThis commit adds support for a lock-free (linked list based) stack to the stack API. This behavior is selected through a new rte_stack_create() flag, RTE_STACK_F_LF.
The stack consists of a linked list of elements, each containing a data pointer and a next pointer, and an atomic stack depth counter.
The lock-free push operation enqueues a linked list of pointers by pointing the tail of the list to the current stack head, and using a CAS to swing the stack head pointer to the head of the list. The operation retries if it is unsuccessful (i.e. the list changed between reading the head and modifying it), else it adjusts the stack length and returns.
The lock-free pop operation first reserves num elements by adjusting the stack length, to ensure the dequeue operation will succeed without blocking. It then dequeues pointers by walking the list -- starting from the head -- then swinging the head pointer (using a CAS as well). While walking the list, the data pointers are recorded in an object table.
This algorithm stack uses a 128-bit compare-and-swap instruction, which atomically updates the stack top pointer and a modification counter, to protect against the ABA problem.
The linked list elements themselves are maintained in a lock-free LIFO list, and are allocated before stack pushes and freed after stack pops. Since the stack has a fixed maximum depth, these elements do not need to be dynamically created.
--- doc/guides/prog_guide/stack_lib.rst | 61 +++++++++++- doc/guides/rel_notes/release_19_05.rst | 3 + lib/librte_stack/Makefile | 7 +- lib/librte_stack/meson.build | 7 +- lib/librte_stack/rte_stack.c | 28 ++++-- lib/librte_stack/rte_stack.h | 62 +++++++++++- lib/librte_stack/rte_stack_lf.c | 31 ++++++ lib/librte_stack/rte_stack_lf.h | 102 ++++++++++++++++++++ lib/librte_stack/rte_stack_lf_generic.h | 164 ++++++++++++++++++++++++++++++++ 9 files changed, 446 insertions(+), 19 deletions(-) create mode 100644 lib/librte_stack/rte_stack_lf.c create mode 100644 lib/librte_stack/rte_stack_lf.h create mode 100644 lib/librte_stack/rte_stack_lf_generic.h
doc/guides/prog_guide/stack_lib.rst +58 -3
@@ -10,7 +10,8 @@ stack of pointers. The stack library provides the following basic operations: * Create a uniquely named stack of a user-specified size and using a - user-specified socket. + user-specified socket, with either standard (lock-based) or lock-free + behavior. * Push and pop a burst of one or more stack objects (pointers). These function are multi-threading safe.
@@ -24,5 +25,59 @@ The stack library provides the following basic operations: Implementation ~~~~~~~~~~~~~~ -The stack consists of a contiguous array of pointers, a current index, and a -spinlock. Accesses to the stack are made multi-thread safe by the spinlock. +The library supports two types of stacks: standard (lock-based) and lock-free. +Both types use the same set of interfaces, but their implementations differ. + +Lock-based Stack +---------------- + +The lock-based stack consists of a contiguous array of pointers, a current +index, and a spinlock. Accesses to the stack are made multi-thread safe by the +spinlock. + +Lock-free Stack +------------------ + +The lock-free stack consists of a linked list of elements, each containing a +data pointer and a next pointer, and an atomic stack depth counter. The +lock-free property means that multiple threads can push and pop simultaneously, +and one thread being preempted/delayed in a push or pop operation will not +impede the forward progress of any other thread. + +The lock-free push operation enqueues a linked list of pointers by pointing the +list's tail to the current stack head, and using a CAS to swing the stack head +pointer to the head of the list. The operation retries if it is unsuccessful +(i.e. the list changed between reading the head and modifying it), else it +adjusts the stack length and returns. + +The lock-free pop operation first reserves one or more list elements by +adjusting the stack length, to ensure the dequeue operation will succeed +without blocking. It then dequeues pointers by walking the list -- starting +from the head -- then swinging the head pointer (using a CAS as well). While +walking the list, the data pointers are recorded in an object table. + +The linked list elements themselves are maintained in a lock-free LIFO, and are +allocated before stack pushes and freed after stack pops. Since the stack has a +fixed maximum depth, these elements do not need to be dynamically created. + +The lock-free behavior is selected by passing the *RTE_STACK_F_LF* flag to +rte_stack_create(). + +Preventing the ABA Problem +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To prevent the ABA problem, this algorithm stack uses a 128-bit +compare-and-swap instruction to atomically update both the stack top pointer +and a modification counter. The ABA problem can occur without a modification +counter if, for example: + +1. Thread A reads head pointer X and stores the pointed-to list element. +2. Other threads modify the list such that the head pointer is once again X, + but its pointed-to data is different than what thread A read. +3. Thread A changes the head pointer with a compare-and-swap and succeeds. + +In this case thread A would not detect that the list had changed, and would +both pop stale data and incorrect change the head pointer. By adding a +modification counter that is updated on every push and pop as part of the +compare-and-swap, the algorithm can detect when the list changes even if the +head pointer remains the same.
doc/guides/rel_notes/release_19_05.rst +3 -0
@@ -127,6 +127,9 @@ New Features pointers. The API provides MT-safe push and pop operations that can operate on one or more pointers per operation. + The library supports two stack implementations: standard (lock-based) and lock-free. + The lock-free implementation is currently limited to x86-64 platforms. + Removed Items -------------
lib/librte_stack/Makefile +5 -2
@@ -16,10 +16,13 @@ LIBABIVER := 1 # all source are stored in SRCS-y SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \ - rte_stack_std.c + rte_stack_std.c \ + rte_stack_lf.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ - rte_stack_std.h + rte_stack_std.h \ + rte_stack_lf.h \ + rte_stack_lf_generic.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +5 -2
@@ -4,5 +4,8 @@ allow_experimental_apis = true version = 1 -sources = files('rte_stack.c', 'rte_stack_std.c') -headers = files('rte_stack.h', 'rte_stack_std.h') +sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c') +headers = files('rte_stack.h', + 'rte_stack_std.h', + 'rte_stack_lf.h', + 'rte_stack_lf_generic.h')
lib/librte_stack/rte_stack.c +21 -7
@@ -25,18 +25,25 @@ static struct rte_tailq_elem rte_stack_tailq = { };EAL_REGISTER_TAILQ (rte_stack_tailq) + static void -rte_stack_init (struct rte_stack *s) +rte_stack_init (struct rte_stack *s,unsigned int count,uint32_t flags) {memset (s,0 ,sizeof (*s)); -rte_stack_std_init (s); +if (flags &RTE_STACK_F_LF ) +rte_stack_lf_init (s, count); +else +rte_stack_std_init (s); }static ssize_t -rte_stack_get_memsize (unsigned int count) +rte_stack_get_memsize (unsigned int count,uint32_t flags) { -return rte_stack_std_get_memsize (count); +if (flags &RTE_STACK_F_LF ) +return rte_stack_lf_get_memsize (count); +else +return rte_stack_std_get_memsize (count); } struct rte_stack *
@@ -51,9 +58,16 @@ rte_stack_create(const char *name, unsigned int count, int socket_id,unsigned int sz;int ret; -RTE_SET_USED (flags); +#ifdef RTE_ARCH_64 +RTE_BUILD_BUG_ON (sizeof (struct rte_stack_lf_head ) !=16 ); +#else +if (flags &RTE_STACK_F_LF ) { +STACK_LOG_ERR ("Lock-free stack is not supported on your platform\n" ); +return NULL ; + } +#endif - sz =rte_stack_get_memsize (count); + sz =rte_stack_get_memsize (count, flags); ret =snprintf (mz_name,sizeof (mz_name),"%s%s" ,RTE_STACK_MZ_PREFIX , name);
@@ -82,7 +96,7 @@ rte_stack_create(const char *name, unsigned int count, int socket_id, s = mz->addr; -rte_stack_init (s); +rte_stack_init (s, count, flags);/* Store the name for later lookups */ ret =snprintf (s->name,sizeof (s->name),"%s" , name);
lib/librte_stack/rte_stack.h +57 -5
101 lines not on this page · raw message
lib/librte_stack/rte_stack_lf.c new file +31 -0
@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2019 Intel Corporation +*/ + +#include "rte_stack.h" + +void +rte_stack_lf_init (struct rte_stack *s,unsigned int count) +{ +struct rte_stack_lf_elem *elems = s->stack_lf.elems; +unsigned int i; + +for (i =0 ; i < count; i++) +__rte_stack_lf_push_elems (&s->stack_lf.free, + &elems[i], &elems[i],1 ); +} + +ssize_t +rte_stack_lf_get_memsize (unsigned int count) +{ +ssize_t sz =sizeof (struct rte_stack ); + + sz +=RTE_CACHE_LINE_ROUNDUP (count *sizeof (struct rte_stack_lf_elem )); + +/* Add padding to avoid false sharing conflicts caused by +* next-line hardware prefetchers. +*/ + sz +=2 *RTE_CACHE_LINE_SIZE ; + +return sz; +}
lib/librte_stack/rte_stack_lf.h new file +102 -0
102 lines not on this page · raw message
lib/librte_stack/rte_stack_lf_generic.h new file +164 -0
164 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v8 6/8 stack: add C11 atomic implementation
In reply to Gage EadsThis commit adds an implementation of the lock-free stack push, pop, and length functions that use __atomic builtins, for systems that benefit from the finer-grained memory ordering control.
--- lib/librte_stack/Makefile | 3 +- lib/librte_stack/meson.build | 3 +- lib/librte_stack/rte_stack_lf.h | 4 + lib/librte_stack/rte_stack_lf_c11.h | 175 ++++++++++++++++++++++++++++++++++++ 4 files changed, 183 insertions(+), 2 deletions(-) create mode 100644 lib/librte_stack/rte_stack_lf_c11.h
lib/librte_stack/Makefile +2 -1
@@ -23,6 +23,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_STACK) := rte_stack.c \ SYMLINK-$(CONFIG_RTE_LIBRTE_STACK)-include := rte_stack.h \ rte_stack_std.h \ rte_stack_lf.h \ - rte_stack_lf_generic.h + rte_stack_lf_generic.h \ + rte_stack_lf_c11.h include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_stack/meson.build +2 -1
@@ -8,4 +8,5 @@ sources = files('rte_stack.c', 'rte_stack_std.c', 'rte_stack_lf.c') headers = files('rte_stack.h', 'rte_stack_std.h', 'rte_stack_lf.h', - 'rte_stack_lf_generic.h') + 'rte_stack_lf_generic.h', + 'rte_stack_lf_c11.h')
lib/librte_stack/rte_stack_lf.h +4 -0
@@ -5,7 +5,11 @@#ifndef _RTE_STACK_LF_H_#define _RTE_STACK_LF_H_ +#ifdef RTE_USE_C11_MEM_MODEL +#include "rte_stack_lf_c11.h" +#else #include "rte_stack_lf_generic.h" +#endif /** * @internal Push several objects on the lock-free stack (MT-safe).
lib/librte_stack/rte_stack_lf_c11.h new file +175 -0
175 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH v8 7/8 test/stack: add lock-free stack tests
In reply to Gage EadsThis commit adds lock-free stack variants of stack_autotest (stack_lf_autotest) and stack_perf_autotest (stack_lf_perf_autotest), which differ only in that the lock-free versions pass the RTE_STACK_F_LF flag to all rte_stack_create() calls.
--- app/test/meson.build | 2 ++ app/test/test_stack.c | 41 +++++++++++++++++++++++++++-------------- app/test/test_stack_perf.c | 17 +++++++++++++++-- 3 files changed, 44 insertions(+), 16 deletions(-)
app/test/meson.build +2 -0
@@ -178,6 +178,7 @@ fast_parallel_test_names = [ 'sched_autotest', 'spinlock_autotest', 'stack_autotest', + 'stack_nb_autotest', 'string_autotest', 'table_autotest', 'tailq_autotest',
@@ -243,6 +244,7 @@ perf_test_names = [ 'ring_pmd_perf_autotest', 'pmd_perf_autotest', 'stack_perf_autotest', + 'stack_nb_perf_autotest', ] # All test cases in driver_test_names list are non-parallel
app/test/test_stack.c +27 -14
112 lines not on this page · raw message
app/test/test_stack_perf.c +15 -2
@@ -297,14 +297,14 @@ test_bulk_push_pop(struct rte_stack *s) }static int -test_stack_perf (void ) +__test_stack_perf (uint32_t flags) {struct lcore_pair cores;struct rte_stack *s;rte_atomic32_init (&lcore_barrier); - s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (),0 ); + s =rte_stack_create (STACK_NAME ,STACK_SIZE ,rte_socket_id (), flags);if (s ==NULL ) {printf ("[%s():%u] failed to create a stack\n" , __func__, __LINE__);
@@ -340,4 +340,17 @@ test_stack_perf(void)return 0 ; } +static int +test_stack_perf (void ) +{ +return __test_stack_perf (0 ); +} + +static int +test_lf_stack_perf (void ) +{ +return __test_stack_perf (RTE_STACK_F_LF ); +} +REGISTER_TEST_COMMAND (stack_perf_autotest, test_stack_perf); +REGISTER_TEST_COMMAND (stack_lf_perf_autotest, test_lf_stack_perf);
Signature · 1 line
2.13.6
PATCH v8 8/8 mempool/stack: add lock-free stack mempool handler
In reply to Gage EadsThis commit adds support for lock-free (linked list based) stack mempool handler.
In mempool_perf_autotest the lock-based stack outperforms the lock-free handler for certain lcore/alloc count/free count combinations*, however:
- For applications with preemptible pthreads, a standard (lock-based) stack's worst-case performance (i.e. one thread being preempted while holding the spinlock) is much worse than the lock-free stack's.
- Using per-thread mempool caches will largely mitigate the performance difference.
*Test setup: x86_64 build with default config, dual-socket Xeon E5-2699 v4, running on isolcpus cores with a tickless scheduler. The lock-based stack's rate_persec was 0.6x-3.5x the lock-free stack's.
--- doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++ doc/guides/rel_notes/release_19_05.rst | 5 +++++ drivers/mempool/stack/rte_mempool_stack.c | 26 +++++++++++++++++++++++-- 3 files changed, 39 insertions(+), 2 deletions(-)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
@@ -581,6 +581,16 @@ Known Issues 5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR. + Alternatively, applications can use the lock-free stack mempool handler. When + considering this handler, note that: + + - It is currently limited to the x86_64 platform, because it uses an + instruction (16-byte compare-and-swap) that is not yet available on other + platforms. + - It has worse average-case performance than the non-preemptive rte_ring, but + software caching (e.g. the mempool cache) can mitigate this by reducing the + number of stack accesses. + + rte_timer Running ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed.
doc/guides/rel_notes/release_19_05.rst +5 -0
@@ -130,6 +130,11 @@ New Features The library supports two stack implementations: standard (lock-based) and lock-free. The lock-free implementation is currently limited to x86-64 platforms. +* **Added Lock-Free Stack Mempool Handler.** + + Added a new lock-free stack handler, which uses the newly added stack + library. + Removed Items -------------
drivers/mempool/stack/rte_mempool_stack.c +24 -2
@@ -7,7 +7,7 @@#include <rte_stack.h> static int -stack_alloc (struct rte_mempool *mp) +__stack_alloc (struct rte_mempool *mp,uint32_t flags) {char name[RTE_STACK_NAMESIZE ];struct rte_stack *s;
@@ -20,7 +20,7 @@ stack_alloc(struct rte_mempool *mp)return -rte_errno; } - s =rte_stack_create (name, mp->size, mp->socket_id,0 ); + s =rte_stack_create (name, mp->size, mp->socket_id, flags);if (s ==NULL )return -rte_errno;
@@ -30,6 +30,18 @@ stack_alloc(struct rte_mempool *mp) }static int +stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,0 ); +} + +static int +lf_stack_alloc (struct rte_mempool *mp) +{ +return __stack_alloc (mp,RTE_STACK_F_LF ); +} + +static int stack_enqueue (struct rte_mempool *mp,void *const *obj_table,unsigned int n) {
@@ -72,4 +84,14 @@ static struct rte_mempool_ops ops_stack = { .get_count = stack_get_count }; +static struct rte_mempool_ops ops_lf_stack = { + .name ="lf_stack" , + .alloc = lf_stack_alloc, + .free = stack_free, + .enqueue = stack_enqueue, + .dequeue = stack_dequeue, + .get_count = stack_get_count +}; +MEMPOOL_REGISTER_OPS (ops_stack); +MEMPOOL_REGISTER_OPS (ops_lf_stack);
Signature · 1 line
2.13.6
v9 cover Add stack library and new mempool handler
In reply to Gage EadsThis patchset introduces a stack library, supporting both lock-based and lock-free stacks, and a lock-free stack mempool handler.
The lock-based stack code is derived from the existing stack mempool handler, and that handler is refactored to use the stack library.
The lock-free stack mempool handler is intended for usages where the rte ring's "non-preemptive" constraint is not acceptable; for example, if the application uses a mixture of pinned high-priority threads and multiplexed low-priority threads that share a mempool.
Note that the lock-free algorithm relies on a 128-bit compare-and-swap[1], so it is currently limited to the x86_64 platform.
This patchset is the successor to a patchset containing only the new mempool handler[2].
[1] http://mails.dpdk.org/archives/dev/2019-April/129014.html
[2] http://mails.dpdk.org/archives/dev/2019-January/123555.html
---
v9:
- Add rte_atomic.h includes to rte_stack.h, test_stack.c, and test_stack_perf.c to fix ARM builds
v8:
- Add rte_debug.h include to rte_stack.h for RTE_ASSERT()
v7:
- Add rte_branch_prediction.h include to rte_stack_std.h for unlikely()
- Add rte_compat.h include to rte_stack.h for __rte_experimental
v6:
- Add load-acquire fence to the lock-free push function
- Correct generic implementation's pop_elems 128b CAS success and failure memorder to match those in the C11 implementation.
v5:
- Add comment to explain padding in *_get_memsize() functions
- Prefix internal functions with '__'
- Use RTE_ASSERT for performance critical run-time checks
- Don't use __atomic_load in the C11 pop_elems function, and put an acquire thread fence at the start of the 2nd do-while loop
- Change pop_elems 128b CAS success memorder to RELEASE and failure memorder to RELAXED
- Change compile-time assertion to run for all 64-bit architectures
- Reorganize the code with standard and lock-free .c and .h files
v4:
- Fix 32-bit build error in test_stack.c by using %zu format specifier for size_t
- Rebase onto master
v3:
- Rebase patchset onto master (test/test/ -> app/test/)
- Fix rte_stack_std_push() segfault introduced in v2
v2:
- Reworked structure and function naming to use rte_stack_{std, lf}_...
- Updated to the latest rte_atomic128_cmp_exchange() interface.
- Rename STACK_F_NB -> RTE_STACK_F_LF.
- Remove rte_rmb() and rte_wmb() from the generic push and pop implementations. These are obviated by rte_atomic128_cmp_exchange()'s two memorder arguments.
- Edit stack_lib.rst text to 80 chars/line.
- Fix rte_stack.h doxygen formatting.
- Allocate popped_objs array from the heap
- Fix stack_thread_push_pop bug ("&t->sz" -> "t->sz")
- Remove unnecessary NULL check from test_stack_basic
- Properly terminate the name string in test_stack_name_length
- Add an empty array of struct rte_nb_lifo_elem elements
- In rte_nb_lifo_push(), retrieve the last element from __nb_lifo_pop()
- Split C11 implementation into a separate patchset
Gage Eads (8):
stack: introduce rte stack library mempool/stack: convert mempool to use rte stack test/stack: add stack test
test/stack: add stack perf test
stack: add lock-free stack implementation
stack: add C11 atomic implementation test/stack: add lock-free stack tests mempool/stack: add lock-free stack mempool handler
MAINTAINERS | 9 +- app/test/Makefile | 3 + app/test/meson.build | 7 + app/test/test_stack.c | 424 ++++++++++++++++++++++++ app/test/test_stack_perf.c | 358 ++++++++++++++++++++ config/common_base | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/prog_guide/env_abstraction_layer.rst | 10 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/stack_lib.rst | 83 +++++ doc/guides/rel_notes/release_19_05.rst | 13 + drivers/mempool/stack/Makefile | 3 +- drivers/mempool/stack/meson.build | 6 +- drivers/mempool/stack/rte_mempool_stack.c | 115 +++---- lib/Makefile | 2 + lib/librte_stack/Makefile | 29 ++ lib/librte_stack/meson.build | 12 + lib/librte_stack/rte_stack.c | 196 +++++++++++ lib/librte_stack/rte_stack.h | 262 +++++++++++++++ lib/librte_stack/rte_stack_lf.c | 31 ++ lib/librte_stack/rte_stack_lf.h | 106 ++++++ lib/librte_stack/rte_stack_lf_c11.h | 175 ++++++++++ lib/librte_stack/rte_stack_lf_generic.h | 164 +++++++++ lib/librte_stack/rte_stack_pvt.h | 34 ++ lib/librte_stack/rte_stack_std.c | 26 ++ lib/librte_stack/rte_stack_std.h | 121 +++++++ lib/librte_stack/rte_stack_version.map | 9 + lib/meson.build | 2 +- mk/rte.app.mk | 1 + 30 files changed, 2137 insertions(+), 72 deletions(-) create mode 100644 app/test/test_stack.c create mode 100644 app/test/test_stack_perf.c create mode 100644 doc/guides/prog_guide/stack_lib.rst create mode 100644 lib/librte_stack/Makefile create mode 100644 lib/librte_stack/meson.build create mode 100644 lib/librte_stack/rte_stack.c create mode 100644 lib/librte_stack/rte_stack.h create mode 100644 lib/librte_stack/rte_stack_lf.c create mode 100644 lib/librte_stack/rte_stack_lf.h create mode 100644 lib/librte_stack/rte_stack_lf_c11.h create mode 100644 lib/librte_stack/rte_stack_lf_generic.h create mode 100644 lib/librte_stack/rte_stack_pvt.h create mode 100644 lib/librte_stack/rte_stack_std.c create mode 100644 lib/librte_stack/rte_stack_std.h create mode 100644 lib/librte_stack/rte_stack_version.map
Signature · 1 line
2.13.6