Skip to contentSkip to conversation

examples: add performance thread sample application

PATCH Superseded ibetts 1 participant 2 messages Dec 2, 2015

v4 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

ibetts

author
#0 lore raw

PATCH v4 1/2 examples: add performance thread sample application

From: Ian Betts <[email protected]>

This example comprises a layer 3 forwarding derivative intended to facilitate characterization of performance with different threading models, specifically:-

1. EAL threads running on different physical cores
2. EAL threads running on the same physical core
3. Lightweight threads running in an EAL thread

Purpose and justification

Since dpdk 2.0 it has been possible to assign multiple EAL threads to a physical core ( case 2 above ).
Currently no example application has focused on demonstrating the performance constraints of differing threading models.

Whilst purpose built applications that fully comprehend the DPDK single threaded programming model will always yield superior performance, the desire to preserve ROI in legacy code written for multithreaded operating environments makes lightweight threads (case 3 above) worthy of consideration.

As well as aiding with legacy code reuse, it is anticipated that lightweight threads will make it possible to scale a multithreaded application with fine granularity allowing an application to more easily take advantage of headroom on EAL cores, or conversely occupy more cores, as dictated by system load.

To explore performance with lightweight threads a simple cooperative scheduler subsystem is being included in this example application. If the expected benefits and use cases prove to be of value, it is anticipated that this lightweight thread subsystem would become a library in some future DPDK release.

Changes in this version:-
* Copyright updated for 2015
* fix TLS destructor handling

---
 config/common_linuxapp                             |    1 +
 config/defconfig_i686-native-linuxapp-gcc          |    1 +
 config/defconfig_i686-native-linuxapp-icc          |    1 +
 config/defconfig_x86_64-native-linuxapp-gcc        |    3 +
 config/defconfig_x86_64-native-linuxapp-icc        |    3 +
 doc/guides/sample_app_ug/performance_thread.rst    | 1149 ++++++
 examples/Makefile                                  |    2 +
 examples/performance-thread/Makefile               |   45 +
 .../performance-thread/common/arch/x86/atomic.h    |   59 +
 examples/performance-thread/common/arch/x86/ctx.c  |   93 +
 examples/performance-thread/common/arch/x86/ctx.h  |   57 +
 examples/performance-thread/common/common.mk       |   40 +
 examples/performance-thread/common/lthread.c       |  530 +++
 examples/performance-thread/common/lthread.h       |   99 +
 examples/performance-thread/common/lthread_api.h   |  829 +++++
 examples/performance-thread/common/lthread_cond.c  |  241 ++
 examples/performance-thread/common/lthread_cond.h  |   77 +
 examples/performance-thread/common/lthread_diag.c  |  321 ++
 examples/performance-thread/common/lthread_diag.h  |  129 +
 .../performance-thread/common/lthread_diag_api.h   |  319 ++
 examples/performance-thread/common/lthread_int.h   |  212 ++
 examples/performance-thread/common/lthread_mutex.c |  256 ++
 examples/performance-thread/common/lthread_mutex.h |   52 +
 .../performance-thread/common/lthread_objcache.h   |  160 +
 examples/performance-thread/common/lthread_pool.h  |  333 ++
 examples/performance-thread/common/lthread_queue.h |  303 ++
 examples/performance-thread/common/lthread_sched.c |  600 ++++
 examples/performance-thread/common/lthread_sched.h |  152 +
 examples/performance-thread/common/lthread_timer.h |   47 +
 examples/performance-thread/common/lthread_tls.c   |  254 ++
 examples/performance-thread/common/lthread_tls.h   |   57 +
 examples/performance-thread/l3fwd-thread/Makefile  |   57 +
 examples/performance-thread/l3fwd-thread/main.c    | 3641 ++++++++++++++++++++
 33 files changed, 10123 insertions(+)
 create mode 100644 doc/guides/sample_app_ug/performance_thread.rst
 create mode 100644 examples/performance-thread/Makefile
 create mode 100644 examples/performance-thread/common/arch/x86/atomic.h
 create mode 100644 examples/performance-thread/common/arch/x86/ctx.c
 create mode 100644 examples/performance-thread/common/arch/x86/ctx.h
 create mode 100644 examples/performance-thread/common/common.mk
 create mode 100644 examples/performance-thread/common/lthread.c
 create mode 100644 examples/performance-thread/common/lthread.h
 create mode 100644 examples/performance-thread/common/lthread_api.h
 create mode 100644 examples/performance-thread/common/lthread_cond.c
 create mode 100644 examples/performance-thread/common/lthread_cond.h
 create mode 100644 examples/performance-thread/common/lthread_diag.c
 create mode 100644 examples/performance-thread/common/lthread_diag.h
 create mode 100644 examples/performance-thread/common/lthread_diag_api.h
 create mode 100644 examples/performance-thread/common/lthread_int.h
 create mode 100644 examples/performance-thread/common/lthread_mutex.c
 create mode 100644 examples/performance-thread/common/lthread_mutex.h
 create mode 100644 examples/performance-thread/common/lthread_objcache.h
 create mode 100644 examples/performance-thread/common/lthread_pool.h
 create mode 100644 examples/performance-thread/common/lthread_queue.h
 create mode 100644 examples/performance-thread/common/lthread_sched.c
 create mode 100644 examples/performance-thread/common/lthread_sched.h
 create mode 100644 examples/performance-thread/common/lthread_timer.h
 create mode 100644 examples/performance-thread/common/lthread_tls.c
 create mode 100644 examples/performance-thread/common/lthread_tls.h
 create mode 100644 examples/performance-thread/l3fwd-thread/Makefile
 create mode 100644 examples/performance-thread/l3fwd-thread/main.c
config/common_linuxapp +1 -0
index 2866986..95da485 100644
@@ -526,3 +526,4 @@ CONFIG_RTE_APP_TEST=y
 CONFIG_RTE_TEST_PMD=y
 CONFIG_RTE_TEST_PMD_RECORD_CORE_CYCLES=n
 CONFIG_RTE_TEST_PMD_RECORD_BURST_STATS=n
+
config/defconfig_i686-native-linuxapp-gcc +1 -0
index a90de9b..c101942 100644
@@ -49,3 +49,4 @@ CONFIG_RTE_LIBRTE_KNI=n
 # Vectorized PMD is not supported on 32-bit
 #
 CONFIG_RTE_IXGBE_INC_VECTOR=n
+
config/defconfig_i686-native-linuxapp-icc +1 -0
index c021321..915bfa0 100644
@@ -49,3 +49,4 @@ CONFIG_RTE_LIBRTE_KNI=n
 # Vectorized PMD is not supported on 32-bit
 #
 CONFIG_RTE_IXGBE_INC_VECTOR=n
+
config/defconfig_x86_64-native-linuxapp-gcc +3 -0
index 60baf5b..76d6b10 100644
@@ -40,3 +40,6 @@ CONFIG_RTE_ARCH_64=y
 
 CONFIG_RTE_TOOLCHAIN="gcc"
 CONFIG_RTE_TOOLCHAIN_GCC=y
+
+CONFIG_RTE_PERFORMANCE_THREAD=y
+
config/defconfig_x86_64-native-linuxapp-icc +3 -0
index 71d1e28..58a1e09 100644
@@ -40,3 +40,6 @@ CONFIG_RTE_ARCH_64=y
 
 CONFIG_RTE_TOOLCHAIN="icc"
 CONFIG_RTE_TOOLCHAIN_ICC=y
+
+CONFIG_RTE_PERFORMANCE_THREAD=y
+
doc/guides/sample_app_ug/performance_thread.rst new file +1149 -0

1,149 lines not on this page · raw message

examples/Makefile +2 -0
index 5dd2c53..aeb10f3 100644
@@ -77,5 +77,7 @@ DIRS-y += vmdq
 DIRS-y += vmdq_dcb
 DIRS-$(CONFIG_RTE_LIBRTE_POWER) += vm_power_manager
 DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += l2fwd-crypto
+DIRS-$(CONFIG_RTE_PERFORMANCE_THREAD) += performance-thread
+
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
examples/performance-thread/Makefile new file +45 -0

45 lines not on this page · raw message

examples/performance-thread/common/arch/x86/atomic.h new file +59 -0

59 lines not on this page · raw message

examples/performance-thread/common/arch/x86/ctx.c new file +93 -0

93 lines not on this page · raw message

examples/performance-thread/common/arch/x86/ctx.h new file +57 -0

57 lines not on this page · raw message

examples/performance-thread/common/common.mk new file +40 -0
new file mode 100644
index 0000000..6e0bc87
@@ -0,0 +1,40 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2014 6WIND S.A.
+#
+#   Redistribution and use in source and binary forms, with or without
+#   modification, are permitted provided that the following conditions
+#   are met:
+#
+#     * Redistributions of source code must retain the above copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of 6WIND S.A. nor the names of its
+#       contributors may be used to endorse or promote products derived
+#       from this software without specific prior written permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+# list the C files belonhing to the lthread subsystem, these are common to all lthread apps
+SRCS-y +=	../common/lthread.c \
+			../common/lthread_sched.c \
+			../common/lthread_cond.c \
+			../common/lthread_tls.c \
+			../common/lthread_mutex.c \
+			../common/lthread_diag.c \
+			../common/arch/x86/ctx.c
+
+INCLUDES += -I$(RTE_SDK)/examples/performance-thread/common/ -I$(RTE_SDK)/examples/performance-thread/common/arch/x86/
examples/performance-thread/common/lthread.c new file +530 -0

530 lines not on this page · raw message

examples/performance-thread/common/lthread.h new file +99 -0

99 lines not on this page · raw message

examples/performance-thread/common/lthread_api.h new file +829 -0

829 lines not on this page · raw message

examples/performance-thread/common/lthread_cond.c new file +241 -0

68 lines not on this page · raw message

examples/performance-thread/common/lthread_cond.h new file +77 -0

77 lines not on this page · raw message

examples/performance-thread/common/lthread_diag.c new file +321 -0

321 lines not on this page · raw message

examples/performance-thread/common/lthread_diag.h new file +129 -0

129 lines not on this page · raw message

examples/performance-thread/common/lthread_diag_api.h new file +319 -0

319 lines not on this page · raw message

examples/performance-thread/common/lthread_int.h new file +212 -0

212 lines not on this page · raw message

examples/performance-thread/common/lthread_mutex.c new file +256 -0

256 lines not on this page · raw message

examples/performance-thread/common/lthread_mutex.h new file +52 -0

52 lines not on this page · raw message

examples/performance-thread/common/lthread_objcache.h new file +160 -0

160 lines not on this page · raw message

examples/performance-thread/common/lthread_pool.h new file +333 -0

333 lines not on this page · raw message

examples/performance-thread/common/lthread_queue.h new file +303 -0

303 lines not on this page · raw message

examples/performance-thread/common/lthread_sched.c new file +600 -0

600 lines not on this page · raw message

examples/performance-thread/common/lthread_sched.h new file +152 -0

152 lines not on this page · raw message

examples/performance-thread/common/lthread_timer.h new file +47 -0

47 lines not on this page · raw message

examples/performance-thread/common/lthread_tls.c new file +254 -0

254 lines not on this page · raw message

examples/performance-thread/common/lthread_tls.h new file +57 -0

57 lines not on this page · raw message

examples/performance-thread/l3fwd-thread/Makefile new file +57 -0

57 lines not on this page · raw message

examples/performance-thread/l3fwd-thread/main.c new file +3641 -0

3,641 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.1.4

Part of the diff is not shown here. View the raw message.

author
#1 lore raw

PATCH v4 2/2 examples: add pthread-shim in performance-thread sample app

From: Ian Betts <[email protected]>

This commit adds a simple pthread_shim example for the cooperative scheduler included with this patchset.

The shim demonstrates a way in which legacy code writtem for pthreads could be adapted to lighweight threads.

---
 doc/guides/sample_app_ug/performance_thread.rst    | 114 ++++
 examples/performance-thread/Makefile               |   2 +
 examples/performance-thread/pthread_shim/Makefile  |  60 ++
 examples/performance-thread/pthread_shim/main.c    | 284 ++++++++
 .../performance-thread/pthread_shim/pthread_shim.c | 714 +++++++++++++++++++++
 .../performance-thread/pthread_shim/pthread_shim.h | 113 ++++
 6 files changed, 1287 insertions(+)
 create mode 100644 examples/performance-thread/pthread_shim/Makefile
 create mode 100644 examples/performance-thread/pthread_shim/main.c
 create mode 100644 examples/performance-thread/pthread_shim/pthread_shim.c
 create mode 100644 examples/performance-thread/pthread_shim/pthread_shim.h
doc/guides/sample_app_ug/performance_thread.rst +114 -0
index 6ea83cc..d71bb84 100644
@@ -1102,6 +1102,120 @@ it the local data it needs, and pick up the new logical core specific values
 from pthread local storage at its new home.
 
 
+.. _pthread_shim:
+
+Pthread shim
+~~~~~~~~~~~~
+
+A convenient way to get something working with legacy code can be to use a
+shim that adapts pthread API calls to the corresponding L-thread ones.
+This approach will not mitigate any of the porting considerations mentioned
+in the previous sections, but it will reduce the amount of code churn that
+would otherwise been involved. It is a reasonable approach to evaluate
+L-threads, before investing effort in porting to the native L-thread APIs.
+
+
+Overview
+^^^^^^^^
+The L-thread subsystem includes an example pthread shim. This is a partial
+implementation but does contain the API stubs needed to get basic applications
+running. There is a simple "hello world" application that demonstrates the
+use of the pthread shim.
+
+A subtlety of working with a shim is that the application will still need
+to make use of the genuine pthread library functions, at the very least in
+order to create the EAL threads in which the L-thread schedulers will run.
+This is the case with DPDK initialization, and exit.
+
+To deal with the initialization and shutdown scenarios, the shim is capable of
+switching on or off its adaptor functionality, an application can control this
+behavior by the calling the function ``pt_override_set()``. The default state
+is disabled.
+
+The pthread shim uses the dynamic linker loader and saves the loaded addresses
+of the genuine pthread API functions in an internal table, when the shim
+functionality is enabled it performs the adaptor function, when disabled it
+invokes the genuine pthread function.
+
+The function ``pthread_exit()`` has additional special handling. The standard
+system header file pthread.h declares ``pthread_exit()`` with
+``__attribute__((noreturn))`` this is an optimization that is possible because
+the pthread is terminating and this enables the compiler to omit the normal
+handling of stack and protection of registers since the function is not
+expected to return, and in fact the thread is being destroyed. These
+optimizations are applied in both the callee and the caller of the
+``pthread_exit()`` function.
+
+In our cooperative scheduling environment this behavior is inadmissible. The
+pthread is the L-thread scheduler thread, and, although an L-thread is
+terminating, there must be a return to the scheduler in order that the system
+can continue to run. Further, returning from a function with attribute
+``noreturn`` is invalid and may result in undefined behavior.
+
+The solution is to redefine the ``pthread_exit`` function with a macro,
+causing it to be mapped to a stub function in the shim that does not have the
+``noreturn`` attribute. This macro is defined in the file
+``pthread_shim.h``. The stub function is otherwise no different than any of
+the other stub functions in the shim, and will switch between the real
+``pthread_exit()`` function or the ``lthread_exit()`` function as
+required. The only difference is that the mapping to the stub by macro
+substitution.
+
+A consequence of this is that the file ``pthread_shim.h`` must be included in
+legacy code wishing to make use of the shim. It also means that dynamic
+linkage of a pre-compiled binary that did not include pthread_shim.h is not be
+supported.
+
+Given the requirements for porting legacy code outlined in
+:ref:`porting_legacy_code_to_run_on_lthreads` most applications will require at
+least some minimal adjustment and recompilation to run on L-threads so
+pre-compiled binaries are unlikely to be met in practice.
+
+In summary the shim approach adds some overhead but can be a useful tool to help
+establish the feasibility of a code reuse project. It is also a fairly
+straightforward task to extend the shim if necessary.
+
+**Note:** Bearing in mind the preceding discussions about the impact of making
+blocking calls then switching the shim in and out on the fly to invoke any
+pthread API this might block is something that should typically be avoided.
+
+
+Building and running the pthread shim
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The shim example application is located in the sample application
+in the performance-thread folder
+
+To build and run the pthread shim example
+
+#. Go to the example applications folder
+
+   .. code-block:: console
+
+       export RTE_SDK=/path/to/rte_sdk
+       cd ${RTE_SDK}/examples/performance-thread/pthread_shim
+
+
+#. Set the target (a default target is used if not specified). For example:
+
+   .. code-block:: console
+
+       export RTE_TARGET=x86_64-native-linuxapp-gcc
+
+   See the DPDK Getting Started Guide for possible RTE_TARGET values.
+
+#. Build the application:
+
+   .. code-block:: console
+
+       make
+
+#. To run the pthread_shim example
+
+   .. code-block:: console
+
+       lthread-pthread-shim -c core_mask -n number_of_channels
+
 .. _lthread_diagnostics:
 
 L-thread Diagnostics
examples/performance-thread/Makefile +2 -0
index fce4f79..1be67e9 100644
@@ -39,6 +39,8 @@ RTE_TARGET ?= x86_64-native-linuxapp-gcc
 include $(RTE_SDK)/mk/rte.vars.mk
 
 DIRS-$(CONFIG_RTE_PERFORMANCE_THREAD) += l3fwd-thread
+DIRS-$(CONFIG_RTE_PERFORMANCE_THREAD) += pthread_shim
+
 
 
 
examples/performance-thread/pthread_shim/Makefile new file +60 -0

60 lines not on this page · raw message

examples/performance-thread/pthread_shim/main.c new file +284 -0

284 lines not on this page · raw message

examples/performance-thread/pthread_shim/pthread_shim.c new file +714 -0

714 lines not on this page · raw message

examples/performance-thread/pthread_shim/pthread_shim.h new file +113 -0

113 lines not on this page · raw message

Signature · 1 line
2.1.4