Skip to contentSkip to conversation

examples: add performance thread sample application

PATCH Superseded ibetts 3 participants 6 messages Nov 17, 2015 – Nov 18, 2015

v3 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

ibetts

author
#0 lore raw

PATCH v3 1/2 examples: add performance thread sample application

From: Ian Betts <[email protected]>

This example comprises a layer 3 forwarding derivative intended to facilitate characterization of performance with different threading models, specifically:-

1. EAL threads running on different physical cores
2. EAL threads running on the same physical core
3. Lightweight threads running in an EAL thread

Purpose and justification

Since dpdk 2.0 it has been possible to assign multiple EAL threads to a physical core ( case 2 above ).
Currently no example application has focused on demonstrating the performance constraints of differing threading models.

Whilst purpose built applications that fully comprehend the DPDK single threaded programming model will always yield superior performance, the desire to preserve ROI in legacy code written for multithreaded operating environments makes lightweight threads (case 3 above) worthy of consideration.

As well as aiding with legacy code reuse, it is anticipated that lightweight threads will make it possible to scale a multithreaded application with fine granularity allowing an application to more easily take advantage of headroom on EAL cores, or conversely occupy more cores, as dictated by system load.

To explore performance with lightweight threads a simple cooperative scheduler subsystem is being included in this example application. If the expected benefits and use cases prove to be of value, it is anticipated that this lightweight thread subsystem would become a library in some future DPDK release.

---
 config/common_linuxapp                             |    5 +
 config/defconfig_i686-native-linuxapp-gcc          |    5 +
 config/defconfig_i686-native-linuxapp-icc          |    5 +
 config/defconfig_x86_x32-native-linuxapp-gcc       |    5 +
 doc/guides/rel_notes/release_2_2.rst               |   11 +
 .../sample_app_ug/img/performance_thread_1.svg     |  799 +++++
 .../sample_app_ug/img/performance_thread_2.svg     |  803 +++++
 doc/guides/sample_app_ug/index.rst                 |    1 +
 doc/guides/sample_app_ug/performance_thread.rst    | 1150 +++++++
 examples/Makefile                                  |    1 +
 examples/performance-thread/Makefile               |   44 +
 .../performance-thread/common/arch/x86/atomic.h    |   59 +
 examples/performance-thread/common/arch/x86/ctx.c  |   93 +
 examples/performance-thread/common/arch/x86/ctx.h  |   57 +
 examples/performance-thread/common/common.mk       |   40 +
 examples/performance-thread/common/lthread.c       |  546 +++
 examples/performance-thread/common/lthread.h       |   99 +
 examples/performance-thread/common/lthread_api.h   |  822 +++++
 examples/performance-thread/common/lthread_cond.c  |  240 ++
 examples/performance-thread/common/lthread_cond.h  |   77 +
 examples/performance-thread/common/lthread_diag.c  |  321 ++
 examples/performance-thread/common/lthread_diag.h  |  129 +
 .../performance-thread/common/lthread_diag_api.h   |  319 ++
 examples/performance-thread/common/lthread_int.h   |  212 ++
 examples/performance-thread/common/lthread_mutex.c |  255 ++
 examples/performance-thread/common/lthread_mutex.h |   52 +
 .../performance-thread/common/lthread_objcache.h   |  160 +
 examples/performance-thread/common/lthread_pool.h  |  333 ++
 examples/performance-thread/common/lthread_queue.h |  303 ++
 examples/performance-thread/common/lthread_sched.c |  598 ++++
 examples/performance-thread/common/lthread_sched.h |  152 +
 examples/performance-thread/common/lthread_timer.h |   47 +
 examples/performance-thread/common/lthread_tls.c   |  242 ++
 examples/performance-thread/common/lthread_tls.h   |   64 +
 examples/performance-thread/l3fwd-thread/Makefile  |   57 +
 examples/performance-thread/l3fwd-thread/main.c    | 3628 ++++++++++++++++++++
 36 files changed, 11734 insertions(+)
 create mode 100644 doc/guides/sample_app_ug/img/performance_thread_1.svg
 create mode 100644 doc/guides/sample_app_ug/img/performance_thread_2.svg
 create mode 100644 doc/guides/sample_app_ug/performance_thread.rst
 create mode 100644 examples/performance-thread/Makefile
 create mode 100644 examples/performance-thread/common/arch/x86/atomic.h
 create mode 100644 examples/performance-thread/common/arch/x86/ctx.c
 create mode 100644 examples/performance-thread/common/arch/x86/ctx.h
 create mode 100644 examples/performance-thread/common/common.mk
 create mode 100644 examples/performance-thread/common/lthread.c
 create mode 100644 examples/performance-thread/common/lthread.h
 create mode 100644 examples/performance-thread/common/lthread_api.h
 create mode 100644 examples/performance-thread/common/lthread_cond.c
 create mode 100644 examples/performance-thread/common/lthread_cond.h
 create mode 100644 examples/performance-thread/common/lthread_diag.c
 create mode 100644 examples/performance-thread/common/lthread_diag.h
 create mode 100644 examples/performance-thread/common/lthread_diag_api.h
 create mode 100644 examples/performance-thread/common/lthread_int.h
 create mode 100644 examples/performance-thread/common/lthread_mutex.c
 create mode 100644 examples/performance-thread/common/lthread_mutex.h
 create mode 100644 examples/performance-thread/common/lthread_objcache.h
 create mode 100644 examples/performance-thread/common/lthread_pool.h
 create mode 100644 examples/performance-thread/common/lthread_queue.h
 create mode 100644 examples/performance-thread/common/lthread_sched.c
 create mode 100644 examples/performance-thread/common/lthread_sched.h
 create mode 100644 examples/performance-thread/common/lthread_timer.h
 create mode 100644 examples/performance-thread/common/lthread_tls.c
 create mode 100644 examples/performance-thread/common/lthread_tls.h
 create mode 100644 examples/performance-thread/l3fwd-thread/Makefile
 create mode 100644 examples/performance-thread/l3fwd-thread/main.c
config/common_linuxapp +5 -0
index 52173d5..dec5950 100644
@@ -481,3 +481,8 @@ CONFIG_RTE_APP_TEST=y
 CONFIG_RTE_TEST_PMD=y
 CONFIG_RTE_TEST_PMD_RECORD_CORE_CYCLES=n
 CONFIG_RTE_TEST_PMD_RECORD_BURST_STATS=n
+
+#
+# Compile the performance thread sample application
+#
+CONFIG_RTE_PERFORMANCE_THREAD=y
config/defconfig_i686-native-linuxapp-gcc +5 -0
index a90de9b..06c9ad0 100644
@@ -49,3 +49,8 @@ CONFIG_RTE_LIBRTE_KNI=n
 # Vectorized PMD is not supported on 32-bit
 #
 CONFIG_RTE_IXGBE_INC_VECTOR=n
+
+#
+# Performance thread sample application is not supported on 32-bit
+#
+CONFIG_RTE_PERFORMANCE_THREAD=n
config/defconfig_i686-native-linuxapp-icc +5 -0
index c021321..e4610d4 100644
@@ -49,3 +49,8 @@ CONFIG_RTE_LIBRTE_KNI=n
 # Vectorized PMD is not supported on 32-bit
 #
 CONFIG_RTE_IXGBE_INC_VECTOR=n
+
+#
+# Performance thread sample application is not supported on 32-bit
+#
+CONFIG_RTE_PERFORMANCE_THREAD=n
config/defconfig_x86_x32-native-linuxapp-gcc +5 -0
index fb0afc4..5680328 100644
@@ -44,3 +44,8 @@ CONFIG_RTE_TOOLCHAIN_GCC=y
 # KNI is not supported on 32-bit
 #
 CONFIG_RTE_LIBRTE_KNI=n
+
+#
+# Performance thread sample application is not supported on 32-bit
+#
+CONFIG_RTE_PERFORMANCE_THREAD=n
doc/guides/rel_notes/release_2_2.rst +11 -0
index 0781ae6..6649b5f 100644
@@ -207,6 +207,12 @@ Libraries
 Examples
 ~~~~~~~~
 
+* **examples: Introducing a performance thread example**
+
+  This an l3fwd derivative focused to enable characterization of performance
+  with different threading models, including multiple EAL threads per physical
+  core, and multiple Lightweight threads running in an EAL thread.
+  The examples includes a simple cooperative scheduler.
 
 Other
 ~~~~~
@@ -214,6 +220,11 @@ Other
 
 Known Issues
 ------------
+* When running the performance thread application in configurations with more than
+  two EAL threads per phsycial core, then forwarding throughput may drop suddenly
+  to a low level. Stopping and restarting traffic restores correct operation.
+  This problem does not occur when running with multiple lightweight threads per
+  physical core.
 
 
 API Changes
doc/guides/sample_app_ug/img/performance_thread_1.svg new file +799 -0

799 lines not on this page · raw message

doc/guides/sample_app_ug/img/performance_thread_2.svg new file +803 -0

803 lines not on this page · raw message

doc/guides/sample_app_ug/index.rst +1 -0
index 8ae86c0..a0449a0 100644
@@ -74,6 +74,7 @@ Sample Applications User Guide
     tep_termination
     proc_info
     ptpclient
+    performance_thread
 
 **Figures**
 
doc/guides/sample_app_ug/performance_thread.rst new file +1150 -0

1,150 lines not on this page · raw message

examples/Makefile +1 -0
index 830e31a..5f94f7f 100644
@@ -75,5 +75,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_XEN_DOM0) += vhost_xen
 DIRS-y += vmdq
 DIRS-y += vmdq_dcb
 DIRS-$(CONFIG_RTE_LIBRTE_POWER) += vm_power_manager
+DIRS-$(CONFIG_RTE_PERFORMANCE_THREAD) += performance-thread
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
examples/performance-thread/Makefile new file +44 -0

44 lines not on this page · raw message

examples/performance-thread/common/arch/x86/atomic.h new file +59 -0

59 lines not on this page · raw message

examples/performance-thread/common/arch/x86/ctx.c new file +93 -0

77 lines not on this page · raw message

examples/performance-thread/common/arch/x86/ctx.h new file +57 -0

57 lines not on this page · raw message

examples/performance-thread/common/common.mk new file +40 -0
new file mode 100644
index 0000000..6e0bc87

Lines of this file are not shown. View the raw message.

examples/performance-thread/common/lthread.c new file +546 -0

546 lines not on this page · raw message

examples/performance-thread/common/lthread.h new file +99 -0

99 lines not on this page · raw message

examples/performance-thread/common/lthread_api.h new file +822 -0

822 lines not on this page · raw message

examples/performance-thread/common/lthread_cond.c new file +240 -0

240 lines not on this page · raw message

examples/performance-thread/common/lthread_cond.h new file +77 -0

77 lines not on this page · raw message

examples/performance-thread/common/lthread_diag.c new file +321 -0

321 lines not on this page · raw message

examples/performance-thread/common/lthread_diag.h new file +129 -0

129 lines not on this page · raw message

examples/performance-thread/common/lthread_diag_api.h new file +319 -0

319 lines not on this page · raw message

examples/performance-thread/common/lthread_int.h new file +212 -0

212 lines not on this page · raw message

examples/performance-thread/common/lthread_mutex.c new file +255 -0

255 lines not on this page · raw message

examples/performance-thread/common/lthread_mutex.h new file +52 -0

52 lines not on this page · raw message

examples/performance-thread/common/lthread_objcache.h new file +160 -0

160 lines not on this page · raw message

examples/performance-thread/common/lthread_pool.h new file +333 -0

333 lines not on this page · raw message

examples/performance-thread/common/lthread_queue.h new file +303 -0

303 lines not on this page · raw message

examples/performance-thread/common/lthread_sched.c new file +598 -0

598 lines not on this page · raw message

examples/performance-thread/common/lthread_sched.h new file +152 -0

152 lines not on this page · raw message

examples/performance-thread/common/lthread_timer.h new file +47 -0

47 lines not on this page · raw message

examples/performance-thread/common/lthread_tls.c new file +242 -0

242 lines not on this page · raw message

examples/performance-thread/common/lthread_tls.h new file +64 -0

64 lines not on this page · raw message

examples/performance-thread/l3fwd-thread/Makefile new file +57 -0

57 lines not on this page · raw message

examples/performance-thread/l3fwd-thread/main.c new file +3628 -0

3,628 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
1.9.3

Part of the diff is not shown here. View the raw message.

author
#1 lore raw

PATCH v3 2/2 examples: add pthread-shim in performance-thread sample app

From: Ian Betts <[email protected]>

This commit adds a simple pthread_shim example for the cooperative scheduler included with this patchset.

The shim demonstrates a way in which legacy code writtem for pthreads could be adapted to lighweight threads.

---
 doc/guides/sample_app_ug/performance_thread.rst    | 113 ++++
 examples/performance-thread/Makefile               |   1 +
 examples/performance-thread/pthread_shim/Makefile  |  60 ++
 examples/performance-thread/pthread_shim/main.c    | 284 ++++++++
 .../performance-thread/pthread_shim/pthread_shim.c | 714 +++++++++++++++++++++
 .../performance-thread/pthread_shim/pthread_shim.h | 113 ++++
 6 files changed, 1285 insertions(+)
 create mode 100644 examples/performance-thread/pthread_shim/Makefile
 create mode 100644 examples/performance-thread/pthread_shim/main.c
 create mode 100644 examples/performance-thread/pthread_shim/pthread_shim.c
 create mode 100644 examples/performance-thread/pthread_shim/pthread_shim.h
doc/guides/sample_app_ug/performance_thread.rst +113 -0
index cc46fb9..d71bb84 100644
@@ -1102,6 +1102,119 @@ it the local data it needs, and pick up the new logical core specific values
 from pthread local storage at its new home.
 
 
+.. _pthread_shim:
+
+Pthread shim
+~~~~~~~~~~~~
+
+A convenient way to get something working with legacy code can be to use a
+shim that adapts pthread API calls to the corresponding L-thread ones.
+This approach will not mitigate any of the porting considerations mentioned
+in the previous sections, but it will reduce the amount of code churn that
+would otherwise been involved. It is a reasonable approach to evaluate
+L-threads, before investing effort in porting to the native L-thread APIs.
+
+
+Overview
+^^^^^^^^
+The L-thread subsystem includes an example pthread shim. This is a partial
+implementation but does contain the API stubs needed to get basic applications
+running. There is a simple "hello world" application that demonstrates the
+use of the pthread shim.
+
+A subtlety of working with a shim is that the application will still need
+to make use of the genuine pthread library functions, at the very least in
+order to create the EAL threads in which the L-thread schedulers will run.
+This is the case with DPDK initialization, and exit.
+
+To deal with the initialization and shutdown scenarios, the shim is capable of
+switching on or off its adaptor functionality, an application can control this
+behavior by the calling the function ``pt_override_set()``. The default state
+is disabled.
+
+The pthread shim uses the dynamic linker loader and saves the loaded addresses
+of the genuine pthread API functions in an internal table, when the shim
+functionality is enabled it performs the adaptor function, when disabled it
+invokes the genuine pthread function.
+
+The function ``pthread_exit()`` has additional special handling. The standard
+system header file pthread.h declares ``pthread_exit()`` with
+``__attribute__((noreturn))`` this is an optimization that is possible because
+the pthread is terminating and this enables the compiler to omit the normal
+handling of stack and protection of registers since the function is not
+expected to return, and in fact the thread is being destroyed. These
+optimizations are applied in both the callee and the caller of the
+``pthread_exit()`` function.
+
+In our cooperative scheduling environment this behavior is inadmissible. The
+pthread is the L-thread scheduler thread, and, although an L-thread is
+terminating, there must be a return to the scheduler in order that the system
+can continue to run. Further, returning from a function with attribute
+``noreturn`` is invalid and may result in undefined behavior.
+
+The solution is to redefine the ``pthread_exit`` function with a macro,
+causing it to be mapped to a stub function in the shim that does not have the
+``noreturn`` attribute. This macro is defined in the file
+``pthread_shim.h``. The stub function is otherwise no different than any of
+the other stub functions in the shim, and will switch between the real
+``pthread_exit()`` function or the ``lthread_exit()`` function as
+required. The only difference is that the mapping to the stub by macro
+substitution.
+
+A consequence of this is that the file ``pthread_shim.h`` must be included in
+legacy code wishing to make use of the shim. It also means that dynamic
+linkage of a pre-compiled binary that did not include pthread_shim.h is not be
+supported.
+
+Given the requirements for porting legacy code outlined in
+:ref:`porting_legacy_code_to_run_on_lthreads` most applications will require at
+least some minimal adjustment and recompilation to run on L-threads so
+pre-compiled binaries are unlikely to be met in practice.
+
+In summary the shim approach adds some overhead but can be a useful tool to help
+establish the feasibility of a code reuse project. It is also a fairly
+straightforward task to extend the shim if necessary.
+
+**Note:** Bearing in mind the preceding discussions about the impact of making
+blocking calls then switching the shim in and out on the fly to invoke any
+pthread API this might block is something that should typically be avoided.
+
+
+Building and running the pthread shim
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The shim example application is located in the sample application
+in the performance-thread folder
+
+To build and run the pthread shim example
+
+#. Go to the example applications folder
+
+   .. code-block:: console
+
+       export RTE_SDK=/path/to/rte_sdk
+       cd ${RTE_SDK}/examples/performance-thread/pthread_shim
+
+
+#. Set the target (a default target is used if not specified). For example:
+
+   .. code-block:: console
+
+       export RTE_TARGET=x86_64-native-linuxapp-gcc
+
+   See the DPDK Getting Started Guide for possible RTE_TARGET values.
+
+#. Build the application:
+
+   .. code-block:: console
+
+       make
+
+#. To run the pthread_shim example
+
+   .. code-block:: console
+
+       lthread-pthread-shim -c core_mask -n number_of_channels
 
 .. _lthread_diagnostics:
 
examples/performance-thread/Makefile +1 -0
index 9e02718..2045f14 100644
@@ -39,6 +39,7 @@ RTE_TARGET ?= x86_64-native-linuxapp-gcc
 include $(RTE_SDK)/mk/rte.vars.mk
 
 DIRS-$(CONFIG_RTE_PERFORMANCE_THREAD) += l3fwd-thread
+DIRS-$(CONFIG_RTE_PERFORMANCE_THREAD) += pthread_shim
 
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
examples/performance-thread/pthread_shim/Makefile new file +60 -0

60 lines not on this page · raw message

examples/performance-thread/pthread_shim/main.c new file +284 -0

284 lines not on this page · raw message

examples/performance-thread/pthread_shim/pthread_shim.c new file +714 -0

714 lines not on this page · raw message

examples/performance-thread/pthread_shim/pthread_shim.h new file +113 -0

113 lines not on this page · raw message

Signature · 1 line
1.9.3
#2 lore raw

Re: [PATCH v3 2/2] examples: add pthread-shim in performance-thread sample app

Quoted 9 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of ibetts
Sent: Tuesday, November 17, 2015 12:11 PM
To: [email protected]
Cc: Betts, Ian
Subject: [dpdk-dev] [PATCH v3 2/2] examples: add pthread-shim in performance-thread sample app

Hi Ian,

For some reason it looks like [PATCH v3 1/2] didn't make it to the list.

John.

author
#3 lore raw

Re: [PATCH v3 2/2] examples: add pthread-shim in performance-thread sample app

Quoted 7 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of ibetts
Sent: Tuesday, November 17, 2015 12:11 PM
To: [email protected]
Cc: Betts, Ian
Subject: [dpdk-dev] [PATCH v3 2/2] examples: add pthread-shim in performance-thread sample app

Hi Ian,

For some reason it looks like [PATCH v3 1/2] didn't make it to the list.

John.

Signature · 11 lines
It is blocked due to exceeding 300kb limit ( it is actually 400kb )
I requested admin to unblock
--
Ian

Ian Betts
Intel Shannon Limited
Registered in Ireland
Registered Office Collinstown Industrial Park, Leixlip, County Kildare
Registered Number:308263
Business address: Dromore House, East Park, Shannon, Co. Clare
Acked-by on v3 2/2
#4 lore raw

Re: [PATCH v3 2/2] examples: add pthread-shim in performance-thread sample app

In reply to ibetts
Quoted 12 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of ibetts
Sent: Tuesday, November 17, 2015 13:11
To: [email protected]
Cc: Betts, Ian
Subject: [dpdk-dev] [PATCH v3 2/2] examples: add pthread-shim in performance-thread sample app

From: Ian Betts <[email protected]>

Quoted 8 lines

This commit adds a simple pthread_shim example for the cooperative scheduler included with this patchset.

The shim demonstrates a way in which legacy code writtem for pthreads could be adapted to lighweight threads.

---

Acked-by on v3 1/2
#5 lore raw
In reply to ibetts
Quoted 39 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of ibetts
Sent: Tuesday, November 17, 2015 13:11
To: [email protected]
Cc: Betts, Ian
Subject: [dpdk-dev] [PATCH v3 1/2] examples: add performance thread sample application

From: Ian Betts <[email protected]>

This example comprises a layer 3 forwarding derivative intended to facilitate characterization of performance with different threading models, specifically:-

1. EAL threads running on different physical cores
2. EAL threads running on the same physical core
3. Lightweight threads running in an EAL thread

Purpose and justification

Since dpdk 2.0 it has been possible to assign multiple EAL threads to a physical core ( case 2 above ).
Currently no example application has focused on demonstrating the performance constraints of differing threading models.

Whilst purpose built applications that fully comprehend the DPDK single threaded programming model will always yield superior performance, the desire to preserve ROI in legacy code written for multithreaded operating environments makes lightweight threads (case 3 above) worthy of consideration.

As well as aiding with legacy code reuse, it is anticipated that lightweight threads will make it possible to scale a multithreaded application with fine granularity allowing an application to more easily take advantage of headroom on EAL cores, or conversely occupy more cores, as dictated by system load.

Quoted 8 lines

To explore performance with lightweight threads a simple cooperative scheduler subsystem is being included in this example application. If the expected benefits and use cases prove to be of value, it is anticipated that this lightweight thread subsystem would become a library in some future DPDK release.

---