examples: add performance thread sample application
PATCH v3 1/2 examples: add performance thread sample application
From: Ian Betts <[email protected]>
This example comprises a layer 3 forwarding derivative intended to facilitate characterization of performance with different threading models, specifically:-
1. EAL threads running on different physical cores
2. EAL threads running on the same physical core
3. Lightweight threads running in an EAL thread
Purpose and justification
Since dpdk 2.0 it has been possible to assign multiple EAL threads to a physical core ( case 2 above ).
Currently no example application has focused on demonstrating the performance constraints of differing threading models.
Whilst purpose built applications that fully comprehend the DPDK single threaded programming model will always yield superior performance, the desire to preserve ROI in legacy code written for multithreaded operating environments makes lightweight threads (case 3 above) worthy of consideration.
As well as aiding with legacy code reuse, it is anticipated that lightweight threads will make it possible to scale a multithreaded application with fine granularity allowing an application to more easily take advantage of headroom on EAL cores, or conversely occupy more cores, as dictated by system load.
To explore performance with lightweight threads a simple cooperative scheduler subsystem is being included in this example application. If the expected benefits and use cases prove to be of value, it is anticipated that this lightweight thread subsystem would become a library in some future DPDK release.
--- config/common_linuxapp | 5 + config/defconfig_i686-native-linuxapp-gcc | 5 + config/defconfig_i686-native-linuxapp-icc | 5 + config/defconfig_x86_x32-native-linuxapp-gcc | 5 + doc/guides/rel_notes/release_2_2.rst | 11 + .../sample_app_ug/img/performance_thread_1.svg | 799 +++++ .../sample_app_ug/img/performance_thread_2.svg | 803 +++++ doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/performance_thread.rst | 1150 +++++++ examples/Makefile | 1 + examples/performance-thread/Makefile | 44 + .../performance-thread/common/arch/x86/atomic.h | 59 + examples/performance-thread/common/arch/x86/ctx.c | 93 + examples/performance-thread/common/arch/x86/ctx.h | 57 + examples/performance-thread/common/common.mk | 40 + examples/performance-thread/common/lthread.c | 546 +++ examples/performance-thread/common/lthread.h | 99 + examples/performance-thread/common/lthread_api.h | 822 +++++ examples/performance-thread/common/lthread_cond.c | 240 ++ examples/performance-thread/common/lthread_cond.h | 77 + examples/performance-thread/common/lthread_diag.c | 321 ++ examples/performance-thread/common/lthread_diag.h | 129 + .../performance-thread/common/lthread_diag_api.h | 319 ++ examples/performance-thread/common/lthread_int.h | 212 ++ examples/performance-thread/common/lthread_mutex.c | 255 ++ examples/performance-thread/common/lthread_mutex.h | 52 + .../performance-thread/common/lthread_objcache.h | 160 + examples/performance-thread/common/lthread_pool.h | 333 ++ examples/performance-thread/common/lthread_queue.h | 303 ++ examples/performance-thread/common/lthread_sched.c | 598 ++++ examples/performance-thread/common/lthread_sched.h | 152 + examples/performance-thread/common/lthread_timer.h | 47 + examples/performance-thread/common/lthread_tls.c | 242 ++ examples/performance-thread/common/lthread_tls.h | 64 + examples/performance-thread/l3fwd-thread/Makefile | 57 + examples/performance-thread/l3fwd-thread/main.c | 3628 ++++++++++++++++++++ 36 files changed, 11734 insertions(+) create mode 100644 doc/guides/sample_app_ug/img/performance_thread_1.svg create mode 100644 doc/guides/sample_app_ug/img/performance_thread_2.svg create mode 100644 doc/guides/sample_app_ug/performance_thread.rst create mode 100644 examples/performance-thread/Makefile create mode 100644 examples/performance-thread/common/arch/x86/atomic.h create mode 100644 examples/performance-thread/common/arch/x86/ctx.c create mode 100644 examples/performance-thread/common/arch/x86/ctx.h create mode 100644 examples/performance-thread/common/common.mk create mode 100644 examples/performance-thread/common/lthread.c create mode 100644 examples/performance-thread/common/lthread.h create mode 100644 examples/performance-thread/common/lthread_api.h create mode 100644 examples/performance-thread/common/lthread_cond.c create mode 100644 examples/performance-thread/common/lthread_cond.h create mode 100644 examples/performance-thread/common/lthread_diag.c create mode 100644 examples/performance-thread/common/lthread_diag.h create mode 100644 examples/performance-thread/common/lthread_diag_api.h create mode 100644 examples/performance-thread/common/lthread_int.h create mode 100644 examples/performance-thread/common/lthread_mutex.c create mode 100644 examples/performance-thread/common/lthread_mutex.h create mode 100644 examples/performance-thread/common/lthread_objcache.h create mode 100644 examples/performance-thread/common/lthread_pool.h create mode 100644 examples/performance-thread/common/lthread_queue.h create mode 100644 examples/performance-thread/common/lthread_sched.c create mode 100644 examples/performance-thread/common/lthread_sched.h create mode 100644 examples/performance-thread/common/lthread_timer.h create mode 100644 examples/performance-thread/common/lthread_tls.c create mode 100644 examples/performance-thread/common/lthread_tls.h create mode 100644 examples/performance-thread/l3fwd-thread/Makefile create mode 100644 examples/performance-thread/l3fwd-thread/main.c
config/common_linuxapp +5 -0
@@ -481,3 +481,8 @@ CONFIG_RTE_APP_TEST=y CONFIG_RTE_TEST_PMD=y CONFIG_RTE_TEST_PMD_RECORD_CORE_CYCLES=n CONFIG_RTE_TEST_PMD_RECORD_BURST_STATS=n + +# +# Compile the performance thread sample application +# +CONFIG_RTE_PERFORMANCE_THREAD=y
config/defconfig_i686-native-linuxapp-gcc +5 -0
@@ -49,3 +49,8 @@ CONFIG_RTE_LIBRTE_KNI=n # Vectorized PMD is not supported on 32-bit # CONFIG_RTE_IXGBE_INC_VECTOR=n + +# +# Performance thread sample application is not supported on 32-bit +# +CONFIG_RTE_PERFORMANCE_THREAD=n
config/defconfig_i686-native-linuxapp-icc +5 -0
@@ -49,3 +49,8 @@ CONFIG_RTE_LIBRTE_KNI=n # Vectorized PMD is not supported on 32-bit # CONFIG_RTE_IXGBE_INC_VECTOR=n + +# +# Performance thread sample application is not supported on 32-bit +# +CONFIG_RTE_PERFORMANCE_THREAD=n
config/defconfig_x86_x32-native-linuxapp-gcc +5 -0
@@ -44,3 +44,8 @@ CONFIG_RTE_TOOLCHAIN_GCC=y # KNI is not supported on 32-bit # CONFIG_RTE_LIBRTE_KNI=n + +# +# Performance thread sample application is not supported on 32-bit +# +CONFIG_RTE_PERFORMANCE_THREAD=n
doc/guides/rel_notes/release_2_2.rst +11 -0
@@ -207,6 +207,12 @@ Libraries Examples ~~~~~~~~ +* **examples: Introducing a performance thread example** + + This an l3fwd derivative focused to enable characterization of performance + with different threading models, including multiple EAL threads per physical + core, and multiple Lightweight threads running in an EAL thread. + The examples includes a simple cooperative scheduler. Other ~~~~~
@@ -214,6 +220,11 @@ Other Known Issues ------------ +* When running the performance thread application in configurations with more than + two EAL threads per phsycial core, then forwarding throughput may drop suddenly + to a low level. Stopping and restarting traffic restores correct operation. + This problem does not occur when running with multiple lightweight threads per + physical core. API Changes
doc/guides/sample_app_ug/img/performance_thread_1.svg new file +799 -0
799 lines not on this page · raw message
doc/guides/sample_app_ug/img/performance_thread_2.svg new file +803 -0
803 lines not on this page · raw message
doc/guides/sample_app_ug/index.rst +1 -0
@@ -74,6 +74,7 @@ Sample Applications User Guide tep_termination proc_info ptpclient + performance_thread **Figures**
doc/guides/sample_app_ug/performance_thread.rst new file +1150 -0
1,150 lines not on this page · raw message
examples/Makefile +1 -0
@@ -75,5 +75,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_XEN_DOM0) += vhost_xen DIRS-y += vmdq DIRS-y += vmdq_dcb DIRS-$(CONFIG_RTE_LIBRTE_POWER) += vm_power_manager +DIRS-$(CONFIG_RTE_PERFORMANCE_THREAD) += performance-thread include $(RTE_SDK)/mk/rte.extsubdir.mk
examples/performance-thread/Makefile new file +44 -0
44 lines not on this page · raw message
examples/performance-thread/common/arch/x86/atomic.h new file +59 -0
59 lines not on this page · raw message
examples/performance-thread/common/arch/x86/ctx.c new file +93 -0
77 lines not on this page · raw message
examples/performance-thread/common/arch/x86/ctx.h new file +57 -0
57 lines not on this page · raw message
examples/performance-thread/common/common.mk new file +40 -0
Lines of this file are not shown. View the raw message.
examples/performance-thread/common/lthread.c new file +546 -0
546 lines not on this page · raw message
examples/performance-thread/common/lthread.h new file +99 -0
99 lines not on this page · raw message
examples/performance-thread/common/lthread_api.h new file +822 -0
822 lines not on this page · raw message
examples/performance-thread/common/lthread_cond.c new file +240 -0
240 lines not on this page · raw message
examples/performance-thread/common/lthread_cond.h new file +77 -0
77 lines not on this page · raw message
examples/performance-thread/common/lthread_diag.c new file +321 -0
321 lines not on this page · raw message
examples/performance-thread/common/lthread_diag.h new file +129 -0
129 lines not on this page · raw message
examples/performance-thread/common/lthread_diag_api.h new file +319 -0
319 lines not on this page · raw message
examples/performance-thread/common/lthread_int.h new file +212 -0
212 lines not on this page · raw message
examples/performance-thread/common/lthread_mutex.c new file +255 -0
255 lines not on this page · raw message
examples/performance-thread/common/lthread_mutex.h new file +52 -0
52 lines not on this page · raw message
examples/performance-thread/common/lthread_objcache.h new file +160 -0
160 lines not on this page · raw message
examples/performance-thread/common/lthread_pool.h new file +333 -0
333 lines not on this page · raw message
examples/performance-thread/common/lthread_queue.h new file +303 -0
303 lines not on this page · raw message
examples/performance-thread/common/lthread_sched.c new file +598 -0
598 lines not on this page · raw message
examples/performance-thread/common/lthread_sched.h new file +152 -0
152 lines not on this page · raw message
examples/performance-thread/common/lthread_timer.h new file +47 -0
47 lines not on this page · raw message
examples/performance-thread/common/lthread_tls.c new file +242 -0
242 lines not on this page · raw message
examples/performance-thread/common/lthread_tls.h new file +64 -0
64 lines not on this page · raw message
examples/performance-thread/l3fwd-thread/Makefile new file +57 -0
57 lines not on this page · raw message
examples/performance-thread/l3fwd-thread/main.c new file +3628 -0
3,628 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
1.9.3
Part of the diff is not shown here. View the raw message.
PATCH v3 2/2 examples: add pthread-shim in performance-thread sample app
From: Ian Betts <[email protected]>
This commit adds a simple pthread_shim example for the cooperative scheduler included with this patchset.
The shim demonstrates a way in which legacy code writtem for pthreads could be adapted to lighweight threads.
--- doc/guides/sample_app_ug/performance_thread.rst | 113 ++++ examples/performance-thread/Makefile | 1 + examples/performance-thread/pthread_shim/Makefile | 60 ++ examples/performance-thread/pthread_shim/main.c | 284 ++++++++ .../performance-thread/pthread_shim/pthread_shim.c | 714 +++++++++++++++++++++ .../performance-thread/pthread_shim/pthread_shim.h | 113 ++++ 6 files changed, 1285 insertions(+) create mode 100644 examples/performance-thread/pthread_shim/Makefile create mode 100644 examples/performance-thread/pthread_shim/main.c create mode 100644 examples/performance-thread/pthread_shim/pthread_shim.c create mode 100644 examples/performance-thread/pthread_shim/pthread_shim.h
doc/guides/sample_app_ug/performance_thread.rst +113 -0
@@ -1102,6 +1102,119 @@ it the local data it needs, and pick up the new logical core specific values from pthread local storage at its new home. +.. _pthread_shim: + +Pthread shim +~~~~~~~~~~~~ + +A convenient way to get something working with legacy code can be to use a +shim that adapts pthread API calls to the corresponding L-thread ones. +This approach will not mitigate any of the porting considerations mentioned +in the previous sections, but it will reduce the amount of code churn that +would otherwise been involved. It is a reasonable approach to evaluate +L-threads, before investing effort in porting to the native L-thread APIs. + + +Overview +^^^^^^^^ +The L-thread subsystem includes an example pthread shim. This is a partial +implementation but does contain the API stubs needed to get basic applications +running. There is a simple "hello world" application that demonstrates the +use of the pthread shim. + +A subtlety of working with a shim is that the application will still need +to make use of the genuine pthread library functions, at the very least in +order to create the EAL threads in which the L-thread schedulers will run. +This is the case with DPDK initialization, and exit. + +To deal with the initialization and shutdown scenarios, the shim is capable of +switching on or off its adaptor functionality, an application can control this +behavior by the calling the function ``pt_override_set()``. The default state +is disabled. + +The pthread shim uses the dynamic linker loader and saves the loaded addresses +of the genuine pthread API functions in an internal table, when the shim +functionality is enabled it performs the adaptor function, when disabled it +invokes the genuine pthread function. + +The function ``pthread_exit()`` has additional special handling. The standard +system header file pthread.h declares ``pthread_exit()`` with +``__attribute__((noreturn))`` this is an optimization that is possible because +the pthread is terminating and this enables the compiler to omit the normal +handling of stack and protection of registers since the function is not +expected to return, and in fact the thread is being destroyed. These +optimizations are applied in both the callee and the caller of the +``pthread_exit()`` function. + +In our cooperative scheduling environment this behavior is inadmissible. The +pthread is the L-thread scheduler thread, and, although an L-thread is +terminating, there must be a return to the scheduler in order that the system +can continue to run. Further, returning from a function with attribute +``noreturn`` is invalid and may result in undefined behavior. + +The solution is to redefine the ``pthread_exit`` function with a macro, +causing it to be mapped to a stub function in the shim that does not have the +``noreturn`` attribute. This macro is defined in the file +``pthread_shim.h``. The stub function is otherwise no different than any of +the other stub functions in the shim, and will switch between the real +``pthread_exit()`` function or the ``lthread_exit()`` function as +required. The only difference is that the mapping to the stub by macro +substitution. + +A consequence of this is that the file ``pthread_shim.h`` must be included in +legacy code wishing to make use of the shim. It also means that dynamic +linkage of a pre-compiled binary that did not include pthread_shim.h is not be +supported. + +Given the requirements for porting legacy code outlined in +:ref:`porting_legacy_code_to_run_on_lthreads` most applications will require at +least some minimal adjustment and recompilation to run on L-threads so +pre-compiled binaries are unlikely to be met in practice. + +In summary the shim approach adds some overhead but can be a useful tool to help +establish the feasibility of a code reuse project. It is also a fairly +straightforward task to extend the shim if necessary. + +**Note:** Bearing in mind the preceding discussions about the impact of making +blocking calls then switching the shim in and out on the fly to invoke any +pthread API this might block is something that should typically be avoided. + + +Building and running the pthread shim +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The shim example application is located in the sample application +in the performance-thread folder + +To build and run the pthread shim example + +#. Go to the example applications folder + + .. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + cd ${RTE_SDK}/examples/performance-thread/pthread_shim + + +#. Set the target (a default target is used if not specified). For example: + + .. code-block:: console + + export RTE_TARGET=x86_64-native-linuxapp-gcc + + See the DPDK Getting Started Guide for possible RTE_TARGET values. + +#. Build the application: + + .. code-block:: console + + make + +#. To run the pthread_shim example + + .. code-block:: console + + lthread-pthread-shim -c core_mask -n number_of_channels .. _lthread_diagnostics:
examples/performance-thread/Makefile +1 -0
@@ -39,6 +39,7 @@ RTE_TARGET ?= x86_64-native-linuxapp-gcc include $(RTE_SDK)/mk/rte.vars.mk DIRS-$(CONFIG_RTE_PERFORMANCE_THREAD) += l3fwd-thread +DIRS-$(CONFIG_RTE_PERFORMANCE_THREAD) += pthread_shim include $(RTE_SDK)/mk/rte.extsubdir.mk
examples/performance-thread/pthread_shim/Makefile new file +60 -0
60 lines not on this page · raw message
examples/performance-thread/pthread_shim/main.c new file +284 -0
284 lines not on this page · raw message
examples/performance-thread/pthread_shim/pthread_shim.c new file +714 -0
714 lines not on this page · raw message
examples/performance-thread/pthread_shim/pthread_shim.h new file +113 -0
113 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [PATCH v3 2/2] examples: add pthread-shim in performance-thread sample app
Quoted 9 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of ibetts
Sent: Tuesday, November 17, 2015 12:11 PM
To: [email protected]
Cc: Betts, Ian
Subject: [dpdk-dev] [PATCH v3 2/2] examples: add pthread-shim in performance-thread sample app
Hi Ian,
For some reason it looks like [PATCH v3 1/2] didn't make it to the list.
John.
Re: [PATCH v3 2/2] examples: add pthread-shim in performance-thread sample app
Quoted 7 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of ibetts
Sent: Tuesday, November 17, 2015 12:11 PM
To: [email protected]
Cc: Betts, Ian
Subject: [dpdk-dev] [PATCH v3 2/2] examples: add pthread-shim in performance-thread sample app
Hi Ian,
For some reason it looks like [PATCH v3 1/2] didn't make it to the list.
John.
Signature · 11 lines
It is blocked due to exceeding 300kb limit ( it is actually 400kb ) I requested admin to unblock -- Ian Ian Betts Intel Shannon Limited Registered in Ireland Registered Office Collinstown Industrial Park, Leixlip, County Kildare Registered Number:308263 Business address: Dromore House, East Park, Shannon, Co. Clare
Re: [PATCH v3 2/2] examples: add pthread-shim in performance-thread sample app
In reply to ibettsQuoted 12 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of ibetts
Sent: Tuesday, November 17, 2015 13:11
To: [email protected]
Cc: Betts, Ian
Subject: [dpdk-dev] [PATCH v3 2/2] examples: add pthread-shim in performance-thread sample appFrom: Ian Betts <[email protected]>
Quoted 8 lines
This commit adds a simple
pthread_shimexample for the cooperative scheduler included with this patchset.The shim demonstrates a way in which legacy code writtem for pthreads could be adapted to lighweight threads.
Signed-off-by Ian Betts <[email protected]>---
Quoted 39 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of ibetts
Sent: Tuesday, November 17, 2015 13:11
To: [email protected]
Cc: Betts, Ian
Subject: [dpdk-dev] [PATCH v3 1/2] examples: add performance thread sample applicationFrom: Ian Betts <[email protected]>
This example comprises a layer 3 forwarding derivative intended to facilitate characterization of performance with different threading models, specifically:-
1. EAL threads running on different physical cores
2. EAL threads running on the same physical core
3. Lightweight threads running in an EAL threadPurpose and justification
Since dpdk 2.0 it has been possible to assign multiple EAL threads to a physical core ( case 2 above ).
Currently no example application has focused on demonstrating the performance constraints of differing threading models.Whilst purpose built applications that fully comprehend the DPDK single threaded programming model will always yield superior performance, the desire to preserve ROI in legacy code written for multithreaded operating environments makes lightweight threads (case 3 above) worthy of consideration.
As well as aiding with legacy code reuse, it is anticipated that lightweight threads will make it possible to scale a multithreaded application with fine granularity allowing an application to more easily take advantage of headroom on EAL cores, or conversely occupy more cores, as dictated by system load.
Quoted 8 lines
To explore performance with lightweight threads a simple cooperative scheduler subsystem is being included in this example application. If the expected benefits and use cases prove to be of value, it is anticipated that this lightweight thread subsystem would become a library in some future DPDK release.
Signed-off-by Ian Betts <[email protected]>---