Skip to contentSkip to conversation

eal/windows: Switched to an external pthreads library

PATCH In review John Alexander 2 participants 2 messages Sep 24, 2020 – Jun 14, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 eal/windows: Switched to an external pthreads library

The Windows EAL build now pulls in the pthreads4w project as an external subproject. The pthreads4w subproject does not currently provide a meson build so the project has been patched with one.

Removed the placeholder librte_eal\windows\include\pthread.h and sched.h header files as these are superseded by the pthreads4w implementation.

rte_eal_init() in the Windows EAL has been modified to use pthread_create() and pthread_setaffinity_np() to setup the secondary lcores. eal_thread_create() has been removed as it is no longer required.

rte_eal_init() in the Windows EAL now calls SetPriorityClass() to set the process class to real-time in a singular location.

eal_thread_loop() in the Windows EAL now calls SetThreadPriority() to set the thread priority to real-time when the thread commences execution. This function now uses pthreads functions to test and compare pthread_t handles.

Added the source file librte_eal\windows\windows_eal_impl.c. This source file serves as a location for the implementation of auxiliary functionality provided by the Windows EAL that would otherwise be provided by the UNIX runtime.

Added the pthreads4w shared library as a dependency to EAL library builds for Windows builds only. The meson install step will install the pthreads4w.dll that is built by the meson external subproject build step. The shared library has been selected for both static and shared library builds to avoid constructior initialisation issues observed when using pthreads4w and static linkage within the DPDK project.

Added the subprojects folder to the .gitignore file such that authored content is not ignored and downloaded content is ignored (as that's where meson stores the subprojects when they are cloned).

Incremented the minimum meson version number to 0.55.0. This is required to use the subproject folder patch feature.

Updated MAINTAINERS to reflect new additions.

---
 .gitignore                                    |   6 +
 MAINTAINERS                                   |   3 +
 lib/librte_eal/windows/eal.c                  |  21 ++-
 lib/librte_eal/windows/eal_thread.c           |  27 +---
 lib/librte_eal/windows/eal_windows.h          |  10 --
 lib/librte_eal/windows/include/pthread.h      | 146 ------------------
 lib/librte_eal/windows/include/rte_os.h       |  15 +-
 lib/librte_eal/windows/include/sched.h        |  92 -----------
 lib/librte_eal/windows/meson.build            |   1 +
 lib/librte_eal/windows/windows_eal_impl.c     |  53 +++++++
 lib/meson.build                               |  14 +-
 meson.build                                   |   7 +-
 .../packagefiles/pthreads4w-meson/VERSION     |   1 +
 .../packagefiles/pthreads4w-meson/meson.build |  49 ++++++
 subprojects/pthreads4w-code.wrap              |   8 +
 15 files changed, 178 insertions(+), 275 deletions(-)
 delete mode 100644 lib/librte_eal/windows/include/pthread.h
 delete mode 100644 lib/librte_eal/windows/include/sched.h
 create mode 100644 lib/librte_eal/windows/windows_eal_impl.c
 create mode 100644 subprojects/packagefiles/pthreads4w-meson/VERSION
 create mode 100644 subprojects/packagefiles/pthreads4w-meson/meson.build
 create mode 100644 subprojects/pthreads4w-code.wrap
.gitignore +6 -0
index f73d93ca5..3792a843b 100644
@@ -33,3 +33,9 @@ build-*
 # ignore other build directory patterns
 *-gcc*
 *-clang*
+
+# ignore sub-directories of the subprojects directory excluding wrap files and the packagefiles directory
+subprojects/**/*
+!subprojects/*.wrap
+!subprojects/packagefiles/
+!subprojects/packagefiles/**
MAINTAINERS +3 -0
index c0abbe0fc..c33c1bc72 100644
@@ -318,10 +318,13 @@ M: Dmitry Kozlyuk <[email protected]>
 M: Narcisa Ana Maria Vasile <[email protected]>
 M: Dmitry Malloy <[email protected]>
 M: Pallavi Kadam <[email protected]>
+M: John Alexander <[email protected]>
 F: lib/librte_eal/windows/
+F: lib/librte_eal/windows/windows_eal_impl.c
 F: lib/librte_eal/rte_eal_exports.def
 F: buildtools/map_to_win.py
 F: doc/guides/windows_gsg/
+F: subprojects/
 
 Windows memory allocation
 M: Dmitry Kozlyuk <[email protected]>
lib/librte_eal/windows/eal.c +19 -2
index bc48f27ab..28320a769 100644
@@ -260,7 +260,7 @@ rte_eal_cleanup(void)
 int
 rte_eal_init(int argc, char **argv)
 {
-	int i, fctret, bscan;
+	int i, fctret, bscan, ret;
 	const struct rte_config *config = rte_eal_get_configuration();
 	struct internal_config *internal_conf =
 		eal_get_internal_configuration();
@@ -360,6 +360,15 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+	/*
+	 * We require real-time priority threads.  To achieve this on Windows we must
+	 * set the current process class to real-time.  Setting the process class to
+	 * real-time requires elevated privileges (admin user) otherwise the maximum
+	 * achievable thread priority will still only be 15 (out of 31) even with
+	 * real-time thread priority and real-time process class set.
+	 */
+	SetPriorityClass(GetCurrentProcess(), REALTIME_PRIORITY_CLASS);
+
 	RTE_LCORE_FOREACH_SLAVE(i) {
 
 		/*
@@ -376,8 +385,16 @@ rte_eal_init(int argc, char **argv)
 		lcore_config[i].state = WAIT;
 
 		/* create a thread for each lcore */
-		if (eal_thread_create(&lcore_config[i].thread_id) != 0)
+		/* create a thread for each lcore */
+		ret = pthread_create(&lcore_config[i].thread_id, NULL,
+					eal_thread_loop, NULL);
+		if (ret != 0)
 			rte_panic("Cannot create thread\n");
+
+		ret = pthread_setaffinity_np(lcore_config[i].thread_id,
+		sizeof(rte_cpuset_t), &lcore_config[i].cpuset);
+		if (ret != 0)
+			rte_panic("Cannot set affinity\n");
 	}
 
 	/* Initialize services so drivers can register services during probe. */
lib/librte_eal/windows/eal_thread.c +6 -21
index 20889b619..24b68d184 100644
@@ -3,6 +3,7 @@
  */
 
 #include <io.h>
+#include <pthread.h>
 
 #include <rte_atomic.h>
 #include <rte_debug.h>
@@ -66,9 +67,11 @@ eal_thread_loop(void *arg __rte_unused)
 
 	thread_id = pthread_self();
 
+	SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_TIME_CRITICAL);
+
 	/* retrieve our lcore_id from the configuration structure */
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (thread_id == lcore_config[lcore_id].thread_id)
+		if (pthread_equal(thread_id, lcore_config[lcore_id].thread_id))
 			break;
 	}
 	if (lcore_id == RTE_MAX_LCORE)
@@ -79,8 +82,8 @@ eal_thread_loop(void *arg __rte_unused)
 
 	__rte_thread_init(lcore_id, &lcore_config[lcore_id].cpuset);
 
-	RTE_LOG(DEBUG, EAL, "lcore %u is ready (tid=%zx;cpuset=[%s])\n",
-		lcore_id, (uintptr_t)thread_id, cpuset);
+	RTE_LOG(DEBUG, EAL, "lcore %u is ready (tid=%p;cpuset=[%s])\n",
+		lcore_id, thread_id, cpuset);
 
 	/* read on our pipe to get commands */
 	while (1) {
@@ -122,24 +125,6 @@ eal_thread_loop(void *arg __rte_unused)
 	}
 }
 
-/* function to create threads */
-int
-eal_thread_create(pthread_t *thread)
-{
-	HANDLE th;
-
-	th = CreateThread(NULL, 0,
-		(LPTHREAD_START_ROUTINE)(ULONG_PTR)eal_thread_loop,
-						NULL, 0, (LPDWORD)thread);
-	if (!th)
-		return -1;
-
-	SetPriorityClass(GetCurrentProcess(), REALTIME_PRIORITY_CLASS);
-	SetThreadPriority(th, THREAD_PRIORITY_TIME_CRITICAL);
-
-	return 0;
-}
-
 /* get current thread ID */
 int
 rte_sys_gettid(void)
lib/librte_eal/windows/eal_windows.h +0 -10
index d48ee0a12..dfd3801ff 100644
@@ -35,16 +35,6 @@
  */
 int eal_create_cpu_map(void);
 
-/**
- * Create a thread.
- *
- * @param thread
- *   The location to store the thread id if successful.
- * @return
- *   0 for success, -1 if the thread is not created.
- */
-int eal_thread_create(pthread_t *thread);
-
 /**
  * Get system NUMA node number for a socket ID.
  *
lib/librte_eal/windows/include/pthread.h deleted +0 -146

146 lines not on this page · raw message

lib/librte_eal/windows/include/rte_os.h +14 -1
index 569ed92d5..09d2aa17e 100644
@@ -15,6 +15,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
+#include <sched.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -43,10 +44,22 @@ extern "C" {
 #define unlink _unlink
 
 /* cpu_set macros implementation */
+typedef cpu_set_t rte_cpuset_t;
 #define RTE_CPU_AND(dst, src1, src2) CPU_AND(dst, src1, src2)
 #define RTE_CPU_OR(dst, src1, src2) CPU_OR(dst, src1, src2)
 #define RTE_CPU_FILL(set) CPU_FILL(set)
-#define RTE_CPU_NOT(dst, src) CPU_NOT(dst, src)
+void RTE_CPU_ANDNOT(cpu_set_t *dst, cpu_set_t *src);
+void RTE_CPU_COPY(cpu_set_t *from, cpu_set_t *to);
+void RTE_CPU_FILL(cpu_set_t *pdestset);
+#define RTE_CPU_NOT(dst, src) do \
+{ \
+	cpu_set_t tmp; \
+	RTE_CPU_FILL(&tmp); \
+	RTE_CPU_ANDNOT(&tmp, src); \
+	RTE_CPU_COPY(&tmp, dst); \
+} while (0)
+
+
 
 /* as in <windows.h> */
 typedef long long ssize_t;
lib/librte_eal/windows/include/sched.h deleted +0 -92

92 lines not on this page · raw message

lib/librte_eal/windows/meson.build +1 -0
index b690bc6b0..9f5b7839a 100644
@@ -18,6 +18,7 @@ sources += files(
 	'eal_timer.c',
 	'fnmatch.c',
 	'getopt.c',
+	'windows_eal_impl.c'
 )
 
 dpdk_conf.set10('RTE_EAL_NUMA_AWARE_HUGEPAGES', true)
lib/librte_eal/windows/windows_eal_impl.c new file +53 -0

53 lines not on this page · raw message

lib/meson.build +12 -2
index d8b358e5f..fd154ec5d 100644
@@ -43,6 +43,11 @@ if is_windows
 		'ring',
 		'mempool', 'mbuf', 'net', 'meter', 'ethdev', 'pci',
 	] # only supported libraries for windows
+
+	# Windows doesn't natively support pthreads so we pull down an external project
+	#   using the meson subproject feature.
+	pthreads4w_proj = subproject('pthreads4w-code')
+	pthreads4w_shared_dep = pthreads4w_proj.get_variable('pthreads4w_shared_dep')
 endif
 
 default_cflags = machine_args
@@ -82,6 +87,12 @@ foreach l:libraries
 	if build
 		shared_deps = ext_deps
 		static_deps = ext_deps
+		if is_windows
+			# Initialisation issues with the external pthreads library when linked with
+			#  DPDK require us to always use the shared build of the pthreads4w library.
+			shared_deps += pthreads4w_shared_dep
+			static_deps += pthreads4w_shared_dep
+		endif
 		foreach d:deps
 			if not is_variable('shared_rte_' + d)
 				error('Missing internal dependency "@0@" for @1@ [@2@]'
@@ -153,8 +164,7 @@ foreach l:libraries
 				output: '@0@_mingw.map'.format(libname))
 
 			if is_ms_linker
-				lk_args = ['-Wl,/def:' + def_file.full_path(),
-					'-Wl,/implib:lib\\' + implib]
+				lk_args = ['-Wl,/def:' + def_file.full_path()]
 			else
 				if is_windows
 					lk_args = ['-Wl,--version-script=' + mingw_map.full_path()]
meson.build +6 -1
index 61d9a4f5f..9e061c73b 100644
@@ -8,7 +8,7 @@ project('DPDK', 'C',
 		files('VERSION')).stdout().strip(),
 	license: 'BSD',
 	default_options: ['buildtype=release', 'default_library=static'],
-	meson_version: '>= 0.47.1'
+	meson_version: '>= 0.55.0'
 )
 
 # set up some global vars for compiler, platform, configuration, etc.
@@ -44,6 +44,11 @@ global_inc = include_directories('.', 'config',
 subdir('buildtools')
 subdir('config')
 
+if is_windows
+	# Windows builds do not define this but it is required so we define it globally here.
+	add_global_arguments('-D_POSIX_C_SOURCE=200809L', language: 'c')
+endif
+
 # build libs and drivers
 subdir('buildtools/pmdinfogen')
 subdir('lib')
subprojects/packagefiles/pthreads4w-meson/VERSION new file +1 -0
new file mode 100644
index 000000000..4a36342fc
@@ -0,0 +1 @@
+3.0.0
subprojects/packagefiles/pthreads4w-meson/meson.build new file +49 -0

49 lines not on this page · raw message

subprojects/pthreads4w-code.wrap new file +8 -0
new file mode 100644
index 000000000..d2215e5ce
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2020 Datapath Limited
+
+[wrap-git]
+directory=pthreads4w
+url=git://git.code.sf.net/p/pthreads4w/code
+revision=Version-3-0-0-release
+patch_directory=pthreads4w-meson
Signature · 1 line
2.17.1
#1 lore raw
Quoted 57 lines · On Thu, 24 Sep 2020 12:41:01 +0100 John Alexander…

The Windows EAL build now pulls in the pthreads4w project as an external subproject. The pthreads4w subproject does not currently provide a meson build so the project has been patched with one.

Removed the placeholder librte_eal\windows\include\pthread.h and sched.h header files as these are superseded by the pthreads4w implementation.

rte_eal_init() in the Windows EAL has been modified to use pthread_create() and pthread_setaffinity_np() to setup the secondary lcores. eal_thread_create() has been removed as it is no longer required.

rte_eal_init() in the Windows EAL now calls SetPriorityClass() to set the process class to real-time in a singular location.

eal_thread_loop() in the Windows EAL now calls SetThreadPriority() to set the thread priority to real-time when the thread commences execution. This function now uses pthreads functions to test and compare pthread_t handles.

Added the source file librte_eal\windows\windows_eal_impl.c. This source file serves as a location for the implementation of auxiliary functionality provided by the Windows EAL that would otherwise be provided by the UNIX runtime.

Added the pthreads4w shared library as a dependency to EAL library builds for Windows builds only. The meson install step will install the pthreads4w.dll that is built by the meson external subproject build step. The shared library has been selected for both static and shared library builds to avoid constructior initialisation issues observed when using pthreads4w and static linkage within the DPDK project.

Added the subprojects folder to the .gitignore file such that authored content is not ignored and downloaded content is ignored (as that's where meson stores the subprojects when they are cloned).

Incremented the minimum meson version number to 0.55.0. This is required to use the subproject folder patch feature.

Updated MAINTAINERS to reflect new additions.

---
 .gitignore                                    |   6 +
 MAINTAINERS                                   |   3 +
 lib/librte_eal/windows/eal.c                  |  21 ++-
 lib/librte_eal/windows/eal_thread.c           |  27 +---
 lib/librte_eal/windows/eal_windows.h          |  10 --
 lib/librte_eal/windows/include/pthread.h      | 146 ------------------
 lib/librte_eal/windows/include/rte_os.h       |  15 +-
 lib/librte_eal/windows/include/sched.h        |  92 -----------
 lib/librte_eal/windows/meson.build            |   1 +
 lib/librte_eal/windows/windows_eal_impl.c     |  53 +++++++
 lib/meson.build                               |  14 +-
 meson.build                                   |   7 +-
 .../packagefiles/pthreads4w-meson/VERSION     |   1 +
 .../packagefiles/pthreads4w-meson/meson.build |  49 ++++++
Quoted 8 lines
 subprojects/pthreads4w-code.wrap              |   8 +
 15 files changed, 178 insertions(+), 275 deletions(-)
 delete mode 100644 lib/librte_eal/windows/include/pthread.h
 delete mode 100644 lib/librte_eal/windows/include/sched.h
 create mode 100644 lib/librte_eal/windows/windows_eal_impl.c
 create mode 100644 subprojects/packagefiles/pthreads4w-meson/VERSION
 create mode 100644 subprojects/packagefiles/pthreads4w-meson/meson.build
 create mode 100644 subprojects/pthreads4w-code.wrap

This patch is really old by now.
The current windows port uses native windows threads in current code. Supporting an external package is not necessary.