Skip to contentSkip to conversation

add thread lifetime and attributes API

PATCH Superseded Tyler Retzlaff 6 participants 69 messages Jun 9, 2022 – Mar 1, 2023 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

cover add thread lifetime and attributes API

add rte thread lifetime and attributes api. with these api additions there is now sufficient platform abstracted thread api to remove the use of pthread in the unit tests.

Tyler Retzlaff (6):
eal: add thread attributes
eal: add thread lifetime management
eal: add basic rte thread ID equal API test/threads: add tests for thread lifetime API test/threads: add tests for thread attributes API test/threads: remove unit test use of pthread

 app/test/test_threads.c         | 130 ++++++++++++++++++++++--
 lib/eal/common/meson.build      |   1 +
 lib/eal/common/rte_thread.c     |  58 +++++++++++
 lib/eal/include/rte_thread.h    | 177 ++++++++++++++++++++++++++++++++
 lib/eal/unix/rte_thread.c       | 108 ++++++++++++++++++++
 lib/eal/version.map             |   8 ++
 lib/eal/windows/include/sched.h |   2 +-
 lib/eal/windows/rte_thread.c    | 217 ++++++++++++++++++++++++++++++++--------
 8 files changed, 654 insertions(+), 47 deletions(-)
 create mode 100644 lib/eal/common/rte_thread.c
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH 1/6 eal: add thread attributes

Implement thread attributes for:

* thread affinity
* thread priority

Implement functions for managing thread attributes.

Priority is represented through an enum that allows for two levels:

* RTE_THREAD_PRIORITY_NORMAL
* RTE_THREAD_PRIORITY_REALTIME_CRITICAL

Affinity is described by the rte_cpuset_t type.

An rte_thread_attr_t object can be set to the default values by calling rte_thread_attr_init().

---
 lib/eal/common/meson.build   |  1 +
 lib/eal/common/rte_thread.c  | 52 +++++++++++++++++++++++++
 lib/eal/include/rte_thread.h | 93 ++++++++++++++++++++++++++++++++++++++++++++
 lib/eal/version.map          |  4 ++
 4 files changed, 150 insertions(+)
 create mode 100644 lib/eal/common/rte_thread.c
lib/eal/common/meson.build +1 -0
index 917758c..1e77dba 100644
@@ -36,6 +36,7 @@ sources += files(
         'rte_random.c',
         'rte_reciprocal.c',
         'rte_service.c',
+        'rte_thread.c',
         'rte_version.c',
 )
 if is_linux or is_windows
lib/eal/common/rte_thread.c new file +52 -0

52 lines not on this page · raw message

lib/eal/include/rte_thread.h +93 -0

105 lines not on this page · raw message

lib/eal/version.map +4 -0
index ab27a4b..72e0e14 100644
@@ -422,6 +422,10 @@ EXPERIMENTAL {
 	rte_intr_type_set;
 
 	# added in 22.07
+	rte_thread_attr_get_affinity;
+	rte_thread_attr_init;
+	rte_thread_attr_set_affinity;
+	rte_thread_attr_set_priority;
 	rte_thread_get_affinity_by_id;
 	rte_thread_get_priority;
 	rte_thread_self;
Signature · 1 line
1.8.3.1
#2 lore raw

PATCH 2/6 eal: add thread lifetime management

In reply to Tyler Retzlaff

The *rte_thread_create()* function can optionally receive an rte_thread_attr_t object that will cause the thread to be created with the affinity and priority described by the attributes object. If no rte_thread_attr_t is passed (parameter is NULL), the default affinity and priority are used.

On Windows, the function executed by a thread when the thread starts is represeneted by a function pointer of type DWORD (*func) (void*). On other platforms, the function pointer is a void* (*func) (void*).

Performing a cast between these two types of function pointers to uniformize the API on all platforms may result in undefined behavior. TO fix this issue, a wrapper that respects the signature required by CreateThread() has been created on Windows.

---
 lib/eal/include/rte_thread.h    |  65 ++++++++++++
 lib/eal/unix/rte_thread.c       | 108 ++++++++++++++++++++
 lib/eal/version.map             |   3 +
 lib/eal/windows/include/sched.h |   2 +-
 lib/eal/windows/rte_thread.c    | 217 ++++++++++++++++++++++++++++++++--------
 5 files changed, 354 insertions(+), 41 deletions(-)
lib/eal/include/rte_thread.h +65 -0
index 062308d..321fb59 100644
@@ -30,6 +30,8 @@
 	uintptr_t opaque_id; /**< thread identifier */
 } rte_thread_t;
 
+typedef void * (*rte_thread_func) (void *);
+
 /**
  * Thread priority values.
  */
@@ -61,6 +63,69 @@ enum rte_thread_priority {
  * @warning
  * @b EXPERIMENTAL: this API may change without prior notice.
  *
+ * Create a new thread that will invoke the 'thread_func' routine.
+ *
+ * @param thread_id
+ *    A pointer that will store the id of the newly created thread.
+ *
+ * @param thread_attr
+ *    Attributes that are used at the creation of the new thread.
+ *
+ * @param thread_func
+ *    The routine that the new thread will invoke when starting execution.
+ *
+ * @param args
+ *    Arguments to be passed to the 'thread_func' routine.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_create(rte_thread_t *thread_id,
+		const rte_thread_attr_t *thread_attr,
+		rte_thread_func thread_func, void *arg);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Waits for the thread identified by 'thread_id' to terminate
+ *
+ * @param thread_id
+ *    The identifier of the thread.
+ *
+ * @param value_ptr
+ *    Stores the exit status of the thread.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_join(rte_thread_t thread_id, unsigned long *value_ptr);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Indicate that the return value of the thread is not needed and
+ * all thread resources should be release when the thread terminates.
+ *
+ * @param thread_id
+ *    The id of the thread to be detached.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_detach(rte_thread_t thread_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
  * Get the id of the calling thread.
  *
  * @return
lib/eal/unix/rte_thread.c +108 -0

114 lines not on this page · raw message

lib/eal/version.map +3 -0
index 72e0e14..22e5c85 100644
@@ -426,8 +426,11 @@ EXPERIMENTAL {
 	rte_thread_attr_init;
 	rte_thread_attr_set_affinity;
 	rte_thread_attr_set_priority;
+	rte_thread_create;
+	rte_thread_detach;
 	rte_thread_get_affinity_by_id;
 	rte_thread_get_priority;
+	rte_thread_join;
 	rte_thread_self;
 	rte_thread_set_affinity_by_id;
 	rte_thread_set_priority;
lib/eal/windows/include/sched.h +1 -1
index bc31cc8..912fed1 100644
@@ -44,7 +44,7 @@
 	(1LL << _WHICH_BIT(b))) != 0LL)
 
 static inline int
-count_cpu(rte_cpuset_t *s)
+count_cpu(const rte_cpuset_t *s)
 {
 	unsigned int _i;
 	int count = 0;
lib/eal/windows/rte_thread.c +177 -40

235 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#3 lore raw

PATCH 3/6 eal: add basic rte thread ID equal API

In reply to Tyler Retzlaff

Add rte_thread_equal() that tests if two rte_thread_id are equal.

---
 lib/eal/common/rte_thread.c  |  6 ++++++
 lib/eal/include/rte_thread.h | 19 +++++++++++++++++++
 lib/eal/version.map          |  1 +
 3 files changed, 26 insertions(+)
lib/eal/common/rte_thread.c +6 -0
index 10d6652..21ed042 100644
@@ -6,6 +6,12 @@
 #include <rte_thread.h>
 
 int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return t1.opaque_id == t2.opaque_id;
+}
+
+int
 rte_thread_attr_init(rte_thread_attr_t *attr)
 {
 	RTE_VERIFY(attr != NULL);
lib/eal/include/rte_thread.h +19 -0
index 321fb59..32ab745 100644
@@ -134,6 +134,25 @@ int rte_thread_create(rte_thread_t *thread_id,
 __rte_experimental
 rte_thread_t rte_thread_self(void);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Check if 2 thread ids are equal.
+ *
+ * @param t1
+ *   First thread id.
+ *
+ * @param t2
+ *   Second thread id.
+ *
+ * @return
+ *   If the ids are equal, return nonzero.
+ *   Otherwise, return 0.
+ */
+__rte_experimental
+int rte_thread_equal(rte_thread_t t1, rte_thread_t t2);
+
 #ifdef RTE_HAS_CPUSET
 
 /**
lib/eal/version.map +1 -0
index 22e5c85..4a52484 100644
@@ -428,6 +428,7 @@ EXPERIMENTAL {
 	rte_thread_attr_set_priority;
 	rte_thread_create;
 	rte_thread_detach;
+	rte_thread_equal;
 	rte_thread_get_affinity_by_id;
 	rte_thread_get_priority;
 	rte_thread_join;
Signature · 1 line
1.8.3.1
#4 lore raw

Re: [PATCH 3/6] eal: add basic rte thread ID equal API

09/06/2022 14:58, Tyler Retzlaff пишет:

Quoted 15 lines

Add rte_thread_equal() that tests if two rte_thread_id are equal.

---
  lib/eal/common/rte_thread.c  |  6 ++++++
  lib/eal/include/rte_thread.h | 19 +++++++++++++++++++
  lib/eal/version.map          |  1 +
  3 files changed, 26 insertions(+)
lib/eal/common/rte_thread.c +0 -0
index 10d6652..21ed042 100644

lib/eal/common/rte_thread.c line 11

Quoted 6 lines
lib/eal/common/rte_thread.c +3 -0
@@ -6,6 +6,12 @@
  #include <rte_thread.h>
 
  int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return t1.opaque_id == t2.opaque_id;

for posix systems, why not:
return pthread_equal(t1.opaque_id, t2.opaque_id);
?

Quoted 48 lines
lib/eal/common/rte_thread.c +3 -0
+}
+
+int
  rte_thread_attr_init(rte_thread_attr_t *attr)
  {
  	RTE_VERIFY(attr != NULL);
lib/eal/include/rte_thread.h +19 -0
index 321fb59..32ab745 100644
@@ -134,6 +134,25 @@ int rte_thread_create(rte_thread_t *thread_id,
  __rte_experimental
  rte_thread_t rte_thread_self(void);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Check if 2 thread ids are equal.
+ *
+ * @param t1
+ *   First thread id.
+ *
+ * @param t2
+ *   Second thread id.
+ *
+ * @return
+ *   If the ids are equal, return nonzero.
+ *   Otherwise, return 0.
+ */
+__rte_experimental
+int rte_thread_equal(rte_thread_t t1, rte_thread_t t2);
+
  #ifdef RTE_HAS_CPUSET
 
  /**
lib/eal/version.map +1 -0
index 22e5c85..4a52484 100644
@@ -428,6 +428,7 @@ EXPERIMENTAL {
  	rte_thread_attr_set_priority;
  	rte_thread_create;
  	rte_thread_detach;
+	rte_thread_equal;
  	rte_thread_get_affinity_by_id;
  	rte_thread_get_priority;
  	rte_thread_join;
#5 lore raw

Re: [PATCH 3/6] eal: add basic rte thread ID equal API

Quoted 11 lines · Konstantin Ananyev

09/06/2022 14:58, Tyler Retzlaff пишет:

Quoted 10 lines

Add rte_thread_equal() that tests if two rte_thread_id are equal.

---
 lib/eal/common/rte_thread.c  |  6 ++++++
 lib/eal/include/rte_thread.h | 19 +++++++++++++++++++
 lib/eal/version.map          |  1 +
 3 files changed, 26 insertions(+)
Quoted 13 lines

lib/eal/common/rte_thread.c line 10

Quoted 10 lines
lib/eal/common/rte_thread.c +3 -0
index 10d6652..21ed042 100644
@@ -6,6 +6,12 @@
 #include <rte_thread.h>
 int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return t1.opaque_id == t2.opaque_id;

for posix systems, why not:
return pthread_equal(t1.opaque_id, t2.opaque_id);

because it would require 2 implementations when this works for both windows and posix platforms. (less code to maintain, no functional difference).

#6 lore raw

Re: [PATCH 3/6] eal: add basic rte thread ID equal API

Quoted 25 lines
Quoted 11 lines
Quoted 11 lines

Add rte_thread_equal() that tests if two rte_thread_id are equal.

---
  lib/eal/common/rte_thread.c  |  6 ++++++
  lib/eal/include/rte_thread.h | 19 +++++++++++++++++++
  lib/eal/version.map          |  1 +
  3 files changed, 26 insertions(+)
Quoted 13 lines

lib/eal/common/rte_thread.c line 10

Quoted 10 lines
lib/eal/common/rte_thread.c +3 -0
index 10d6652..21ed042 100644
@@ -6,6 +6,12 @@
  #include <rte_thread.h>
  int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return t1.opaque_id == t2.opaque_id;

for posix systems, why not:
return pthread_equal(t1.opaque_id, t2.opaque_id);

Quoted 1 line

because it would require 2 implementations

We already have plenty of such cases for rte_thread implementation. Why it became a problem here?

when this works for both

Quoted 3 lines

windows and posix platforms. (less code to maintain, no functional difference).

Well posix insists that the only safe way for applications to directly compare two pthread_t values is to call pthread_equal(). So I'd suggest we do what is recommended.

#7 lore raw

Re: [PATCH 3/6] eal: add basic rte thread ID equal API

Quoted 30 lines · Konstantin Ananyev
Quoted 25 lines
Quoted 11 lines
Quoted 11 lines

Add rte_thread_equal() that tests if two rte_thread_id are equal.

---
 lib/eal/common/rte_thread.c  |  6 ++++++
 lib/eal/include/rte_thread.h | 19 +++++++++++++++++++
 lib/eal/version.map          |  1 +
 3 files changed, 26 insertions(+)
Quoted 13 lines

lib/eal/common/rte_thread.c line 10

Quoted 10 lines
lib/eal/common/rte_thread.c +3 -0
index 10d6652..21ed042 100644
@@ -6,6 +6,12 @@
 #include <rte_thread.h>
 int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return t1.opaque_id == t2.opaque_id;

for posix systems, why not:
return pthread_equal(t1.opaque_id, t2.opaque_id);

Quoted 1 line

because it would require 2 implementations

We already have plenty of such cases for rte_thread implementation. Why it became a problem here?

Quoted 8 lines

when this works for both

Quoted 3 lines

windows and posix platforms. (less code to maintain, no functional difference).

Well posix insists that the only safe way for applications to directly compare two pthread_t values is to call pthread_equal(). So I'd suggest we do what is recommended.

agreed, will provide a v2 that uses pthread_equal().

Acked-by on v1 3/6
#8 lore raw

Re: [PATCH 3/6] eal: add basic rte thread ID equal API

In reply to Tyler Retzlaff
Quoted 70 lines · Tyler Retzlaff

Add rte_thread_equal() that tests if two rte_thread_id are equal.

---
 lib/eal/common/rte_thread.c  |  6 ++++++
 lib/eal/include/rte_thread.h | 19 +++++++++++++++++++
 lib/eal/version.map          |  1 +
 3 files changed, 26 insertions(+)
lib/eal/common/rte_thread.c +6 -0
index 10d6652..21ed042 100644
@@ -6,6 +6,12 @@
 #include <rte_thread.h>
 
 int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return t1.opaque_id == t2.opaque_id;
+}
+
+int
 rte_thread_attr_init(rte_thread_attr_t *attr)
 {
 	RTE_VERIFY(attr != NULL);
lib/eal/include/rte_thread.h +19 -0
index 321fb59..32ab745 100644
@@ -134,6 +134,25 @@ int rte_thread_create(rte_thread_t *thread_id,
 __rte_experimental
 rte_thread_t rte_thread_self(void);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Check if 2 thread ids are equal.
+ *
+ * @param t1
+ *   First thread id.
+ *
+ * @param t2
+ *   Second thread id.
+ *
+ * @return
+ *   If the ids are equal, return nonzero.
+ *   Otherwise, return 0.
+ */
+__rte_experimental
+int rte_thread_equal(rte_thread_t t1, rte_thread_t t2);
+
 #ifdef RTE_HAS_CPUSET
 
 /**
lib/eal/version.map +1 -0
index 22e5c85..4a52484 100644
@@ -428,6 +428,7 @@ EXPERIMENTAL {
 	rte_thread_attr_set_priority;
 	rte_thread_create;
 	rte_thread_detach;
+	rte_thread_equal;
 	rte_thread_get_affinity_by_id;
 	rte_thread_get_priority;
 	rte_thread_join;
#9 lore raw

PATCH 4/6 test/threads: add tests for thread lifetime API

In reply to Tyler Retzlaff

test basic functionality and demonstrate use of following thread lifetime api.

* rte_thread_create
* rte_thread_detach
* rte_thread_join

---
 app/test/test_threads.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)
app/test/test_threads.c +50 -0
index b9d8b4e..9a30af5 100644
@@ -27,6 +27,54 @@
 }
 
 static int
+test_thread_create_join(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_t thread_main_id;
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, &thread_main_id) == 0,
+		"Failed to create thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_equal(thread_id, thread_main_id) != 0,
+		"Unexpected thread id.");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	RTE_TEST_ASSERT(rte_thread_join(thread_id, NULL) == 0,
+		"Failed to join thread.");
+
+	return 0;
+}
+
+static int
+test_thread_create_detach(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_t thread_main_id;
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main,
+		&thread_main_id) == 0, "Failed to create thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_equal(thread_id, thread_main_id) != 0,
+		"Unexpected thread id.");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	RTE_TEST_ASSERT(rte_thread_detach(thread_id) == 0,
+		"Failed to detach thread.");
+
+	return 0;
+}
+
+static int
 test_thread_priority(void)
 {
 	pthread_t id;
@@ -123,6 +171,8 @@
 	.setup = NULL,
 	.teardown = NULL,
 	.unit_test_cases = {
+		TEST_CASE(test_thread_create_join),
+		TEST_CASE(test_thread_create_detach),
 		TEST_CASE(test_thread_affinity),
 		TEST_CASE(test_thread_priority),
 		TEST_CASES_END()
Signature · 1 line
1.8.3.1
#10 lore raw

PATCH 5/6 test/threads: add tests for thread attributes API

In reply to Tyler Retzlaff

test basic functionality and demonstrate use of following thread attributes api. additionally, test attributes are processed when supplied to rte_thread_create().

* rte_thread_attr_init
* rte_thread_attr_set_affinity
* rte_thread_attr_get_affinity
* rte_thread_attr_set_priority

---
 app/test/test_threads.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 72 insertions(+), 1 deletion(-)
app/test/test_threads.c +72 -1
index 9a30af5..92a149a 100644
@@ -17,7 +17,9 @@
 static void *
 thread_main(void *arg)
 {
-	*(rte_thread_t *)arg = rte_thread_self();
+	if (arg != NULL)
+		*(rte_thread_t *)arg = rte_thread_self();
+
 	__atomic_store_n(&thread_id_ready, 1, __ATOMIC_RELEASE);
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 1)
@@ -166,6 +168,73 @@
 	return 0;
 }
 
+static int
+test_thread_attributes_affinity(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_attr_t attr;
+	rte_cpuset_t cpuset0;
+	rte_cpuset_t cpuset1;
+
+	RTE_TEST_ASSERT(rte_thread_attr_init(&attr) == 0,
+		"Failed to initialize thread attributes");
+
+	CPU_ZERO(&cpuset0);
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(rte_thread_self(), &cpuset0) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(rte_thread_attr_set_affinity(&attr, &cpuset0) == 0,
+		"Failed to set thread attributes affinity");
+	RTE_TEST_ASSERT(rte_thread_attr_get_affinity(&attr, &cpuset1) == 0,
+		"Failed to get thread attributes affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, &attr, thread_main, NULL) == 0,
+		"Failed to create attributes affinity thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get attributes thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Failed to apply affinity attributes");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	return 0;
+}
+
+static int
+test_thread_attributes_priority(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_attr_t attr;
+	enum rte_thread_priority priority;
+
+	RTE_TEST_ASSERT(rte_thread_attr_init(&attr) == 0,
+		"Failed to initialize thread attributes");
+	RTE_TEST_ASSERT(rte_thread_attr_set_priority(&attr, RTE_THREAD_PRIORITY_NORMAL) == 0,
+		"Failed to set thread attributes priority");
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, &attr, thread_main, NULL) == 0,
+		"Failed to create attributes priority thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_get_priority(thread_id, &priority) == 0,
+		"Failed to get thread priority");
+	RTE_TEST_ASSERT(priority == RTE_THREAD_PRIORITY_NORMAL,
+		"Failed to apply priority attributes");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	return 0;
+}
+
 static struct unit_test_suite threads_test_suite = {
 	.suite_name = "threads autotest",
 	.setup = NULL,
@@ -175,6 +244,8 @@
 		TEST_CASE(test_thread_create_detach),
 		TEST_CASE(test_thread_affinity),
 		TEST_CASE(test_thread_priority),
+		TEST_CASE(test_thread_attributes_affinity),
+		TEST_CASE(test_thread_attributes_priority),
 		TEST_CASES_END()
 	}
 };
Signature · 1 line
1.8.3.1
#11 lore raw

PATCH 6/6 test/threads: remove unit test use of pthread

In reply to Tyler Retzlaff

now that rte_thread provides thread lifetime functions stop using pthread in unit tests.

---
 app/test/test_threads.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
app/test/test_threads.c +2 -5
index 92a149a..034f82d 100644
@@ -3,7 +3,6 @@
  */
 
 #include <string.h>
-#include <pthread.h>
 
 #include <rte_thread.h>
 #include <rte_debug.h>
@@ -79,12 +78,11 @@
 static int
 test_thread_priority(void)
 {
-	pthread_t id;
 	rte_thread_t thread_id;
 	enum rte_thread_priority priority;
 
 	thread_id_ready = 0;
-	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, NULL) == 0,
 		"Failed to create thread");
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
@@ -131,13 +129,12 @@
 static int
 test_thread_affinity(void)
 {
-	pthread_t id;
 	rte_thread_t thread_id;
 	rte_cpuset_t cpuset0;
 	rte_cpuset_t cpuset1;
 
 	thread_id_ready = 0;
-	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, NULL) == 0,
 		"Failed to create thread");
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
Signature · 1 line
1.8.3.1
#12 lore raw

v2 cover add thread lifetime and attributes API

In reply to Tyler Retzlaff

add rte thread lifetime and attributes api. with these api additions there is now sufficient platform abstracted thread api to remove the use of pthread in the unit tests.

v2:
* split implementation of rte_thread_equal for windows / posix and use pthread_equal for posix platforms.
* remove parameter validation assertions and instead return EINVAL for mandatory pointers to type that are NULL.
* correct doxygen comment parameter name args -> arg

Tyler Retzlaff (6):
eal: add thread attributes
eal: add thread lifetime management
eal: add basic rte thread ID equal API test/threads: add tests for thread lifetime API test/threads: add tests for thread attributes API test/threads: remove unit test use of pthread

 app/test/test_threads.c         | 130 +++++++++++++++++++++--
 lib/eal/common/meson.build      |   1 +
 lib/eal/common/rte_thread.c     |  60 +++++++++++
 lib/eal/include/rte_thread.h    | 177 +++++++++++++++++++++++++++++++
 lib/eal/unix/rte_thread.c       | 114 ++++++++++++++++++++
 lib/eal/version.map             |   8 ++
 lib/eal/windows/include/sched.h |   2 +-
 lib/eal/windows/rte_thread.c    | 223 +++++++++++++++++++++++++++++++++-------
 8 files changed, 668 insertions(+), 47 deletions(-)
 create mode 100644 lib/eal/common/rte_thread.c
Signature · 1 line
1.8.3.1
#13 lore raw

PATCH v2 1/6 eal: add thread attributes

Implement thread attributes for:

* thread affinity
* thread priority

Implement functions for managing thread attributes.

Priority is represented through an enum that allows for two levels:

* RTE_THREAD_PRIORITY_NORMAL
* RTE_THREAD_PRIORITY_REALTIME_CRITICAL

Affinity is described by the rte_cpuset_t type.

An rte_thread_attr_t object can be set to the default values by calling rte_thread_attr_init().

---
 lib/eal/common/meson.build   |  1 +
 lib/eal/common/rte_thread.c  | 60 ++++++++++++++++++++++++++++
 lib/eal/include/rte_thread.h | 93 ++++++++++++++++++++++++++++++++++++++++++++
 lib/eal/version.map          |  4 ++
 4 files changed, 158 insertions(+)
 create mode 100644 lib/eal/common/rte_thread.c
lib/eal/common/meson.build +1 -0
index 917758c..1e77dba 100644
@@ -36,6 +36,7 @@ sources += files(
         'rte_random.c',
         'rte_reciprocal.c',
         'rte_service.c',
+        'rte_thread.c',
         'rte_version.c',
 )
 if is_linux or is_windows
lib/eal/common/rte_thread.c new file +60 -0

60 lines not on this page · raw message

lib/eal/include/rte_thread.h +93 -0

105 lines not on this page · raw message

lib/eal/version.map +4 -0
index ab27a4b..72e0e14 100644
@@ -422,6 +422,10 @@ EXPERIMENTAL {
 	rte_intr_type_set;
 
 	# added in 22.07
+	rte_thread_attr_get_affinity;
+	rte_thread_attr_init;
+	rte_thread_attr_set_affinity;
+	rte_thread_attr_set_priority;
 	rte_thread_get_affinity_by_id;
 	rte_thread_get_priority;
 	rte_thread_self;
Signature · 1 line
1.8.3.1
#14 lore raw

PATCH v2 2/6 eal: add thread lifetime management

In reply to Tyler Retzlaff

The *rte_thread_create()* function can optionally receive an rte_thread_attr_t object that will cause the thread to be created with the affinity and priority described by the attributes object. If no rte_thread_attr_t is passed (parameter is NULL), the default affinity and priority are used.

On Windows, the function executed by a thread when the thread starts is represeneted by a function pointer of type DWORD (*func) (void*). On other platforms, the function pointer is a void* (*func) (void*).

Performing a cast between these two types of function pointers to uniformize the API on all platforms may result in undefined behavior. TO fix this issue, a wrapper that respects the signature required by CreateThread() has been created on Windows.

---
 lib/eal/include/rte_thread.h    |  65 ++++++++++++
 lib/eal/unix/rte_thread.c       | 108 ++++++++++++++++++++
 lib/eal/version.map             |   3 +
 lib/eal/windows/include/sched.h |   2 +-
 lib/eal/windows/rte_thread.c    | 217 ++++++++++++++++++++++++++++++++--------
 5 files changed, 354 insertions(+), 41 deletions(-)
lib/eal/include/rte_thread.h +65 -0
index 062308d..c27e580 100644
@@ -30,6 +30,8 @@
 	uintptr_t opaque_id; /**< thread identifier */
 } rte_thread_t;
 
+typedef void * (*rte_thread_func) (void *);
+
 /**
  * Thread priority values.
  */
@@ -61,6 +63,69 @@ enum rte_thread_priority {
  * @warning
  * @b EXPERIMENTAL: this API may change without prior notice.
  *
+ * Create a new thread that will invoke the 'thread_func' routine.
+ *
+ * @param thread_id
+ *    A pointer that will store the id of the newly created thread.
+ *
+ * @param thread_attr
+ *    Attributes that are used at the creation of the new thread.
+ *
+ * @param thread_func
+ *    The routine that the new thread will invoke when starting execution.
+ *
+ * @param arg
+ *    Argument to be passed to the 'thread_func' routine.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_create(rte_thread_t *thread_id,
+		const rte_thread_attr_t *thread_attr,
+		rte_thread_func thread_func, void *arg);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Waits for the thread identified by 'thread_id' to terminate
+ *
+ * @param thread_id
+ *    The identifier of the thread.
+ *
+ * @param value_ptr
+ *    Stores the exit status of the thread.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_join(rte_thread_t thread_id, unsigned long *value_ptr);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Indicate that the return value of the thread is not needed and
+ * all thread resources should be release when the thread terminates.
+ *
+ * @param thread_id
+ *    The id of the thread to be detached.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_detach(rte_thread_t thread_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
  * Get the id of the calling thread.
  *
  * @return
lib/eal/unix/rte_thread.c +108 -0

114 lines not on this page · raw message

lib/eal/version.map +3 -0
index 72e0e14..22e5c85 100644
@@ -426,8 +426,11 @@ EXPERIMENTAL {
 	rte_thread_attr_init;
 	rte_thread_attr_set_affinity;
 	rte_thread_attr_set_priority;
+	rte_thread_create;
+	rte_thread_detach;
 	rte_thread_get_affinity_by_id;
 	rte_thread_get_priority;
+	rte_thread_join;
 	rte_thread_self;
 	rte_thread_set_affinity_by_id;
 	rte_thread_set_priority;
lib/eal/windows/include/sched.h +1 -1
index bc31cc8..912fed1 100644
@@ -44,7 +44,7 @@
 	(1LL << _WHICH_BIT(b))) != 0LL)
 
 static inline int
-count_cpu(rte_cpuset_t *s)
+count_cpu(const rte_cpuset_t *s)
 {
 	unsigned int _i;
 	int count = 0;
lib/eal/windows/rte_thread.c +177 -40

235 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#15 lore raw

Re: [PATCH v2 2/6] eal: add thread lifetime management

2022-06-14 16:47 (UTC-0700), Tyler Retzlaff:

Quoted 8 lines

On Windows, the function executed by a thread when the thread starts is represeneted by a function pointer of type DWORD (*func) (void*). On other platforms, the function pointer is a void* (*func) (void*).

Performing a cast between these two types of function pointers to uniformize the API on all platforms may result in undefined behavior. TO fix this issue, a wrapper that respects the signature required by CreateThread() has been created on Windows.

The interface issue is still there:
rte_thread_func allows the thread routine to have a pointer-sized result. rte_thread_join() allows to obtain this value as unsigned long, which is pointer-sized on 32-bit platforms
and less than that on 64-bit platforms.
This can lead to issues when developers assume they can return a pointer and this works on 32 bits, but doesn't work on 64 bits.
If you want to keep API as close to phtread as possible,
the limitation must be at least clearly documented in Doxygen (rte_thread_func is undocumented BTW).
I also suggest using uint32_t instead of unsigned long precisely to avoid "is it pointer-sized?" doubts.
(I don't see much benefit in keeping pthread-like signature. When moving from pthread to rte_thread,
it is as trivial to change the thread routine return type.)

Quoted 5 lines
lib/eal/unix/rte_thread.c +5 -0
+int
+rte_thread_create(rte_thread_t *thread_id,
+		const rte_thread_attr_t *thread_attr,
+		rte_thread_func thread_func, void *args)
+{

lib/eal/unix/rte_thread.c struct eal_tls_key { · line 120

Quoted 10 lines
lib/eal/unix/rte_thread.c +9 -0
[...]
+		if (thread_attr->priority ==
+				RTE_THREAD_PRIORITY_REALTIME_CRITICAL) {
+			ret = ENOTSUP;
+			goto cleanup;
+		}
+		ret = thread_map_priority_to_os_value(thread_attr->priority,
+				&param.sched_priority, &policy);
+		if (ret != 0)
+			goto cleanup;

thread_map_priority_to_os_value() already checks for unsupported values, why not let it do this particular check?

Quoted 11 lines
lib/eal/unix/rte_thread.c +11 -0
+int
+rte_thread_join(rte_thread_t thread_id, unsigned long *value_ptr)
+{
+	int ret = 0;
+	void *res = NULL;
+	void **pres = NULL;
+
+	if (value_ptr != NULL)
+		pres = &res;
+
+	ret = pthread_join((pthread_t)thread_id.opaque_id, pres);

lib/eal/unix/rte_thread.c struct eal_tls_key { · line 178

Quoted 10 lines
lib/eal/unix/rte_thread.c +10 -0
+	if (ret != 0) {
+		RTE_LOG(DEBUG, EAL, "pthread_join failed\n");
+		return ret;
+	}
+
+	if (value_ptr != NULL && *pres != NULL)
+		*value_ptr = *(unsigned long *)(*pres);
+
+	return 0;
+}

What makes *pres == NULL special?

lib/eal/windows/rte_thread.c struct eal_tls_key { · line 170

Quoted 8 lines
lib/eal/windows/rte_thread.c +8 -0
+static DWORD
+thread_func_wrapper(void *args)
+{
+	struct thread_routine_ctx *pctx = args;
+	struct thread_routine_ctx ctx;
+
+	ctx.thread_func = pctx->thread_func;
+	ctx.routine_args = pctx->routine_args;

ctx = *pctx?

Quoted 22 lines
lib/eal/windows/rte_thread.c +22 -0
+
+	free(pctx);
+
+	return (DWORD)(uintptr_t)ctx.thread_func(ctx.routine_args);
+}
+
+int
+rte_thread_create(rte_thread_t *thread_id,
+		  const rte_thread_attr_t *thread_attr,
+		  rte_thread_func thread_func, void *args)
+{
+	int ret = 0;
+	DWORD tid;
+	HANDLE thread_handle = NULL;
+	GROUP_AFFINITY thread_affinity;
+	struct thread_routine_ctx *ctx = NULL;
+
+	ctx = calloc(1, sizeof(*ctx));
+	if (ctx == NULL) {
+		RTE_LOG(DEBUG, EAL, "Insufficient memory for thread context allocations\n");
+		ret = ENOMEM;
+		goto cleanup;

lib/eal/windows/rte_thread.c struct eal_tls_key { · line 202

Quoted 10 lines
lib/eal/windows/rte_thread.c +10 -0
+	}
+	ctx->routine_args = args;
+	ctx->thread_func = thread_func;
+
+	thread_handle = CreateThread(NULL, 0, thread_func_wrapper, ctx,
+		CREATE_SUSPENDED, &tid);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("CreateThread()");
+		free(ctx);
+		goto cleanup;

Missing free(ctx) from other error paths below.

Quoted 41 lines
lib/eal/windows/rte_thread.c +41 -0
+	}
+	thread_id->opaque_id = tid;
+
+	if (thread_attr != NULL) {
+		if (CPU_COUNT(&thread_attr->cpuset) > 0) {
+			ret = convert_cpuset_to_affinity(
+							&thread_attr->cpuset,
+							&thread_affinity
+							);
+			if (ret != 0) {
+				RTE_LOG(DEBUG, EAL, "Unable to convert cpuset to thread affinity\n");
+				goto cleanup;
+			}
+
+			if (!SetThreadGroupAffinity(thread_handle,
+						    &thread_affinity, NULL)) {
+				ret = thread_log_last_error("SetThreadGroupAffinity()");
+				goto cleanup;
+			}
+		}
+		ret = rte_thread_set_priority(*thread_id,
+				thread_attr->priority);
+		if (ret != 0) {
+			RTE_LOG(DEBUG, EAL, "Unable to set thread priority\n");
+			goto cleanup;
+		}
+	}
+
+	if (ResumeThread(thread_handle) == (DWORD)-1) {
+		ret = thread_log_last_error("ResumeThread()");
+		goto cleanup;
+	}
+
+cleanup:
+	if (thread_handle != NULL) {
+		CloseHandle(thread_handle);
+		thread_handle = NULL;
+	}
+
+	return ret;
+}
#16 lore raw

Re: [PATCH v2 2/6] eal: add thread lifetime management

Quoted 10 lines · Dmitry Kozlyuk

2022-06-14 16:47 (UTC-0700), Tyler Retzlaff:

Quoted 8 lines

On Windows, the function executed by a thread when the thread starts is represeneted by a function pointer of type DWORD (*func) (void*). On other platforms, the function pointer is a void* (*func) (void*).

Performing a cast between these two types of function pointers to uniformize the API on all platforms may result in undefined behavior. TO fix this issue, a wrapper that respects the signature required by CreateThread() has been created on Windows.

Quoted 15 lines

The interface issue is still there:
rte_thread_func allows the thread routine to have a pointer-sized result. rte_thread_join() allows to obtain this value as unsigned long, which is pointer-sized on 32-bit platforms
and less than that on 64-bit platforms.
This can lead to issues when developers assume they can return a pointer and this works on 32 bits, but doesn't work on 64 bits.
If you want to keep API as close to phtread as possible,
the limitation must be at least clearly documented in Doxygen (rte_thread_func is undocumented BTW).
I also suggest using uint32_t instead of unsigned long precisely to avoid "is it pointer-sized?" doubts.
(I don't see much benefit in keeping pthread-like signature. When moving from pthread to rte_thread,
it is as trivial to change the thread routine return type.)

thanks Dmitry, i wasn't aware this feedback had been given previously. let me digest the problem and we'll figure out how we can address it.

Quoted 11 lines
Quoted 11 lines
lib/eal/unix/rte_thread.c +9 -0
+int
+rte_thread_create(rte_thread_t *thread_id,
+		const rte_thread_attr_t *thread_attr,
+		rte_thread_func thread_func, void *args)
+{
[...]
+		if (thread_attr->priority ==
+				RTE_THREAD_PRIORITY_REALTIME_CRITICAL) {
+			ret = ENOTSUP;
+			goto cleanup;
Quoted 8 lines

lib/eal/unix/rte_thread.c struct eal_tls_key { · line 120

Quoted 5 lines
lib/eal/unix/rte_thread.c +5 -0
+		}
+		ret = thread_map_priority_to_os_value(thread_attr->priority,
+				&param.sched_priority, &policy);
+		if (ret != 0)
+			goto cleanup;

thread_map_priority_to_os_value() already checks for unsupported values, why not let it do this particular check?

hm, i hadn't noticed that was duplicated. let me try to eliminate it.

Quoted 25 lines
Quoted 12 lines
lib/eal/unix/rte_thread.c +11 -0
+int
+rte_thread_join(rte_thread_t thread_id, unsigned long *value_ptr)
+{
+	int ret = 0;
+	void *res = NULL;
+	void **pres = NULL;
+
+	if (value_ptr != NULL)
+		pres = &res;
+
+	ret = pthread_join((pthread_t)thread_id.opaque_id, pres);

lib/eal/unix/rte_thread.c struct eal_tls_key { · line 178

Quoted 10 lines
lib/eal/unix/rte_thread.c +10 -0
+	if (ret != 0) {
+		RTE_LOG(DEBUG, EAL, "pthread_join failed\n");
+		return ret;
+	}
+
+	if (value_ptr != NULL && *pres != NULL)
+		*value_ptr = *(unsigned long *)(*pres);
+
+	return 0;
+}

What makes *pres == NULL special?

Quoted 10 lines

lib/eal/windows/rte_thread.c struct eal_tls_key { · line 170

Quoted 8 lines
lib/eal/windows/rte_thread.c +8 -0
+static DWORD
+thread_func_wrapper(void *args)
+{
+	struct thread_routine_ctx *pctx = args;
+	struct thread_routine_ctx ctx;
+
+	ctx.thread_func = pctx->thread_func;
+	ctx.routine_args = pctx->routine_args;

ctx = *pctx?

ack

Quoted 27 lines
Quoted 27 lines
lib/eal/windows/rte_thread.c +26 -0
+
+	free(pctx);
+
+	return (DWORD)(uintptr_t)ctx.thread_func(ctx.routine_args);
+}
+
+int
+rte_thread_create(rte_thread_t *thread_id,
+		  const rte_thread_attr_t *thread_attr,
+		  rte_thread_func thread_func, void *args)
+{
+	int ret = 0;
+	DWORD tid;
+	HANDLE thread_handle = NULL;
+	GROUP_AFFINITY thread_affinity;
+	struct thread_routine_ctx *ctx = NULL;
+
+	ctx = calloc(1, sizeof(*ctx));
+	if (ctx == NULL) {
+		RTE_LOG(DEBUG, EAL, "Insufficient memory for thread context allocations\n");
+		ret = ENOMEM;
+		goto cleanup;
+	}
+	ctx->routine_args = args;
+	ctx->thread_func = thread_func;
+
Quoted 8 lines

lib/eal/windows/rte_thread.c struct eal_tls_key { · line 202

Quoted 6 lines
lib/eal/windows/rte_thread.c +6 -0
+	thread_handle = CreateThread(NULL, 0, thread_func_wrapper, ctx,
+		CREATE_SUSPENDED, &tid);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("CreateThread()");
+		free(ctx);
+		goto cleanup;

Missing free(ctx) from other error paths below.

ack

Quoted 34 lines
Quoted 34 lines
lib/eal/windows/rte_thread.c +33 -0
+	}
+	thread_id->opaque_id = tid;
+
+	if (thread_attr != NULL) {
+		if (CPU_COUNT(&thread_attr->cpuset) > 0) {
+			ret = convert_cpuset_to_affinity(
+							&thread_attr->cpuset,
+							&thread_affinity
+							);
+			if (ret != 0) {
+				RTE_LOG(DEBUG, EAL, "Unable to convert cpuset to thread affinity\n");
+				goto cleanup;
+			}
+
+			if (!SetThreadGroupAffinity(thread_handle,
+						    &thread_affinity, NULL)) {
+				ret = thread_log_last_error("SetThreadGroupAffinity()");
+				goto cleanup;
+			}
+		}
+		ret = rte_thread_set_priority(*thread_id,
+				thread_attr->priority);
+		if (ret != 0) {
+			RTE_LOG(DEBUG, EAL, "Unable to set thread priority\n");
+			goto cleanup;
+		}
+	}
+
+	if (ResumeThread(thread_handle) == (DWORD)-1) {
+		ret = thread_log_last_error("ResumeThread()");
+		goto cleanup;
+	}
+
Quoted 8 lines

lib/eal/windows/rte_thread.c struct eal_tls_key { · line 243

Quoted 8 lines
lib/eal/windows/rte_thread.c +8 -0
+cleanup:
+	if (thread_handle != NULL) {
+		CloseHandle(thread_handle);
+		thread_handle = NULL;
+	}
+
+	return ret;
+}

thanks let me tidy those bits up, i'll submit a revision this week.

#17 lore raw

Re: [PATCH v2 2/6] eal: add thread lifetime management

In reply to Dmitry Kozlyuk
Quoted 12 lines · Dmitry Kozlyuk

2022-06-14 16:47 (UTC-0700), Tyler Retzlaff:

Quoted 10 lines
lib/eal/unix/rte_thread.c +9 -0
+int
+rte_thread_create(rte_thread_t *thread_id,
+		const rte_thread_attr_t *thread_attr,
+		rte_thread_func thread_func, void *args)
+{
[...]
+		if (thread_attr->priority ==
+				RTE_THREAD_PRIORITY_REALTIME_CRITICAL) {
+			ret = ENOTSUP;
+			goto cleanup;
Quoted 8 lines

lib/eal/unix/rte_thread.c struct eal_tls_key { · line 120

Quoted 5 lines
lib/eal/unix/rte_thread.c +5 -0
+		}
+		ret = thread_map_priority_to_os_value(thread_attr->priority,
+				&param.sched_priority, &policy);
+		if (ret != 0)
+			goto cleanup;

thread_map_priority_to_os_value() already checks for unsupported values, why not let it do this particular check?

okay, so i looked at this more closely.

* thread_map_priority_to_os_value() just does mapping and RTE_THREAD_PRIORITY_REALTIME_CRITICAL is a valid mapping so it does not fail. by design it does one thing and one thing only perform the value mapping. admittedly it does not map every valid value right now, maybe it should?

for consistency in the behavior of the function i'd suggest we don't make it have special cases in case we later decide to expand for additional valid mappings.

* rte_thread_set_priority() does fail with ENOTSUP on non-windows if provided RTE_THREAD_PRIORITY_REALTIME_CRITICAL however it cannot be used at this point since the thread has not been created.

if no further follow up on this i'm going to leave it as is.

thanks

#18 lore raw

Re: [PATCH v2 2/6] eal: add thread lifetime management

In reply to Dmitry Kozlyuk
Quoted 10 lines · Dmitry Kozlyuk

2022-06-14 16:47 (UTC-0700), Tyler Retzlaff:

Quoted 8 lines

On Windows, the function executed by a thread when the thread starts is represeneted by a function pointer of type DWORD (*func) (void*). On other platforms, the function pointer is a void* (*func) (void*).

Performing a cast between these two types of function pointers to uniformize the API on all platforms may result in undefined behavior. TO fix this issue, a wrapper that respects the signature required by CreateThread() has been created on Windows.

Quoted 15 lines

The interface issue is still there:
rte_thread_func allows the thread routine to have a pointer-sized result. rte_thread_join() allows to obtain this value as unsigned long, which is pointer-sized on 32-bit platforms
and less than that on 64-bit platforms.
This can lead to issues when developers assume they can return a pointer and this works on 32 bits, but doesn't work on 64 bits.
If you want to keep API as close to phtread as possible,
the limitation must be at least clearly documented in Doxygen (rte_thread_func is undocumented BTW).
I also suggest using uint32_t instead of unsigned long precisely to avoid "is it pointer-sized?" doubts.
(I don't see much benefit in keeping pthread-like signature. When moving from pthread to rte_thread,
it is as trivial to change the thread routine return type.)

i'll alter the rte api to use fixed width uint32_t for thread result. it seems like an unnecessary feature to return a pointer and it can be accomplished through void *arg if necessary.

Quoted 16 lines
Quoted 16 lines
lib/eal/unix/rte_thread.c +15 -0
+int
+rte_thread_join(rte_thread_t thread_id, unsigned long *value_ptr)
+{
+	int ret = 0;
+	void *res = NULL;
+	void **pres = NULL;
+
+	if (value_ptr != NULL)
+		pres = &res;
+
+	ret = pthread_join((pthread_t)thread_id.opaque_id, pres);
+	if (ret != 0) {
+		RTE_LOG(DEBUG, EAL, "pthread_join failed\n");
+		return ret;
+	}
Quoted 8 lines

lib/eal/unix/rte_thread.c struct eal_tls_key { · line 178

Quoted 6 lines
lib/eal/unix/rte_thread.c +6 -0
+
+	if (value_ptr != NULL && *pres != NULL)
+		*value_ptr = *(unsigned long *)(*pres);
+
+	return 0;
+}

What makes *pres == NULL special?

it's not clear what you mean, can you explain? maybe there is some context i am missing from the original patch series?

Quoted 20 lines
Quoted 20 lines
lib/eal/windows/rte_thread.c +20 -0
+int
+rte_thread_create(rte_thread_t *thread_id,
+		  const rte_thread_attr_t *thread_attr,
+		  rte_thread_func thread_func, void *args)
+{
+	int ret = 0;
+	DWORD tid;
+	HANDLE thread_handle = NULL;
+	GROUP_AFFINITY thread_affinity;
+	struct thread_routine_ctx *ctx = NULL;
+
+	ctx = calloc(1, sizeof(*ctx));
+	if (ctx == NULL) {
+		RTE_LOG(DEBUG, EAL, "Insufficient memory for thread context allocations\n");
+		ret = ENOMEM;
+		goto cleanup;
+	}
+	ctx->routine_args = args;
+	ctx->thread_func = thread_func;
+
Quoted 8 lines

lib/eal/windows/rte_thread.c struct eal_tls_key { · line 202

Quoted 6 lines
lib/eal/windows/rte_thread.c +6 -0
+	thread_handle = CreateThread(NULL, 0, thread_func_wrapper, ctx,
+		CREATE_SUSPENDED, &tid);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("CreateThread()");
+		free(ctx);
+		goto cleanup;

Missing free(ctx) from other error paths below.

beyond this point free(ctx) will happen in thread_func_wrapper. i will add a comment to make it clear.

thanks.

#19 lore raw

Re: [PATCH v2 2/6] eal: add thread lifetime management

2022-06-21 11:51 (UTC-0700), Tyler Retzlaff:

Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
lib/eal/unix/rte_thread.c +15 -0
+int
+rte_thread_join(rte_thread_t thread_id, unsigned long *value_ptr)
+{
+	int ret = 0;
+	void *res = NULL;
+	void **pres = NULL;
+
+	if (value_ptr != NULL)
+		pres = &res;
+
+	ret = pthread_join((pthread_t)thread_id.opaque_id, pres);
+	if (ret != 0) {
+		RTE_LOG(DEBUG, EAL, "pthread_join failed\n");
+		return ret;
+	}
Quoted 8 lines

lib/eal/unix/rte_thread.c struct eal_tls_key { · line 178

Quoted 6 lines
lib/eal/unix/rte_thread.c +6 -0
+
+	if (value_ptr != NULL && *pres != NULL)
+		*value_ptr = *(unsigned long *)(*pres);
+
+	return 0;
+}  

What makes *pres == NULL special?

Quoted 2 lines

it's not clear what you mean, can you explain? maybe there is some context i am missing from the original patch series?

There's no previous context.
After ptread_join(), *pres holds the return value of the thread routine. You only assign *value_ptr if value_ptr is not NULL (obviously correct) and if *pres != NULL, that is, if the thread returned a non-NULL value. But this value is opaque, why do you filter NULL?
Perhaps you meant if (pres != NULL), no dereference?

Quoted 29 lines
Quoted 20 lines
Quoted 20 lines
lib/eal/windows/rte_thread.c +20 -0
+int
+rte_thread_create(rte_thread_t *thread_id,
+		  const rte_thread_attr_t *thread_attr,
+		  rte_thread_func thread_func, void *args)
+{
+	int ret = 0;
+	DWORD tid;
+	HANDLE thread_handle = NULL;
+	GROUP_AFFINITY thread_affinity;
+	struct thread_routine_ctx *ctx = NULL;
+
+	ctx = calloc(1, sizeof(*ctx));
+	if (ctx == NULL) {
+		RTE_LOG(DEBUG, EAL, "Insufficient memory for thread context allocations\n");
+		ret = ENOMEM;
+		goto cleanup;
+	}
+	ctx->routine_args = args;
+	ctx->thread_func = thread_func;
+
Quoted 8 lines

lib/eal/windows/rte_thread.c struct eal_tls_key { · line 202

Quoted 6 lines
lib/eal/windows/rte_thread.c +6 -0
+	thread_handle = CreateThread(NULL, 0, thread_func_wrapper, ctx,
+		CREATE_SUSPENDED, &tid);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("CreateThread()");
+		free(ctx);
+		goto cleanup;  

Missing free(ctx) from other error paths below.

Quoted 2 lines

beyond this point free(ctx) will happen in thread_func_wrapper. i will add a comment to make it clear.

Not if you exit before ResumeThread()
and thread_func_wrapper() will never execute to call free().

#20 lore raw

Re: [PATCH v2 2/6] eal: add thread lifetime management

Quoted 25 lines · Dmitry Kozlyuk

2022-06-21 11:51 (UTC-0700), Tyler Retzlaff:

Quoted 24 lines
Quoted 23 lines
Quoted 11 lines
lib/eal/unix/rte_thread.c +11 -0
+int
+rte_thread_join(rte_thread_t thread_id, unsigned long *value_ptr)
+{
+	int ret = 0;
+	void *res = NULL;
+	void **pres = NULL;
+
+	if (value_ptr != NULL)
+		pres = &res;
+
+	ret = pthread_join((pthread_t)thread_id.opaque_id, pres);

lib/eal/unix/rte_thread.c struct eal_tls_key { · line 178

Quoted 10 lines
lib/eal/unix/rte_thread.c +10 -0
+	if (ret != 0) {
+		RTE_LOG(DEBUG, EAL, "pthread_join failed\n");
+		return ret;
+	}
+
+	if (value_ptr != NULL && *pres != NULL)
+		*value_ptr = *(unsigned long *)(*pres);
+
+	return 0;
+}  

What makes *pres == NULL special?

Quoted 8 lines
Quoted 2 lines

it's not clear what you mean, can you explain? maybe there is some context i am missing from the original patch series?

There's no previous context.
After ptread_join(), *pres holds the return value of the thread routine. You only assign *value_ptr if value_ptr is not NULL (obviously correct) and if *pres != NULL, that is, if the thread returned a non-NULL value. But this value is opaque, why do you filter NULL?

i don't think it is opaque here? unsigned long * value_ptr says we have to store an integer. which leads to a discussion of what should get stored at the value_ptr location if pthread_join() itself returns no result but the caller of rte_thread_join() requests the result.

Quoted 1 line

Perhaps you meant if (pres != NULL), no dereference?

that i think is just a repeat of a test checking if the caller of rte_thread_join is interested in the result?
i.e. value_ptr != NULL -> pres != NULL

both pres and *pres are dereferenced so it seems to track that prior to those dereferences they have to be validated as being non-NULL. i don't see how we could avoid dereferencing **pres to satisfy the calling contract when the result is requested.

now if value_ptr was unsigned long ** i guess i'd understand. i could always be reading the code wrong. but thinking about further there is another problem with this in that we really don't know what is being aliased in *pres when using the pthread implementation, since pthread could be returning a pointer to something narrow or with unknown layout where later dereferencing it as something wider or in this case specifically as unsigned long * would have horrible consequences.

i think this ends up semi-related to your other comment about what the result type from rte_thread_func is, we can discuss offline and post details back to the list.

Quoted 30 lines
Quoted 30 lines
Quoted 29 lines
Quoted 17 lines
lib/eal/windows/rte_thread.c +16 -0
+int
+rte_thread_create(rte_thread_t *thread_id,
+		  const rte_thread_attr_t *thread_attr,
+		  rte_thread_func thread_func, void *args)
+{
+	int ret = 0;
+	DWORD tid;
+	HANDLE thread_handle = NULL;
+	GROUP_AFFINITY thread_affinity;
+	struct thread_routine_ctx *ctx = NULL;
+
+	ctx = calloc(1, sizeof(*ctx));
+	if (ctx == NULL) {
+		RTE_LOG(DEBUG, EAL, "Insufficient memory for thread context allocations\n");
+		ret = ENOMEM;
+		goto cleanup;

lib/eal/windows/rte_thread.c struct eal_tls_key { · line 202

Quoted 10 lines
lib/eal/windows/rte_thread.c +10 -0
+	}
+	ctx->routine_args = args;
+	ctx->thread_func = thread_func;
+
+	thread_handle = CreateThread(NULL, 0, thread_func_wrapper, ctx,
+		CREATE_SUSPENDED, &tid);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("CreateThread()");
+		free(ctx);
+		goto cleanup;  

Missing free(ctx) from other error paths below.

Quoted 5 lines
Quoted 2 lines

beyond this point free(ctx) will happen in thread_func_wrapper. i will add a comment to make it clear.

Not if you exit before ResumeThread()
and thread_func_wrapper() will never execute to call free().

yes, you are right i forgot that this thread is created suspended.

#21 lore raw

Re: [PATCH v2 2/6] eal: add thread lifetime management

2022-06-21 14:28 (UTC-0700), Tyler Retzlaff:

Quoted 35 lines · Dmitry Kozlyuk
Quoted 25 lines

2022-06-21 11:51 (UTC-0700), Tyler Retzlaff:

Quoted 24 lines
Quoted 23 lines
Quoted 11 lines
lib/eal/unix/rte_thread.c +11 -0
+int
+rte_thread_join(rte_thread_t thread_id, unsigned long *value_ptr)
+{
+	int ret = 0;
+	void *res = NULL;
+	void **pres = NULL;
+
+	if (value_ptr != NULL)
+		pres = &res;
+
+	ret = pthread_join((pthread_t)thread_id.opaque_id, pres);

lib/eal/unix/rte_thread.c struct eal_tls_key { · line 178

Quoted 10 lines
lib/eal/unix/rte_thread.c +10 -0
+	if (ret != 0) {
+		RTE_LOG(DEBUG, EAL, "pthread_join failed\n");
+		return ret;
+	}
+
+	if (value_ptr != NULL && *pres != NULL)
+		*value_ptr = *(unsigned long *)(*pres);
+
+	return 0;
+}    

What makes *pres == NULL special?

Quoted 8 lines
Quoted 2 lines

it's not clear what you mean, can you explain? maybe there is some context i am missing from the original patch series?

There's no previous context.
After ptread_join(), *pres holds the return value of the thread routine. You only assign *value_ptr if value_ptr is not NULL (obviously correct) and if *pres != NULL, that is, if the thread returned a non-NULL value. But this value is opaque, why do you filter NULL?

Quoted 4 lines

i don't think it is opaque here? unsigned long * value_ptr says we have to store an integer. which leads to a discussion of what should get stored at the value_ptr location if pthread_join() itself returns no result but the caller of rte_thread_join() requests the result.

There is no question. If pthread_join() fails, the function exits early and *value_ptr remains unmodified. If pthread_join() succeeds with a non-NULL second argument (pres), *pres aka res is always filled. NULL can be placed there too if that's what the thread routine has returned.

Quoted 12 lines
Quoted 2 lines

Perhaps you meant if (pres != NULL), no dereference?

that i think is just a repeat of a test checking if the caller of rte_thread_join is interested in the result?
i.e. value_ptr != NULL -> pres != NULL

both pres and *pres are dereferenced so it seems to track that prior to those dereferences they have to be validated as being non-NULL. i don't see how we could avoid dereferencing **pres to satisfy the calling contract when the result is requested.

Quoted 7 lines

now if value_ptr was unsigned long ** i guess i'd understand. i could always be reading the code wrong. but thinking about further there is another problem with this in that we really don't know what is being aliased in *pres when using the pthread implementation, since pthread could be returning a pointer to something narrow or with unknown layout where later dereferencing it as something wider or in this case specifically as unsigned long * would have horrible consequences.

Sorry, no, it's just all wrong.
We get a pointer from pthread_join() and store it in res, which can also be accessed as *pres, because pres == &res. Then we store the lower bits of res in an *value_ptr. We don't care what res points to because it is never dereferenced.

Quoted 4 lines

i think this ends up semi-related to your other comment about what the result type from rte_thread_func is, we can discuss offline and post details back to the list.

#22 lore raw

Re: [PATCH v2 2/6] eal: add thread lifetime management

Quoted 41 lines · Dmitry Kozlyuk

2022-06-21 14:28 (UTC-0700), Tyler Retzlaff:

Quoted 35 lines · Dmitry Kozlyuk
Quoted 25 lines

2022-06-21 11:51 (UTC-0700), Tyler Retzlaff:

Quoted 24 lines
Quoted 23 lines
Quoted 11 lines
lib/eal/unix/rte_thread.c +11 -0
+int
+rte_thread_join(rte_thread_t thread_id, unsigned long *value_ptr)
+{
+	int ret = 0;
+	void *res = NULL;
+	void **pres = NULL;
+
+	if (value_ptr != NULL)
+		pres = &res;
+
+	ret = pthread_join((pthread_t)thread_id.opaque_id, pres);

lib/eal/unix/rte_thread.c struct eal_tls_key { · line 178

Quoted 10 lines
lib/eal/unix/rte_thread.c +10 -0
+	if (ret != 0) {
+		RTE_LOG(DEBUG, EAL, "pthread_join failed\n");
+		return ret;
+	}
+
+	if (value_ptr != NULL && *pres != NULL)
+		*value_ptr = *(unsigned long *)(*pres);
+
+	return 0;
+}    

What makes *pres == NULL special?

Quoted 8 lines
Quoted 2 lines

it's not clear what you mean, can you explain? maybe there is some context i am missing from the original patch series?

There's no previous context.
After ptread_join(), *pres holds the return value of the thread routine. You only assign *value_ptr if value_ptr is not NULL (obviously correct) and if *pres != NULL, that is, if the thread returned a non-NULL value. But this value is opaque, why do you filter NULL?

Quoted 4 lines

i don't think it is opaque here? unsigned long * value_ptr says we have to store an integer. which leads to a discussion of what should get stored at the value_ptr location if pthread_join() itself returns no result but the caller of rte_thread_join() requests the result.

Quoted 4 lines

There is no question. If pthread_join() fails, the function exits early and *value_ptr remains unmodified. If pthread_join() succeeds with a non-NULL second argument (pres), *pres aka res is always filled. NULL can be placed there too if that's what the thread routine has returned.

okay, discussed offline it was just my misunderstanding of impact on the conditional block in the presence of the NULL check.

as discussed we'll rewrite the check as.

if (value_ptr != NULL)

    *value_ptr = (uint32_t)(uintptr_t)res;

this doesn't double dereference pres as the original code was and therefore the *pres NULL check is unnecessary.

thanks!

#23 lore raw

PATCH v2 3/6 eal: add basic rte thread ID equal API

In reply to Tyler Retzlaff

Add rte_thread_equal() that tests if two rte_thread_id are equal.

---
 lib/eal/include/rte_thread.h | 19 +++++++++++++++++++
 lib/eal/unix/rte_thread.c    |  6 ++++++
 lib/eal/version.map          |  1 +
 lib/eal/windows/rte_thread.c |  6 ++++++
 4 files changed, 32 insertions(+)
lib/eal/include/rte_thread.h +19 -0
index c27e580..de0486d 100644
@@ -134,6 +134,25 @@ int rte_thread_create(rte_thread_t *thread_id,
 __rte_experimental
 rte_thread_t rte_thread_self(void);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Check if 2 thread ids are equal.
+ *
+ * @param t1
+ *   First thread id.
+ *
+ * @param t2
+ *   Second thread id.
+ *
+ * @return
+ *   If the ids are equal, return nonzero.
+ *   Otherwise, return 0.
+ */
+__rte_experimental
+int rte_thread_equal(rte_thread_t t1, rte_thread_t t2);
+
 #ifdef RTE_HAS_CPUSET
 
 /**
lib/eal/unix/rte_thread.c +6 -0
index 19c7b80..0304d53 100644
@@ -183,6 +183,12 @@ struct eal_tls_key {
 	return pthread_detach((pthread_t)thread_id.opaque_id);
 }
 
+int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return pthread_equal((pthread_t)t1.opaque_id, (pthread_t)t2.opaque_id);
+}
+
 rte_thread_t
 rte_thread_self(void)
 {
lib/eal/version.map +1 -0
index 22e5c85..4a52484 100644
@@ -428,6 +428,7 @@ EXPERIMENTAL {
 	rte_thread_attr_set_priority;
 	rte_thread_create;
 	rte_thread_detach;
+	rte_thread_equal;
 	rte_thread_get_affinity_by_id;
 	rte_thread_get_priority;
 	rte_thread_join;
lib/eal/windows/rte_thread.c +6 -0
index b5f2b04..1352513 100644
@@ -291,6 +291,12 @@ struct thread_routine_ctx {
 	return 0;
 }
 
+int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return t1.opaque_id == t2.opaque_id;
+}
+
 rte_thread_t
 rte_thread_self(void)
 {
Signature · 1 line
1.8.3.1
Acked-by on v2 3/6
#24 lore raw

Re: [PATCH v2 3/6] eal: add basic rte thread ID equal API

Quoted 79 lines

Add rte_thread_equal() that tests if two rte_thread_id are equal.

---
  lib/eal/include/rte_thread.h | 19 +++++++++++++++++++
  lib/eal/unix/rte_thread.c    |  6 ++++++
  lib/eal/version.map          |  1 +
  lib/eal/windows/rte_thread.c |  6 ++++++
  4 files changed, 32 insertions(+)
lib/eal/include/rte_thread.h +19 -0
index c27e580..de0486d 100644
@@ -134,6 +134,25 @@ int rte_thread_create(rte_thread_t *thread_id,
  __rte_experimental
  rte_thread_t rte_thread_self(void);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Check if 2 thread ids are equal.
+ *
+ * @param t1
+ *   First thread id.
+ *
+ * @param t2
+ *   Second thread id.
+ *
+ * @return
+ *   If the ids are equal, return nonzero.
+ *   Otherwise, return 0.
+ */
+__rte_experimental
+int rte_thread_equal(rte_thread_t t1, rte_thread_t t2);
+
  #ifdef RTE_HAS_CPUSET
 
  /**
lib/eal/unix/rte_thread.c +6 -0
index 19c7b80..0304d53 100644
@@ -183,6 +183,12 @@ struct eal_tls_key {
  	return pthread_detach((pthread_t)thread_id.opaque_id);
  }
 
+int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return pthread_equal((pthread_t)t1.opaque_id, (pthread_t)t2.opaque_id);
+}
+
  rte_thread_t
  rte_thread_self(void)
  {
lib/eal/version.map +1 -0
index 22e5c85..4a52484 100644
@@ -428,6 +428,7 @@ EXPERIMENTAL {
  	rte_thread_attr_set_priority;
  	rte_thread_create;
  	rte_thread_detach;
+	rte_thread_equal;
  	rte_thread_get_affinity_by_id;
  	rte_thread_get_priority;
  	rte_thread_join;
lib/eal/windows/rte_thread.c +0 -0
index b5f2b04..1352513 100644
@@ -291,6 +291,12 @@ struct thread_routine_ctx {
  	return 0;
  }

lib/eal/windows/rte_thread.c struct thread_routine_ctx { · line 302

Quoted 10 lines
lib/eal/windows/rte_thread.c +6 -0
 
+int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return t1.opaque_id == t2.opaque_id;
+}
+
  rte_thread_t
  rte_thread_self(void)
  {
#25 lore raw

PATCH v2 4/6 test/threads: add tests for thread lifetime API

In reply to Tyler Retzlaff

test basic functionality and demonstrate use of following thread lifetime api.

* rte_thread_create
* rte_thread_detach
* rte_thread_join

---
 app/test/test_threads.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)
app/test/test_threads.c +50 -0
index b9d8b4e..9a30af5 100644
@@ -27,6 +27,54 @@
 }
 
 static int
+test_thread_create_join(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_t thread_main_id;
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, &thread_main_id) == 0,
+		"Failed to create thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_equal(thread_id, thread_main_id) != 0,
+		"Unexpected thread id.");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	RTE_TEST_ASSERT(rte_thread_join(thread_id, NULL) == 0,
+		"Failed to join thread.");
+
+	return 0;
+}
+
+static int
+test_thread_create_detach(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_t thread_main_id;
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main,
+		&thread_main_id) == 0, "Failed to create thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_equal(thread_id, thread_main_id) != 0,
+		"Unexpected thread id.");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	RTE_TEST_ASSERT(rte_thread_detach(thread_id) == 0,
+		"Failed to detach thread.");
+
+	return 0;
+}
+
+static int
 test_thread_priority(void)
 {
 	pthread_t id;
@@ -123,6 +171,8 @@
 	.setup = NULL,
 	.teardown = NULL,
 	.unit_test_cases = {
+		TEST_CASE(test_thread_create_join),
+		TEST_CASE(test_thread_create_detach),
 		TEST_CASE(test_thread_affinity),
 		TEST_CASE(test_thread_priority),
 		TEST_CASES_END()
Signature · 1 line
1.8.3.1
#26 lore raw

PATCH v2 5/6 test/threads: add tests for thread attributes API

In reply to Tyler Retzlaff

test basic functionality and demonstrate use of following thread attributes api. additionally, test attributes are processed when supplied to rte_thread_create().

* rte_thread_attr_init
* rte_thread_attr_set_affinity
* rte_thread_attr_get_affinity
* rte_thread_attr_set_priority

---
 app/test/test_threads.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 72 insertions(+), 1 deletion(-)
app/test/test_threads.c +72 -1
index 9a30af5..92a149a 100644
@@ -17,7 +17,9 @@
 static void *
 thread_main(void *arg)
 {
-	*(rte_thread_t *)arg = rte_thread_self();
+	if (arg != NULL)
+		*(rte_thread_t *)arg = rte_thread_self();
+
 	__atomic_store_n(&thread_id_ready, 1, __ATOMIC_RELEASE);
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 1)
@@ -166,6 +168,73 @@
 	return 0;
 }
 
+static int
+test_thread_attributes_affinity(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_attr_t attr;
+	rte_cpuset_t cpuset0;
+	rte_cpuset_t cpuset1;
+
+	RTE_TEST_ASSERT(rte_thread_attr_init(&attr) == 0,
+		"Failed to initialize thread attributes");
+
+	CPU_ZERO(&cpuset0);
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(rte_thread_self(), &cpuset0) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(rte_thread_attr_set_affinity(&attr, &cpuset0) == 0,
+		"Failed to set thread attributes affinity");
+	RTE_TEST_ASSERT(rte_thread_attr_get_affinity(&attr, &cpuset1) == 0,
+		"Failed to get thread attributes affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, &attr, thread_main, NULL) == 0,
+		"Failed to create attributes affinity thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get attributes thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Failed to apply affinity attributes");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	return 0;
+}
+
+static int
+test_thread_attributes_priority(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_attr_t attr;
+	enum rte_thread_priority priority;
+
+	RTE_TEST_ASSERT(rte_thread_attr_init(&attr) == 0,
+		"Failed to initialize thread attributes");
+	RTE_TEST_ASSERT(rte_thread_attr_set_priority(&attr, RTE_THREAD_PRIORITY_NORMAL) == 0,
+		"Failed to set thread attributes priority");
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, &attr, thread_main, NULL) == 0,
+		"Failed to create attributes priority thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_get_priority(thread_id, &priority) == 0,
+		"Failed to get thread priority");
+	RTE_TEST_ASSERT(priority == RTE_THREAD_PRIORITY_NORMAL,
+		"Failed to apply priority attributes");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	return 0;
+}
+
 static struct unit_test_suite threads_test_suite = {
 	.suite_name = "threads autotest",
 	.setup = NULL,
@@ -175,6 +244,8 @@
 		TEST_CASE(test_thread_create_detach),
 		TEST_CASE(test_thread_affinity),
 		TEST_CASE(test_thread_priority),
+		TEST_CASE(test_thread_attributes_affinity),
+		TEST_CASE(test_thread_attributes_priority),
 		TEST_CASES_END()
 	}
 };
Signature · 1 line
1.8.3.1
#27 lore raw

PATCH v2 6/6 test/threads: remove unit test use of pthread

In reply to Tyler Retzlaff

now that rte_thread provides thread lifetime functions stop using pthread in unit tests.

---
 app/test/test_threads.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
app/test/test_threads.c +2 -5
index 92a149a..034f82d 100644
@@ -3,7 +3,6 @@
  */
 
 #include <string.h>
-#include <pthread.h>
 
 #include <rte_thread.h>
 #include <rte_debug.h>
@@ -79,12 +78,11 @@
 static int
 test_thread_priority(void)
 {
-	pthread_t id;
 	rte_thread_t thread_id;
 	enum rte_thread_priority priority;
 
 	thread_id_ready = 0;
-	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, NULL) == 0,
 		"Failed to create thread");
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
@@ -131,13 +129,12 @@
 static int
 test_thread_affinity(void)
 {
-	pthread_t id;
 	rte_thread_t thread_id;
 	rte_cpuset_t cpuset0;
 	rte_cpuset_t cpuset1;
 
 	thread_id_ready = 0;
-	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, NULL) == 0,
 		"Failed to create thread");
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
Signature · 1 line
1.8.3.1
#28 lore raw

v3 cover add thread lifetime and attributes API

In reply to Tyler Retzlaff

add rte thread lifetime and attributes api. with these api additions there is now sufficient platform abstracted thread api to remove the use of pthread in the unit tests.

v3:
* change rte_thread_func return type to uint32_t for exit value.
* change rte_thread_join retval to be uint32_t (matched with the return value from rte_thread_func.
* introduce a wrapper for rte_thread_func on posix platforms to adapt differences between rte_thread_func and pthread start_routine.
* remove interpretation / dereference of result from pthread_join in posix implementation of rte_thread_join.
* fix leak of dynamically allocated thread_routine_ctx on windows in error paths.
* don't cast and truncate NULL to integer value for rte_thread_join when pthread_join returns no result.

v2:
* split implementation of rte_thread_equal for windows / posix and use pthread_equal for posix platforms.
* remove parameter validation assertions and instead return EINVAL for mandatory pointers to type that are NULL.
* correct doxygen comment parameter name args -> arg

Tyler Retzlaff (6):
eal: add thread attributes
eal: add thread lifetime management
eal: add basic rte thread ID equal API test/threads: add tests for thread lifetime API test/threads: add tests for thread attributes API test/threads: remove unit test use of pthread

 app/test/test_threads.c         | 134 ++++++++++++++++++++++--
 lib/eal/common/meson.build      |   1 +
 lib/eal/common/rte_thread.c     |  60 +++++++++++
 lib/eal/include/rte_thread.h    | 187 ++++++++++++++++++++++++++++++++++
 lib/eal/unix/rte_thread.c       | 141 ++++++++++++++++++++++++++
 lib/eal/version.map             |   8 ++
 lib/eal/windows/include/sched.h |   2 +-
 lib/eal/windows/rte_thread.c    | 219 ++++++++++++++++++++++++++++++++--------
 8 files changed, 703 insertions(+), 49 deletions(-)
 create mode 100644 lib/eal/common/rte_thread.c
Signature · 1 line
1.8.3.1
#29 lore raw

PATCH v3 1/6 eal: add thread attributes

Implement thread attributes for:

* thread affinity
* thread priority

Implement functions for managing thread attributes.

Priority is represented through an enum that allows for two levels:

* RTE_THREAD_PRIORITY_NORMAL
* RTE_THREAD_PRIORITY_REALTIME_CRITICAL

Affinity is described by the rte_cpuset_t type.

An rte_thread_attr_t object can be set to the default values by calling rte_thread_attr_init().

---
 lib/eal/common/meson.build   |  1 +
 lib/eal/common/rte_thread.c  | 60 ++++++++++++++++++++++++++++
 lib/eal/include/rte_thread.h | 93 ++++++++++++++++++++++++++++++++++++++++++++
 lib/eal/version.map          |  4 ++
 4 files changed, 158 insertions(+)
 create mode 100644 lib/eal/common/rte_thread.c
lib/eal/common/meson.build +1 -0
index 917758c..1e77dba 100644
@@ -36,6 +36,7 @@ sources += files(
         'rte_random.c',
         'rte_reciprocal.c',
         'rte_service.c',
+        'rte_thread.c',
         'rte_version.c',
 )
 if is_linux or is_windows
lib/eal/common/rte_thread.c new file +60 -0

60 lines not on this page · raw message

lib/eal/include/rte_thread.h +93 -0

105 lines not on this page · raw message

lib/eal/version.map +4 -0
index 5e6d851..ba03528 100644
@@ -423,6 +423,10 @@ EXPERIMENTAL {
 
 	# added in 22.07
 	rte_drand;
+	rte_thread_attr_get_affinity;
+	rte_thread_attr_init;
+	rte_thread_attr_set_affinity;
+	rte_thread_attr_set_priority;
 	rte_thread_get_affinity_by_id;
 	rte_thread_get_priority;
 	rte_thread_self;
Signature · 1 line
1.8.3.1
#30 lore raw

PATCH v3 2/6 eal: add thread lifetime management

In reply to Tyler Retzlaff

The *rte_thread_create()* function can optionally receive an rte_thread_attr_t object that will cause the thread to be created with the affinity and priority described by the attributes object. If no rte_thread_attr_t is passed (parameter is NULL), the default affinity and priority are used.

On Windows, the function executed by a thread when the thread starts is represeneted by a function pointer of type DWORD (*func) (void*). On other platforms, the function pointer is a void* (*func) (void*).

Performing a cast between these two types of function pointers to uniformize the API on all platforms may result in undefined behavior. TO fix this issue, a wrapper that respects the signature required by CreateThread() has been created on Windows.

---
 lib/eal/include/rte_thread.h    |  75 ++++++++++++++
 lib/eal/unix/rte_thread.c       | 135 +++++++++++++++++++++++++
 lib/eal/version.map             |   3 +
 lib/eal/windows/include/sched.h |   2 +-
 lib/eal/windows/rte_thread.c    | 213 ++++++++++++++++++++++++++++++++--------
 5 files changed, 387 insertions(+), 41 deletions(-)
lib/eal/include/rte_thread.h +75 -0
index 062308d..9e80270 100644
@@ -31,6 +31,18 @@
 } rte_thread_t;
 
 /**
+ * Thread function
+ *
+ * Function pointer to thread start routine.
+ *
+ * @param arg
+ *   Argument passed to rte_thread_create().
+ * @return
+ *   Thread function exit value.
+ */
+typedef uint32_t (*rte_thread_func) (void *);
+
+/**
  * Thread priority values.
  */
 enum rte_thread_priority {
@@ -61,6 +73,69 @@ enum rte_thread_priority {
  * @warning
  * @b EXPERIMENTAL: this API may change without prior notice.
  *
+ * Create a new thread that will invoke the 'thread_func' routine.
+ *
+ * @param thread_id
+ *    A pointer that will store the id of the newly created thread.
+ *
+ * @param thread_attr
+ *    Attributes that are used at the creation of the new thread.
+ *
+ * @param thread_func
+ *    The routine that the new thread will invoke when starting execution.
+ *
+ * @param arg
+ *    Argument to be passed to the 'thread_func' routine.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_create(rte_thread_t *thread_id,
+		const rte_thread_attr_t *thread_attr,
+		rte_thread_func thread_func, void *arg);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Waits for the thread identified by 'thread_id' to terminate
+ *
+ * @param thread_id
+ *    The identifier of the thread.
+ *
+ * @param value_ptr
+ *    Stores the exit status of the thread.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_join(rte_thread_t thread_id, uint32_t *value_ptr);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Indicate that the return value of the thread is not needed and
+ * all thread resources should be release when the thread terminates.
+ *
+ * @param thread_id
+ *    The id of the thread to be detached.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_detach(rte_thread_t thread_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
  * Get the id of the calling thread.
  *
  * @return
lib/eal/unix/rte_thread.c +135 -0

147 lines not on this page · raw message

lib/eal/version.map +3 -0
index ba03528..0b26652 100644
@@ -427,8 +427,11 @@ EXPERIMENTAL {
 	rte_thread_attr_init;
 	rte_thread_attr_set_affinity;
 	rte_thread_attr_set_priority;
+	rte_thread_create;
+	rte_thread_detach;
 	rte_thread_get_affinity_by_id;
 	rte_thread_get_priority;
+	rte_thread_join;
 	rte_thread_self;
 	rte_thread_set_affinity_by_id;
 	rte_thread_set_priority;
lib/eal/windows/include/sched.h +1 -1
index bc31cc8..912fed1 100644
@@ -44,7 +44,7 @@
 	(1LL << _WHICH_BIT(b))) != 0LL)
 
 static inline int
-count_cpu(rte_cpuset_t *s)
+count_cpu(const rte_cpuset_t *s)
 {
 	unsigned int _i;
 	int count = 0;
lib/eal/windows/rte_thread.c +173 -40

231 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#31 lore raw

PATCH v3 3/6 eal: add basic rte thread ID equal API

In reply to Tyler Retzlaff

Add rte_thread_equal() that tests if two rte_thread_id are equal.

---
 lib/eal/include/rte_thread.h | 19 +++++++++++++++++++
 lib/eal/unix/rte_thread.c    |  6 ++++++
 lib/eal/version.map          |  1 +
 lib/eal/windows/rte_thread.c |  6 ++++++
 4 files changed, 32 insertions(+)
lib/eal/include/rte_thread.h +19 -0
index 9e80270..ca6d9a2 100644
@@ -144,6 +144,25 @@ int rte_thread_create(rte_thread_t *thread_id,
 __rte_experimental
 rte_thread_t rte_thread_self(void);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Check if 2 thread ids are equal.
+ *
+ * @param t1
+ *   First thread id.
+ *
+ * @param t2
+ *   Second thread id.
+ *
+ * @return
+ *   If the ids are equal, return nonzero.
+ *   Otherwise, return 0.
+ */
+__rte_experimental
+int rte_thread_equal(rte_thread_t t1, rte_thread_t t2);
+
 #ifdef RTE_HAS_CPUSET
 
 /**
lib/eal/unix/rte_thread.c +6 -0
index d4c1a7f..37ebfcf 100644
@@ -210,6 +210,12 @@ struct thread_routine_ctx {
 	return pthread_detach((pthread_t)thread_id.opaque_id);
 }
 
+int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return pthread_equal((pthread_t)t1.opaque_id, (pthread_t)t2.opaque_id);
+}
+
 rte_thread_t
 rte_thread_self(void)
 {
lib/eal/version.map +1 -0
index 0b26652..9096075 100644
@@ -429,6 +429,7 @@ EXPERIMENTAL {
 	rte_thread_attr_set_priority;
 	rte_thread_create;
 	rte_thread_detach;
+	rte_thread_equal;
 	rte_thread_get_affinity_by_id;
 	rte_thread_get_priority;
 	rte_thread_join;
lib/eal/windows/rte_thread.c +6 -0
index ad71be4..1bc648e 100644
@@ -287,6 +287,12 @@ struct thread_routine_ctx {
 	return 0;
 }
 
+int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return t1.opaque_id == t2.opaque_id;
+}
+
 rte_thread_t
 rte_thread_self(void)
 {
Signature · 1 line
1.8.3.1
#32 lore raw

PATCH v3 4/6 test/threads: add tests for thread lifetime API

In reply to Tyler Retzlaff

test basic functionality and demonstrate use of following thread lifetime api.

* rte_thread_create
* rte_thread_detach
* rte_thread_join

---
 app/test/test_threads.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 52 insertions(+), 2 deletions(-)
app/test/test_threads.c +52 -2
index b9d8b4e..1077373 100644
@@ -14,7 +14,7 @@
 
 static uint32_t thread_id_ready;
 
-static void *
+static uint32_t
 thread_main(void *arg)
 {
 	*(rte_thread_t *)arg = rte_thread_self();
@@ -23,7 +23,55 @@
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 1)
 		;
 
-	return NULL;
+	return 0;
+}
+
+static int
+test_thread_create_join(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_t thread_main_id;
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, &thread_main_id) == 0,
+		"Failed to create thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_equal(thread_id, thread_main_id) != 0,
+		"Unexpected thread id.");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	RTE_TEST_ASSERT(rte_thread_join(thread_id, NULL) == 0,
+		"Failed to join thread.");
+
+	return 0;
+}
+
+static int
+test_thread_create_detach(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_t thread_main_id;
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main,
+		&thread_main_id) == 0, "Failed to create thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_equal(thread_id, thread_main_id) != 0,
+		"Unexpected thread id.");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	RTE_TEST_ASSERT(rte_thread_detach(thread_id) == 0,
+		"Failed to detach thread.");
+
+	return 0;
 }
 
 static int
@@ -123,6 +171,8 @@
 	.setup = NULL,
 	.teardown = NULL,
 	.unit_test_cases = {
+		TEST_CASE(test_thread_create_join),
+		TEST_CASE(test_thread_create_detach),
 		TEST_CASE(test_thread_affinity),
 		TEST_CASE(test_thread_priority),
 		TEST_CASES_END()
Signature · 1 line
1.8.3.1
#33 lore raw

PATCH v3 5/6 test/threads: add tests for thread attributes API

In reply to Tyler Retzlaff

test basic functionality and demonstrate use of following thread attributes api. additionally, test attributes are processed when supplied to rte_thread_create().

* rte_thread_attr_init
* rte_thread_attr_set_affinity
* rte_thread_attr_get_affinity
* rte_thread_attr_set_priority

---
 app/test/test_threads.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 72 insertions(+), 1 deletion(-)
app/test/test_threads.c +72 -1
index 1077373..3c22cec 100644
@@ -17,7 +17,9 @@
 static uint32_t
 thread_main(void *arg)
 {
-	*(rte_thread_t *)arg = rte_thread_self();
+	if (arg != NULL)
+		*(rte_thread_t *)arg = rte_thread_self();
+
 	__atomic_store_n(&thread_id_ready, 1, __ATOMIC_RELEASE);
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 1)
@@ -166,6 +168,73 @@
 	return 0;
 }
 
+static int
+test_thread_attributes_affinity(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_attr_t attr;
+	rte_cpuset_t cpuset0;
+	rte_cpuset_t cpuset1;
+
+	RTE_TEST_ASSERT(rte_thread_attr_init(&attr) == 0,
+		"Failed to initialize thread attributes");
+
+	CPU_ZERO(&cpuset0);
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(rte_thread_self(), &cpuset0) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(rte_thread_attr_set_affinity(&attr, &cpuset0) == 0,
+		"Failed to set thread attributes affinity");
+	RTE_TEST_ASSERT(rte_thread_attr_get_affinity(&attr, &cpuset1) == 0,
+		"Failed to get thread attributes affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, &attr, thread_main, NULL) == 0,
+		"Failed to create attributes affinity thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get attributes thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Failed to apply affinity attributes");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	return 0;
+}
+
+static int
+test_thread_attributes_priority(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_attr_t attr;
+	enum rte_thread_priority priority;
+
+	RTE_TEST_ASSERT(rte_thread_attr_init(&attr) == 0,
+		"Failed to initialize thread attributes");
+	RTE_TEST_ASSERT(rte_thread_attr_set_priority(&attr, RTE_THREAD_PRIORITY_NORMAL) == 0,
+		"Failed to set thread attributes priority");
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, &attr, thread_main, NULL) == 0,
+		"Failed to create attributes priority thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_get_priority(thread_id, &priority) == 0,
+		"Failed to get thread priority");
+	RTE_TEST_ASSERT(priority == RTE_THREAD_PRIORITY_NORMAL,
+		"Failed to apply priority attributes");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	return 0;
+}
+
 static struct unit_test_suite threads_test_suite = {
 	.suite_name = "threads autotest",
 	.setup = NULL,
@@ -175,6 +244,8 @@
 		TEST_CASE(test_thread_create_detach),
 		TEST_CASE(test_thread_affinity),
 		TEST_CASE(test_thread_priority),
+		TEST_CASE(test_thread_attributes_affinity),
+		TEST_CASE(test_thread_attributes_priority),
 		TEST_CASES_END()
 	}
 };
Signature · 1 line
1.8.3.1
#34 lore raw

PATCH v3 6/6 test/threads: remove unit test use of pthread

In reply to Tyler Retzlaff

now that rte_thread provides thread lifetime functions stop using pthread in unit tests.

---
 app/test/test_threads.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
app/test/test_threads.c +2 -5
index 3c22cec..e0f18e4 100644
@@ -3,7 +3,6 @@
  */
 
 #include <string.h>
-#include <pthread.h>
 
 #include <rte_thread.h>
 #include <rte_debug.h>
@@ -79,12 +78,11 @@
 static int
 test_thread_priority(void)
 {
-	pthread_t id;
 	rte_thread_t thread_id;
 	enum rte_thread_priority priority;
 
 	thread_id_ready = 0;
-	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, NULL) == 0,
 		"Failed to create thread");
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
@@ -131,13 +129,12 @@
 static int
 test_thread_affinity(void)
 {
-	pthread_t id;
 	rte_thread_t thread_id;
 	rte_cpuset_t cpuset0;
 	rte_cpuset_t cpuset1;
 
 	thread_id_ready = 0;
-	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, NULL) == 0,
 		"Failed to create thread");
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
Signature · 1 line
1.8.3.1
#35 lore raw

v4 cover add thread lifetime and attributes API

In reply to Tyler Retzlaff

add rte thread lifetime and attributes api. with these api additions there is now sufficient platform abstracted thread api to remove the use of pthread in the unit tests.

v4:
* update version.map to show api from series added in 22.11 instead

    of 22.07.

* fix missing parameter name in rte_thread_func declaration causing doxygen ci failure.

v3:
* change rte_thread_func return type to uint32_t for exit value.
* change rte_thread_join retval to be uint32_t (matched with the return value from rte_thread_func).
* introduce a wrapper for rte_thread_func on posix platforms to adapt differences between rte_thread_func and pthread start_routine.
* remove interpretation / dereference of result from pthread_join in posix implementation of rte_thread_join.
* fix leak of dynamically allocated thread_routine_ctx on windows in error paths.
* don't cast and truncate NULL to integer value for rte_thread_join when pthread_join returns no result.

v2:
* split implementation of rte_thread_equal for windows / posix and use pthread_equal for posix platforms.
* remove parameter validation assertions and instead return EINVAL for mandatory pointers to type that are NULL.
* correct doxygen comment parameter name args -> arg

Tyler Retzlaff (6):
eal: add thread attributes
eal: add thread lifetime management
eal: add basic rte thread ID equal API test/threads: add tests for thread lifetime API test/threads: add tests for thread attributes API test/threads: remove unit test use of pthread

 app/test/test_threads.c         | 134 ++++++++++++++++++++++--
 lib/eal/common/meson.build      |   1 +
 lib/eal/common/rte_thread.c     |  60 +++++++++++
 lib/eal/include/rte_thread.h    | 187 ++++++++++++++++++++++++++++++++++
 lib/eal/unix/rte_thread.c       | 141 ++++++++++++++++++++++++++
 lib/eal/version.map             |  10 ++
 lib/eal/windows/include/sched.h |   2 +-
 lib/eal/windows/rte_thread.c    | 219 ++++++++++++++++++++++++++++++++--------
 8 files changed, 705 insertions(+), 49 deletions(-)
 create mode 100644 lib/eal/common/rte_thread.c
Signature · 1 line
1.8.3.1
#36 lore raw

PATCH v4 1/6 eal: add thread attributes

Implement thread attributes for:

* thread affinity
* thread priority

Implement functions for managing thread attributes.

Priority is represented through an enum that allows for two levels:

* RTE_THREAD_PRIORITY_NORMAL
* RTE_THREAD_PRIORITY_REALTIME_CRITICAL

Affinity is described by the rte_cpuset_t type.

An rte_thread_attr_t object can be set to the default values by calling rte_thread_attr_init().

---
 lib/eal/common/meson.build   |  1 +
 lib/eal/common/rte_thread.c  | 60 ++++++++++++++++++++++++++++
 lib/eal/include/rte_thread.h | 93 ++++++++++++++++++++++++++++++++++++++++++++
 lib/eal/version.map          |  4 ++
 4 files changed, 158 insertions(+)
 create mode 100644 lib/eal/common/rte_thread.c
lib/eal/common/meson.build +1 -0
index 917758c..1e77dba 100644
@@ -36,6 +36,7 @@ sources += files(
         'rte_random.c',
         'rte_reciprocal.c',
         'rte_service.c',
+        'rte_thread.c',
         'rte_version.c',
 )
 if is_linux or is_windows
lib/eal/common/rte_thread.c new file +60 -0

60 lines not on this page · raw message

lib/eal/include/rte_thread.h +93 -0

105 lines not on this page · raw message

lib/eal/version.map +4 -0
index c2a2ceb..6d8f89a 100644
@@ -419,6 +419,10 @@ EXPERIMENTAL {
 
 	# added in 22.07
 	rte_drand;
+	rte_thread_attr_get_affinity;
+	rte_thread_attr_init;
+	rte_thread_attr_set_affinity;
+	rte_thread_attr_set_priority;
 	rte_thread_get_affinity_by_id;
 	rte_thread_get_priority;
 	rte_thread_self;
Signature · 1 line
1.8.3.1
#37 lore raw

PATCH v4 2/6 eal: add thread lifetime management

In reply to Tyler Retzlaff

The *rte_thread_create()* function can optionally receive an rte_thread_attr_t object that will cause the thread to be created with the affinity and priority described by the attributes object. If no rte_thread_attr_t is passed (parameter is NULL), the default affinity and priority are used.

On Windows, the function executed by a thread when the thread starts is represeneted by a function pointer of type DWORD (*func) (void*). On other platforms, the function pointer is a void* (*func) (void*).

Performing a cast between these two types of function pointers to uniformize the API on all platforms may result in undefined behavior. TO fix this issue, a wrapper that respects the signature required by CreateThread() has been created on Windows.

---
 lib/eal/include/rte_thread.h    |  75 ++++++++++++++
 lib/eal/unix/rte_thread.c       | 135 +++++++++++++++++++++++++
 lib/eal/version.map             |   5 +
 lib/eal/windows/include/sched.h |   2 +-
 lib/eal/windows/rte_thread.c    | 213 ++++++++++++++++++++++++++++++++--------
 5 files changed, 389 insertions(+), 41 deletions(-)
lib/eal/include/rte_thread.h +75 -0
index 062308d..3c8ff50 100644
@@ -31,6 +31,18 @@
 } rte_thread_t;
 
 /**
+ * Thread function
+ *
+ * Function pointer to thread start routine.
+ *
+ * @param arg
+ *   Argument passed to rte_thread_create().
+ * @return
+ *   Thread function exit value.
+ */
+typedef uint32_t (*rte_thread_func) (void *arg);
+
+/**
  * Thread priority values.
  */
 enum rte_thread_priority {
@@ -61,6 +73,69 @@ enum rte_thread_priority {
  * @warning
  * @b EXPERIMENTAL: this API may change without prior notice.
  *
+ * Create a new thread that will invoke the 'thread_func' routine.
+ *
+ * @param thread_id
+ *    A pointer that will store the id of the newly created thread.
+ *
+ * @param thread_attr
+ *    Attributes that are used at the creation of the new thread.
+ *
+ * @param thread_func
+ *    The routine that the new thread will invoke when starting execution.
+ *
+ * @param arg
+ *    Argument to be passed to the 'thread_func' routine.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_create(rte_thread_t *thread_id,
+		const rte_thread_attr_t *thread_attr,
+		rte_thread_func thread_func, void *arg);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Waits for the thread identified by 'thread_id' to terminate
+ *
+ * @param thread_id
+ *    The identifier of the thread.
+ *
+ * @param value_ptr
+ *    Stores the exit status of the thread.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_join(rte_thread_t thread_id, uint32_t *value_ptr);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Indicate that the return value of the thread is not needed and
+ * all thread resources should be release when the thread terminates.
+ *
+ * @param thread_id
+ *    The id of the thread to be detached.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_detach(rte_thread_t thread_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
  * Get the id of the calling thread.
  *
  * @return
lib/eal/unix/rte_thread.c +135 -0

147 lines not on this page · raw message

lib/eal/version.map +5 -0
index 6d8f89a..b404343 100644
@@ -428,6 +428,11 @@ EXPERIMENTAL {
 	rte_thread_self;
 	rte_thread_set_affinity_by_id;
 	rte_thread_set_priority;
+
+	# added in 22.11
+	rte_thread_create;
+	rte_thread_detach;
+	rte_thread_join;
 };
 
 INTERNAL {
lib/eal/windows/include/sched.h +1 -1
index bc31cc8..912fed1 100644
@@ -44,7 +44,7 @@
 	(1LL << _WHICH_BIT(b))) != 0LL)
 
 static inline int
-count_cpu(rte_cpuset_t *s)
+count_cpu(const rte_cpuset_t *s)
 {
 	unsigned int _i;
 	int count = 0;
lib/eal/windows/rte_thread.c +173 -40

231 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#38 lore raw

PATCH v4 3/6 eal: add basic rte thread ID equal API

In reply to Tyler Retzlaff

Add rte_thread_equal() that tests if two rte_thread_id are equal.

---
 lib/eal/include/rte_thread.h | 19 +++++++++++++++++++
 lib/eal/unix/rte_thread.c    |  6 ++++++
 lib/eal/version.map          |  1 +
 lib/eal/windows/rte_thread.c |  6 ++++++
 4 files changed, 32 insertions(+)
lib/eal/include/rte_thread.h +19 -0
index 3c8ff50..8359c1c 100644
@@ -144,6 +144,25 @@ int rte_thread_create(rte_thread_t *thread_id,
 __rte_experimental
 rte_thread_t rte_thread_self(void);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Check if 2 thread ids are equal.
+ *
+ * @param t1
+ *   First thread id.
+ *
+ * @param t2
+ *   Second thread id.
+ *
+ * @return
+ *   If the ids are equal, return nonzero.
+ *   Otherwise, return 0.
+ */
+__rte_experimental
+int rte_thread_equal(rte_thread_t t1, rte_thread_t t2);
+
 #ifdef RTE_HAS_CPUSET
 
 /**
lib/eal/unix/rte_thread.c +6 -0
index d4c1a7f..37ebfcf 100644
@@ -210,6 +210,12 @@ struct thread_routine_ctx {
 	return pthread_detach((pthread_t)thread_id.opaque_id);
 }
 
+int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return pthread_equal((pthread_t)t1.opaque_id, (pthread_t)t2.opaque_id);
+}
+
 rte_thread_t
 rte_thread_self(void)
 {
lib/eal/version.map +1 -0
index b404343..b5dc7f1 100644
@@ -432,6 +432,7 @@ EXPERIMENTAL {
 	# added in 22.11
 	rte_thread_create;
 	rte_thread_detach;
+	rte_thread_equal;
 	rte_thread_join;
 };
 
lib/eal/windows/rte_thread.c +6 -0
index ad71be4..1bc648e 100644
@@ -287,6 +287,12 @@ struct thread_routine_ctx {
 	return 0;
 }
 
+int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return t1.opaque_id == t2.opaque_id;
+}
+
 rte_thread_t
 rte_thread_self(void)
 {
Signature · 1 line
1.8.3.1
#39 lore raw

PATCH v4 4/6 test/threads: add tests for thread lifetime API

In reply to Tyler Retzlaff

test basic functionality and demonstrate use of following thread lifetime api.

* rte_thread_create
* rte_thread_detach
* rte_thread_join

---
 app/test/test_threads.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 52 insertions(+), 2 deletions(-)
app/test/test_threads.c +52 -2
index b9d8b4e..1077373 100644
@@ -14,7 +14,7 @@
 
 static uint32_t thread_id_ready;
 
-static void *
+static uint32_t
 thread_main(void *arg)
 {
 	*(rte_thread_t *)arg = rte_thread_self();
@@ -23,7 +23,55 @@
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 1)
 		;
 
-	return NULL;
+	return 0;
+}
+
+static int
+test_thread_create_join(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_t thread_main_id;
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, &thread_main_id) == 0,
+		"Failed to create thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_equal(thread_id, thread_main_id) != 0,
+		"Unexpected thread id.");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	RTE_TEST_ASSERT(rte_thread_join(thread_id, NULL) == 0,
+		"Failed to join thread.");
+
+	return 0;
+}
+
+static int
+test_thread_create_detach(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_t thread_main_id;
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main,
+		&thread_main_id) == 0, "Failed to create thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_equal(thread_id, thread_main_id) != 0,
+		"Unexpected thread id.");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	RTE_TEST_ASSERT(rte_thread_detach(thread_id) == 0,
+		"Failed to detach thread.");
+
+	return 0;
 }
 
 static int
@@ -123,6 +171,8 @@
 	.setup = NULL,
 	.teardown = NULL,
 	.unit_test_cases = {
+		TEST_CASE(test_thread_create_join),
+		TEST_CASE(test_thread_create_detach),
 		TEST_CASE(test_thread_affinity),
 		TEST_CASE(test_thread_priority),
 		TEST_CASES_END()
Signature · 1 line
1.8.3.1
#40 lore raw

PATCH v4 5/6 test/threads: add tests for thread attributes API

In reply to Tyler Retzlaff

test basic functionality and demonstrate use of following thread attributes api. additionally, test attributes are processed when supplied to rte_thread_create().

* rte_thread_attr_init
* rte_thread_attr_set_affinity
* rte_thread_attr_get_affinity
* rte_thread_attr_set_priority

---
 app/test/test_threads.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 72 insertions(+), 1 deletion(-)
app/test/test_threads.c +72 -1
index 1077373..3c22cec 100644
@@ -17,7 +17,9 @@
 static uint32_t
 thread_main(void *arg)
 {
-	*(rte_thread_t *)arg = rte_thread_self();
+	if (arg != NULL)
+		*(rte_thread_t *)arg = rte_thread_self();
+
 	__atomic_store_n(&thread_id_ready, 1, __ATOMIC_RELEASE);
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 1)
@@ -166,6 +168,73 @@
 	return 0;
 }
 
+static int
+test_thread_attributes_affinity(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_attr_t attr;
+	rte_cpuset_t cpuset0;
+	rte_cpuset_t cpuset1;
+
+	RTE_TEST_ASSERT(rte_thread_attr_init(&attr) == 0,
+		"Failed to initialize thread attributes");
+
+	CPU_ZERO(&cpuset0);
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(rte_thread_self(), &cpuset0) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(rte_thread_attr_set_affinity(&attr, &cpuset0) == 0,
+		"Failed to set thread attributes affinity");
+	RTE_TEST_ASSERT(rte_thread_attr_get_affinity(&attr, &cpuset1) == 0,
+		"Failed to get thread attributes affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, &attr, thread_main, NULL) == 0,
+		"Failed to create attributes affinity thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get attributes thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Failed to apply affinity attributes");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	return 0;
+}
+
+static int
+test_thread_attributes_priority(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_attr_t attr;
+	enum rte_thread_priority priority;
+
+	RTE_TEST_ASSERT(rte_thread_attr_init(&attr) == 0,
+		"Failed to initialize thread attributes");
+	RTE_TEST_ASSERT(rte_thread_attr_set_priority(&attr, RTE_THREAD_PRIORITY_NORMAL) == 0,
+		"Failed to set thread attributes priority");
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, &attr, thread_main, NULL) == 0,
+		"Failed to create attributes priority thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_get_priority(thread_id, &priority) == 0,
+		"Failed to get thread priority");
+	RTE_TEST_ASSERT(priority == RTE_THREAD_PRIORITY_NORMAL,
+		"Failed to apply priority attributes");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	return 0;
+}
+
 static struct unit_test_suite threads_test_suite = {
 	.suite_name = "threads autotest",
 	.setup = NULL,
@@ -175,6 +244,8 @@
 		TEST_CASE(test_thread_create_detach),
 		TEST_CASE(test_thread_affinity),
 		TEST_CASE(test_thread_priority),
+		TEST_CASE(test_thread_attributes_affinity),
+		TEST_CASE(test_thread_attributes_priority),
 		TEST_CASES_END()
 	}
 };
Signature · 1 line
1.8.3.1
#41 lore raw

PATCH v4 6/6 test/threads: remove unit test use of pthread

In reply to Tyler Retzlaff

now that rte_thread provides thread lifetime functions stop using pthread in unit tests.

---
 app/test/test_threads.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
app/test/test_threads.c +2 -5
index 3c22cec..e0f18e4 100644
@@ -3,7 +3,6 @@
  */
 
 #include <string.h>
-#include <pthread.h>
 
 #include <rte_thread.h>
 #include <rte_debug.h>
@@ -79,12 +78,11 @@
 static int
 test_thread_priority(void)
 {
-	pthread_t id;
 	rte_thread_t thread_id;
 	enum rte_thread_priority priority;
 
 	thread_id_ready = 0;
-	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, NULL) == 0,
 		"Failed to create thread");
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
@@ -131,13 +129,12 @@
 static int
 test_thread_affinity(void)
 {
-	pthread_t id;
 	rte_thread_t thread_id;
 	rte_cpuset_t cpuset0;
 	rte_cpuset_t cpuset1;
 
 	thread_id_ready = 0;
-	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, NULL) == 0,
 		"Failed to create thread");
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
Signature · 1 line
1.8.3.1
#43 lore raw
In reply to Tyler Retzlaff
Quoted 8 lines · Tyler Retzlaff

add rte thread lifetime and attributes api. with these api additions there is now sufficient platform abstracted thread api to remove the use of pthread in the unit tests.

v4:
* update version.map to show api from series added in 22.11 instead

    of 22.07.

Patch 1 still adds symbols in 22.07 section of the experimental block.

Compilation is broken when stopping at patch 4.

Newly added code can go to eal_common_thread.c rather than introduce a new common/rte_thread.c file (or is there a rationale for this?).

There are guards for the new structs against RTE_HAS_CPUSET being defined. Can you elaborate why those guards are needed?

Commitlogs / comments sentences must start with a capital letter.

Can you prepare a new revision for -rc1?

Thanks.

Quoted 2 lines

* fix missing parameter name in rte_thread_func declaration causing doxygen ci failure.

Signature · 1 line
David Marchand
#44 lore raw
Quoted 19 lines · David Marchand
Quoted 8 lines · Tyler Retzlaff

add rte thread lifetime and attributes api. with these api additions there is now sufficient platform abstracted thread api to remove the use of pthread in the unit tests.

v4:
* update version.map to show api from series added in 22.11 instead

    of 22.07.

Patch 1 still adds symbols in 22.07 section of the experimental block.

Compilation is broken when stopping at patch 4.

Newly added code can go to eal_common_thread.c rather than introduce a new common/rte_thread.c file (or is there a rationale for this?).

Quoted 6 lines

There are guards for the new structs against RTE_HAS_CPUSET being defined. Can you elaborate why those guards are needed?

Commitlogs / comments sentences must start with a capital letter.

Can you prepare a new revision for -rc1?

Any update?

Signature · 1 line
David Marchand
#45 lore raw
In reply to David Marchand

hi David,

Quoted 7 lines · David Marchand
Quoted 5 lines · Tyler Retzlaff

add rte thread lifetime and attributes api. with these api additions there is now sufficient platform abstracted thread api to remove the use of pthread in the unit tests.

Quoted 5 lines
Quoted 3 lines

v4:
* update version.map to show api from series added in 22.11 instead

    of 22.07.

Patch 1 still adds symbols in 22.07 section of the experimental block.

this looks like a goofup on my part, i will fix in the next revision.

Quoted 2 lines

Compilation is broken when stopping at patch 4.

this was because commit 72b452c5f2599f970f47fd17d3e8e5d60bfebe7a removed #include of <errno.h> from rte_common.h

i will fix this in the next revision.

Quoted 3 lines

Newly added code can go to eal_common_thread.c rather than introduce a new common/rte_thread.c file (or is there a rationale for this?).

i will make this change in the next revision. if anyone does object i hope they will do so quickly.

Quoted 3 lines

There are guards for the new structs against RTE_HAS_CPUSET being defined. Can you elaborate why those guards are needed?

eal uses this guard in a couple of places so the best explanation i can provide is an existing pattern was being followed. that being said it's clear that this code won't work without RTE_HAS_CPUSET.

i will remove the guard and test if the build breaks for the platforms i have toolchains available. if it builds cleanly i'll remove the guard in the next revision.

Quoted 2 lines

Commitlogs / comments sentences must start with a capital letter.

i will fix in next revision, but ew caps.

Quoted 2 lines

Can you prepare a new revision for -rc1?

assuming the above actions to address feedback are acceptable the new revision should be up today (PST).

Quoted 2 lines

Thanks.

thank you!

ty

#46 lore raw
Quoted 9 lines · Tyler Retzlaff

hi David,

Quoted 3 lines

Newly added code can go to eal_common_thread.c rather than introduce a new common/rte_thread.c file (or is there a rationale for this?).

i will make this change in the next revision. if anyone does object i hope they will do so quickly.

looking at this more closely i'm going to back away from making the adjustment here. if Thomas and/or Dmitry could comment it would be appreciated.

it appears that functions placed in eal_common_xxx files are consumed internally by the eal where rte_xxx files are functions that are exposed through public api.

since these additions are public api it seems they should remain in rte_thread.c

i won't change this in the next revision, but please do correct me if i'm still not on track.

#47 lore raw
Quoted 20 lines · Tyler Retzlaff
Quoted 9 lines · Tyler Retzlaff

hi David,

Quoted 3 lines

Newly added code can go to eal_common_thread.c rather than introduce a new common/rte_thread.c file (or is there a rationale for this?).

i will make this change in the next revision. if anyone does object i hope they will do so quickly.

looking at this more closely i'm going to back away from making the adjustment here. if Thomas and/or Dmitry could comment it would be appreciated.

it appears that functions placed in eal_common_xxx files are consumed internally by the eal where rte_xxx files are functions that are exposed through public api.

Quoted 5 lines

since these additions are public api it seems they should remain in rte_thread.c

i won't change this in the next revision, but please do correct me if i'm still not on track.

This seems a fair argument.

I'll reply on the v5 series: compilation is still broken at patch 4.

Signature · 1 line
David Marchand
#48 lore raw
Quoted 25 lines · David Marchand
Quoted 23 lines · Tyler Retzlaff
Quoted 9 lines · Tyler Retzlaff

hi David,

Quoted 3 lines

Newly added code can go to eal_common_thread.c rather than introduce a new common/rte_thread.c file (or is there a rationale for this?).

i will make this change in the next revision. if anyone does object i hope they will do so quickly.

looking at this more closely i'm going to back away from making the adjustment here. if Thomas and/or Dmitry could comment it would be appreciated.

it appears that functions placed in eal_common_xxx files are consumed internally by the eal where rte_xxx files are functions that are exposed through public api.

since these additions are public api it seems they should remain in rte_thread.c

Quoted 7 lines
Quoted 2 lines

i won't change this in the next revision, but please do correct me if i'm still not on track.

This seems a fair argument.

I'll reply on the v5 series: compilation is still broken at patch 4.

can you share what the break is? and which platform/target/toolchain?

the only thing I found was that the removal of errno.h broke compilation but that is fixed in v5. the CI run for the series doesn't indicate any compilation failures. assuming i'm looking at the correct results.

https://lab.dpdk.org/results/dashboard/patchsets/23789/

it does show a unit test failure but it looks like it is related to a service test that you mailed about within the past few days.

thanks

#49 lore raw
In reply to Tyler Retzlaff

05/10/2022 18:34, Tyler Retzlaff:

Quoted 7 lines · Tyler Retzlaff
Quoted 5 lines
Quoted 2 lines

Newly added code can go to eal_common_thread.c rather than introduce a new common/rte_thread.c file (or is there a rationale for this?).

i will make this change in the next revision. if anyone does object i hope they will do so quickly.

Quoted 7 lines

looking at this more closely i'm going to back away from making the adjustment here. if Thomas and/or Dmitry could comment it would be appreciated.

it appears that functions placed in eal_common_xxx files are consumed internally by the eal where rte_xxx files are functions that are exposed through public api.

It is not so clear.
There is already eal_common_thread.c which implements the same kind of functions, so I think you should move your new functions here.

Quoted 2 lines

since these additions are public api it seems they should remain in rte_thread.c

Let's not have 2 .c files for the same purpose in the same directory.

Quoted 2 lines

i won't change this in the next revision, but please do correct me if i'm still not on track.

Please I would prefer to keep all in one file: eal_common_thread.c Thank you

#50 lore raw
Quoted 16 lines · Thomas Monjalon

05/10/2022 18:34, Tyler Retzlaff:

Quoted 7 lines · Tyler Retzlaff
Quoted 5 lines
Quoted 2 lines

Newly added code can go to eal_common_thread.c rather than introduce a new common/rte_thread.c file (or is there a rationale for this?).

i will make this change in the next revision. if anyone does object i hope they will do so quickly.

Quoted 7 lines

looking at this more closely i'm going to back away from making the adjustment here. if Thomas and/or Dmitry could comment it would be appreciated.

it appears that functions placed in eal_common_xxx files are consumed internally by the eal where rte_xxx files are functions that are exposed through public api.

Quoted 8 lines

It is not so clear.
There is already eal_common_thread.c which implements the same kind of functions, so I think you should move your new functions here.

Quoted 2 lines

since these additions are public api it seems they should remain in rte_thread.c

Let's not have 2 .c files for the same purpose in the same directory.

just as another point there seem to be several other rte_xxx.c files here can we clarify why they were not subject to the same requirement? as a follow on does it mean that the code in those files should also be moved to eal_common_xxx files?

please let me know if the justification is not the same i'll move the functions to the eal_common file as requested. i just want to make sure it is being done for the consistent/correct reason.

thanks.

#51 lore raw

06/10/2022 17:10, Tyler Retzlaff:

Quoted 24 lines · Thomas Monjalon
Quoted 23 lines

05/10/2022 18:34, Tyler Retzlaff:

Quoted 7 lines · Tyler Retzlaff
Quoted 5 lines
Quoted 2 lines

Newly added code can go to eal_common_thread.c rather than introduce a new common/rte_thread.c file (or is there a rationale for this?).

i will make this change in the next revision. if anyone does object i hope they will do so quickly.

Quoted 7 lines

looking at this more closely i'm going to back away from making the adjustment here. if Thomas and/or Dmitry could comment it would be appreciated.

it appears that functions placed in eal_common_xxx files are consumed internally by the eal where rte_xxx files are functions that are exposed through public api.

It is not so clear.
There is already eal_common_thread.c which implements the same kind of functions, so I think you should move your new functions here.

Quoted 2 lines

since these additions are public api it seems they should remain in rte_thread.c

Quoted 6 lines
Quoted 1 line

Let's not have 2 .c files for the same purpose in the same directory.

just as another point there seem to be several other rte_xxx.c files here can we clarify why they were not subject to the same requirement? as a follow on does it mean that the code in those files should also be moved to eal_common_xxx files?

That's just history.

Quoted 3 lines

please let me know if the justification is not the same i'll move the functions to the eal_common file as requested. i just want to make sure it is being done for the consistent/correct reason.

Some file names are not correct, we could rename them.

I think David is already doing the last minor changes on this series while merging, so no need to do anything on your side.

#52 lore raw
Quoted 34 lines · Thomas Monjalon

06/10/2022 17:10, Tyler Retzlaff:

Quoted 24 lines · Thomas Monjalon
Quoted 23 lines

05/10/2022 18:34, Tyler Retzlaff:

Quoted 7 lines · Tyler Retzlaff
Quoted 5 lines
Quoted 2 lines

Newly added code can go to eal_common_thread.c rather than introduce a new common/rte_thread.c file (or is there a rationale for this?).

i will make this change in the next revision. if anyone does object i hope they will do so quickly.

Quoted 7 lines

looking at this more closely i'm going to back away from making the adjustment here. if Thomas and/or Dmitry could comment it would be appreciated.

it appears that functions placed in eal_common_xxx files are consumed internally by the eal where rte_xxx files are functions that are exposed through public api.

It is not so clear.
There is already eal_common_thread.c which implements the same kind of functions, so I think you should move your new functions here.

Quoted 2 lines

since these additions are public api it seems they should remain in rte_thread.c

Quoted 6 lines
Quoted 1 line

Let's not have 2 .c files for the same purpose in the same directory.

just as another point there seem to be several other rte_xxx.c files here can we clarify why they were not subject to the same requirement? as a follow on does it mean that the code in those files should also be moved to eal_common_xxx files?

That's just history.

Quoted 9 lines
Quoted 3 lines

please let me know if the justification is not the same i'll move the functions to the eal_common file as requested. i just want to make sure it is being done for the consistent/correct reason.

Some file names are not correct, we could rename them.

I think David is already doing the last minor changes on this series while merging, so no need to do anything on your side.

Thomas, David with this just a final confirmation no need for a v6? you're tweaking the series as is for final comments?

thanks

#53 lore raw
Quoted 40 lines · Tyler Retzlaff
Quoted 38 lines · Thomas Monjalon

06/10/2022 17:10, Tyler Retzlaff:

Quoted 24 lines · Thomas Monjalon
Quoted 23 lines

05/10/2022 18:34, Tyler Retzlaff:

Quoted 7 lines · Tyler Retzlaff
Quoted 5 lines
Quoted 2 lines

Newly added code can go to eal_common_thread.c rather than introduce a new common/rte_thread.c file (or is there a rationale for this?).

i will make this change in the next revision. if anyone does object i hope they will do so quickly.

Quoted 7 lines

looking at this more closely i'm going to back away from making the adjustment here. if Thomas and/or Dmitry could comment it would be appreciated.

it appears that functions placed in eal_common_xxx files are consumed internally by the eal where rte_xxx files are functions that are exposed through public api.

It is not so clear.
There is already eal_common_thread.c which implements the same kind of functions, so I think you should move your new functions here.

Quoted 2 lines

since these additions are public api it seems they should remain in rte_thread.c

Quoted 6 lines
Quoted 1 line

Let's not have 2 .c files for the same purpose in the same directory.

just as another point there seem to be several other rte_xxx.c files here can we clarify why they were not subject to the same requirement? as a follow on does it mean that the code in those files should also be moved to eal_common_xxx files?

That's just history.

Quoted 3 lines

please let me know if the justification is not the same i'll move the functions to the eal_common file as requested. i just want to make sure it is being done for the consistent/correct reason.

Quoted 8 lines
Quoted 5 lines

Some file names are not correct, we could rename them.

I think David is already doing the last minor changes on this series while merging, so no need to do anything on your side.

Thomas, David with this just a final confirmation no need for a v6? you're tweaking the series as is for final comments?

No need for a v6, the code move is trivial, and for the rest, I'm finished. I'll restart the per patch build all over again to be sure, and then merge the series probably tonight (CET+2).

Signature · 1 line
David Marchand
#54 lore raw
Quoted 48 lines · David Marchand
Quoted 46 lines · Tyler Retzlaff
Quoted 43 lines · Thomas Monjalon

06/10/2022 17:10, Tyler Retzlaff:

Quoted 24 lines · Thomas Monjalon
Quoted 23 lines

05/10/2022 18:34, Tyler Retzlaff:

Quoted 7 lines · Tyler Retzlaff
Quoted 5 lines
Quoted 2 lines

Newly added code can go to eal_common_thread.c rather than introduce a new common/rte_thread.c file (or is there a rationale for this?).

i will make this change in the next revision. if anyone does object i hope they will do so quickly.

Quoted 7 lines

looking at this more closely i'm going to back away from making the adjustment here. if Thomas and/or Dmitry could comment it would be appreciated.

it appears that functions placed in eal_common_xxx files are consumed internally by the eal where rte_xxx files are functions that are exposed through public api.

It is not so clear.
There is already eal_common_thread.c which implements the same kind of functions, so I think you should move your new functions here.

Quoted 2 lines

since these additions are public api it seems they should remain in rte_thread.c

Quoted 6 lines
Quoted 1 line

Let's not have 2 .c files for the same purpose in the same directory.

just as another point there seem to be several other rte_xxx.c files here can we clarify why they were not subject to the same requirement? as a follow on does it mean that the code in those files should also be moved to eal_common_xxx files?

That's just history.

Quoted 3 lines

please let me know if the justification is not the same i'll move the functions to the eal_common file as requested. i just want to make sure it is being done for the consistent/correct reason.

Some file names are not correct, we could rename them.

I think David is already doing the last minor changes on this series while merging, so no need to do anything on your side.

Quoted 6 lines
Quoted 2 lines

Thomas, David with this just a final confirmation no need for a v6? you're tweaking the series as is for final comments?

No need for a v6, the code move is trivial, and for the rest, I'm finished. I'll restart the per patch build all over again to be sure, and then merge the series probably tonight (CET+2).

acknowledged.

thanks very much!

Quoted 4 lines
Signature · 2 lines
-- 
David Marchand
#55 lore raw

v5 cover add thread lifetime and attributes API

In reply to Tyler Retzlaff

add rte thread lifetime and attributes api. with these api additions there is now sufficient platform abstracted thread api to remove the use of pthread in the unit tests.

v5:
* include errno.h in rte_thread.c since errno.h is no longer included in rte_common.h
* move rte_thread_attr symbols from 22.07 to 22.11 section of version.map.
* remove RTE_HAS_CPUSET guards from rte_thread.h
* change case of characters in commit message for patches {6,5,4} of the series.

v4:
* update version.map to show api from series added in 22.11 instead

    of 22.07.

* fix missing parameter name in rte_thread_func declaration causing doxygen ci failure.

v3:
* change rte_thread_func return type to uint32_t for exit value.
* change rte_thread_join retval to be uint32_t (matched with the return value from rte_thread_func).
* introduce a wrapper for rte_thread_func on posix platforms to adapt differences between rte_thread_func and pthread start_routine.
* remove interpretation / dereference of result from pthread_join in posix implementation of rte_thread_join.
* fix leak of dynamically allocated thread_routine_ctx on windows in error paths.
* don't cast and truncate NULL to integer value for rte_thread_join when pthread_join returns no result.

v2:
* split implementation of rte_thread_equal for windows / posix and use pthread_equal for posix platforms.
* remove parameter validation assertions and instead return EINVAL for mandatory pointers to type that are NULL.
* correct doxygen comment parameter name args -> arg

Tyler Retzlaff (6):
eal: add thread attributes
eal: add thread lifetime management
eal: add basic rte thread ID equal API test/threads: add tests for thread lifetime API test/threads: add tests for thread attributes API test/threads: remove unit test use of pthread

 app/test/test_threads.c         | 134 ++++++++++++++++++++++--
 lib/eal/common/meson.build      |   1 +
 lib/eal/common/rte_thread.c     |  62 ++++++++++++
 lib/eal/include/rte_thread.h    | 185 ++++++++++++++++++++++++++++++++-
 lib/eal/unix/rte_thread.c       | 141 ++++++++++++++++++++++++++
 lib/eal/version.map             |  10 ++
 lib/eal/windows/include/sched.h |   2 +-
 lib/eal/windows/rte_thread.c    | 219 ++++++++++++++++++++++++++++++++--------
 8 files changed, 702 insertions(+), 52 deletions(-)
 create mode 100644 lib/eal/common/rte_thread.c
Signature · 1 line
1.8.3.1
#56 lore raw

PATCH v5 1/6 eal: add thread attributes

Implement thread attributes for:

* thread affinity
* thread priority

Implement functions for managing thread attributes.

Priority is represented through an enum that allows for two levels:

* RTE_THREAD_PRIORITY_NORMAL
* RTE_THREAD_PRIORITY_REALTIME_CRITICAL

Affinity is described by the rte_cpuset_t type.

An rte_thread_attr_t object can be set to the default values by calling rte_thread_attr_init().

---
 lib/eal/common/meson.build   |   1 +
 lib/eal/common/rte_thread.c  |  62 ++++++++++++++++++++++++
 lib/eal/include/rte_thread.h | 110 +++++++++++++++++++++++++++++++++++++++++--
 lib/eal/version.map          |   6 +++
 4 files changed, 176 insertions(+), 3 deletions(-)
 create mode 100644 lib/eal/common/rte_thread.c
lib/eal/common/meson.build +1 -0
index 917758c..1e77dba 100644
@@ -36,6 +36,7 @@ sources += files(
         'rte_random.c',
         'rte_reciprocal.c',
         'rte_service.c',
+        'rte_thread.c',
         'rte_version.c',
 )
 if is_linux or is_windows
lib/eal/common/rte_thread.c new file +62 -0

62 lines not on this page · raw message

lib/eal/include/rte_thread.h +107 -3

128 lines not on this page · raw message

lib/eal/version.map +6 -0
index e617ba7..e9df18d 100644
@@ -430,6 +430,12 @@ EXPERIMENTAL {
 	rte_thread_self;
 	rte_thread_set_affinity_by_id;
 	rte_thread_set_priority;
+
+	# added in 22.11
+	rte_thread_attr_get_affinity;
+	rte_thread_attr_init;
+	rte_thread_attr_set_affinity;
+	rte_thread_attr_set_priority;
 };
 
 INTERNAL {
Signature · 1 line
1.8.3.1
#57 lore raw

Re: [PATCH v5 1/6] eal: add thread attributes

Quoted 13 lines · Tyler Retzlaff
lib/eal/include/rte_thread.h +8 -1
@@ -57,7 +65,105 @@ enum rte_thread_priority {
 __rte_experimental
 rte_thread_t rte_thread_self(void);
 
-#ifdef RTE_HAS_CPUSET
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Check if 2 thread ids are equal.
+ *
+ * @param t1
+ *   First thread id.

lib/eal/include/rte_thread.h enum rte_thread_priority { · line 85

Quoted 10 lines
lib/eal/include/rte_thread.h +10 -0
+ *
+ * @param t2
+ *   Second thread id.
+ *
+ * @return
+ *   If the ids are equal, return nonzero.
+ *   Otherwise, return 0.
+ */
+__rte_experimental
+int rte_thread_equal(rte_thread_t t1, rte_thread_t t2);

Rebase damage, I'll fix when applying.

Signature · 1 line
David Marchand
#58 lore raw

PATCH v5 2/6 eal: add thread lifetime management

In reply to Tyler Retzlaff

The *rte_thread_create()* function can optionally receive an rte_thread_attr_t object that will cause the thread to be created with the affinity and priority described by the attributes object. If no rte_thread_attr_t is passed (parameter is NULL), the default affinity and priority are used.

On Windows, the function executed by a thread when the thread starts is represeneted by a function pointer of type DWORD (*func) (void*). On other platforms, the function pointer is a void* (*func) (void*).

Performing a cast between these two types of function pointers to uniformize the API on all platforms may result in undefined behavior. TO fix this issue, a wrapper that respects the signature required by CreateThread() has been created on Windows.

---
 lib/eal/include/rte_thread.h    |  75 ++++++++++++++
 lib/eal/unix/rte_thread.c       | 135 +++++++++++++++++++++++++
 lib/eal/version.map             |   3 +
 lib/eal/windows/include/sched.h |   2 +-
 lib/eal/windows/rte_thread.c    | 213 ++++++++++++++++++++++++++++++++--------
 5 files changed, 387 insertions(+), 41 deletions(-)
lib/eal/include/rte_thread.h +75 -0
index a3802de..65cdcfa 100644
@@ -31,6 +31,18 @@
 } rte_thread_t;
 
 /**
+ * Thread function
+ *
+ * Function pointer to thread start routine.
+ *
+ * @param arg
+ *   Argument passed to rte_thread_create().
+ * @return
+ *   Thread function exit value.
+ */
+typedef uint32_t (*rte_thread_func) (void *arg);
+
+/**
  * Thread priority values.
  */
 enum rte_thread_priority {
@@ -57,6 +69,69 @@ enum rte_thread_priority {
  * @warning
  * @b EXPERIMENTAL: this API may change without prior notice.
  *
+ * Create a new thread that will invoke the 'thread_func' routine.
+ *
+ * @param thread_id
+ *    A pointer that will store the id of the newly created thread.
+ *
+ * @param thread_attr
+ *    Attributes that are used at the creation of the new thread.
+ *
+ * @param thread_func
+ *    The routine that the new thread will invoke when starting execution.
+ *
+ * @param arg
+ *    Argument to be passed to the 'thread_func' routine.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_create(rte_thread_t *thread_id,
+		const rte_thread_attr_t *thread_attr,
+		rte_thread_func thread_func, void *arg);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Waits for the thread identified by 'thread_id' to terminate
+ *
+ * @param thread_id
+ *    The identifier of the thread.
+ *
+ * @param value_ptr
+ *    Stores the exit status of the thread.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_join(rte_thread_t thread_id, uint32_t *value_ptr);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Indicate that the return value of the thread is not needed and
+ * all thread resources should be release when the thread terminates.
+ *
+ * @param thread_id
+ *    The id of the thread to be detached.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_detach(rte_thread_t thread_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
  * Get the id of the calling thread.
  *
  * @return
lib/eal/unix/rte_thread.c +135 -0

147 lines not on this page · raw message

lib/eal/version.map +3 -0
index e9df18d..5fd84b6 100644
@@ -436,6 +436,9 @@ EXPERIMENTAL {
 	rte_thread_attr_init;
 	rte_thread_attr_set_affinity;
 	rte_thread_attr_set_priority;
+	rte_thread_create;
+	rte_thread_detach;
+	rte_thread_join;
 };
 
 INTERNAL {
lib/eal/windows/include/sched.h +1 -1
index bc31cc8..912fed1 100644
@@ -44,7 +44,7 @@
 	(1LL << _WHICH_BIT(b))) != 0LL)
 
 static inline int
-count_cpu(rte_cpuset_t *s)
+count_cpu(const rte_cpuset_t *s)
 {
 	unsigned int _i;
 	int count = 0;
lib/eal/windows/rte_thread.c +173 -40

231 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#59 lore raw

Re: [PATCH v5 2/6] eal: add thread lifetime management

Hello Tyler,

Quoted 111 lines · Tyler Retzlaff
lib/eal/unix/rte_thread.c +96 -0
index 9126595..d4c1a7f 100644
@@ -16,6 +16,11 @@ struct eal_tls_key {
        pthread_key_t thread_index;
 };
 
+struct thread_routine_ctx {
+       rte_thread_func thread_func;
+       void *routine_args;
+};
+
 static int
 thread_map_priority_to_os_value(enum rte_thread_priority eal_pri, int *os_pri,
        int *pol)
@@ -75,6 +80,136 @@ struct eal_tls_key {
        return 0;
 }
 
+static void *
+thread_func_wrapper(void *arg)
+{
+       struct thread_routine_ctx ctx = *(struct thread_routine_ctx *)arg;
+
+       free(arg);
+
+       return (void *)(uintptr_t)ctx.thread_func(ctx.routine_args);
+}
+
+int
+rte_thread_create(rte_thread_t *thread_id,
+               const rte_thread_attr_t *thread_attr,
+               rte_thread_func thread_func, void *args)
+{
+       int ret = 0;
+       pthread_attr_t attr;
+       pthread_attr_t *attrp = NULL;
+       struct thread_routine_ctx *ctx;
+       struct sched_param param = {
+               .sched_priority = 0,
+       };
+       int policy = SCHED_OTHER;
+
+       ctx = calloc(1, sizeof(*ctx));
+       if (ctx == NULL) {
+               RTE_LOG(DEBUG, EAL, "Insufficient memory for thread context allocations\n");
+               ret = ENOMEM;
+               goto cleanup;
+       }
+       ctx->routine_args = args;
+       ctx->thread_func = thread_func;
+
+       if (thread_attr != NULL) {
+               ret = pthread_attr_init(&attr);
+               if (ret != 0) {
+                       RTE_LOG(DEBUG, EAL, "pthread_attr_init failed\n");
+                       goto cleanup;
+               }
+
+               attrp = &attr;
+
+               /*
+                * Set the inherit scheduler parameter to explicit,
+                * otherwise the priority attribute is ignored.
+                */
+               ret = pthread_attr_setinheritsched(attrp,
+                               PTHREAD_EXPLICIT_SCHED);
+               if (ret != 0) {
+                       RTE_LOG(DEBUG, EAL, "pthread_attr_setinheritsched failed\n");
+                       goto cleanup;
+               }
+
+
+               if (thread_attr->priority ==
+                               RTE_THREAD_PRIORITY_REALTIME_CRITICAL) {
+                       ret = ENOTSUP;
+                       goto cleanup;
+               }
+               ret = thread_map_priority_to_os_value(thread_attr->priority,
+                               &param.sched_priority, &policy);
+               if (ret != 0)
+                       goto cleanup;
+
+               ret = pthread_attr_setschedpolicy(attrp, policy);
+               if (ret != 0) {
+                       RTE_LOG(DEBUG, EAL, "pthread_attr_setschedpolicy failed\n");
+                       goto cleanup;
+               }
+
+               ret = pthread_attr_setschedparam(attrp, &param);
+               if (ret != 0) {
+                       RTE_LOG(DEBUG, EAL, "pthread_attr_setschedparam failed\n");
+                       goto cleanup;
+               }
+       }
+
+       ret = pthread_create((pthread_t *)&thread_id->opaque_id, attrp,
+               thread_func_wrapper, ctx);
+       if (ret != 0) {
+               RTE_LOG(DEBUG, EAL, "pthread_create failed\n");
+               goto cleanup;
+       }
+
+       if (thread_attr != NULL && CPU_COUNT(&thread_attr->cpuset) > 0) {
+               ret = rte_thread_set_affinity_by_id(*thread_id,
+                       &thread_attr->cpuset);
+               if (ret != 0) {
+                       RTE_LOG(DEBUG, EAL, "rte_thread_set_affinity_by_id failed\n");
+                       goto cleanup;
+               }

lib/eal/unix/rte_thread.c struct eal_tls_key { · line 183

Quoted 10 lines
lib/eal/unix/rte_thread.c +10 -0
+       }
+
+       ctx = NULL;
+cleanup:
+       free(ctx);
+       if (attrp != NULL)
+               pthread_attr_destroy(&attr);
+
+       return ret;
+}

I am looking back at potential races in our code while reviewing the ctrl thread creation fix, and I stopped at this patch.

What happens if rte_thread_set_affinity_by_id() fails?
A new thread got started, but I don't see it being terminated in the cleanup phase.

In this same situation, we may have a small race for a double free on ctx since thread_func_wrapper() may have already been called from the new thread.

I have the same concerns with the windows implementation.

Signature · 1 line
David Marchand
#60 lore raw

Re: [PATCH v5 2/6] eal: add thread lifetime management

Quoted 120 lines · David Marchand

Hello Tyler,

Quoted 116 lines · Tyler Retzlaff
lib/eal/unix/rte_thread.c +101 -0
index 9126595..d4c1a7f 100644
@@ -16,6 +16,11 @@ struct eal_tls_key {
        pthread_key_t thread_index;
 };
 
+struct thread_routine_ctx {
+       rte_thread_func thread_func;
+       void *routine_args;
+};
+
 static int
 thread_map_priority_to_os_value(enum rte_thread_priority eal_pri, int *os_pri,
        int *pol)
@@ -75,6 +80,136 @@ struct eal_tls_key {
        return 0;
 }
 
+static void *
+thread_func_wrapper(void *arg)
+{
+       struct thread_routine_ctx ctx = *(struct thread_routine_ctx *)arg;
+
+       free(arg);
+
+       return (void *)(uintptr_t)ctx.thread_func(ctx.routine_args);
+}
+
+int
+rte_thread_create(rte_thread_t *thread_id,
+               const rte_thread_attr_t *thread_attr,
+               rte_thread_func thread_func, void *args)
+{
+       int ret = 0;
+       pthread_attr_t attr;
+       pthread_attr_t *attrp = NULL;
+       struct thread_routine_ctx *ctx;
+       struct sched_param param = {
+               .sched_priority = 0,
+       };
+       int policy = SCHED_OTHER;
+
+       ctx = calloc(1, sizeof(*ctx));
+       if (ctx == NULL) {
+               RTE_LOG(DEBUG, EAL, "Insufficient memory for thread context allocations\n");
+               ret = ENOMEM;
+               goto cleanup;
+       }
+       ctx->routine_args = args;
+       ctx->thread_func = thread_func;
+
+       if (thread_attr != NULL) {
+               ret = pthread_attr_init(&attr);
+               if (ret != 0) {
+                       RTE_LOG(DEBUG, EAL, "pthread_attr_init failed\n");
+                       goto cleanup;
+               }
+
+               attrp = &attr;
+
+               /*
+                * Set the inherit scheduler parameter to explicit,
+                * otherwise the priority attribute is ignored.
+                */
+               ret = pthread_attr_setinheritsched(attrp,
+                               PTHREAD_EXPLICIT_SCHED);
+               if (ret != 0) {
+                       RTE_LOG(DEBUG, EAL, "pthread_attr_setinheritsched failed\n");
+                       goto cleanup;
+               }
+
+
+               if (thread_attr->priority ==
+                               RTE_THREAD_PRIORITY_REALTIME_CRITICAL) {
+                       ret = ENOTSUP;
+                       goto cleanup;
+               }
+               ret = thread_map_priority_to_os_value(thread_attr->priority,
+                               &param.sched_priority, &policy);
+               if (ret != 0)
+                       goto cleanup;
+
+               ret = pthread_attr_setschedpolicy(attrp, policy);
+               if (ret != 0) {
+                       RTE_LOG(DEBUG, EAL, "pthread_attr_setschedpolicy failed\n");
+                       goto cleanup;
+               }
+
+               ret = pthread_attr_setschedparam(attrp, &param);
+               if (ret != 0) {
+                       RTE_LOG(DEBUG, EAL, "pthread_attr_setschedparam failed\n");
+                       goto cleanup;
+               }
+       }
+
+       ret = pthread_create((pthread_t *)&thread_id->opaque_id, attrp,
+               thread_func_wrapper, ctx);
+       if (ret != 0) {
+               RTE_LOG(DEBUG, EAL, "pthread_create failed\n");
+               goto cleanup;
+       }
+
+       if (thread_attr != NULL && CPU_COUNT(&thread_attr->cpuset) > 0) {
+               ret = rte_thread_set_affinity_by_id(*thread_id,
+                       &thread_attr->cpuset);
+               if (ret != 0) {
+                       RTE_LOG(DEBUG, EAL, "rte_thread_set_affinity_by_id failed\n");
+                       goto cleanup;
+               }
+       }
+
+       ctx = NULL;
+cleanup:
+       free(ctx);
Quoted 8 lines

lib/eal/unix/rte_thread.c struct eal_tls_key { · line 183

Quoted 5 lines
lib/eal/unix/rte_thread.c +5 -0
+       if (attrp != NULL)
+               pthread_attr_destroy(&attr);
+
+       return ret;
+}

I am looking back at potential races in our code while reviewing the ctrl thread creation fix, and I stopped at this patch.

Thank you for doing this, this series I actually did not start I only took over so the review is welcome.

Quoted 9 lines

What happens if rte_thread_set_affinity_by_id() fails?
A new thread got started, but I don't see it being terminated in the cleanup phase.

In this same situation, we may have a small race for a double free on ctx since thread_func_wrapper() may have already been called from the new thread.

I will review this and get back with fixes if necessary.

Quoted 3 lines

I have the same concerns with the windows implementation.

I will check it too.

Quoted 4 lines
Signature · 2 lines
-- 
David Marchand
#61 lore raw

PATCH v5 3/6 eal: add basic rte thread ID equal API

In reply to Tyler Retzlaff

Add rte_thread_equal() that tests if two rte_thread_id are equal.

---
 lib/eal/unix/rte_thread.c    | 6 ++++++
 lib/eal/version.map          | 1 +
 lib/eal/windows/rte_thread.c | 6 ++++++
 3 files changed, 13 insertions(+)
lib/eal/unix/rte_thread.c +6 -0
index d4c1a7f..37ebfcf 100644
@@ -210,6 +210,12 @@ struct thread_routine_ctx {
 	return pthread_detach((pthread_t)thread_id.opaque_id);
 }
 
+int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return pthread_equal((pthread_t)t1.opaque_id, (pthread_t)t2.opaque_id);
+}
+
 rte_thread_t
 rte_thread_self(void)
 {
lib/eal/version.map +1 -0
index 5fd84b6..2d64a25 100644
@@ -438,6 +438,7 @@ EXPERIMENTAL {
 	rte_thread_attr_set_priority;
 	rte_thread_create;
 	rte_thread_detach;
+	rte_thread_equal;
 	rte_thread_join;
 };
 
lib/eal/windows/rte_thread.c +6 -0
index e153278..1c1e9d0 100644
@@ -289,6 +289,12 @@ struct thread_routine_ctx {
 	return 0;
 }
 
+int
+rte_thread_equal(rte_thread_t t1, rte_thread_t t2)
+{
+	return t1.opaque_id == t2.opaque_id;
+}
+
 rte_thread_t
 rte_thread_self(void)
 {
Signature · 1 line
1.8.3.1
#62 lore raw

PATCH v5 4/6 test/threads: add tests for thread lifetime API

In reply to Tyler Retzlaff

Test basic functionality and demonstrate use of following thread lifetime api.

* rte_thread_create
* rte_thread_detach
* rte_thread_join

---
 app/test/test_threads.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 52 insertions(+), 2 deletions(-)
app/test/test_threads.c +52 -2
index b9d8b4e..1077373 100644
@@ -14,7 +14,7 @@
 
 static uint32_t thread_id_ready;
 
-static void *
+static uint32_t
 thread_main(void *arg)
 {
 	*(rte_thread_t *)arg = rte_thread_self();
@@ -23,7 +23,55 @@
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 1)
 		;
 
-	return NULL;
+	return 0;
+}
+
+static int
+test_thread_create_join(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_t thread_main_id;
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, &thread_main_id) == 0,
+		"Failed to create thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_equal(thread_id, thread_main_id) != 0,
+		"Unexpected thread id.");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	RTE_TEST_ASSERT(rte_thread_join(thread_id, NULL) == 0,
+		"Failed to join thread.");
+
+	return 0;
+}
+
+static int
+test_thread_create_detach(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_t thread_main_id;
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main,
+		&thread_main_id) == 0, "Failed to create thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_equal(thread_id, thread_main_id) != 0,
+		"Unexpected thread id.");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	RTE_TEST_ASSERT(rte_thread_detach(thread_id) == 0,
+		"Failed to detach thread.");
+
+	return 0;
 }
 
 static int
@@ -123,6 +171,8 @@
 	.setup = NULL,
 	.teardown = NULL,
 	.unit_test_cases = {
+		TEST_CASE(test_thread_create_join),
+		TEST_CASE(test_thread_create_detach),
 		TEST_CASE(test_thread_affinity),
 		TEST_CASE(test_thread_priority),
 		TEST_CASES_END()
Signature · 1 line
1.8.3.1
#63 lore raw

Re: [PATCH v5 4/6] test/threads: add tests for thread lifetime API

Quoted 6 lines · Tyler Retzlaff

Test basic functionality and demonstrate use of following thread lifetime api.

* rte_thread_create
* rte_thread_detach

And, to some extent, rte_thread_equal.

Quoted 13 lines

* rte_thread_join

---
 app/test/test_threads.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 52 insertions(+), 2 deletions(-)
app/test/test_threads.c +0 -0
index b9d8b4e..1077373 100644

app/test/test_threads.c line 18

Quoted 6 lines
app/test/test_threads.c +1 -1
@@ -14,7 +14,7 @@
 
 static uint32_t thread_id_ready;
 
-static void *
+static uint32_t
 thread_main(void *arg)

Stopping at this patch, I still see a build failure. This prototype change makes it uncompatible with remaining calls to pthread_create.

This is fixed in patch 6 which I intend to squash here (and adding the check on arg != NULL that is in patch 5).
Deal?

Signature · 1 line
David Marchand
#64 lore raw

Re: [PATCH v5 4/6] test/threads: add tests for thread lifetime API

Quoted 27 lines · David Marchand
Quoted 6 lines · Tyler Retzlaff

Test basic functionality and demonstrate use of following thread lifetime api.

* rte_thread_create
* rte_thread_detach

And, to some extent, rte_thread_equal.

Quoted 16 lines

* rte_thread_join

---
 app/test/test_threads.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 52 insertions(+), 2 deletions(-)
app/test/test_threads.c +0 -0
index b9d8b4e..1077373 100644
@@ -14,7 +14,7 @@
 
 static uint32_t thread_id_ready;
 
Quoted 7 lines

app/test/test_threads.c line 18

Quoted 3 lines
app/test/test_threads.c +1 -1
-static void *
+static uint32_t
 thread_main(void *arg)

Stopping at this patch, I still see a build failure. This prototype change makes it uncompatible with remaining calls to pthread_create.

hmm, i must have made a mistake somewhere in the rebase. now i see what you mean about build failure @ patch 4 i was considering it only with the entire series applied i had not been testing individual patches in the series. sorry about that.

Quoted 4 lines

This is fixed in patch 6 which I intend to squash here (and adding the check on arg != NULL that is in patch 5).
Deal?

i have no problem with that.

now that i've read this can you confirm the only outstanding issue is whether to move code to eal_common_thread? i posted a follow up to thomas and i'm just waiting on his reply.

if the reply is still a positive to move the code i can fix the above in a v6.

Thomas waiting on your follow up regarding code move.

thanks

#65 lore raw

PATCH v5 5/6 test/threads: add tests for thread attributes API

In reply to Tyler Retzlaff

Test basic functionality and demonstrate use of following thread attributes api. Additionally, test attributes are processed when supplied to rte_thread_create().

* rte_thread_attr_init
* rte_thread_attr_set_affinity
* rte_thread_attr_get_affinity
* rte_thread_attr_set_priority

---
 app/test/test_threads.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 72 insertions(+), 1 deletion(-)
app/test/test_threads.c +72 -1
index 1077373..3c22cec 100644
@@ -17,7 +17,9 @@
 static uint32_t
 thread_main(void *arg)
 {
-	*(rte_thread_t *)arg = rte_thread_self();
+	if (arg != NULL)
+		*(rte_thread_t *)arg = rte_thread_self();
+
 	__atomic_store_n(&thread_id_ready, 1, __ATOMIC_RELEASE);
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 1)
@@ -166,6 +168,73 @@
 	return 0;
 }
 
+static int
+test_thread_attributes_affinity(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_attr_t attr;
+	rte_cpuset_t cpuset0;
+	rte_cpuset_t cpuset1;
+
+	RTE_TEST_ASSERT(rte_thread_attr_init(&attr) == 0,
+		"Failed to initialize thread attributes");
+
+	CPU_ZERO(&cpuset0);
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(rte_thread_self(), &cpuset0) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(rte_thread_attr_set_affinity(&attr, &cpuset0) == 0,
+		"Failed to set thread attributes affinity");
+	RTE_TEST_ASSERT(rte_thread_attr_get_affinity(&attr, &cpuset1) == 0,
+		"Failed to get thread attributes affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, &attr, thread_main, NULL) == 0,
+		"Failed to create attributes affinity thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get attributes thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Failed to apply affinity attributes");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	return 0;
+}
+
+static int
+test_thread_attributes_priority(void)
+{
+	rte_thread_t thread_id;
+	rte_thread_attr_t attr;
+	enum rte_thread_priority priority;
+
+	RTE_TEST_ASSERT(rte_thread_attr_init(&attr) == 0,
+		"Failed to initialize thread attributes");
+	RTE_TEST_ASSERT(rte_thread_attr_set_priority(&attr, RTE_THREAD_PRIORITY_NORMAL) == 0,
+		"Failed to set thread attributes priority");
+
+	thread_id_ready = 0;
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, &attr, thread_main, NULL) == 0,
+		"Failed to create attributes priority thread.");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_get_priority(thread_id, &priority) == 0,
+		"Failed to get thread priority");
+	RTE_TEST_ASSERT(priority == RTE_THREAD_PRIORITY_NORMAL,
+		"Failed to apply priority attributes");
+
+	__atomic_store_n(&thread_id_ready, 2, __ATOMIC_RELEASE);
+
+	return 0;
+}
+
 static struct unit_test_suite threads_test_suite = {
 	.suite_name = "threads autotest",
 	.setup = NULL,
@@ -175,6 +244,8 @@
 		TEST_CASE(test_thread_create_detach),
 		TEST_CASE(test_thread_affinity),
 		TEST_CASE(test_thread_priority),
+		TEST_CASE(test_thread_attributes_affinity),
+		TEST_CASE(test_thread_attributes_priority),
 		TEST_CASES_END()
 	}
 };
Signature · 1 line
1.8.3.1
#66 lore raw

PATCH v5 6/6 test/threads: remove unit test use of pthread

In reply to Tyler Retzlaff

Now that rte_thread provides thread lifetime functions stop using pthread in unit tests.

---
 app/test/test_threads.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
app/test/test_threads.c +2 -5
index 3c22cec..e0f18e4 100644
@@ -3,7 +3,6 @@
  */
 
 #include <string.h>
-#include <pthread.h>
 
 #include <rte_thread.h>
 #include <rte_debug.h>
@@ -79,12 +78,11 @@
 static int
 test_thread_priority(void)
 {
-	pthread_t id;
 	rte_thread_t thread_id;
 	enum rte_thread_priority priority;
 
 	thread_id_ready = 0;
-	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, NULL) == 0,
 		"Failed to create thread");
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
@@ -131,13 +129,12 @@
 static int
 test_thread_affinity(void)
 {
-	pthread_t id;
 	rte_thread_t thread_id;
 	rte_cpuset_t cpuset0;
 	rte_cpuset_t cpuset1;
 
 	thread_id_ready = 0;
-	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+	RTE_TEST_ASSERT(rte_thread_create(&thread_id, NULL, thread_main, NULL) == 0,
 		"Failed to create thread");
 
 	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
Signature · 1 line
1.8.3.1
#67 lore raw
In reply to Tyler Retzlaff
Quoted 11 lines · Tyler Retzlaff

add rte thread lifetime and attributes api. with these api additions there is now sufficient platform abstracted thread api to remove the use of pthread in the unit tests.

v5:
* include errno.h in rte_thread.c since errno.h is no longer included in rte_common.h
* move rte_thread_attr symbols from 22.07 to 22.11 section of version.map.
* remove RTE_HAS_CPUSET guards from rte_thread.h

Here is a summary of what I did:
- I had to rollback part of RTE_HAS_CPUSET guards.
I had forgotten about the reason for those guards: the "culprit" is
Alpine Linux / musl where cpuset is not exposed unless an application
is built with _GNU_SOURCE.
See: https://git.dpdk.org/dpdk/commit/lib/librte_eal/include/rte_thread.h?id=e0473c6d5b18560dd11fd4d7ebc81dea6774f33e
- I squashed patch 6 in patch 4 to fix the compilation.
- I moved rte_thread.c content to eal_common_thread.c.
- I still added Dmitry acks from v4, as the functional parts/API did not change.

Series applied, thanks.

Signature · 1 line
David Marchand
#68 lore raw

thanks David,

appreciate it. it's nice to finally get this series cleared.

Quoted 30 lines · David Marchand
Quoted 11 lines · Tyler Retzlaff

add rte thread lifetime and attributes api. with these api additions there is now sufficient platform abstracted thread api to remove the use of pthread in the unit tests.

v5:
* include errno.h in rte_thread.c since errno.h is no longer included in rte_common.h
* move rte_thread_attr symbols from 22.07 to 22.11 section of version.map.
* remove RTE_HAS_CPUSET guards from rte_thread.h

Here is a summary of what I did:
- I had to rollback part of RTE_HAS_CPUSET guards.
I had forgotten about the reason for those guards: the "culprit" is
Alpine Linux / musl where cpuset is not exposed unless an application
is built with _GNU_SOURCE.
See: https://git.dpdk.org/dpdk/commit/lib/librte_eal/include/rte_thread.h?id=e0473c6d5b18560dd11fd4d7ebc81dea6774f33e
- I squashed patch 6 in patch 4 to fix the compilation.
- I moved rte_thread.c content to eal_common_thread.c.
- I still added Dmitry acks from v4, as the functional parts/API did not change.

Series applied, thanks.

Signature · 1 line
David Marchand