Skip to contentSkip to conversation

cyclictest: Delay setting of main_affinity_mask after creating threads.

PATCH In review Sebastian Andrzej Siewior 3 participants 8 messages Jun 15, 2022 – Jun 29, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cyclictest: Delay setting of main_affinity_mask after creating threads.

Assuming the current affinity mask of the task is 0x31 (CPUs 0, 4, 5). Starting cyclictest with '-S' option will fork the following threads:
- monitor, mask 0x31
- measure thread 1, mask 0x01
- measure thread 2, mask 0x10
- measure thread 3, mask 0x20

works as expected. Using the options '-S --mainaffinity=0' leads to:
- monitor, mask 0x01
- measure thread 1, mask 0x01
- measure thread 2, mask 0x01
- measure thread 3, mask 0x01

because the mask of the main thread has been reset early to 0x01 and does not allow a CPU mask outside of this mask while setting the affinity for the new threads.

Delay setting the affinity of the main/ monitor thread after the measuring threads have been deployed. This leads to the following state:
- monitor, mask 0x01
- measure thread 1, mask 0x01
- measure thread 2, mask 0x10
- measure thread 3, mask 0x20

---
 src/cyclictest/cyclictest.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
src/cyclictest/cyclictest.c +5 -4
index be8285a072b4e..cbf8d00293ec6 100644
@@ -1816,10 +1816,7 @@ int main(int argc, char **argv)
 		printf("Online CPUs = %d\n", online_cpus);
 	}
 
-	/* Restrict the main pid to the affinity specified by the user */
-	if (main_affinity_mask != NULL) {
-		set_main_thread_affinity(main_affinity_mask);
-	} else if (affinity_mask != NULL) {
+	if (affinity_mask != NULL) {
 		set_main_thread_affinity(affinity_mask);
 		if (verbose)
 			printf("Using %u cpus.\n",
@@ -2094,6 +2091,10 @@ int main(int argc, char **argv)
 			fatal("failed to create thread %d: %s\n", i, strerror(status));
 
 	}
+	/* Restrict the main pid to the affinity specified by the user */
+	if (main_affinity_mask != NULL)
+		set_main_thread_affinity(main_affinity_mask);
+
 	if (use_fifo) {
 		status = pthread_create(&fifo_threadid, NULL, fifothread, NULL);
 		if (status)
Signature · 1 line
2.36.1
#1 lore raw

Hello Sebastian,

Quoted 13 lines · Sebastian Andrzej Siewior

Assuming the current affinity mask of the task is 0x31 (CPUs 0, 4, 5). Starting cyclictest with '-S' option will fork the following threads:
- monitor, mask 0x31
- measure thread 1, mask 0x01
- measure thread 2, mask 0x10
- measure thread 3, mask 0x20

works as expected. Using the options '-S --mainaffinity=0' leads to:
- monitor, mask 0x01
- measure thread 1, mask 0x01
- measure thread 2, mask 0x01
- measure thread 3, mask 0x01

Quoted 6 lines

because the mask of the main thread has been reset early to 0x01 and does not allow a CPU mask outside of this mask while setting the affinity for the new threads.

Delay setting the affinity of the main/ monitor thread after the measuring threads have been deployed. This leads to the following

Since starting the threads represents some work that can disturb the threads, shouldn't the main thread be migrated to the 'mainaffinity' mask early as it is right now ?

I think it would be better to:
1. Save the initial mask of the main thread somewhere
2. Set the affinity of the main thread
3. Use the initial mask when setting the affinity of the threads

Regards,
Pierre

Quoted 38 lines

state:
- monitor, mask 0x01
- measure thread 1, mask 0x01
- measure thread 2, mask 0x10
- measure thread 3, mask 0x20

---
  src/cyclictest/cyclictest.c | 9 +++++----
  1 file changed, 5 insertions(+), 4 deletions(-)
src/cyclictest/cyclictest.c +5 -4
index be8285a072b4e..cbf8d00293ec6 100644
@@ -1816,10 +1816,7 @@ int main(int argc, char **argv)
  		printf("Online CPUs = %d\n", online_cpus);
  	}
 
-	/* Restrict the main pid to the affinity specified by the user */
-	if (main_affinity_mask != NULL) {
-		set_main_thread_affinity(main_affinity_mask);
-	} else if (affinity_mask != NULL) {
+	if (affinity_mask != NULL) {
  		set_main_thread_affinity(affinity_mask);
  		if (verbose)
  			printf("Using %u cpus.\n",
@@ -2094,6 +2091,10 @@ int main(int argc, char **argv)
  			fatal("failed to create thread %d: %s\n", i, strerror(status));
 
  	}
+	/* Restrict the main pid to the affinity specified by the user */
+	if (main_affinity_mask != NULL)
+		set_main_thread_affinity(main_affinity_mask);
+
  	if (use_fifo) {
  		status = pthread_create(&fifo_threadid, NULL, fifothread, NULL);
  		if (status)
#2 lore raw
Quoted 1 line · Pierre Gondois

Hello Sebastian,

Hi Pierre,

Quoted 3 lines

Since starting the threads represents some work that can disturb the threads, shouldn't the main thread be migrated to the 'mainaffinity' mask early as it is right now ?

I doubt this such a huge impact given that once the threads are created, they do rt_numa_set_numa_run_on_node() and pthread_setaffinity_np() among other things.

Quoted 4 lines

I think it would be better to:
1. Save the initial mask of the main thread somewhere
2. Set the affinity of the main thread
3. Use the initial mask when setting the affinity of the threads

If this is *that* important I would lean towards always doing the globalt_barr so all threads start once everything is set up. If I remember correctly, the first few cycles are not counted and just fill the cache.

Quoted 2 lines

Regards,
Pierre

Sebastian

#3 lore raw
Quoted 17 lines · Sebastian Andrzej Siewior
Quoted 1 line · Pierre Gondois

Hello Sebastian,

Hi Pierre,

Quoted 3 lines

Since starting the threads represents some work that can disturb the threads, shouldn't the main thread be migrated to the 'mainaffinity' mask early as it is right now ?

I doubt this such a huge impact given that once the threads are created, they do rt_numa_set_numa_run_on_node() and pthread_setaffinity_np() among other things.

Quoted 4 lines

I think it would be better to:
1. Save the initial mask of the main thread somewhere
2. Set the affinity of the main thread
3. Use the initial mask when setting the affinity of the threads

Quoted 4 lines

If this is *that* important I would lean towards always doing the globalt_barr so all threads start once everything is set up. If I remember correctly, the first few cycles are not counted and just fill the cache.

Ok yes, I didn't see the globalt_barr barrier. FWIW I tested the patch and it works well.

Quoted 5 lines
Quoted 3 lines

Regards,
Pierre

Sebastian

#4 lore raw
Quoted 7 lines · Pierre Gondois
Quoted 4 lines

If this is *that* important I would lean towards always doing the globalt_barr so all threads start once everything is set up. If I remember correctly, the first few cycles are not counted and just fill the cache.

Ok yes, I didn't see the globalt_barr barrier. FWIW I tested the patch and it works well.

The globalt_barr isn't enabled by default. If that is important then I recommend doing this by default. I can post patch if you want me to.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Regards,
Pierre

Sebastian

#5 lore raw
Quoted 11 lines · Sebastian Andrzej Siewior
Quoted 7 lines · Pierre Gondois
Quoted 4 lines

If this is *that* important I would lean towards always doing the globalt_barr so all threads start once everything is set up. If I remember correctly, the first few cycles are not counted and just fill the cache.

Ok yes, I didn't see the globalt_barr barrier. FWIW I tested the patch and it works well.

The globalt_barr isn't enabled by default. If that is important then I recommend doing this by default. I can post patch if you want me to.

Starting all the threads at the same time seems sensible to me, but I don't have any numbers to prove this is useful.

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Regards,
Pierre

Sebastian

#6 lore raw
In reply to Sebastian Andrzej Siewior
Quoted 35 lines · Sebastian Andrzej Siewior

Assuming the current affinity mask of the task is 0x31 (CPUs 0, 4, 5). Starting cyclictest with '-S' option will fork the following threads:
- monitor, mask 0x31
- measure thread 1, mask 0x01
- measure thread 2, mask 0x10
- measure thread 3, mask 0x20

works as expected. Using the options '-S --mainaffinity=0' leads to:
- monitor, mask 0x01
- measure thread 1, mask 0x01
- measure thread 2, mask 0x01
- measure thread 3, mask 0x01

because the mask of the main thread has been reset early to 0x01 and does not allow a CPU mask outside of this mask while setting the affinity for the new threads.

Delay setting the affinity of the main/ monitor thread after the measuring threads have been deployed. This leads to the following state:
- monitor, mask 0x01
- measure thread 1, mask 0x01
- measure thread 2, mask 0x10
- measure thread 3, mask 0x20

---
  src/cyclictest/cyclictest.c | 9 +++++----
  1 file changed, 5 insertions(+), 4 deletions(-)
src/cyclictest/cyclictest.c +0 -0
index be8285a072b4e..cbf8d00293ec6 100644

src/cyclictest/cyclictest.c int main(int argc, char **argv) · line 1820

Quoted 9 lines
src/cyclictest/cyclictest.c +1 -4
@@ -1816,10 +1816,7 @@ int main(int argc, char **argv)
  		printf("Online CPUs = %d\n", online_cpus);
  	}
 
-	/* Restrict the main pid to the affinity specified by the user */
-	if (main_affinity_mask != NULL) {
-		set_main_thread_affinity(main_affinity_mask);
-	} else if (affinity_mask != NULL) {
+	if (affinity_mask != NULL) {
  		set_main_thread_affinity(affinity_mask);

2 other remarks (sorry for the spam):
- In the documentation, '-a' seems to indicate a mask for the threads, not for the main process. Should the main thread be restricted to the mask obtained for '-a' ?
- I think there is a bug (unrelated to the patch) when issuing:
./cyclictest -l20000 -v -a -t10
This is due to trying to print a NULL the affinity_mask at:
@@ -1088,9 +1093,6 @@ static void process_options(int argc, char *argv[], int max_cpus)

                        if (setaffinity == AFFINITY_SPECIFIED && !affinity_mask)
                                display_help(1);
-                       if (verbose)
-                               printf("Using %u cpus.\n",
-                                       numa_bitmask_weight(affinity_mask));
                        break;
                case 'A':
                case OPT_ALIGNED:

The command:
./cyclictest -l20000 -a -t10 -v
will work since '-v' is parsed at last and the 'if (verbose)' statement is ignored.

Quoted 13 lines
src/cyclictest/cyclictest.c +4 -0
  		if (verbose)
  			printf("Using %u cpus.\n",
@@ -2094,6 +2091,10 @@ int main(int argc, char **argv)
  			fatal("failed to create thread %d: %s\n", i, strerror(status));
 
  	}
+	/* Restrict the main pid to the affinity specified by the user */
+	if (main_affinity_mask != NULL)
+		set_main_thread_affinity(main_affinity_mask);
+
  	if (use_fifo) {
  		status = pthread_create(&fifo_threadid, NULL, fifothread, NULL);
  		if (status)
#7 lore raw
In reply to Sebastian Andrzej Siewior
Quoted 61 lines · Sebastian Andrzej Siewior

Assuming the current affinity mask of the task is 0x31 (CPUs 0, 4, 5). Starting cyclictest with '-S' option will fork the following threads:
- monitor, mask 0x31
- measure thread 1, mask 0x01
- measure thread 2, mask 0x10
- measure thread 3, mask 0x20

works as expected. Using the options '-S --mainaffinity=0' leads to:
- monitor, mask 0x01
- measure thread 1, mask 0x01
- measure thread 2, mask 0x01
- measure thread 3, mask 0x01

because the mask of the main thread has been reset early to 0x01 and does not allow a CPU mask outside of this mask while setting the affinity for the new threads.

Delay setting the affinity of the main/ monitor thread after the measuring threads have been deployed. This leads to the following state:
- monitor, mask 0x01
- measure thread 1, mask 0x01
- measure thread 2, mask 0x10
- measure thread 3, mask 0x20

---
 src/cyclictest/cyclictest.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
src/cyclictest/cyclictest.c +5 -4
index be8285a072b4e..cbf8d00293ec6 100644
@@ -1816,10 +1816,7 @@ int main(int argc, char **argv)
 		printf("Online CPUs = %d\n", online_cpus);
 	}
 
-	/* Restrict the main pid to the affinity specified by the user */
-	if (main_affinity_mask != NULL) {
-		set_main_thread_affinity(main_affinity_mask);
-	} else if (affinity_mask != NULL) {
+	if (affinity_mask != NULL) {
 		set_main_thread_affinity(affinity_mask);
 		if (verbose)
 			printf("Using %u cpus.\n",
@@ -2094,6 +2091,10 @@ int main(int argc, char **argv)
 			fatal("failed to create thread %d: %s\n", i, strerror(status));
 
 	}
+	/* Restrict the main pid to the affinity specified by the user */
+	if (main_affinity_mask != NULL)
+		set_main_thread_affinity(main_affinity_mask);
+
 	if (use_fifo) {
 		status = pthread_create(&fifo_threadid, NULL, fifothread, NULL);
 		if (status)
Signature · 1 line
2.36.1