rt-tests: cyclictest: Add option to specify main pid affinity

PATCH Superseded Jonathan Schwender 2 participants 4 messages Mar 25, 2021 – May 7, 2021

lore

Share & tools

lore

Jonathan Schwender

author

2021-03-25 20:19 UTC #0 lore raw

v3 cover rt-tests: cyclictest: Add option to specify main pid affinity

Hi John,

Changes in v3:
Rebased onto latest commit 95066b1a1a0a ("cyclicdeadline.c: Fix printf format specifier") in unstable/devel/latest.

This patch adds the option --mainaffinity to specify the affinity of the main pid.
This is mainly useful if you want to bind the main thread to a different (e.g. housekeeping ) CPU than the measurement threads.

Some numbers on potential latency benefits are linked in the
v1 thread at https://lore.kernel.org/linux-rt-users/[email protected]/

Regards

Jonathan

Jonathan Schwender (2):
cyclictest: Move main pid setaffinity handling into a function
cyclictest: Add --mainaffinity=[CPUSET] option.

 src/cyclictest/cyclictest.c | 28 +++++++++++++++++++---------
 src/include/rt-numa.h       |  2 ++
 src/lib/rt-numa.c           | 11 +++++++++++
 3 files changed, 32 insertions(+), 9 deletions(-)

Signature · 1 line

2.30.2

Jonathan Schwender

author

2021-03-25 20:19 UTC #1 lore raw

PATCH v3 1/2 cyclictest: Move main pid setaffinity handling into a function

Move error handling for setting the affinity of the main pid into a separate function.
This prevents duplicating the code in the next commit, where the main thread pid can be restricted to one of two bitmasks depending on the passed parameters.

After feedback from John Kacur, the function is now located in src/lib/rt-numa.c.
This allows other tests to reuse this, if they also prefer to warn if numa_sched_setaffinity fails.

Signed-off-by Jonathan Schwender <[email protected]>

---
 src/cyclictest/cyclictest.c |  8 +-------
 src/include/rt-numa.h       |  2 ++
 src/lib/rt-numa.c           | 11 +++++++++++
 3 files changed, 14 insertions(+), 7 deletions(-)

src/cyclictest/cyclictest.c +1 -7

index dca9610..460f6ae 100644
@@ -1791,13 +1791,7 @@ int main(int argc, char **argv)
 
 	/* Restrict the main pid to the affinity specified by the user */
 	if (affinity_mask != NULL) {
-		int res;
-
-		errno = 0;
-		res = numa_sched_setaffinity(getpid(), affinity_mask);
-		if (res != 0)
-			warn("Couldn't setaffinity in main thread: %s\n", strerror(errno));
-
+		try_numa_sched_setaffinity(getpid(), affinity_mask);
 		if (verbose)
 			printf("Using %u cpus.\n",
 				numa_bitmask_weight(affinity_mask));

src/include/rt-numa.h +2 -0

index ca86a45..e9fa312 100644
@@ -18,4 +18,6 @@ int cpu_for_thread_ua(int thread_num, int max_cpus);
 
 int parse_cpumask(char *str, int max_cpus, struct bitmask **cpumask);
 
+void try_numa_sched_setaffinity(__pid_t pid, struct bitmask *cpumask);
+
 #endif

src/lib/rt-numa.c +11 -0

index babcc63..9a81cfb 100644
@@ -138,3 +138,14 @@ int parse_cpumask(char *str, int max_cpus, struct bitmask **cpumask)
 
 	return 0;
 }
+
+void try_numa_sched_setaffinity(pid_t pid, struct bitmask *cpumask)
+{
+	int res;
+
+	errno = 0;
+	res = numa_sched_setaffinity(pid, cpumask);
+	if (res != 0)
+		warn("Couldn't setaffinity for thread %d: %s\n", pid,
+		     strerror(errno));
+}

Signature · 1 line

2.30.2

John Kacur

2021-05-07 17:07 UTC #2 lore raw

Re: [PATCH v3 1/2] cyclictest: Move main pid setaffinity handling into a function

Quoted 71 lines · Jonathan Schwender

After feedback from John Kacur, the function is now located in src/lib/rt-numa.c.
This allows other tests to reuse this, if they also prefer to warn if numa_sched_setaffinity fails.

Signed-off-by Jonathan Schwender <[email protected]>

---
 src/cyclictest/cyclictest.c |  8 +-------
 src/include/rt-numa.h       |  2 ++
 src/lib/rt-numa.c           | 11 +++++++++++
 3 files changed, 14 insertions(+), 7 deletions(-)

src/cyclictest/cyclictest.c +1 -7

index dca9610..460f6ae 100644
@@ -1791,13 +1791,7 @@ int main(int argc, char **argv)
 
 	/* Restrict the main pid to the affinity specified by the user */
 	if (affinity_mask != NULL) {
-		int res;
-
-		errno = 0;
-		res = numa_sched_setaffinity(getpid(), affinity_mask);
-		if (res != 0)
-			warn("Couldn't setaffinity in main thread: %s\n", strerror(errno));
-
+		try_numa_sched_setaffinity(getpid(), affinity_mask);
 		if (verbose)
 			printf("Using %u cpus.\n",
 				numa_bitmask_weight(affinity_mask));

src/include/rt-numa.h +2 -0

index ca86a45..e9fa312 100644
@@ -18,4 +18,6 @@ int cpu_for_thread_ua(int thread_num, int max_cpus);
 
 int parse_cpumask(char *str, int max_cpus, struct bitmask **cpumask);
 
+void try_numa_sched_setaffinity(__pid_t pid, struct bitmask *cpumask);
+
 #endif

src/lib/rt-numa.c +11 -0

index babcc63..9a81cfb 100644
@@ -138,3 +138,14 @@ int parse_cpumask(char *str, int max_cpus, struct bitmask **cpumask)
 
 	return 0;
 }
+
+void try_numa_sched_setaffinity(pid_t pid, struct bitmask *cpumask)
+{
+	int res;
+
+	errno = 0;
+	res = numa_sched_setaffinity(pid, cpumask);
+	if (res != 0)
+		warn("Couldn't setaffinity for thread %d: %s\n", pid,
+		     strerror(errno));
+}

Signature · 1 line

2.30.2

Yuck. My bad, this is some pretty inelegant error code wrapped around a numa call, I don't think it really warrents trying to make an API out of it. That said, I might need to do this anyway, when I put back code that can run when numa is not available on a target machine.

For now, can you just leave this inline and respin?

Thanks

John Kacur

Jonathan Schwender

author

2021-03-25 20:19 UTC #3 lore raw

PATCH v3 2/2 cyclictest: Add --mainaffinity=[CPUSET] option.

In reply to Jonathan Schwender

This allows the user to specify a separate cpuset for the main pid, e.g. on a housekeeping CPU.
If --mainaffinity is not specified, but --affinity is, then the current behaviour is preserved and the main pid is bound to the cpuset specified by --affinity

Signed-off-by Jonathan Schwender <[email protected]>

---
 src/cyclictest/cyclictest.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

src/cyclictest/cyclictest.c +18 -2

index 460f6ae..b7ca8de 100644
@@ -836,6 +836,8 @@ static void display_help(int error)
 	       "	 --laptop	   Save battery when running cyclictest\n"
 	       "			   This will give you poorer realtime results\n"
 	       "			   but will not drain your battery so quickly\n"
+	       "         --mainaffinity=[CPUSET]  Run the main thread on CPU #N. This only affects\n"
+	       "                           the main thread and not the measurement threads\n"
 	       "-m       --mlockall        lock current and future memory allocations\n"
 	       "-M       --refresh_on_max  delay updating the screen until a new max\n"
 	       "			   latency is hit. Useful for low bandwidth.\n"

@@ -891,6 +893,7 @@ static int quiet;
 static int interval = DEFAULT_INTERVAL;
 static int distance = -1;
 static struct bitmask *affinity_mask = NULL;
+static struct bitmask *main_affinity_mask = NULL;
 static int smp = 0;
 static int setaffinity = AFFINITY_UNSPECIFIED;

@@ -944,7 +947,7 @@ enum option_values {
 	OPT_AFFINITY=1, OPT_BREAKTRACE, OPT_CLOCK,
 	OPT_DISTANCE, OPT_DURATION, OPT_LATENCY,
 	OPT_FIFO, OPT_HISTOGRAM, OPT_HISTOFALL, OPT_HISTFILE,
-	OPT_INTERVAL, OPT_LOOPS, OPT_MLOCKALL, OPT_REFRESH,
+	OPT_INTERVAL, OPT_LOOPS, OPT_MAINAFFINITY, OPT_MLOCKALL, OPT_REFRESH,
 	OPT_NANOSLEEP, OPT_NSECS, OPT_OSCOPE, OPT_PRIORITY,
 	OPT_QUIET, OPT_PRIOSPREAD, OPT_RELATIVE, OPT_RESOLUTION,
 	OPT_SYSTEM, OPT_SMP, OPT_THREADS, OPT_TRIGGER,

@@ -981,6 +984,7 @@ static void process_options(int argc, char *argv[], int max_cpus)
 			{"interval",         required_argument, NULL, OPT_INTERVAL },
 			{"laptop",	     no_argument,	NULL, OPT_LAPTOP },
 			{"loops",            required_argument, NULL, OPT_LOOPS },
+			{"mainaffinity",     required_argument, NULL, OPT_MAINAFFINITY},
 			{"mlockall",         no_argument,       NULL, OPT_MLOCKALL },
 			{"refresh_on_max",   no_argument,       NULL, OPT_REFRESH },
 			{"nsecs",            no_argument,       NULL, OPT_NSECS },

@@ -1083,6 +1087,16 @@ static void process_options(int argc, char *argv[], int max_cpus)
 		case 'l':
 		case OPT_LOOPS:
 			max_cycles = atoi(optarg); break;
+		case OPT_MAINAFFINITY:
+			if (optarg) {
+				parse_cpumask(optarg, max_cpus, &main_affinity_mask);
+			} else if (optind < argc &&
+			           (atoi(argv[optind]) ||
+			            argv[optind][0] == '0' ||
+			            argv[optind][0] == '!')) {
+				parse_cpumask(argv[optind], max_cpus, &main_affinity_mask);
+			}
+			break;
 		case 'm':
 		case OPT_MLOCKALL:
 			lockall = 1; break;

@@ -1790,7 +1804,9 @@ int main(int argc, char **argv)
 	}
 
 	/* Restrict the main pid to the affinity specified by the user */
-	if (affinity_mask != NULL) {
+	if (main_affinity_mask != NULL) {
+		try_numa_sched_setaffinity(getpid(), main_affinity_mask);
+	} else if (affinity_mask != NULL) {
 		try_numa_sched_setaffinity(getpid(), affinity_mask);
 		if (verbose)
 			printf("Using %u cpus.\n",

Signature · 1 line

2.30.2