Skip to contentSkip to conversation

Affinity devel and fixes

PATCH New John Kacur 1 participant 7 messages Mar 2, 2020

lore
Share & tools
lore

John Kacur

author
#0 lore raw

cover Affinity devel and fixes

These are changes to improve affinity on cyclictest

There are some fixes to the parsing of -a CPUSET that occurred in a few exceptional instances.

Also, cyclictest was written as if it owned the whole machine. So before these changes we would run on all cpus or on the CPUSET that the user passed with the command line option even if those cpus were not in the CPUSET from the affinity in the runtime environment. These changes here fix that. For example,

Before the changes
taskset 4-7 cyclictest -t -a 1-6
the taskset 4-7 would only apply to the main cyclictest thread, and then the 1-6 would only apply to every newly created thread.

After the changes
taskset 4-7 cyclictest -t -a 1-6
Now, 4-7 still applies to the main thread, but all newly created threads run on the interesection of the cpus 4-7 and 1-6, in other words on 4,5 and 6

In addition if the user doesn't pass and affinity with the command line option, the runtime environment is respected.

For example

Before
taskset 0-4 cyclictest -t
Only the main cyclictest thread would run on cpus 0-4 and the other threads could run on any cpu

After
taskset 0-4 cyclictest -t
All cyclictest threads run on 0-4

John Kacur (6):
rt-tests: cyclictest: Set errno to 0 before call to mlock
rt-tests: cyclictest: Report all errors from pthread_setaffinity_np
rt-tests: cyclictest: Fix parsing affinity with a space and a leading
zero
rt-tests: cyclictest: Fix parsing affinity with leading exclamation
mark
rt-tests: cyclictest: Only run on available cpus according to the
affinity
rt-tests: cyclictest: Only run on runtime affinity and user supplied
affinity

 src/cyclictest/cyclictest.c | 78 +++++++++++++++++++++++++++++++++----
 1 file changed, 71 insertions(+), 7 deletions(-)
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH 1/6 rt-tests: cyclictest: Set errno to 0 before call to mlock

Set errno to 0 before the call to mlock in rstat_mlock()

---
 src/cyclictest/cyclictest.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
src/cyclictest/cyclictest.c +1 -2
index 01ae72f8cdf1..c66755dd4d87 100644
@@ -1868,9 +1868,8 @@ static int rstat_mlock(void *mptr)
 {
 	int err;
 
-	err = mlock(mptr, _SC_PAGE_SIZE);
-
 	errno = 0;
+	err = mlock(mptr, _SC_PAGE_SIZE);
 	if (err == -1) {
 		fprintf(stderr, "ERROR, mlock %s\n", strerror(errno));
 	}
Signature · 1 line
2.20.1
author
#2 lore raw

PATCH 2/6 rt-tests: cyclictest: Report all errors from pthread_setaffinity_np

In reply to John Kacur

On error pthread_setaffinity_np returns a non-zero number. Make sure cyclictest prints a warning in all such cases.

---
 src/cyclictest/cyclictest.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
src/cyclictest/cyclictest.c +1 -1
index c66755dd4d87..bd1fcd1092aa 100644
@@ -650,7 +650,7 @@ static void *timerthread(void *param)
 		CPU_ZERO(&mask);
 		CPU_SET(par->cpu, &mask);
 		thread = pthread_self();
-		if (pthread_setaffinity_np(thread, sizeof(mask), &mask) == -1)
+		if (pthread_setaffinity_np(thread, sizeof(mask), &mask) != 0)
 			warn("Could not set CPU affinity to CPU #%d\n",
 			     par->cpu);
 	}
Signature · 1 line
2.20.1
author
#3 lore raw

PATCH 3/6 rt-tests: cyclictest: Fix parsing affinity with a space and a leading zero

In reply to John Kacur

-/cyclictest -t -a1-4
./cyclictest -t -a 1-4
./cyclictest -t -a0-4
all work as expected, but
./cyclictest -t -a 0-4
did not work, and instead AFFINITY_USEALL was set.

This is because atoi(argv[optind]) returns 0 for parsing non-numbers, and this was treated as an error.

This missed the case where the user intentionally passes a 0 which should not be treated as an error.

Fix this by testing for this case.

---
 src/cyclictest/cyclictest.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
src/cyclictest/cyclictest.c +1 -1
index bd1fcd1092aa..1d2962fda777 100644
@@ -1199,7 +1199,7 @@ static void process_options (int argc, char *argv[], int max_cpus)
 			if (optarg != NULL) {
 				parse_cpumask(optarg, max_cpus);
 				setaffinity = AFFINITY_SPECIFIED;
-			} else if (optind<argc && atoi(argv[optind])) {
+			} else if (optind<argc && (atoi(argv[optind]) || argv[optind][0] == '0')) {
 				parse_cpumask(argv[optind], max_cpus);
 				setaffinity = AFFINITY_SPECIFIED;
 			} else {
Signature · 1 line
2.20.1
author
#4 lore raw

PATCH 4/6 rt-tests: cyclictest: Fix parsing affinity with leading exclamation mark

In reply to John Kacur

The case in which the user specifies the affinity with a space and a leading exclamation mark was not working. For example

./cyclictest -t -a'!4-5'
was working correctly, but the following was not.
./cyclictest -t -a '!4-5'

Fix this by recognizing this input as a non-error case.

---
 src/cyclictest/cyclictest.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
src/cyclictest/cyclictest.c +1 -1
index 1d2962fda777..18c383689dc2 100644
@@ -1199,7 +1199,7 @@ static void process_options (int argc, char *argv[], int max_cpus)
 			if (optarg != NULL) {
 				parse_cpumask(optarg, max_cpus);
 				setaffinity = AFFINITY_SPECIFIED;
-			} else if (optind<argc && (atoi(argv[optind]) || argv[optind][0] == '0')) {
+			} else if (optind<argc && (atoi(argv[optind]) || argv[optind][0] == '0' || argv[optind][0] == '!')) {
 				parse_cpumask(argv[optind], max_cpus);
 				setaffinity = AFFINITY_SPECIFIED;
 			} else {
Signature · 1 line
2.20.1
author
#5 lore raw

PATCH 5/6 rt-tests: cyclictest: Only run on available cpus according to the affinity

In reply to John Kacur

Right now cyclictest acts as if it owns the machine. If you don't specify the affinity with -a [CPUSET] it will run threads on all cpus.

Change this to only run on cpus according to the affinity.

For example.

taskset -c 0-3 ./cyclictest -t
will only affect the main thread and limit it that to cpus 0-3 but the threads that are spun off can run anywhere, with this change all threads will be limited to cpus 0-3

---
 src/cyclictest/cyclictest.c | 39 ++++++++++++++++++++++++++++++++++---
 1 file changed, 36 insertions(+), 3 deletions(-)
src/cyclictest/cyclictest.c +36 -3
index 18c383689dc2..ce93ad0643cd 100644
@@ -1040,7 +1040,8 @@ static unsigned int is_cpumask_zero(const struct bitmask *mask)
 	return (rt_numa_bitmask_count(mask) == 0);
 }
 
-static int cpu_for_thread(int thread_num, int max_cpus)
+/* cpu_for_thread AFFINITY_SPECIFIED */
+static int cpu_for_thread_sp(int thread_num, int max_cpus)
 {
 	unsigned int m, cpu, i, num_cpus;
 	num_cpus = rt_numa_bitmask_count(affinity_mask);
@@ -1059,6 +1060,36 @@ static int cpu_for_thread(int thread_num, int max_cpus)
 	return 0;
 }
 
+/* cpu_for_thread AFFINITY_USEALL */
+static int cpu_for_thread_ua(int thread_num, int max_cpus)
+{
+	int res, num_cpus, i, m, cpu;
+	pthread_t thread;
+	cpu_set_t cpuset;
+
+	thread = pthread_self();
+	CPU_ZERO(&cpuset);
+
+	res = pthread_getaffinity_np(thread, sizeof(cpu_set_t), &cpuset);
+	if (res != 0) {
+		fatal("pthread_getaffinity_np failed: %s\n", strerror(res));
+	}
+
+	num_cpus = CPU_COUNT(&cpuset);
+	m = thread_num % num_cpus;
+
+	for (i = 0, cpu = 0; i < max_cpus; i++) {
+		if (CPU_ISSET(i, &cpuset)) {
+			if (cpu == m)
+				return i;
+			cpu++;
+		}
+	}
+
+	fprintf(stderr, "Bug in cpu mask handling code.\n");
+	return 0;
+}
+
 
 static void parse_cpumask(const char *option, const int max_cpus)
 {
@@ -2086,11 +2117,13 @@ int main(int argc, char **argv)
 		switch (setaffinity) {
 		case AFFINITY_UNSPECIFIED: cpu = -1; break;
 		case AFFINITY_SPECIFIED:
-			cpu = cpu_for_thread(i, max_cpus);
+			cpu = cpu_for_thread_sp(i, max_cpus);
 			if (verbose)
 				printf("Thread %d using cpu %d.\n", i, cpu);
 			break;
-		case AFFINITY_USEALL: cpu = i % max_cpus; break;
+		case AFFINITY_USEALL:
+			cpu = cpu_for_thread_ua(i, max_cpus);
+			break;
 		default: cpu = -1;
 		}
 
Signature · 1 line
2.20.1
author
#6 lore raw

PATCH 6/6 rt-tests: cyclictest: Only run on runtime affinity and user supplied affinity

In reply to John Kacur

Currently if the user passes the affinity to cyclictest, threads are run there even if they should be excluded according to the affinity of the runtime environment.

For example

taskset -c 4-7 cyclictest -t -a 1-6

Should run on the interesection of this, that is on cpus 4,5,6

Fix this so it works as expected

Note, one caveat, this apply to the threads that cyclictest creates, but only the initial taskset applies to the main cyclictest thread, so the main thread can run on cpus 4,5,6 and 7

---
 src/cyclictest/cyclictest.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
src/cyclictest/cyclictest.c +32 -0
index ce93ad0643cd..13d2b1722890 100644
@@ -1044,8 +1044,13 @@ static unsigned int is_cpumask_zero(const struct bitmask *mask)
 static int cpu_for_thread_sp(int thread_num, int max_cpus)
 {
 	unsigned int m, cpu, i, num_cpus;
+
 	num_cpus = rt_numa_bitmask_count(affinity_mask);
 
+	if (num_cpus == 0) {
+		fatal("No allowable cpus to run on\n");
+	}
+
 	m = thread_num % num_cpus;
 
 	/* there are num_cpus bits set, we want position of m'th one */
@@ -1091,6 +1096,30 @@ static int cpu_for_thread_ua(int thread_num, int max_cpus)
 }
 
 
+/* After this function is called, affinity_mask is the intersection of the user
+ * supplied affinity mask and the affinity mask from the run time environment */
+static void use_current_cpuset(const int max_cpus)
+{
+	int i;
+	pid_t pid;
+	struct bitmask *curmask;
+
+	pid = getpid();
+
+	curmask = numa_bitmask_alloc(sizeof(struct bitmask));
+	numa_sched_getaffinity(pid, curmask);
+
+	/* Clear bits that are not set in both the cpuset from the environment,
+	 * and in the user specified affinity for cyclictest */
+	for (i=0; i < max_cpus; i++) {
+		if ((!rt_numa_bitmask_isbitset(affinity_mask, i)) || (!rt_numa_bitmask_isbitset(curmask, i))) {
+			numa_bitmask_clearbit(affinity_mask, i);
+		}
+	}
+
+	numa_bitmask_free(curmask);
+}
+
 static void parse_cpumask(const char *option, const int max_cpus)
 {
 	affinity_mask = rt_numa_parse_cpustring(option, max_cpus);
@@ -1098,7 +1127,10 @@ static void parse_cpumask(const char *option, const int max_cpus)
 		if (is_cpumask_zero(affinity_mask)) {
 			rt_bitmask_free(affinity_mask);
 			affinity_mask = NULL;
+		} else {
+			use_current_cpuset(max_cpus);
 		}
+
 	}
 	if (!affinity_mask)
 		display_help(1);
Signature · 1 line
2.20.1