Affinity devel and fixes
cover Affinity devel and fixes
These are changes to improve affinity on cyclictest
There are some fixes to the parsing of -a CPUSET that occurred in a few exceptional instances.
Also, cyclictest was written as if it owned the whole machine. So before these changes we would run on all cpus or on the CPUSET that the user passed with the command line option even if those cpus were not in the CPUSET from the affinity in the runtime environment. These changes here fix that. For example,
Before the changes
taskset 4-7 cyclictest -t -a 1-6
the taskset 4-7 would only apply to the main cyclictest thread, and then the 1-6 would only apply to every newly created thread.
After the changes
taskset 4-7 cyclictest -t -a 1-6
Now, 4-7 still applies to the main thread, but all newly created threads run on the interesection of the cpus 4-7 and 1-6, in other words on 4,5 and 6
In addition if the user doesn't pass and affinity with the command line option, the runtime environment is respected.
For example
Before
taskset 0-4 cyclictest -t
Only the main cyclictest thread would run on cpus 0-4 and the other threads could run on any cpu
After
taskset 0-4 cyclictest -t
All cyclictest threads run on 0-4
John Kacur (6):
rt-tests: cyclictest: Set errno to 0 before call to mlock
rt-tests: cyclictest: Report all errors from pthread_setaffinity_np
rt-tests: cyclictest: Fix parsing affinity with a space and a leading
zero
rt-tests: cyclictest: Fix parsing affinity with leading exclamation
mark
rt-tests: cyclictest: Only run on available cpus according to the
affinity
rt-tests: cyclictest: Only run on runtime affinity and user supplied
affinity
src/cyclictest/cyclictest.c | 78 +++++++++++++++++++++++++++++++++---- 1 file changed, 71 insertions(+), 7 deletions(-)
Signature · 1 line
2.20.1
PATCH 1/6 rt-tests: cyclictest: Set errno to 0 before call to mlock
Set errno to 0 before the call to mlock in rstat_mlock()
--- src/cyclictest/cyclictest.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
src/cyclictest/cyclictest.c +1 -2
@@ -1868,9 +1868,8 @@ static int rstat_mlock(void *mptr) {int err; - err =mlock (mptr, _SC_PAGE_SIZE); - errno =0 ; + err =mlock (mptr, _SC_PAGE_SIZE);if (err ==-1 ) {fprintf (stderr,"ERROR, mlock %s\n" ,strerror (errno)); }
Signature · 1 line
2.20.1
PATCH 2/6 rt-tests: cyclictest: Report all errors from pthread_setaffinity_np
In reply to John KacurOn error pthread_setaffinity_np returns a non-zero number. Make sure cyclictest prints a warning in all such cases.
--- src/cyclictest/cyclictest.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
src/cyclictest/cyclictest.c +1 -1
@@ -650,7 +650,7 @@ static void *timerthread(void *param)CPU_ZERO (&mask);CPU_SET (par->cpu, &mask); thread =pthread_self (); -if (pthread_setaffinity_np (thread,sizeof (mask), &mask) ==-1 ) +if (pthread_setaffinity_np (thread,sizeof (mask), &mask) !=0 )warn ("Could not set CPU affinity to CPU #%d\n" , par->cpu); }
Signature · 1 line
2.20.1
PATCH 3/6 rt-tests: cyclictest: Fix parsing affinity with a space and a leading zero
In reply to John Kacur-/cyclictest -t -a1-4
./cyclictest -t -a 1-4
./cyclictest -t -a0-4
all work as expected, but
./cyclictest -t -a 0-4
did not work, and instead AFFINITY_USEALL was set.
This is because atoi(argv[optind]) returns 0 for parsing non-numbers, and this was treated as an error.
This missed the case where the user intentionally passes a 0 which should not be treated as an error.
Fix this by testing for this case.
--- src/cyclictest/cyclictest.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
src/cyclictest/cyclictest.c +1 -1
@@ -1199,7 +1199,7 @@ static void process_options (int argc, char *argv[], int max_cpus)if (optarg !=NULL ) {parse_cpumask (optarg, max_cpus); setaffinity =AFFINITY_SPECIFIED ; - }else if (optind<argc &&atoi (argv[optind])) { + }else if (optind<argc && (atoi (argv[optind]) || argv[optind][0 ] =='0' )) {parse_cpumask (argv[optind], max_cpus); setaffinity =AFFINITY_SPECIFIED ; }else {
Signature · 1 line
2.20.1
PATCH 4/6 rt-tests: cyclictest: Fix parsing affinity with leading exclamation mark
In reply to John KacurThe case in which the user specifies the affinity with a space and a leading exclamation mark was not working. For example
./cyclictest -t -a'!4-5'
was working correctly, but the following was not.
./cyclictest -t -a '!4-5'
Fix this by recognizing this input as a non-error case.
--- src/cyclictest/cyclictest.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
src/cyclictest/cyclictest.c +1 -1
@@ -1199,7 +1199,7 @@ static void process_options (int argc, char *argv[], int max_cpus)if (optarg !=NULL ) {parse_cpumask (optarg, max_cpus); setaffinity =AFFINITY_SPECIFIED ; - }else if (optind<argc && (atoi (argv[optind]) || argv[optind][0 ] =='0' )) { + }else if (optind<argc && (atoi (argv[optind]) || argv[optind][0 ] =='0' || argv[optind][0 ] =='!' )) {parse_cpumask (argv[optind], max_cpus); setaffinity =AFFINITY_SPECIFIED ; }else {
Signature · 1 line
2.20.1
PATCH 5/6 rt-tests: cyclictest: Only run on available cpus according to the affinity
In reply to John KacurRight now cyclictest acts as if it owns the machine. If you don't specify the affinity with -a [CPUSET] it will run threads on all cpus.
Change this to only run on cpus according to the affinity.
For example.
taskset -c 0-3 ./cyclictest -t
will only affect the main thread and limit it that to cpus 0-3 but the threads that are spun off can run anywhere, with this change all threads will be limited to cpus 0-3
--- src/cyclictest/cyclictest.c | 39 ++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-)
src/cyclictest/cyclictest.c +36 -3
@@ -1040,7 +1040,8 @@ static unsigned int is_cpumask_zero(const struct bitmask *mask)return (rt_numa_bitmask_count (mask) ==0 ); } -static int cpu_for_thread (int thread_num,int max_cpus) +/* cpu_for_thread AFFINITY_SPECIFIED */ +static int cpu_for_thread_sp (int thread_num,int max_cpus) {unsigned int m, cpu, i, num_cpus; num_cpus =rt_numa_bitmask_count (affinity_mask);
@@ -1059,6 +1060,36 @@ static int cpu_for_thread(int thread_num, int max_cpus)return 0 ; } +/* cpu_for_thread AFFINITY_USEALL */ +static int cpu_for_thread_ua (int thread_num,int max_cpus) +{ +int res, num_cpus, i, m, cpu; +pthread_t thread; +cpu_set_t cpuset; + + thread =pthread_self (); +CPU_ZERO (&cpuset); + + res =pthread_getaffinity_np (thread,sizeof (cpu_set_t), &cpuset); +if (res !=0 ) { +fatal ("pthread_getaffinity_np failed: %s\n" ,strerror (res)); + } + + num_cpus =CPU_COUNT (&cpuset); + m = thread_num % num_cpus; + +for (i =0 , cpu =0 ; i < max_cpus; i++) { +if (CPU_ISSET (i, &cpuset)) { +if (cpu == m) +return i; + cpu++; + } + } + +fprintf (stderr,"Bug in cpu mask handling code.\n" ); +return 0 ; +} +static void parse_cpumask (const char *option,const int max_cpus) {
@@ -2086,11 +2117,13 @@ int main(int argc, char **argv)switch (setaffinity) {case AFFINITY_UNSPECIFIED : cpu =-1 ;break ;case AFFINITY_SPECIFIED : - cpu =cpu_for_thread (i, max_cpus); + cpu =cpu_for_thread_sp (i, max_cpus);if (verbose)printf ("Thread %d using cpu %d.\n" , i, cpu);break ; -case AFFINITY_USEALL : cpu = i % max_cpus;break ; +case AFFINITY_USEALL : + cpu =cpu_for_thread_ua (i, max_cpus); +break ;default : cpu =-1 ; }
Signature · 1 line
2.20.1
PATCH 6/6 rt-tests: cyclictest: Only run on runtime affinity and user supplied affinity
In reply to John KacurCurrently if the user passes the affinity to cyclictest, threads are run there even if they should be excluded according to the affinity of the runtime environment.
For example
taskset -c 4-7 cyclictest -t -a 1-6
Should run on the interesection of this, that is on cpus 4,5,6
Fix this so it works as expected
Note, one caveat, this apply to the threads that cyclictest creates, but only the initial taskset applies to the main cyclictest thread, so the main thread can run on cpus 4,5,6 and 7
--- src/cyclictest/cyclictest.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
src/cyclictest/cyclictest.c +32 -0
@@ -1044,8 +1044,13 @@ static unsigned int is_cpumask_zero(const struct bitmask *mask)static int cpu_for_thread_sp (int thread_num,int max_cpus) {unsigned int m, cpu, i, num_cpus; + num_cpus =rt_numa_bitmask_count (affinity_mask); +if (num_cpus ==0 ) { +fatal ("No allowable cpus to run on\n" ); + } + m = thread_num % num_cpus;/* there are num_cpus bits set, we want position of m'th one */
@@ -1091,6 +1096,30 @@ static int cpu_for_thread_ua(int thread_num, int max_cpus) } +/* After this function is called, affinity_mask is the intersection of the user +* supplied affinity mask and the affinity mask from the run time environment */ +static void use_current_cpuset (const int max_cpus) +{ +int i; +pid_t pid; +struct bitmask *curmask; + + pid =getpid (); + + curmask =numa_bitmask_alloc (sizeof (struct bitmask )); +numa_sched_getaffinity (pid, curmask); + +/* Clear bits that are not set in both the cpuset from the environment, +* and in the user specified affinity for cyclictest */ +for (i=0 ; i < max_cpus; i++) { +if ((!rt_numa_bitmask_isbitset (affinity_mask, i)) || (!rt_numa_bitmask_isbitset (curmask, i))) { +numa_bitmask_clearbit (affinity_mask, i); + } + } + +numa_bitmask_free (curmask); +} +static void parse_cpumask (const char *option,const int max_cpus) { affinity_mask =rt_numa_parse_cpustring (option, max_cpus);
@@ -1098,7 +1127,10 @@ static void parse_cpumask(const char *option, const int max_cpus)if (is_cpumask_zero (affinity_mask)) {rt_bitmask_free (affinity_mask); affinity_mask =NULL ; + }else { +use_current_cpuset (max_cpus); } + }if (!affinity_mask)display_help (1 );
Signature · 1 line
2.20.1