align thread wakeup times
PATCH 1/1 align thread wakeup times
Hi !
This patch provides and additional -A/--align flag to cyclictest to align
thread wakeup times of all threads as closly defined as possible.
When running multiple threads in cyclictest (-S or -t # option) the threads
are launched in an unsynchronized manner. Basically the creation order and time for thread creation determines the start time. For provoking a maximum congestion situation (e.g. cache evictions) and to improve reproducibility or run conditions the start time should be defined distances appart. The well defined distance is implemented as a offset parameter to -A/--align and will offset each threads start time by the parameter * the sequentially assigned thread number (par->tnum), together with the -d0 (distance in the intervals of the individual threads) this alignment option allows to get the thread wakeup times as closely synchronized as possible.
The method to sync is simply that the thread with par->tnum == 0 is chosen
to set a globally shared timestamp, and all other threads use this timestamp as their starting time rather than each calling clock_gettime() at startup. To ensure synchronization of the thread startup the setting of the global time is guarded by pthread_barriers.
--- src/cyclictest/cyclictest.c | 36 ++++++++++++++++++++++++++++++++++-- 1 files changed, 34 insertions(+), 2 deletions(-)
src/cyclictest/cyclictest.c +34 -2
@@ -138,6 +138,7 @@ struct thread_param {unsigned long interval;int cpu;int node; +int tnum; };/* Struct for statistics */
@@ -178,6 +179,8 @@ static int force_sched_other;static int priospread =0 ;static int check_clock_resolution;static int ct_debug; +static int aligned =0 ; +static int disaligned =0 ;static pthread_cond_t refresh_on_max_cond =PTHREAD_COND_INITIALIZER ;static pthread_mutex_t refresh_on_max_lock =PTHREAD_MUTEX_INITIALIZER ;
@@ -186,6 +189,10 @@ static pthread_mutex_t break_thread_id_lock = PTHREAD_MUTEX_INITIALIZER;static pid_t break_thread_id =0 ;static uint64_t break_thread_value =0 ; +static pthread_barrier_t align_barr; +static pthread_barrier_t globalt_barr; +static struct timespec globalt; +/* Backup of kernel variables that we modify */ static struct kvars { char name[KVARNAMELEN];
@@ -758,7 +765,20 @@ void *timerthread(void *param)fatal ("timerthread%d: failed to set priority to %d\n" , par->cpu, par->prio);/* Get current time */ -clock_gettime (par->clock, &now); +if (aligned){ +pthread_barrier_wait (&globalt_barr); +if (par->tnum==0 ){ +clock_gettime (par->clock, &globalt); + } +pthread_barrier_wait (&align_barr); + now = globalt; +if (disaligned){ + now.tv_nsec += disaligned * par->tnum; +tsnorm (&now); + } + }else { +clock_gettime (par->clock, &now); + } next = now; next.tv_sec += interval.tv_sec;
@@ -951,6 +971,7 @@ static void display_help(int error)"cyclictest <options>\n\n" "-a [NUM] --affinity run thread #N on processor #N, if possible\n" " with NUM pin all threads to the processor NUM\n" +"-A USEC --aligned=USEC allign threads wakeup to a specific offset" "-b USEC --breaktrace=USEC send break trace command when latency > USEC\n" "-B --preemptirqs both preempt and irqsoff tracing (used with -b)\n" "-c CLOCK --clock=CLOCK select clock\n"
@@ -1093,6 +1114,7 @@ static void process_options (int argc, char *argv[])*/ static struct option long_options[] = { {"affinity" , optional_argument,NULL ,'a' }, + {"aligned" , optional_argument,NULL ,'A' }, {"breaktrace" , required_argument,NULL ,'b' }, {"preemptirqs" , no_argument,NULL ,'B' }, {"clock" , required_argument,NULL ,'c' },
@@ -1133,7 +1155,7 @@ static void process_options (int argc, char *argv[]) {"help", no_argument, NULL, '?'}, {NULL, 0, NULL, 0} }; -int c =getopt_long (argc, argv,"a::b:Bc:Cd:D:e:Efh:H:i:Il:MnNo:O:p:PmqQrRsSt::uUvD:wWXT:y:" , +int c =getopt_long (argc, argv,"a::A:b:Bc:Cd:D:e:Efh:H:i:Il:MnNo:O:p:PmqQrRsSt::uUvD:wWXT:y:" , long_options, &option_index);if (c ==-1 )break ;
@@ -1152,6 +1174,10 @@ static void process_options (int argc, char *argv[]) setaffinity =AFFINITY_USEALL ; }break ; +case 'A' : aligned=1 ; +if (optarg !=NULL ) + disaligned =atoi (optarg); +break ;case 'b' : tracelimit =atoi (optarg);break ;case 'B' : tracetype =PREEMPTIRQSOFF ;break ;case 'c' : clocksel =atoi (optarg);break ;
@@ -1318,6 +1344,11 @@ static void process_options (int argc, char *argv[])if (num_threads <1 ) error =1 ; +if (aligned){ +pthread_barrier_init (&globalt_barr,NULL , num_threads); +pthread_barrier_init (&align_barr,NULL , num_threads); + } +if (error)display_help (1 ); }
@@ -1744,6 +1775,7 @@ int main(int argc, char **argv) par->max_cycles = max_cycles; par->stats = stat; par->node = node; + par->tnum = i;switch (setaffinity) {case AFFINITY_UNSPECIFIED : par->cpu =-1 ;break ;case AFFINITY_SPECIFIED : par->cpu = affinity;break ;
Signature · 1 line
1.7.2.5
Quoted 1 line · * Nicholas Mc Guire | 2013-09-09 09:29:48 [+0200]:
Hi !
Hi Nicholas,
Quoted 14 lines
This patch provides and additional -A/--align flag to cyclictest to align
thread wakeup times of all threads as closly defined as possible.When running multiple threads in cyclictest (-S or -t # option) the threads
are launched in an unsynchronized manner. Basically the creation order and time for thread creation determines the start time. For provoking a maximum congestion situation (e.g. cache evictions) and to improve reproducibility or run conditions the start time should be defined distances appart. The well defined distance is implemented as a offset parameter to -A/--align and will offset each threads start time by the parameter * the sequentially assigned thread number (par->tnum), together with the -d0 (distance in the intervals of the individual threads) this alignment option allows to get the thread wakeup times as closely synchronized as possible.
Quoted 5 lines
The method to sync is simply that the thread with
par->tnum== 0 is chosen
to set a globally shared timestamp, and all other threads use this timestamp as their starting time rather than each callingclock_gettime()at startup. To ensure synchronization of the thread startup the setting of the global time is guarded bypthread_barriers.
I would rather fix current behaviour instead introducing yet another option. By using -d0 I assume that all threads wakeup at the same time. According to your patch this does not happen due to the thread creating / starting overhead.
Is there is a reason to keep this "faulty" behavior? If not I would vote to make this what you suggest the default.
Sebastian
Quoted 19 lines · Sebastian Andrzej Siewior
Quoted 1 line · * Nicholas Mc Guire | 2013-09-09 09:29:48 [+0200]:
Hi !
Hi Nicholas,
Quoted 14 lines
This patch provides and additional -A/--align flag to cyclictest to align
thread wakeup times of all threads as closly defined as possible.When running multiple threads in cyclictest (-S or -t # option) the threads
are launched in an unsynchronized manner. Basically the creation order and time for thread creation determines the start time. For provoking a maximum congestion situation (e.g. cache evictions) and to improve reproducibility or run conditions the start time should be defined distances appart. The well defined distance is implemented as a offset parameter to -A/--align and will offset each threads start time by the parameter * the sequentially assigned thread number (par->tnum), together with the -d0 (distance in the intervals of the individual threads) this alignment option allows to get the thread wakeup times as closely synchronized as possible.
Quoted 8 lines
Quoted 5 lines
The method to sync is simply that the thread with
par->tnum== 0 is chosen
to set a globally shared timestamp, and all other threads use this timestamp as their starting time rather than each callingclock_gettime()at startup. To ensure synchronization of the thread startup the setting of the global time is guarded bypthread_barriers.I would rather fix current behaviour instead introducing yet another option. By using -d0 I assume that all threads wakeup at the same time.
Nop they do not -d0 just says that they all use
the same period rather than some offset with multiple threads (e.g. -S)
Quoted 5 lines
According to your patch this does not happen due to the thread creating / starting overhead.
Is there is a reason to keep this "faulty" behavior? If not I would vote to make this what you suggest the default.
its not faulty behavior - its a different case in fact we need both.
same period + "random" start time
same period + synced start time
it makes a difference on some boxes that is significant.
thx!
hofrat
Quoted 12 lines · Nicholas Mc Guire
Quoted 2 lines
I would rather fix current behaviour instead introducing yet another option. By using -d0 I assume that all threads wakeup at the same time.
Nop they do not -d0 just says that they all use
the same period rather than some offset with multiple threads (e.g. -S)Quoted 5 lines
According to your patch this does not happen due to the thread creating / starting overhead.
Is there is a reason to keep this "faulty" behavior? If not I would vote to make this what you suggest the default.
Quoted 7 lines
its not faulty behavior - its a different case in fact we need both.
same period + "random" start time
same period + synced start timeit makes a difference on some boxes that is significant.
So you say with -d0, where d is documented as "distance of thread intervals in us", I should not expect that all threads share the exact same wakeup time (because their distance is 0)?
Quoted 3 lines
thx!
hofrat
Sebastian
Quoted 16 lines · Sebastian Andrzej Siewior
Quoted 15 lines · Nicholas Mc Guire
Quoted 2 lines
I would rather fix current behaviour instead introducing yet another option. By using -d0 I assume that all threads wakeup at the same time.
Nop they do not -d0 just says that they all use
the same period rather than some offset with multiple threads (e.g. -S)Quoted 5 lines
According to your patch this does not happen due to the thread creating / starting overhead.
Is there is a reason to keep this "faulty" behavior? If not I would vote to make this what you suggest the default.its not faulty behavior - its a different case in fact we need both.
Quoted 9 lines
Quoted 4 lines
same period + "random" start time
same period + synced start timeit makes a difference on some boxes that is significant.
So you say with -d0, where d is documented as "distance of thread intervals in us", I should not expect that all threads share the exact same wakeup time (because their distance is 0)?
no why should you - you might want
-i 100 -d 50 -A 20
T1 0 100 200 300
T2 20 170
T3 40 240
T4 60 310
Which is quite a lot different than just stating
that i_1=100 i_2=150, etc... but initial distance "undefined" or random
so in the sense of schedulability tests these are really two seperate parameters and thus should be configurable independently.
That the current documentation might give the impression that they are in sync is a bug in the documentation not in the -d implementation.
hofrat
Quoted 11 lines · Nicholas Mc Guire
no why should you - you might want
-i 100 -d 50 -A 20
T1 0 100 200 300
T2 20 170
T3 40 240
T4 60 310Which is quite a lot different than just stating
Quoted 7 lines
that i_1=100 i_2=150, etc... but initial distance "undefined" or random
so in the sense of schedulability tests these are really two seperate parameters and thus should be configurable independently.
That the current documentation might give the impression that they are in sync is a bug in the documentation not in the -d implementation.
Now, I get it. Okay, so this makes sense.
Quoted 3 lines
hofrat
Sebastian