Allow osnoise tracer to run without workload
Subsystems (3)
cover Allow osnoise tracer to run without workload
The osnoise tracer is not only a tracer and a set of tracepoints, but also a workload dispatcher.
In preparation for having other workloads, e.g., in user-space, add an option to avoid dispatching the workload.
By not dispatching the workload, the osnoise: tracepoints become generic events to measure the execution time of *any* task on Linux.
Daniel Bristot de Oliveira (4):
tracing/osnoise: Fix duration type tracing/osnoise: Add osnoise/options file tracing/osnoise: Add OSNOISE_WORKLOAD option Documentation/osnoise: Add osnoise/options documentation
Documentation/trace/osnoise-tracer.rst | 12 ++ kernel/trace/trace_osnoise.c | 200 ++++++++++++++++++++++++- 2 files changed, 205 insertions(+), 7 deletions(-)
Signature · 1 line
2.32.0
PATCH 1/4 tracing/osnoise: Fix duration type
The duration type is a 64 long value, not an int. This was causing some long noise to report wrong values.
Change the duration to a 64 bits value.
--- kernel/trace/trace_osnoise.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/trace/trace_osnoise.c +3 -3
@@ -917,7 +917,7 @@ void osnoise_trace_irq_entry(int id)void osnoise_trace_irq_exit (int id,const char *desc) {struct osnoise_variables *osn_var =this_cpu_osn_var (); -int duration; +s64 duration;if (!osn_var->sampling)return ;
@@ -1048,7 +1048,7 @@ static void trace_softirq_entry_callback(void *data, unsigned int vec_nr)static void trace_softirq_exit_callback (void *data,unsigned int vec_nr) {struct osnoise_variables *osn_var =this_cpu_osn_var (); -int duration; +s64 duration;if (!osn_var->sampling)return ;
@@ -1144,7 +1144,7 @@ thread_entry(struct osnoise_variables *osn_var, struct task_struct *t)static void thread_exit (struct osnoise_variables *osn_var,struct task_struct *t) { -int duration; +s64 duration;if (!osn_var->sampling)return ;
Signature · 1 line
2.32.0
Re: [PATCH 1/4] tracing/osnoise: Fix duration type
Quoted 4 lines · On Thu, 17 Nov 2022 14:46:17 +0100 Daniel Bristot de Oliveira…
The duration type is a 64 long value, not an int. This was causing some long noise to report wrong values.
Change the duration to a 64 bits value.
This looks good to me.
Thank you!
Quoted 45 lines
Fixes bce29ac9ce0b ("trace: Add osnoise tracer")Cc Daniel Bristot de Oliveira <[email protected]>Cc Steven Rostedt <[email protected]>Cc Masami Hiramatsu <[email protected]>Cc Jonathan Corbet <[email protected]>Signed-off-by Daniel Bristot de Oliveira <[email protected]>--- kernel/trace/trace_osnoise.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)kernel/trace/trace_osnoise.c +3
-3@@ -917,7 +917,7 @@ void osnoise_trace_irq_entry(int id)void osnoise_trace_irq_exit (int id,const char *desc) {struct osnoise_variables *osn_var =this_cpu_osn_var (); -int duration; +s64 duration;if (!osn_var->sampling)return ;@@ -1048,7 +1048,7 @@ static void trace_softirq_entry_callback(void *data, unsigned int vec_nr)static void trace_softirq_exit_callback (void *data,unsigned int vec_nr) {struct osnoise_variables *osn_var =this_cpu_osn_var (); -int duration; +s64 duration;if (!osn_var->sampling)return ;@@ -1144,7 +1144,7 @@ thread_entry(struct osnoise_variables *osn_var, struct task_struct *t)static void thread_exit (struct osnoise_variables *osn_var,struct task_struct *t) { -int duration; +s64 duration;if (!osn_var->sampling)return ;Signature · 1 line
2.32.0
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Re: [PATCH 1/4] tracing/osnoise: Fix duration type
Quoted 10 lines · Masami Hiramatsu (Google)
Quoted 4 lines · On Thu, 17 Nov 2022 14:46:17 +0100 Daniel Bristot de Oliveira…
The duration type is a 64 long value, not an int. This was causing some long noise to report wrong values.
Change the duration to a 64 bits value.
This looks good to me.
Thanks, Masami!
Btw, this patch can be applied independently of the other as a fix. It is only here because I noticed the problem while adding the features in the next patches.
-- Daniel
PATCH 2/4 tracing/osnoise: Add osnoise/options file
In reply to Daniel Bristot de OliveiraAdd the tracing/osnoise/options file to control osnoise/timerlat tracer features. It is a single file to contain multiple features, similar to
the sched/features file.
Reading the file displays a list of options. Writing the OPTION_NAME enables it, writing NO_OPTION_NAME disables it.
The DEAFULTS is a particular option that resets the options to the default ones.
It uses a bitmask to keep track of the status of the option. When needed, we can add a list of static keys, but for now it does not justify the memory increase.
--- kernel/trace/trace_osnoise.c | 170 +++++++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+)
kernel/trace/trace_osnoise.c +170 -0
@@ -48,6 +48,19 @@#define DEFAULT_TIMERLAT_PERIOD 1000/* 1ms */ #define DEFAULT_TIMERLAT_PRIO 95/* FIFO 95 */ +/* +* osnoise/options entries. +*/ +enum osnoise_options_index { +OSN_DEFAULTS =0 , +OSN_MAX +}; + +static const char *const osnoise_options_str[OSN_MAX ] = {"DEFAULTS" }; + +#define OSN_DEFAULT_OPTIONS 0 +unsigned long osnoise_options =OSN_DEFAULT_OPTIONS ; +/* * trace_array of the enabled osnoise/timerlat instances. */
@@ -1860,6 +1873,150 @@ static void osnoise_init_hotplug_support(void) }#endif /* CONFIG_HOTPLUG_CPU */ +/* +* seq file functions for the osnoise/options file. +*/ +static void *s_options_start (struct seq_file *s,loff_t *pos) +{ +int option = *pos; + +mutex_lock (&interface_lock); + +if (option >=OSN_MAX ) +return NULL ; + +return pos; +} + +static void *s_options_next (struct seq_file *s,void *v,loff_t *pos) +{ +int option = ++(*pos); + +if (option >=OSN_MAX ) +return NULL ; + +return pos; +} + +static int s_options_show (struct seq_file *s,void *v) +{ +loff_t *pos = v; +int option = *pos; + +if (option ==OSN_DEFAULTS ) { +if (osnoise_options ==OSN_DEFAULT_OPTIONS ) +seq_printf (s,"%s" , osnoise_options_str[option]); +else +seq_printf (s,"NO_%s" , osnoise_options_str[option]); + goto out; + } + +if (test_bit (option, &osnoise_options)) +seq_printf (s,"%s" , osnoise_options_str[option]); +else +seq_printf (s,"NO_%s" , osnoise_options_str[option]); + +out: +if (option !=OSN_MAX ) +seq_puts (s," " ); + +return 0 ; +} + +static void s_options_stop (struct seq_file *s,void *v) +{ +seq_puts (s,"\n" ); +mutex_unlock (&interface_lock); +} + +static const struct seq_operations osnoise_options_seq_ops = { + .start = s_options_start, + .next = s_options_next, + .show = s_options_show, + .stop = s_options_stop +}; + +static int osnoise_options_open (struct inode *inode,struct file *file) +{ +return seq_open (file, &osnoise_options_seq_ops); +}; + +/** +* osnoise_options_write - Write function for "options" entry +* @filp: The active open file structure +* @ubuf: The user buffer that contains the value to write +* @cnt: The maximum number of bytes to write to "file" +* @ppos: The current position in @file +* +* Writing the option name sets the option, writing the "NO_" +* prefix in front of the option name disables it. +* +* Writing "DEFAULTS" resets the option values to the default ones. +*/ +static ssize_t osnoise_options_write (struct file *filp,const char __user *ubuf, +size_t cnt,loff_t *ppos) +{ +int running, option, enable, retval; +char buf[256 ], *option_str; + +if (cnt >=256 ) +return -EINVAL ; + +if (copy_from_user (buf, ubuf, cnt)) +return -EFAULT ; + + buf[cnt] =0 ; + +if (strncmp (buf,"NO_" ,3 )) { + option_str =strstrip (buf); + enable = true; + }else { + option_str =strstrip (&buf[3 ]); + enable = false; + } + + option =match_string (osnoise_options_str,OSN_MAX , option_str); +if (option <0 ) +return -EINVAL ; + +/* +* trace_types_lock is taken to avoid concurrency on start/stop. +*/ +mutex_lock (&trace_types_lock); + running =osnoise_has_registered_instances (); +if (running) +stop_per_cpu_kthreads (); + +mutex_lock (&interface_lock); +/* +* avoid CPU hotplug operations that might read options. +*/ +cpus_read_lock (); + + retval = cnt; + +if (enable) { +if (option ==OSN_DEFAULTS ) + osnoise_options =OSN_DEFAULT_OPTIONS ; +else +set_bit (option, &osnoise_options); + }else { +if (option ==OSN_DEFAULTS ) + retval = -EINVAL ; +else +clear_bit (option, &osnoise_options); + } + +cpus_read_unlock (); +mutex_unlock (&interface_lock); + +if (running) +start_per_cpu_kthreads (); +mutex_unlock (&trace_types_lock); + +return retval; +} +/* * osnoise_cpus_read - Read function for reading the "cpus" file * @filp: The active open file structure
@@ -2042,6 +2199,14 @@ static const struct file_operations cpus_fops = { .llseek = generic_file_llseek, }; +static const struct file_operations osnoise_options_fops = { + .open = osnoise_options_open, + .read = seq_read, + .llseek = seq_lseek, + .release = seq_release, + .write = osnoise_options_write +}; +#ifdef CONFIG_TIMERLAT_TRACER #ifdef CONFIG_STACKTRACE static int init_timerlat_stack_tracefs (struct dentry *top_dir)
@@ -2128,6 +2293,11 @@ static int init_tracefs(void)if (!tmp) goto err; + tmp =trace_create_file ("options" ,TRACE_MODE_WRITE , top_dir,NULL , + &osnoise_options_fops); +if (!tmp) + goto err; + ret =init_timerlat_tracefs (top_dir);if (ret) goto err;
Signature · 1 line
2.32.0
PATCH 3/4 tracing/osnoise: Add OSNOISE_WORKLOAD option
In reply to Daniel Bristot de OliveiraThe osnoise tracer is not only a tracer, and a set of tracepoints, but also a workload dispatcher.
In preparation for having other workloads, e.g., in user-space, add an option to avoid dispatching the workload.
By not dispatching the workload, the osnoise: tracepoints become generic events to measure the execution time of *any* task on Linux.
For example:
# cd /sys/kernel/tracing/
# cat osnoise/options
DEFAULTS OSNOISE_WORKLOAD
# echo NO_OSNOISE_WORKLOAD > osnoise/options
# cat osnoise/options
NO_DEFAULTS NO_OSNOISE_WORKLOAD
# echo osnoise > set_event
# echo osnoise > current_tracer
# tail -8 trace
make-94722 [002] d..3. 1371.794507: thread_noise: make:94722 start 1371.794302286 duration 200897 ns
sh-121042 [020] d..3. 1371.794534: thread_noise: sh:121042 start 1371.781610976 duration 8943683 ns
make-121097 [005] d..3. 1371.794542: thread_noise: make:121097 start 1371.794481522 duration 60444 ns
<...>-40 [005] d..3. 1371.794550: thread_noise: migration/5:40 start 1371.794542256 duration 7154 ns
<idle>-0 [018] dNh2. 1371.794554: irq_noise: reschedule:253 start 1371.794553547 duration 40 ns
<idle>-0 [018] dNh2. 1371.794561: irq_noise: local_timer:236 start 1371.794556222 duration 4890 ns
<idle>-0 [018] .Ns2. 1371.794563: softirq_noise: SCHED:7 start 1371.794561803 duration 992 ns
<idle>-0 [018] d..3. 1371.794566: thread_noise: swapper/18:0 start 1371.781368110 duration 13191798 nsIn preparation for the rtla exec_time tracer/tool and rtla osnoise --user option.
--- kernel/trace/trace_osnoise.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-)
kernel/trace/trace_osnoise.c +22 -6
@@ -53,12 +53,13 @@*/ enum osnoise_options_index {OSN_DEFAULTS =0 , +OSN_WORKLOAD ,OSN_MAX }; -static const char *const osnoise_options_str[OSN_MAX ] = {"DEFAULTS" }; +static const char *const osnoise_options_str[OSN_MAX ] = {"DEFAULTS" ,"OSNOISE_WORKLOAD" }; -#define OSN_DEFAULT_OPTIONS 0 +#define OSN_DEFAULT_OPTIONS 0x2unsigned long osnoise_options =OSN_DEFAULT_OPTIONS ;/*
@@ -1186,11 +1187,12 @@ trace_sched_switch_callback(void *data, bool preempt, unsigned int prev_state) {struct osnoise_variables *osn_var =this_cpu_osn_var (); +int workload =test_bit (OSN_WORKLOAD , &osnoise_options); -if (p->pid != osn_var->pid) +if ((p->pid != osn_var->pid) || !workload)thread_exit (osn_var, p); -if (n->pid != osn_var->pid) +if ((n->pid != osn_var->pid) || !workload)thread_entry (osn_var, n); }
@@ -1723,9 +1725,16 @@ static void stop_kthread(unsigned int cpu)struct task_struct *kthread; kthread =per_cpu (per_cpu_osnoise_var, cpu).kthread; -if (kthread) +if (kthread) {kthread_stop (kthread); -per_cpu (per_cpu_osnoise_var, cpu).kthread =NULL ; +per_cpu (per_cpu_osnoise_var, cpu).kthread =NULL ; + }else { +if (!test_bit (OSN_WORKLOAD , &osnoise_options)) { +per_cpu (per_cpu_osnoise_var, cpu).sampling = false; +barrier (); +return ; + } + } }/*
@@ -1759,6 +1768,13 @@ static int start_kthread(unsigned int cpu)snprintf (comm,24 ,"timerlat/%d" , cpu); main = timerlat_main; }else { +/* if no workload, just return */ +if (!test_bit (OSN_WORKLOAD , &osnoise_options)) { +per_cpu (per_cpu_osnoise_var, cpu).sampling = true; +barrier (); +return 0 ; + } +snprintf (comm,24 ,"osnoise/%d" , cpu); }
Signature · 1 line
2.32.0
PATCH 4/4 Documentation/osnoise: Add osnoise/options documentation
In reply to Daniel Bristot de OliveiraAdd the documentation about the osnoise/options file, along with an explanation about the OSNOISE_WORKLOAD option.
--- Documentation/trace/osnoise-tracer.rst | 12 ++++++++++++ 1 file changed, 12 insertions(+)
Documentation/trace/osnoise-tracer.rst +12 -0
@@ -109,6 +109,11 @@ The tracer has a set of options inside the osnoise directory, they are: - tracing_threshold: the minimum delta between two time() reads to be considered as noise, in us. When set to 0, the default value will be used, which is currently 5 us. + - osnoise/options: a set of on/off options that can be enabled by + writing the option name to the file or disabled by writing the option + name preceded with the 'NO_' prefix. For example, writing + NO_OSNOISE_WORKLOAD disables the OSNOISE_WORKLOAD option. The + special DEAFAULTS option resets all options to the default value. Additional Tracing ------------------
@@ -150,3 +155,10 @@ tracepoints is smaller than eight us reported in the sample_threshold. The reason roots in the overhead of the entry and exit code that happens before and after any interference execution. This justifies the dual approach: measuring thread and tracing. + +Running osnoise tracer without workload +--------------------------------------- + +By enabling the osnoise tracer with the NO_OSNOISE_WORKLOAD option set, +the osnoise: tracepoints serve to measure the execution time of +any type of Linux task, free from the interference of other tasks.
Signature · 1 line
2.32.0