rtla/timerlat: Support actions on threshold and on end
Subsystems (3)
Tomas Glozar
maintainerv2 cover rtla/timerlat: Support actions on threshold and on end
This series adds a feature that allows to user to specify certain kinds of "actions" to be executed at one of two places in the rtla
measurement: when tracing is stopped on latency threshold, and at the end of tracing.
Two new options are added: --on-threshold, and --on-end, taking the action as an argument. For example:
$ rtla timerlat hist -T 10 \ --on-threshold shell,command="echo Threshold" --on-end shell,command="echo Tracing stopped"
will print "Threshold" if a thread latency higher than 10 microseconds is reached, and "Tracing stopped" always at the end.
The list of possible actions is extensible and is covered in the commit messages. Later, a documentation patch series will be sent with clear explanation of every action and its syntax.
Notably, a special action "continue" resumes tracing. For example:
$ rtla timerlat hist -T 100 \ --on-threshold shell,command="echo Threshold" \ --on-threshold continue -d 10s
will print "Threshold" as many times as tracing is stopped after thread latency reaches 100us.
The feature was inspired by a case where collecting perf data on rtla latency overflow was required, which can be done by sending a signal to the perf process.
Example of this with Intel PT aux buffer:
$ perf record --cpu 0 -e intel_pt// -S -- rtla timerlat top -q -T 100 \ -c 0 --on-threshold signal,pid=parent,num=12 --on-theshold continue
In general, the feature is aiming to allow integration with external tooling. To implement even more flexibility, passing context to the shell through environmental variables, or even an entire scripting language with access to the rtla internals can be implemented if needed.
Changes since v1:
- Dropped short options on suggestion from Arnaldo.
- Added documentation.
Tomas Glozar (9):
rtla/timerlat: Introduce enum timerlat_tracing_mode rtla/timerlat: Add action on threshold feature rtla/timerlat_bpf: Allow resuming tracing rtla/timerlat: Add continue action rtla/timerlat: Add action on end feature rtla/tests: Check rtla output with grep rtla/tests: Add tests for actions rtla/tests: Limit duration to maximum of 10s Documentation/rtla: Add actions feature
.../tools/rtla/common_timerlat_options.rst | 64 +++++ tools/tracing/rtla/src/Build | 1 + tools/tracing/rtla/src/actions.c | 260 ++++++++++++++++++ tools/tracing/rtla/src/actions.h | 52 ++++ tools/tracing/rtla/src/timerlat.bpf.c | 13 +- tools/tracing/rtla/src/timerlat.c | 24 +- tools/tracing/rtla/src/timerlat.h | 24 +- tools/tracing/rtla/src/timerlat_bpf.c | 13 + tools/tracing/rtla/src/timerlat_bpf.h | 3 + tools/tracing/rtla/src/timerlat_hist.c | 140 +++++++--- tools/tracing/rtla/src/timerlat_top.c | 165 +++++++---- tools/tracing/rtla/tests/engine.sh | 21 +- tools/tracing/rtla/tests/hwnoise.t | 8 +- tools/tracing/rtla/tests/osnoise.t | 4 +- tools/tracing/rtla/tests/timerlat.t | 36 ++- 15 files changed, 712 insertions(+), 116 deletions(-) create mode 100644 tools/tracing/rtla/src/actions.c create mode 100644 tools/tracing/rtla/src/actions.h
Signature · 1 line
2.49.0
Tomas Glozar
maintainerPATCH v2 1/9 rtla/timerlat: Introduce enum timerlat_tracing_mode
After the introduction of BPF-based sample collection, rtla-timerlat effectively runs in one of three modes:
- Pure BPF mode, with tracefs only being used to set up the timerlat tracer. Sample processing and stop on threshold are handled by BPF.
- tracefs mode. BPF is unsupported or kernel is lacking the necessary trace event (osnoise:timerlat_sample). Stop on theshold is handled by timerlat tracer stopping tracing in all instances.
- BPF/tracefs mixed mode - BPF is used for sample collection for top or histogram, tracefs is used for trace output and/or auto-analysis. Stop on threshold is handled both through BPF program, which stops sample collection for top/histogram and wakes up rtla, and by timerlat tracer, which stops tracing for trace output/auto-analysis instances.
Add enum timerlat_tracing_mode, with three values:
- TRACING_MODE_BPF
- TRACING_MODE_TRACEFS
- TRACING_MODE_MIXED
Those represent the modes described above. A field of this type is added to struct timerlat_params, named "mode", replacing the no_bpf variable. params->mode is set in timerlat_{top,hist}_parse_args to TRACING_MODE_BPF or TRACING_MODE_MIXED based on whether trace output and/or auto-analysis is requested. timerlat_{top,hist}_main then checks if BPF is not unavailable or disabled, in that case, it sets params->mode to TRACING_MODE_TRACEFS.
A condition is added to timerlat_apply_config that skips setting timerlat tracer thresholds if params->mode is TRACING_MODE_BPF (those are unnecessary, since they only turn off tracing, which is already turned off in that case, since BPF is used to collect samples).
--- tools/tracing/rtla/src/timerlat.c | 24 +++++++---- tools/tracing/rtla/src/timerlat.h | 18 ++++++++ tools/tracing/rtla/src/timerlat_hist.c | 51 +++++++++++++---------- tools/tracing/rtla/src/timerlat_top.c | 57 +++++++++++++++----------- 4 files changed, 97 insertions(+), 53 deletions(-)
tools/tracing/rtla/src/timerlat.c +15 -9
@@ -40,16 +40,22 @@ timerlat_apply_config(struct osnoise_tool *tool, struct timerlat_params *params)CPU_SET (i, ¶ms->monitored_cpus); } - retval =osnoise_set_stop_us (tool->context, params->stop_us); -if (retval) { -err_msg ("Failed to set stop us\n" ); - goto out_err; - } +if (params->mode !=TRACING_MODE_BPF ) { +/* +* In tracefs and mixed mode, timerlat tracer handles stopping +* on threshold +*/ + retval =osnoise_set_stop_us (tool->context, params->stop_us); +if (retval) { +err_msg ("Failed to set stop us\n" ); + goto out_err; + } - retval =osnoise_set_stop_total_us (tool->context, params->stop_total_us); -if (retval) { -err_msg ("Failed to set stop total us\n" ); - goto out_err; + retval =osnoise_set_stop_total_us (tool->context, params->stop_total_us); +if (retval) { +err_msg ("Failed to set stop total us\n" ); + goto out_err; + } }
tools/tracing/rtla/src/timerlat.h +18 -0
@@ -1,6 +1,23 @@// SPDX-License-Identifier: GPL-2.0 #include "osnoise.h" +/* +* Define timerlat tracing mode. +* +* There are three tracing modes: +* - tracefs-only, used when BPF is unavailable. +* - BPF-only, used when BPF is available and neither trace saving nor +* auto-analysis are enabled. +* - mixed mode, used when BPF is available and either trace saving or +* auto-analysis is enabled (which rely on sample collection through +* tracefs). +*/ +enum timerlat_tracing_mode { +TRACING_MODE_BPF , +TRACING_MODE_TRACEFS , +TRACING_MODE_MIXED , +}; +struct timerlat_params {/* Common params */ char *cpus;
@@ -30,6 +47,7 @@ struct timerlat_params {cpu_set_t hk_cpu_set;struct sched_attr sched_param;struct trace_events *events; +enum timerlat_tracing_mode mode; union { struct { /* top only */
tools/tracing/rtla/src/timerlat_hist.c +30 -21
116 lines not on this page · raw message
tools/tracing/rtla/src/timerlat_top.c +34 -23
118 lines not on this page · raw message
Signature · 1 line
2.49.0
Tomas Glozar
maintainerPATCH v2 2/9 rtla/timerlat: Add action on threshold feature
In reply to Tomas GlozarExtend the functionality provided by the -t/--trace option, which triggers saving the contents of a tracefs buffer after tracing is stopped, to support implementing arbitrary actions.
A new option, --on-threshold, is added, taking an argument that further specifies the action. Actions added in this patch are:
- trace[,file=<filename>]: Saves tracefs buffer, optionally taking a filename.
- signal,num=<sig>,pid=<pid>: Sends signal to process. "parent" might be specified instead of number to send signal to parent process.
- shell,command=<command>: Execute shell command.
Multiple actions may be specified and will be executed in order, including multiple actions of the same type. Trace output requested via -t and -a now adds a trace action to the end of the list.
If an action fails, the following actions are not executed. For example, this command:
$ rtla timerlat -T 20 --on-threshold trace \ --on-threshold shell,command="grep ipi_send timerlat_trace.txt" \ --on-threshold signal,num=2,pid=parent
will send signal 2 (SIGINT) to parent process, but only if saved trace contains the text "ipi_send".
This way, the feature can be used for flexible reactions on latency spikes, and allows combining rtla with other tooling like perf.
--- tools/tracing/rtla/src/Build | 1 + tools/tracing/rtla/src/actions.c | 235 +++++++++++++++++++++++++ tools/tracing/rtla/src/actions.h | 49 ++++++ tools/tracing/rtla/src/timerlat.h | 3 +- tools/tracing/rtla/src/timerlat_hist.c | 37 ++-- tools/tracing/rtla/src/timerlat_top.c | 38 ++-- 6 files changed, 341 insertions(+), 22 deletions(-) create mode 100644 tools/tracing/rtla/src/actions.c create mode 100644 tools/tracing/rtla/src/actions.h
tools/tracing/rtla/src/Build +1 -0
@@ -1,5 +1,6 @@ rtla-y += trace.o rtla-y += utils.o +rtla-y += actions.o rtla-y += osnoise.o rtla-y += osnoise_top.o rtla-y += osnoise_hist.o
tools/tracing/rtla/src/actions.c new file +235 -0
235 lines not on this page · raw message
tools/tracing/rtla/src/actions.h new file +49 -0
49 lines not on this page · raw message
tools/tracing/rtla/src/timerlat.h +2 -1
@@ -1,4 +1,5 @@// SPDX-License-Identifier: GPL-2.0 +#include "actions.h" #include "osnoise.h" /*
@@ -22,7 +23,6 @@ struct timerlat_params {/* Common params */ char *cpus;cpu_set_t monitored_cpus; -char *trace_output;char *cgroup_name;unsigned long long runtime;long long stop_us;
@@ -48,6 +48,7 @@ struct timerlat_params {struct sched_attr sched_param;struct trace_events *events;enum timerlat_tracing_mode mode; +struct actions actions; union { struct { /* top only */
tools/tracing/rtla/src/timerlat_hist.c +27 -10
@@ -757,6 +757,7 @@ static void timerlat_hist_usage(char *usage)" --warm-up s: let the workload run for s seconds before collecting data" ," --trace-buffer-size kB: set the per-cpu trace buffer size in kB" ," --deepest-idle-state n: only go down to idle state n on cpus used by timerlat to reduce exit from idle latency" , +" --on-threshold <action>: define action to be executed at latency threshold, multiple are allowed" , NULL, };
@@ -786,11 +787,14 @@ static struct timerlat_paramsint auto_thresh;int retval;int c; +char *trace_output =NULL ; params =calloc (1 ,sizeof (*params));if (!params)exit (1 ); +actions_init (¶ms->actions); +/* disabled by default */ params->dma_latency =-1 ;
@@ -841,6 +845,7 @@ static struct timerlat_params {"warm-up", required_argument, 0, '\2'}, {"trace-buffer-size", required_argument, 0, '\3'}, {"deepest-idle-state", required_argument, 0, '\4'}, + {"on-threshold", required_argument, 0, '\5'}, {0, 0, 0, 0} };
@@ -866,7 +871,7 @@ static struct timerlat_params params->print_stack = auto_thresh;/* set trace */ - params->trace_output ="timerlat_trace.txt" ; + trace_output ="timerlat_trace.txt" ;break ;case 'c' :
@@ -956,13 +961,13 @@ static struct timerlat_paramscase 't' :if (optarg) {if (optarg[0 ] =='=' ) - params->trace_output = &optarg[1 ]; + trace_output = &optarg[1 ];else - params->trace_output = &optarg[0 ]; + trace_output = &optarg[0 ]; }else if (optind < argc && argv[optind][0 ] !='-' ) - params->trace_output = argv[optind]; + trace_output = argv[optind];else - params->trace_output ="timerlat_trace.txt" ; + trace_output ="timerlat_trace.txt" ;break ;case 'u' : params->user_workload =1 ;
@@ -1032,11 +1037,21 @@ static struct timerlat_paramscase '\4' : params->deepest_idle_state =get_llong_from_str (optarg);break ; +case '\5' : + retval =actions_parse (¶ms->actions, optarg); +if (retval) { +err_msg ("Invalid action %s\n" , optarg); +exit (EXIT_FAILURE ); + } +break ;default :timerlat_hist_usage ("Invalid option" ); } } +if (trace_output) +actions_add_trace_output (¶ms->actions, trace_output); +if (geteuid ()) {err_msg ("rtla needs root permission\n" );exit (EXIT_FAILURE );
@@ -1061,7 +1076,8 @@ static struct timerlat_params* If auto-analysis or trace output is enabled, switch from BPF mode to * mixed mode */ -if (params->mode ==TRACING_MODE_BPF && params->trace_output && !params->no_aa) +if (params->mode ==TRACING_MODE_BPF && + (params->actions.present[ACTION_TRACE_OUTPUT ] || !params->no_aa)) params->mode =TRACING_MODE_MIXED ;return params;
@@ -1254,12 +1270,13 @@ int timerlat_hist_main(int argc, char *argv[]) } } -if (params->trace_output) { +if (params->actions.present[ACTION_TRACE_OUTPUT ]) { record =osnoise_init_trace_tool ("timerlat" );if (!record) {err_msg ("Failed to enable the trace instance\n" ); goto out_free; } + params->actions.trace_output_inst = record->trace.inst;if (params->events) { retval =trace_events_enable (&record->trace, params->events);
@@ -1325,7 +1342,7 @@ int timerlat_hist_main(int argc, char *argv[])* tracing while enabling other instances. The trace instance is the * one with most valuable information. */ -if (params->trace_output) +if (params->actions.present[ACTION_TRACE_OUTPUT ])trace_instance_start (&record->trace);if (!params->no_aa)trace_instance_start (&aa->trace);
@@ -1395,8 +1412,7 @@ int timerlat_hist_main(int argc, char *argv[])if (!params->no_aa)timerlat_auto_analysis (params->stop_us, params->stop_total_us); -save_trace_to_file (record ? record->trace.inst :NULL , - params->trace_output); +actions_perform (¶ms->actions); return_value =FAILED ; }
@@ -1418,6 +1434,7 @@ int timerlat_hist_main(int argc, char *argv[])osnoise_destroy_tool (aa);osnoise_destroy_tool (record);osnoise_destroy_tool (tool); +actions_destroy (¶ms->actions);if (params->mode !=TRACING_MODE_TRACEFS )timerlat_bpf_destroy ();free (params);
tools/tracing/rtla/src/timerlat_top.c +27 -11
@@ -516,6 +516,7 @@ static void timerlat_top_usage(char *usage)" --warm-up s: let the workload run for s seconds before collecting data" ," --trace-buffer-size kB: set the per-cpu trace buffer size in kB" ," --deepest-idle-state n: only go down to idle state n on cpus used by timerlat to reduce exit from idle latency" , +" --on-threshold <action>: define action to be executed at latency threshold, multiple are allowed" , NULL, };
@@ -545,11 +546,14 @@ static struct timerlat_paramslong long auto_thresh;int retval;int c; +char *trace_output =NULL ; params =calloc (1 ,sizeof (*params));if (!params)exit (1 ); +actions_init (¶ms->actions); +/* disabled by default */ params->dma_latency =-1 ;
@@ -592,6 +596,7 @@ static struct timerlat_params {"warm-up", required_argument, 0, '6'}, {"trace-buffer-size", required_argument, 0, '7'}, {"deepest-idle-state", required_argument, 0, '8'}, + {"on-threshold", required_argument, 0, '9'}, {0, 0, 0, 0} };
@@ -617,7 +622,7 @@ static struct timerlat_params params->print_stack = auto_thresh;/* set trace */ - params->trace_output ="timerlat_trace.txt" ; + trace_output ="timerlat_trace.txt" ;break ;case '5' :/* it is here because it is similar to -a */
@@ -712,14 +717,13 @@ static struct timerlat_paramscase 't' :if (optarg) {if (optarg[0 ] =='=' ) - params->trace_output = &optarg[1 ]; + trace_output = &optarg[1 ];else - params->trace_output = &optarg[0 ]; + trace_output = &optarg[0 ]; }else if (optind < argc && argv[optind][0 ] !='-' ) - params->trace_output = argv[optind]; + trace_output = argv[optind];else - params->trace_output ="timerlat_trace.txt" ; - + trace_output ="timerlat_trace.txt" ;break ;case 'u' : params->user_workload = true;
@@ -771,11 +775,21 @@ static struct timerlat_paramscase '8' : params->deepest_idle_state =get_llong_from_str (optarg);break ; +case '9' : + retval =actions_parse (¶ms->actions, optarg); +if (retval) { +err_msg ("Invalid action %s\n" , optarg); +exit (EXIT_FAILURE ); + } +break ;default :timerlat_top_usage ("Invalid option" ); } } +if (trace_output) +actions_add_trace_output (¶ms->actions, trace_output); +if (geteuid ()) {err_msg ("rtla needs root permission\n" );exit (EXIT_FAILURE );
@@ -797,7 +811,8 @@ static struct timerlat_params* If auto-analysis or trace output is enabled, switch from BPF mode to * mixed mode */ -if (params->mode ==TRACING_MODE_BPF && params->trace_output && !params->no_aa) +if (params->mode ==TRACING_MODE_BPF && + (params->actions.present[ACTION_TRACE_OUTPUT ] || !params->no_aa)) params->mode =TRACING_MODE_MIXED ;return params;
@@ -1099,12 +1114,13 @@ int timerlat_top_main(int argc, char *argv[]) } } -if (params->trace_output) { +if (params->actions.present[ACTION_TRACE_OUTPUT ]) { record =osnoise_init_trace_tool ("timerlat" );if (!record) {err_msg ("Failed to enable the trace instance\n" ); goto out_free; } + params->actions.trace_output_inst = record->trace.inst;if (params->events) { retval =trace_events_enable (&record->trace, params->events);
@@ -1171,7 +1187,7 @@ int timerlat_top_main(int argc, char *argv[])* tracing while enabling other instances. The trace instance is the * one with most valuable information. */ -if (params->trace_output) +if (params->actions.present[ACTION_TRACE_OUTPUT ])trace_instance_start (&record->trace);if (!params->no_aa)trace_instance_start (&aa->trace);
@@ -1214,8 +1230,7 @@ int timerlat_top_main(int argc, char *argv[])if (!params->no_aa)timerlat_auto_analysis (params->stop_us, params->stop_total_us); -save_trace_to_file (record ? record->trace.inst :NULL , - params->trace_output); +actions_perform (¶ms->actions); return_value =FAILED ; }else if (params->aa_only) {/*
@@ -1248,6 +1263,7 @@ int timerlat_top_main(int argc, char *argv[])osnoise_destroy_tool (aa);osnoise_destroy_tool (record);osnoise_destroy_tool (top); +actions_destroy (¶ms->actions);if (params->mode !=TRACING_MODE_TRACEFS )timerlat_bpf_destroy ();free (params);
Signature · 1 line
2.49.0
Tomas Glozar
maintainerPATCH v2 3/9 rtla/timerlat_bpf: Allow resuming tracing
In reply to Tomas GlozarCurrently, rtla-timerlat BPF program uses a global variable stored in a .bss section to store whether tracing has been stopped.
Move the information to a separate map, so that it is easily writable from userspace, and add a function that clears the value, resuming tracing after it has been stopped.
--- tools/tracing/rtla/src/timerlat.bpf.c | 13 +++++++++---- tools/tracing/rtla/src/timerlat_bpf.c | 13 +++++++++++++ tools/tracing/rtla/src/timerlat_bpf.h | 3 +++ 3 files changed, 25 insertions(+), 4 deletions(-)
tools/tracing/rtla/src/timerlat.bpf.c +9 -4
@@ -28,6 +28,13 @@ struct { __type(value, unsigned long long); } summary_irq SEC(".maps"), summary_thread SEC(".maps"), summary_user SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, unsigned int); + __type(value, unsigned long long); +} stop_tracing SEC(".maps"); +struct { __uint(type, BPF_MAP_TYPE_RINGBUF); __uint(max_entries, 1);
@@ -41,8 +48,6 @@ const volatile int irq_threshold;const volatile int thread_threshold;const volatile bool aa_only; -int stop_tracing; -nosubprog unsigned long long map_get (void *map,unsigned int key) {
@@ -109,7 +114,7 @@ nosubprog void set_stop_tracing(void)int value =0 ;/* Suppress further sample processing */ - stop_tracing =1 ; +map_set (&stop_tracing,0 ,1 );/* Signal to userspace */ bpf_ringbuf_output (&signal_stop_tracing, &value,sizeof (value),0 );
@@ -121,7 +126,7 @@ int handle_timerlat_sample(struct trace_event_raw_timerlat_sample *tp_args)unsigned long long latency, latency_us;int bucket; -if (stop_tracing) +if (map_get (&stop_tracing,0 ))return 0 ; latency = tp_args->timer_latency / output_divisor;
tools/tracing/rtla/src/timerlat_bpf.c +13 -0
@@ -106,6 +106,19 @@ int timerlat_bpf_wait(int timeout)return retval; } +/* +* timerlat_bpf_restart_tracing - restart stopped tracing +*/ +int timerlat_bpf_restart_tracing (void ) +{ +unsigned int key =0 ; +unsigned long long value =0 ; + +return bpf_map__update_elem (bpf->maps.stop_tracing, + &key,sizeof (key), + &value,sizeof (value),BPF_ANY ); +} + static int get_value(struct bpf_map *map_irq, struct bpf_map *map_thread, struct bpf_map *map_user,
tools/tracing/rtla/src/timerlat_bpf.h +3 -0
@@ -18,6 +18,7 @@ int timerlat_bpf_attach(void);void timerlat_bpf_detach (void );void timerlat_bpf_destroy (void );int timerlat_bpf_wait (int timeout); +int timerlat_bpf_restart_tracing (void ); int timerlat_bpf_get_hist_value(int key, long long *value_irq, long long *value_thread,
@@ -28,6 +29,7 @@ int timerlat_bpf_get_summary_value(enum summary_field key,long long *value_thread, long long *value_user, int cpus); +static inline int have_libbpf_support (void ) {return 1 ; }#else static inline int timerlat_bpf_init (struct timerlat_params *params)
@@ -38,6 +40,7 @@ static inline int timerlat_bpf_attach(void) { return -1; }static inline void timerlat_bpf_detach (void ) { };static inline void timerlat_bpf_destroy (void ) { };static inline int timerlat_bpf_wait (int timeout) {return -1 ; } +static inline int timerlat_bpf_restart_tracing (void ) {return -1 ; }; static inline int timerlat_bpf_get_hist_value(int key, long long *value_irq, long long *value_thread,
Signature · 1 line
2.49.0
Tomas Glozar
maintainerPATCH v2 4/9 rtla/timerlat: Add continue action
In reply to Tomas GlozarIntroduce option to resume tracing after a latency threshold overflow. The option is implemented as an action named "continue".
Example:
$ rtla timerlat top -q -T 200 -d 1s --on-threshold \
exec,command="echo Threshold" --on-threshold continue
Threshold
Threshold
Threshold
Timer Latency
...The feature is supported for both hist and top. After the continue action is executed, processing of the list of actions is stopped and tracing is resumed.
--- tools/tracing/rtla/src/actions.c | 27 +++++++++++- tools/tracing/rtla/src/actions.h | 5 ++- tools/tracing/rtla/src/timerlat_hist.c | 40 +++++++++++++++--- tools/tracing/rtla/src/timerlat_top.c | 57 ++++++++++++++++---------- 4 files changed, 100 insertions(+), 29 deletions(-)
tools/tracing/rtla/src/actions.c +26 -1
@@ -17,6 +17,7 @@ actions_init(struct actions *self) self->size = action_default_size; self->list =calloc (self->size,sizeof (struct action )); self->len =0 ; + self->continue_flag = false;memset (&self->present,0 ,sizeof (self->present));
@@ -108,6 +109,20 @@ actions_add_shell(struct actions *self, const char *command)return 0 ; } +/* +* actions_add_continue - add an action to resume measurement +*/ +int +actions_add_continue (struct actions *self) +{ +struct action *action =actions_new (self); + + self->present[ACTION_CONTINUE ] = true; + action->type =ACTION_CONTINUE ; + +return 0 ; +} +/* * actions_parse - add an action based on text specification */
@@ -133,6 +148,8 @@ actions_parse(struct actions *self, const char *trigger) type =ACTION_SIGNAL ; else if (strcmp(token, "shell") == 0) type =ACTION_SHELL ; + else if (strcmp(token, "continue") == 0) + type = ACTION_CONTINUE; else /* Invalid trigger type */ return -1;
@@ -187,6 +204,11 @@ actions_parse(struct actions *self, const char *trigger)if (strlen (token) >8 &&strncmp (token,"command=" ,8 ) ==0 )return actions_add_shell (self, token +8 );return -1 ; +case ACTION_CONTINUE : +/* Takes no argument */ +if (token !=NULL ) +return -1 ; +return actions_add_continue (self);default :return -1 ; }
@@ -196,7 +218,7 @@ actions_parse(struct actions *self, const char *trigger)* actions_perform - perform all actions */ int -actions_perform (const struct actions *self) +actions_perform (struct actions *self) {int pid, retval;const struct action *action;
@@ -226,6 +248,9 @@ actions_perform(const struct actions *self)if (retval)return retval;break ; +case ACTION_CONTINUE : + self->continue_flag = true; +return 0 ;default :break ; }
tools/tracing/rtla/src/actions.h +4 -1
@@ -7,6 +7,7 @@ enum action_type {ACTION_TRACE_OUTPUT ,ACTION_SIGNAL ,ACTION_SHELL , +ACTION_CONTINUE , ACTION_FIELD_N };
@@ -35,6 +36,7 @@ struct actions {struct action *list;int len, size;bool present[ACTION_FIELD_N ]; +bool continue_flag;/* External dependencies */ struct tracefs_instance *trace_output_inst;
@@ -45,5 +47,6 @@ void actions_destroy(struct actions *self);int actions_add_trace_output (struct actions *self,const char *trace_output);int actions_add_signal (struct actions *self,int signal,int pid);int actions_add_shell (struct actions *self,const char *command); +int actions_add_continue (struct actions *self);int actions_parse (struct actions *self,const char *trigger); -int actions_perform (const struct actions *self); +int actions_perform (struct actions *self);
tools/tracing/rtla/src/timerlat_hist.c +35 -5
@@ -1374,8 +1374,20 @@ int timerlat_hist_main(int argc, char *argv[]) goto out_hist; } -if (osnoise_trace_is_off (tool, record)) -break ; +if (osnoise_trace_is_off (tool, record)) { +actions_perform (¶ms->actions); + +if (!params->actions.continue_flag) +/* continue flag not set, break */ +break ; + +/* continue action reached, re-enable tracing */ +if (params->actions.present[ACTION_TRACE_OUTPUT ]) +trace_instance_start (&record->trace); +if (!params->no_aa) +trace_instance_start (&aa->trace); +trace_instance_start (trace); + }/* is there still any user-threads ? */ if (params->user_workload) {
@@ -1385,8 +1397,27 @@ int timerlat_hist_main(int argc, char *argv[]) } } } - }else -timerlat_bpf_wait (-1 ); + }else { +while (!stop_tracing) { +timerlat_bpf_wait (-1 ); + +if (!stop_tracing) { +/* Threshold overflow, perform actions on threshold */ +actions_perform (¶ms->actions); + +if (!params->actions.continue_flag) +/* continue flag not set, break */ +break ; + +/* continue action reached, re-enable tracing */ +if (params->actions.present[ACTION_TRACE_OUTPUT ]) +trace_instance_start (&record->trace); +if (!params->no_aa) +trace_instance_start (&aa->trace); +timerlat_bpf_restart_tracing (); + } + } + }if (params->mode !=TRACING_MODE_TRACEFS ) {timerlat_bpf_detach ();
@@ -1412,7 +1443,6 @@ int timerlat_hist_main(int argc, char *argv[])if (!params->no_aa)timerlat_auto_analysis (params->stop_us, params->stop_total_us); -actions_perform (¶ms->actions); return_value =FAILED ; }
tools/tracing/rtla/src/timerlat_top.c +35 -22
104 lines not on this page · raw message
Signature · 1 line
2.49.0
Tomas Glozar
maintainerPATCH v2 5/9 rtla/timerlat: Add action on end feature
In reply to Tomas GlozarImplement actions on end next to actions on threshold. A new option, --on-end is added, parallel to --on-threshold. Instead of being executed whenever a latency threshold is reached, it is executed at the end of the measurement.
For example:
$ rtla timerlat hist -d 5s --on-end trace
will save the trace output at the end.
All actions supported by --on-threshold are also supported by --on-end, except for continue, which does nothing with --on-end.
--- tools/tracing/rtla/src/timerlat.h | 5 ++- tools/tracing/rtla/src/timerlat_hist.c | 44 +++++++++++++++++-------- tools/tracing/rtla/src/timerlat_top.c | 45 ++++++++++++++++++-------- 3 files changed, 65 insertions(+), 29 deletions(-)
tools/tracing/rtla/src/timerlat.h +4 -1
@@ -48,7 +48,10 @@ struct timerlat_params {struct sched_attr sched_param;struct trace_events *events;enum timerlat_tracing_mode mode; -struct actions actions; + +struct actions threshold_actions; +struct actions end_actions; + union { struct { /* top only */
tools/tracing/rtla/src/timerlat_hist.c +30 -14
131 lines not on this page · raw message
tools/tracing/rtla/src/timerlat_top.c +31 -14
138 lines not on this page · raw message
Signature · 1 line
2.49.0
Tomas Glozar
maintainerPATCH v2 6/9 rtla/tests: Check rtla output with grep
In reply to Tomas GlozarAdd argument to the check command in the test suite that takes a regular expression that the output of rtla command is checked against. This allows testing for specific information in rtla output in addition to checking the return value.
Two minor improvements are included: running rtla with "eval" so that arguments with spaces can be passed to it via shell quotations, and the stdout of pushd and popd is suppressed to clean up the test output.
--- tools/tracing/rtla/tests/engine.sh | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)
tools/tracing/rtla/tests/engine.sh +17 -4
@@ -11,7 +11,7 @@ test_begin() { reset_osnoise() { # Reset osnoise options to default and remove any dangling instances created # by improperly exited rtla runs. - pushd /sys/kernel/tracing || return 1 + pushd /sys/kernel/tracing >/dev/null || return 1 # Remove dangling instances created by previous rtla run echo 0 > tracing_thresh
@@ -35,11 +35,14 @@ reset_osnoise() { echo 0 > stop_tracing_us echo 1000 > timerlat_period_us - popd + popd >/dev/null } check() { + test_name=$0 + tested_command=$1 expected_exitcode=${3:-0} + expected_output=$4 # Simple check: run rtla with given arguments and test exit code. # If TEST_COUNT is set, run the test. Otherwise, just count. ctr=$(($ctr + 1))
@@ -49,8 +52,16 @@ check() { [ "$NO_RESET_OSNOISE" == 1 ] || reset_osnoise # Run rtla; in case of failure, include its output as comment # in the test results. - result=$(stdbuf -oL $TIMEOUT "$RTLA" $2 2>&1); exitcode=$? - if [ $exitcode -eq $expected_exitcode ] + result=$(eval stdbuf -oL $TIMEOUT "$RTLA" $2 2>&1); exitcode=$? + # Test if the results matches if requested + if [ -n "$expected_output" ] + then + grep -E "$expected_output" <<< "$result" > /dev/null; grep_result=$? + else + grep_result=0 + fi + + if [ $exitcode -eq $expected_exitcode ] && [ $grep_result -eq 0 ] then echo "ok $ctr - $1" else
@@ -58,6 +69,8 @@ check() { # Add rtla output and exit code as comments in case of failure echo "$result" | col -b | while read line; do echo "# $line"; done printf "#\n# exit code %s\n" $exitcode + [ -n "$expected_output" ] && \ + printf "# Output match failed: \"%s\"\n" "$expected_output" fi fi }
Signature · 1 line
2.49.0
Tomas Glozar
maintainerPATCH v2 7/9 rtla/tests: Add tests for actions
In reply to Tomas GlozarAdd a bunch of tests covering most of both --on-threshold and --on-end. Parts sensitive to implementation of hist/top are tested for both.
--- tools/tracing/rtla/tests/timerlat.t | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
tools/tracing/rtla/tests/timerlat.t +28 -0
@@ -18,6 +18,8 @@ fi for option in $no_bpf_options do export RTLA_NO_BPF=$option + +# Basic tests check "verify help page" \ "timerlat --help" check "verify -s/--stack" \
@@ -36,6 +38,32 @@ check "verify -c/--cpus" \ "timerlat hist -c 0 -d 30s" check "hist test in nanoseconds" \ "timerlat hist -i 2 -c 0 -n -d 30s" 2 + +# Actions tests +check "trace output through -t" \ + "timerlat hist -T 2 -t" 2 "^ Saving trace to timerlat_trace.txt$" +check "trace output through -t with custom filename" \ + "timerlat hist -T 2 -t custom_filename.txt" 2 "^ Saving trace to custom_filename.txt$" +check "trace output through -A trace" \ + "timerlat hist -T 2 --on-threshold trace" 2 "^ Saving trace to timerlat_trace.txt$" +check "trace output through -A trace with custom filename" \ + "timerlat hist -T 2 --on-threshold trace,file=custom_filename.txt" 2 "^ Saving trace to custom_filename.txt$" +check "exec command" \ + "timerlat hist -T 2 --on-threshold shell,command='echo TestOutput'" 2 "^TestOutput$" +check "multiple actions" \ + "timerlat hist -T 2 --on-threshold shell,command='echo -n 1' --on-threshold shell,command='echo 2'" 2 "^12$" +check "hist stop at failed action" \ + "timerlat hist -T 2 --on-threshold shell,command='echo -n 1; false' --on-threshold shell,command='echo -n 2'" 2 "^1# RTLA timerlat histogram$" +check "top stop at failed action" \ + "timerlat top -T 2 --on-threshold shell,command='echo -n 1; false' --on-threshold shell,command='echo -n 2'" 2 "^1ALL" +check "hist with continue" \ + "timerlat hist -T 2 -d 1s --on-threshold shell,command='echo TestOutput' --on-threshold continue" 0 "^TestOutput$" +check "top with continue" \ + "timerlat top -q -T 2 -d 1s --on-threshold shell,command='echo TestOutput' --on-threshold continue" 0 "^TestOutput$" +check "hist with trace output at end" \ + "timerlat hist -d 1s --on-end trace" 0 "^ Saving trace to timerlat_trace.txt$" +check "top with trace output at end" \ + "timerlat top -d 1s --on-end trace" 0 "^ Saving trace to timerlat_trace.txt$" done test_end
Signature · 1 line
2.49.0
Tomas Glozar
maintainerPATCH v2 8/9 rtla/tests: Limit duration to maximum of 10s
In reply to Tomas GlozarMany of the original rtla tests included durations of 1 minute and 30 seconds. Experience has shown this is unnecessary, since 10 seconds as waiting time for samples to appear.
Change duration of all rtla tests to at most 10 seconds. This speeds up testing significantly.
Before:
$ make check All tests successful. Files=3, Tests=54, 536 wallclock secs ( 0.03 usr 0.00 sys + 20.31 cusr 22.02 csys = 42.36 CPU) Result: PASS
After:
$ make check ... All tests successful. Files=3, Tests=54, 196 wallclock secs ( 0.03 usr 0.01 sys + 20.28 cusr 20.68 csys = 41.00 CPU) Result: PASS
--- tools/tracing/rtla/tests/hwnoise.t | 8 ++++---- tools/tracing/rtla/tests/osnoise.t | 4 ++-- tools/tracing/rtla/tests/timerlat.t | 8 ++++---- 3 files changed, 10 insertions(+), 10 deletions(-)
tools/tracing/rtla/tests/hwnoise.t +4 -4
@@ -10,12 +10,12 @@ check "verify help page" \ check "detect noise higher than one microsecond" \ "hwnoise -c 0 -T 1 -d 5s -q" check "set the automatic trace mode" \ - "hwnoise -a 5 -d 30s" 2 + "hwnoise -a 5 -d 10s" 2 check "set scheduling param to the osnoise tracer threads" \ - "hwnoise -P F:1 -c 0 -r 900000 -d 1M -q" + "hwnoise -P F:1 -c 0 -r 900000 -d 10s -q" check "stop the trace if a single sample is higher than 1 us" \ - "hwnoise -s 1 -T 1 -t -d 30s" 2 + "hwnoise -s 1 -T 1 -t -d 10s" 2 check "enable a trace event trigger" \ - "hwnoise -t -e osnoise:irq_noise trigger=\"hist:key=desc,duration:sort=desc,duration:vals=hitcount\" -d 1m" + "hwnoise -t -e osnoise:irq_noise trigger=\"hist:key=desc,duration:sort=desc,duration:vals=hitcount\" -d 10s" test_end
tools/tracing/rtla/tests/osnoise.t +2 -2
@@ -8,13 +8,13 @@ set_timeout 2m check "verify help page" \ "osnoise --help" check "verify the --priority/-P param" \ - "osnoise top -P F:1 -c 0 -r 900000 -d 1M -q" + "osnoise top -P F:1 -c 0 -r 900000 -d 10s -q" check "verify the --stop/-s param" \ "osnoise top -s 30 -T 1 -t" 2 check "verify the --trace param" \ "osnoise hist -s 30 -T 1 -t" 2 check "verify the --entries/-E param" \ - "osnoise hist -P F:1 -c 0 -r 900000 -d 1M -b 10 -E 25" + "osnoise hist -P F:1 -c 0 -r 900000 -d 10s -b 10 -E 25" # Test setting default period by putting an absurdly high period # and stopping on threshold.
tools/tracing/rtla/tests/timerlat.t +4 -4
@@ -25,9 +25,9 @@ check "verify help page" \ check "verify -s/--stack" \ "timerlat top -s 3 -T 10 -t" 2 check "verify -P/--priority" \ - "timerlat top -P F:1 -c 0 -d 1M -q" + "timerlat top -P F:1 -c 0 -d 10s -q" check "test in nanoseconds" \ - "timerlat top -i 2 -c 0 -n -d 30s" 2 + "timerlat top -i 2 -c 0 -n -d 10s" 2 check "set the automatic trace mode" \ "timerlat top -a 5 --dump-tasks" 2 check "print the auto-analysis if hits the stop tracing condition" \
@@ -35,9 +35,9 @@ check "print the auto-analysis if hits the stop tracing condition" \ check "disable auto-analysis" \ "timerlat top -s 3 -T 10 -t --no-aa" 2 check "verify -c/--cpus" \ - "timerlat hist -c 0 -d 30s" + "timerlat hist -c 0 -d 10s" check "hist test in nanoseconds" \ - "timerlat hist -i 2 -c 0 -n -d 30s" 2 + "timerlat hist -i 2 -c 0 -n -d 10s" 2 # Actions tests check "trace output through -t" \
Signature · 1 line
2.49.0
Tomas Glozar
maintainerPATCH v2 9/9 Documentation/rtla: Add actions feature
In reply to Tomas GlozarDocument both --on-threshold and --on-end, with examples.
--- .../tools/rtla/common_timerlat_options.rst | 64 +++++++++++++++++++ 1 file changed, 64 insertions(+)
Documentation/tools/rtla/common_timerlat_options.rst +64 -0
@@ -55,3 +55,67 @@ Set timerlat to run without workload, waiting for the user to dispatch a per-cpu task that waits for a new period on the tracing/osnoise/per_cpu/cpu$ID/timerlat_fd. See linux/tools/rtla/sample/timerlat_load.py for an example of user-load code. + +**--on-threshold** *action* + + Defines an action to be executed when tracing is stopped on a latency threshold + specified by **-i/--irq** or **-T/--thread**. + + Multiple --on-threshold actions may be specified, and they will be executed in + the order they are provided. If any action fails, subsequent actions in the list + will not be executed. + + Supported actions are: + + - *trace[,file=<filename>]* + + Saves trace output, optionally taking a filename. Alternative to -t/--trace. + Note that nlike -t/--trace, specifying this multiple times will result in + the trace being saved multiple times. + + - *signal,num=<sig>,pid=<pid>* + + Sends signal to process. "parent" might be specified in place of pid to target + the parent process of rtla. + + - *shell,command=<command>* + + Execute shell command. + + - *continue* + + Continue tracing after actions are executed instead of stopping. + + Example: + + $ rtla timerlat -T 20 --on-threshold trace + --on-threshold shell,command="grep ipi_send timerlat_trace.txt" + --on-threshold signal,num=2,pid=parent + + This will save a trace with the default filename "timerlat_trace.txt", print its + lines that contain the text "ipi_send" on standard output, and send signal 2 + (SIGINT) to the parent process. + + Performance Considerations: + + For time-sensitive actions, it is recommended to run **rtla timerlat** with BPF + support and RT priority. Note that due to implementational limitations, actions + might be delayed up to one second after tracing is stopped if BPF mode is not + available or disabled. + +**--on-end** *action* + + Defines an action to be executed at the end of **rtla timerlat** tracing. + + Multiple --on-end actions can be specified, and they will be executed in the order + they are provided. If any action fails, subsequent actions in the list will not be + executed. + + See the documentation for **--on-threshold** for the list of supported actions, with + the exception that *continue* has no effect. + + Example: + + $ rtla timerlat -d 5s --on-end trace + + This runs rtla timerlat with default options and save trace output at the end.
Signature · 1 line
2.49.0
Steven Rostedt
maintainerRe: [PATCH v2 9/9] Documentation/rtla: Add actions feature
Documentation/tools/rtla/common_timerlat_options.rst line 105
Quoted 5 lines · On Thu, 26 Jun 2025 14:34:05 +0200 Tomas Glozar <[email protected]> wrote:
Documentation/tools/rtla/common_timerlat_options.rst +5
-0+ For time-sensitive actions, it is recommended to run **rtla timerlat** with BPF + support and RT priority. Note that due to implementational limitations, actions + might be delayed up to one second after tracing is stopped if BPF mode is not + available or disabled. +
I'm curious to what is looked for for triggering an action. We can poll on events and get woken when they are triggered. It may be possible to add even more ways to wake a task waiting for something to happen.
-- Steve
Tomas Glozar
maintainerRe: [PATCH v2 9/9] Documentation/rtla: Add actions feature
út 22. 7. 2025 v 0:35 odesílatel Steven Rostedt <[email protected]> napsal:
Quoted 5 lines
I'm curious to what is looked for for triggering an action. We can poll on events and get woken when they are triggered. It may be possible to add even more ways to wake a task waiting for something to happen.
Threshold actions are triggered immediately after a sample over the set threshold is detected by rtla. For BPF mode, this happens almost right after the sample is processed in the BPF program and the scheduler gets to waking up rtla following a BPF ringbuffer write. There is only a short delay (up to tens of microseconds) because the BPF helper defers the wake-up into irq_work.
For non-BPF mode, rtla periodically pulls samples from tracefs, when it does that, it also checks whether tracing has been turned off. If yes, that means there was a threshold overflow, and actions are triggered. Since the period for that is currently set to 1 second, the action might be delayed up to one second from the threshold occurring, That delay might be a problem if you need to collect a lot of data from a ringbuffer in the action, e.g. global Intel PT data collection for precise troubleshooting of difficult latencies.
Of course, this is just an implementational limitation of the timerlat tracer. If timerlat had an event (like osnoise's "sample_threshold") triggered on threshold overflow and if it is possible to wait on it even without BPF, rtla could wait on that for both BPF and non-BPF mode instead of what it is currently doing.
Tomas
Steven Rostedt
maintainerRe: [PATCH v2 9/9] Documentation/rtla: Add actions feature
Quoted 5 lines · On Tue, 22 Jul 2025 09:03:24 +0200 Tomas Glozar <[email protected]> wrote:
Of course, this is just an implementational limitation of the timerlat tracer. If timerlat had an event (like osnoise's "
sample_threshold") triggered on threshold overflow and if it is possible to wait on it even without BPF, rtla could wait on that for both BPF and non-BPF mode instead of what it is currently doing.
Right. Is this something you may want?
-- Steve
Tomas Glozar
maintainerRe: [PATCH v2 9/9] Documentation/rtla: Add actions feature
út 22. 7. 2025 v 17:30 odesílatel Steven Rostedt <[email protected]> napsal:
Quoted 4 lines
On Tue, 22 Jul 2025 09:03:24 +0200
Tomas Glozar <[email protected]> wrote:
Quoted 8 lines
Quoted 5 lines
Of course, this is just an implementational limitation of the timerlat tracer. If timerlat had an event (like osnoise's "
sample_threshold") triggered on threshold overflow and if it is possible to wait on it even without BPF, rtla could wait on that for both BPF and non-BPF mode instead of what it is currently doing.Right. Is this something you may want?
I don't think it is that important. Non-BPF mode is mostly as a fallback for users of rtla on older kernels which don't have the osnoise:timerlat_sample trace event. Those are (I assume) mostly users of LTS distributions who run newer rtla from a container. Adding a new event wouldn't help in their case.
The only users who would benefit from that are those who don't have BPF or libbpf. If there is interest from using low-latency actions on threshold in such settings, I'm not against implementing a threshold overflow tracepoint and supporting it in rtla for triggering actions on threshold.
Tomas