perf_event: introduce 'perf timer' to analyze timer's behavior
Subsystems (3)
cover perf_event: introduce 'perf timer' to analyze timer's behavior
Hi,
We introduce 'perf timer' in this patchset, it can analyze timer latency and timer function handle time, the usage and result is like below:
# perf timer record # perf timer lat --print-lat --print-handle ------------------------------------------------------------------------------------------------------- | Timer | TYPE | Avg-latency | Max-latency | Max-latency-at-TS |Max-lat-at-Task | |0xf7ad1f5c |hrtimer |996068.500 ns|1607650 ns|10270128658526 |init | |0xf7903f04 |timer |0.625 HZ|2 HZ|10270344082394 |swapper | |0xf787a05c |hrtimer |200239.500 ns|359929 ns|10269316024808 |main | |main :[ PROF]|itimer |0.000 HZ|0 HZ|10237021270557 |main | |main :[VIRTUAL]|itimer |0.000 HZ|0 HZ|10257314773501 |main |
......
-------------------------------------------------------------------------------------------------------
| Timer | TYPE | Avg-handle (ms)|Max-handle(ms)| Max-handle-at-TS(s)|Max-lat-at-func | |0xf7ad1f5c |hrtimer |0.025 |0.025 |10270.129 |0xc016b5b0 | |0xf7903f04 |timer |0.009 |0.011 |10260.342 |0xc0159240 | |0xf787a05c |hrtimer |0.031 |0.062 |10267.018 |0xc014cc40 |
And, in current code, it'll complain with below message when we use 'perf timer lat':
# ./perf timer lat
Warning: unknown op '{'
Warning: Error: expected type 5 but read 1
Warning: failed to read event print fmt for hrtimer_start
Warning: unknown op '{'
Warning: Error: expected type 5 but read 1
Warning: failed to read event print fmt for hrtimer_expire_entry It's because perf parse "hrtimer_start" and "hrtimer_expire_entry" fail, but it not hurt using.
See: http://lkml.org/lkml/2009/10/12/726
include/trace/events/timer.h | 8 tools/perf/Documentation/perf-timer.txt | 40 + tools/perf/Makefile | 1 tools/perf/builtin-sched.c | 3 tools/perf/builtin-timer.c | 954 ++++++++++++++++++++++++++++++++ tools/perf/builtin.h | 1 tools/perf/command-list.txt | 1 tools/perf/perf.c | 1 tools/perf/util/trace-event-parse.c | 25 tools/perf/util/trace-event.h | 4 10 files changed, 1030 insertions(+), 8 deletions(-)
PATCH 1/4 trace_event: record task' real_timer in itimer_state tracepoint
When itimer is ITIMER_REAL, we should cooperate with hrtimer's tracepoit to get it's latency, this patch can let us get itimer's hrtimer easy
--- include/trace/events/timer.h | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-)
include/trace/events/timer.h +6 -2
@@ -285,6 +285,7 @@ TRACE_EVENT(itimer_state,TP_STRUCT__entry (__field (int ,which ) + __field( void *, timer )__field ( cputime_t, expires )__field ( long, value_sec ) __field( long, value_usec )
@@ -294,6 +295,8 @@ TRACE_EVENT(itimer_state,TP_fast_assign ( __entry->which = which; + __entry->timer = which ==ITIMER_REAL ? + ¤t->signal->real_timer :NULL ; __entry->expires = expires; __entry->value_sec = value->it_value.tv_sec; __entry->value_usec = value->it_value.tv_usec;
@@ -301,8 +304,9 @@ TRACE_EVENT(itimer_state, __entry->interval_usec = value->it_interval.tv_usec; ), -TP_printk ("which=%d expires=%llu it_value=%ld.%ld it_interval=%ld.%ld" , - __entry->which, (unsigned long long )__entry->expires, +TP_printk ("which=%d timer=%p expires=%llu it_value=%ld.%ld it_interval=" +"%ld.%ld" , __entry->which, __entry->timer, + (unsigned long long )__entry->expires, __entry->value_sec, __entry->value_usec, __entry->interval_sec, __entry->interval_usec) );
Signature · 1 line
1.6.1.2
PATCH 2/4 perf_event: fix getting point
The point that got by raw_field_ptr() is point to @data offset not tracepoint saved.
Introduce raw_field_arry() to get arry data is tracepoint save arry to it.
--- tools/perf/builtin-sched.c | 3 +-- tools/perf/util/trace-event-parse.c | 25 ++++++++++++++++++++++--- tools/perf/util/trace-event.h | 3 +++ 3 files changed, 26 insertions(+), 5 deletions(-)
tools/perf/builtin-sched.c +1 -2
@@ -630,8 +630,7 @@ static void test_calibrations(void)#define FILL_ARRAY (ptr, array, event, data) \ do { \ - void *__array = raw_field_ptr(event, #array, data); \ - memcpy(ptr.array, __array, sizeof(ptr.array)); \ + raw_field_arry(event, #array, data, ptr.array, sizeof(ptr.array));\ } while(0)#define FILL_COMMON_FIELDS (ptr, event, data) \
tools/perf/util/trace-event-parse.c +22 -3
@@ -840,7 +840,7 @@ static int event_read_fields(struct event *event, struct format_field **fields) field->flags |=FIELD_IS_ARRAY ; type =read_token (&token); -while (strcmp (token,"]" ) !=0 ) { +while (strcmp (token,"]" ) !=0 ) {if (last_type ==EVENT_ITEM && type ==EVENT_ITEM ) len =2 ;
@@ -1919,13 +1919,32 @@ raw_field_value(struct event *event, const char *name, void *data)void *raw_field_ptr (struct event *event,const char *name,void *data) { +void *ptr; +unsigned long long value; + + value =raw_field_value (event, name, data); + +if (!value) +return NULL ; + +memcpy (&ptr, &value,sizeof (ptr)); +return ptr; +} + +unsigned long long +raw_field_arry (struct event *event,const char *name,void *data, +void *arry,int arry_size) +{ +int len;struct format_field *field; field =find_any_field (event, name);if (!field) -return NULL ; +return 0ULL ; -return data + field->offset; + len =min (arry_size, field->size); +memcpy (arry, data + field->offset, len); +return len; }static int get_common_info (const char *type,int *offset,int *size)
tools/perf/util/trace-event.h +3 -0
@@ -255,6 +255,9 @@ unsigned long long read_size(void *ptr, int size);unsigned long long raw_field_value (struct event *event,const char *name,void *data);void *raw_field_ptr (struct event *event,const char *name,void *data); +unsigned long long +raw_field_arry (struct event *event,const char *name,void *data, +void *arry,int arry_size);unsigned long long eval_flag (const char *flag);int read_tracing_data (int fd,struct perf_event_attr *pattrs,int nb_events);
Signature · 1 line
1.6.1.2
PATCH 3/4 perf/timer: add document
Add document for 'perf timer'
--- tools/perf/Documentation/perf-timer.txt | 40 +++++++++++++++++++++++++++++++ 1 files changed, 40 insertions(+), 0 deletions(-) create mode 100644 tools/perf/Documentation/perf-timer.txt
tools/perf/Documentation/perf-timer.txt new file +40 -0
@@ -0,0 +1,40 @@ +perf-timer(1) +============== + +NAME +---- +perf-timer - Tool to trace/measure kernel timer properties + +SYNOPSIS +-------- +[verse] +'perf timer' {record|latency} [<options>] + +DESCRIPTION +----------- +There are two variants of perf timer: + + 'perf timer record <command>' to record the timer events + of an arbitrary workload. + + 'perf timer latency' to report kernel timer latency and + timer function handle time. + +OPTIONS +------- +-D:: +--dump-raw-trace=:: + Display verbose dump of the sched data. + +-i <file>:: +--input=<file>:: + Select the input file (default: perf.data) + +-s <key[,key2...]>:: +--sort=<key[,key2...]>:: + Sort the output (default: timer, type, avg-timer-latency, + max-timer-latency, avg-handle-time, max-handle-time) + +SEE ALSO +-------- +linkperf:perf-record[1]
Signature · 1 line
1.6.1.2
PATCH 4/4 perf/timer: 'perf timer' core code
It's the core code of 'perf timer', we can use it to analyse timer's behavior:
# perf timer record # perf timer lat --print-lat --print-handle ------------------------------------------------------------------------------------------------------- | Timer | TYPE | Avg-latency | Max-latency | Max-latency-at-TS |Max-lat-at-Task | |0xf7ad1f5c |hrtimer |996068.500 ns|1607650 ns|10270128658526 |init | |0xf7903f04 |timer |0.625 HZ|2 HZ|10270344082394 |swapper | |0xf787a05c |hrtimer |200239.500 ns|359929 ns|10269316024808 |main | |main :[ PROF]|itimer |0.000 HZ|0 HZ|10237021270557 |main | |main :[VIRTUAL]|itimer |0.000 HZ|0 HZ|10257314773501 |main |
......
-------------------------------------------------------------------------------------------------------
| Timer | TYPE | Avg-handle (ms)|Max-handle(ms)| Max-handle-at-TS(s)|Max-lat-at-func | |0xf7ad1f5c |hrtimer |0.025 |0.025 |10270.129 |0xc016b5b0 | |0xf7903f04 |timer |0.009 |0.011 |10260.342 |0xc0159240 | |0xf787a05c |hrtimer |0.031 |0.062 |10267.018 |0xc014cc40 |
--- tools/perf/Makefile | 1 + tools/perf/builtin-timer.c | 954 +++++++++++++++++++++++++++++++++++++++++++ tools/perf/builtin.h | 1 + tools/perf/command-list.txt | 1 + tools/perf/perf.c | 1 + 5 files changed, 958 insertions(+), 0 deletions(-) create mode 100644 tools/perf/builtin-timer.c
tools/perf/Makefile +1 -0
@@ -439,6 +439,7 @@ BUILTIN_OBJS += builtin-top.o BUILTIN_OBJS += builtin-trace.o BUILTIN_OBJS += builtin-probe.o BUILTIN_OBJS += builtin-kmem.o +BUILTIN_OBJS += builtin-timer.o PERFLIBS = $(LIB_FILE)
tools/perf/builtin-timer.c new file +954 -0
954 lines not on this page · raw message
tools/perf/builtin.h +1 -0
@@ -29,5 +29,6 @@ extern int cmd_trace(int argc, const char **argv, const char *prefix);extern int cmd_version (int argc,const char **argv,const char *prefix);extern int cmd_probe (int argc,const char **argv,const char *prefix);extern int cmd_kmem (int argc,const char **argv,const char *prefix); +extern int cmd_timer (int argc,const char **argv,const char *prefix);#endif
tools/perf/command-list.txt +1 -0
@@ -15,3 +15,4 @@ perf-top mainporcelain common perf-trace mainporcelain common perf-probe mainporcelain common perf-kmem mainporcelain common +perf-timer mainporcelain common
tools/perf/perf.c +1 -0
@@ -300,6 +300,7 @@ static void handle_internal_command(int argc, const char **argv) { "sched", cmd_sched, 0 }, { "probe", cmd_probe, 0 }, { "kmem", cmd_kmem, 0 }, + { "timer", cmd_timer, 0 }, };unsigned int i;static const char ext[] =STRIP_EXTENSION ;
Signature · 1 line
1.6.1.2
Re: [PATCH 2/4] perf_event: fix getting point
In reply to Xiao GuangrongQuoted 5 lines · Xiao Guangrong
The point that got by
raw_field_ptr()is point to @data offset not tracepoint saved.Introduce
raw_field_arry()to get arry data is tracepoint save arry to it.
Could you please rename it into raw_field_array()? I feel like...a kind of loss there :)
Quoted 38 lines
Signed-off-by Xiao Guangrong <[email protected]>--- tools/perf/builtin-sched.c | 3 +-- tools/perf/util/trace-event-parse.c | 25 ++++++++++++++++++++++--- tools/perf/util/trace-event.h | 3 +++ 3 files changed, 26 insertions(+), 5 deletions(-)tools/perf/builtin-sched.c +1
-2@@ -630,8 +630,7 @@ static void test_calibrations(void)#define FILL_ARRAY (ptr, array, event, data) \ do { \ - void *__array = raw_field_ptr(event, #array, data); \ - memcpy(ptr.array, __array, sizeof(ptr.array)); \ + raw_field_arry(event, #array, data, ptr.array, sizeof(ptr.array));\ } while(0)#define FILL_COMMON_FIELDS (ptr, event, data) \tools/perf/util/trace-event-parse.c +1
-1@@ -840,7 +840,7 @@ static int event_read_fields(struct event *event, struct format_field **fields) field->flags |=FIELD_IS_ARRAY ; type =read_token (&token); -while (strcmp (token,"]" ) !=0 ) { +while (strcmp (token,"]" ) !=0 ) {if (last_type ==EVENT_ITEM && type ==EVENT_ITEM ) len =2 ;@@ -1919,13 +1919,32 @@ raw_field_value(struct event *event, const char *name, void *data)void *raw_field_ptr (struct event *event,const char *name,void *data)
tools/perf/util/trace-event-parse.c raw_field_value(struct event *event, const char *name, void *data) · line 1930
Quoted 10 lines
tools/perf/util/trace-event-parse.c +9
-0{ +void *ptr; +unsigned long long value; + + value =raw_field_value (event, name, data); + +if (!value) +return NULL ; + +memcpy (&ptr, &value,sizeof (ptr));
Could you perhaps just do
ptr = (void *)value; ?
This memcpy suggest we are dereferencing some field while actually just copy a pointer.
Thanks!
Quoted 6 lines · Xiao Guangrong
Hi,
We introduce 'perf timer' in this patchset, it can analyze timer latency and timer function handle time, the usage and result is like below:
Quoted 9 lines
# perf timer record # perf timer lat --print-lat --print-handle ------------------------------------------------------------------------------------------------------- | Timer | TYPE | Avg-latency | Max-latency | Max-latency-at-TS |Max-lat-at-Task | |0xf7ad1f5c |hrtimer |996068.500 ns|1607650 ns|10270128658526 |init | |0xf7903f04 |timer |0.625 HZ|2 HZ|10270344082394 |swapper | |0xf787a05c |hrtimer |200239.500 ns|359929 ns|10269316024808 |main | |main :[ PROF]|itimer |0.000 HZ|0 HZ|10237021270557 |main | |main :[VIRTUAL]|itimer |0.000 HZ|0 HZ|10257314773501 |main |
Cool! This is really a good work and a good idea.
Just have some neats in mind. hrtimers and timers don't have the same latency and granularity requirements.
As you show it, timers have an HZ granularity and hrtimers are about nanoseconds, and mixing them up in the same array of latency report is too messy. They don't have the same granularity/latency scope so they should
be reported separately.
Otherwise it's like mixing slothes and flies latency results after a fly swat attack test ;-)
Quoted 27 lines · Xiao Guangrong
Hi,
We introduce 'perf timer' in this patchset, it can analyze timer latency and timer function handle time, the usage and result is like below:
# perf timer record # perf timer lat --print-lat --print-handle ------------------------------------------------------------------------------------------------------- | Timer | TYPE | Avg-latency | Max-latency | Max-latency-at-TS |Max-lat-at-Task | |0xf7ad1f5c |hrtimer |996068.500 ns|1607650 ns|10270128658526 |init | |0xf7903f04 |timer |0.625 HZ|2 HZ|10270344082394 |swapper | |0xf787a05c |hrtimer |200239.500 ns|359929 ns|10269316024808 |main | |main :[ PROF]|itimer |0.000 HZ|0 HZ|10237021270557 |main | |main :[VIRTUAL]|itimer |0.000 HZ|0 HZ|10257314773501 |main |......
-------------------------------------------------------------------------------------------------------
| Timer | TYPE | Avg-handle (ms)|Max-handle(ms)| Max-handle-at-TS(s)|Max-lat-at-func | |0xf7ad1f5c |hrtimer |0.025 |0.025 |10270.129 |0xc016b5b0 | |0xf7903f04 |timer |0.009 |0.011 |10260.342 |0xc0159240 | |0xf787a05c |hrtimer |0.031 |0.062 |10267.018 |0xc014cc40 |And, in current code, it'll complain with below message when we use 'perf timer lat':
Quoted 7 lines
# ./perf timer lat
Warning: unknown op '{'
Warning: Error: expected type 5 but read 1
Warning: failed to read event print fmt for hrtimer_start
Warning: unknown op '{'
Warning: Error: expected type 5 but read 1
Warning: failed to read event print fmt for hrtimer_expire_entryOh and indeed we have some event format complexities to solve there.
cat /debug/tracing/events/timer/hrtimer_start/format
print fmt: "hrtimer=%p function=%pf expires=%llu softexpires=%llu", REC->hrtimer, REC->function, (unsigned long long)(((ktime_t) { .tv64 = REC->expires }).tv64), (unsigned long long)(((ktime_t) { .tv64 = REC->softexpires }).tv64)
We should try to simplify this, may be we should just expose __entry->expires as is.
Re: [PATCH 4/4] perf/timer: 'perf timer' core code
In reply to Xiao GuangrongQuoted 12 lines · Xiao Guangrong
It's the core code of 'perf timer', we can use it to analyse timer's behavior:
# perf timer record # perf timer lat --print-lat --print-handle ------------------------------------------------------------------------------------------------------- | Timer | TYPE | Avg-latency | Max-latency | Max-latency-at-TS |Max-lat-at-Task | |0xf7ad1f5c |hrtimer |996068.500 ns|1607650 ns|10270128658526 |init | |0xf7903f04 |timer |0.625 HZ|2 HZ|10270344082394 |swapper | |0xf787a05c |hrtimer |200239.500 ns|359929 ns|10269316024808 |main | |main :[ PROF]|itimer |0.000 HZ|0 HZ|10237021270557 |main | |main :[VIRTUAL]|itimer |0.000 HZ|0 HZ|10257314773501 |main |
Quoted 7 lines
......
-------------------------------------------------------------------------------------------------------
| Timer | TYPE | Avg-handle (ms)|Max-handle(ms)| Max-handle-at-TS(s)|Max-lat-at-func | |0xf7ad1f5c |hrtimer |0.025 |0.025 |10270.129 |0xc016b5b0 | |0xf7903f04 |timer |0.009 |0.011 |10260.342 |0xc0159240 | |0xf787a05c |hrtimer |0.031 |0.062 |10267.018 |0xc014cc40 |
The output is confusing in several aspects:
1) Different time units:
Please use consistent time units for everything. micro seconds are fine and we definitely do not care about nanosecond fractions.
2) Timer description:
Why do we have hex addresses and process names intermingled ? Why don't we print the process/thread name which owns the timer always ? [PROF/VIRTUAL] is not a property of the Timer, it belongs into type.
3) Max-lat-at-Task:
What does this field tell ?
4) *handle:
That should be a more descriptive name, e.g. function runtime
5) Max-lat-at-func:
Is this the callback function which ran the longest time ? Why is it named latency ? Why is it not decoded into a symbol ?
Btw, fitting the output into 80chars allows to use the tool on a non graphical terminal as well.
Also there are other metrics of timers which are interesting and should be retrieved by such a tool:
number of activated timers
number of canceled timers
number of expired timers
in the form of simple statistics.
The canceled timers are interesting also in a list, so we can see which timers are canceled after which time and how long before the expiry.
tools/perf/builtin-timer.c line 22
Quoted 4 lines
tools/perf/builtin-timer.c new file +4
-0+#ifdef TDEBUG +static char const *timer_state_string[] = { +"TIMER_INIT" , + "TIMER_STARE",
If I _stare_ longer, I might understand it :)
tools/perf/builtin-timer.c line 34
Quoted 3 lines
tools/perf/builtin-timer.c new file +3
-0+#define SORT_KEY \ + "timer, type, avg-timer-latency, max-timer-latency," \ + " avg-handle-time, max-handle-time"
The code has at least two places where these strings are used. They should be synced.
Quoted 14 lines
tools/perf/builtin-timer.c new file +14
-0+static char const *input_name ="perf.data" ; +static char default_sort_order[] =SORT_KEY ; + +static char *sort_order = default_sort_order; +static int profile_cpu =-1 ; +static int print_lat; +static int print_handle; +static int bug_nr; +static struct perf_session *timer_session; + +static const struct option timer_options[] = { +OPT_STRING ('i' ,"input" , &input_name,"file" , +"input file name" ), + OPT_BOOLEAN('v', "verbose", &verbose,
tools/perf/builtin-timer.c line 59
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+ "be more verbose (show symbol address, etc)"), +OPT_BOOLEAN ('D' ,"dump-raw-trace" , &dump_trace, +"dump raw trace in ASCII" ), + OPT_END() +}; + +static const char *const timer_usage[] = { +"perf timer [<options>] {record|latency}" , +NULL +};
Your example above uses "perf timer lat". What's correct ?
tools/perf/builtin-timer.c line 63
Quoted 3 lines
tools/perf/builtin-timer.c new file +3
-0+static const struct option latency_options[] = { +OPT_STRING ('s' ,"sort" , &sort_order,"key[,key2...]" , +"sort by key(s): " SORT_KEY ),
Do we really need a sort order ? A single sort key should be sufficient.
I'd prefer to have a selector instead which lets me filter timer types. If I debug hrtimers then I have no interest in itimers or timer list timers.
tools/perf/builtin-timer.c line 111
Quoted 5 lines
tools/perf/builtin-timer.c new file +5
-0+enum timer_type { +TIMER , +HRTIMER , +ITIMER , +};
Hmm. I don't think that's correct. You should split the itimer type into the various subimplementations.
tools/perf/builtin-timer.c line 117
Quoted 5 lines
tools/perf/builtin-timer.c new file +5
-0+static char const *timer_type_string[] = { +"timer" , +"hrtimer" , +"itimer" , +};
Ditto.
tools/perf/builtin-timer.c line 164
Quoted 7 lines
tools/perf/builtin-timer.c new file +7
-0+struct sort_dimension { +const char *name; +sort_fn_t cmp; +struct list_head list; +}; + +#define CMP_FUNCTION (value) \
Please use something like "field". value confused me for a while, as I expected a real value.
tools/perf/builtin-timer.c line 174
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+static int cmp_##value(struct timer_info *v1, struct timer_info *v2) \ +{ \ + if (v1->value < v2->value) \ + return -1; \ + if (v1->value > v2->value) \ + return 1; \ + return 0; \ +} + +#define DECLARE_SORT_DIMENSION (sort_name, field) \
That should be DEFINE_... You are defining the function, not declaring it.
Quoted 11 lines
tools/perf/builtin-timer.c new file +11
-0+CMP_FUNCTION (field) \ +static struct sort_dimension sort_##field = { \ + .name = sort_name, \ + .cmp = cmp_##field, \ +} + +DECLARE_SORT_DIMENSION ("timer" , timer); +DECLARE_SORT_DIMENSION ("type" , type); +DECLARE_SORT_DIMENSION ("avg-timer-latency" , timer_avg_lat); +DECLARE_SORT_DIMENSION ("max-timer-latency" , timer_max_lat); +DECLARE_SORT_DIMENSION ("avg-handle-time" , handler_avg_time);
tools/perf/builtin-timer.c line 195
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+DECLARE_SORT_DIMENSION ("max-handle-time" , handler_max_time); + +static struct sort_dimension *available_sorts[] = { + &sort_timer, + &sort_type, + &sort_timer_avg_lat, + &sort_timer_max_lat, + &sort_handler_avg_time, + &sort_handler_max_time, +};
This indirection horrible.
#define SORT(field,_name) \
{ .func = cmp_##field, \
.name = _name, }static struct sort_key sorts[] = {
SORT(timer, "timer"),
....
};Quoted 1 line
+#define
NB_AVAILABLE_SORTS(int)(sizeof(available_sorts) /sizeof(struct sort_dimension *))
Please use NR_ as abbreviation for number. Also the size calculation for arrays is:
ARRAY_SIZE(available_sorts)
Quoted 1 line
+static int
sort_dimension__add(const char *tok, struct list_head *list)
No double underscores please.
Quoted 9 lines
tools/perf/builtin-timer.c new file +9
-0+{ +int i; + +for (i =0 ; i <NB_AVAILABLE_SORTS ; i++) { +if (!strcmp (available_sorts[i]->name, tok)) { +list_add_tail (&available_sorts[i]->list, list); + +return 0 ; + }
tools/perf/builtin-timer.c line 217
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+ } + +return -1 ; +} + +static LIST_HEAD (sort_list); + +static void setup_sorting (void ) +{ +char *tmp, *tok, *str =strdup (sort_order);
Please hand sort_order in as an argument.
tools/perf/builtin-timer.c line 230
Quoted 12 lines
tools/perf/builtin-timer.c new file +12
-0+for (tok =strtok_r (str,", " , &tmp); + tok; tok =strtok_r (NULL ,", " , &tmp)) { +if (sort_dimension__add (tok, &sort_list) <0 ) { +error ("Unknown --sort key: `%s'" , tok); +usage_with_options (latency_usage, latency_options); + } + } + +free (str); + +sort_dimension__add ("timer" , &default_cmp); +sort_dimension__add ("itimer-type" , &default_cmp);
That obviously does not match any of the sort types.
Quoted 46 lines
tools/perf/builtin-timer.c new file +46
-0+} + +static struct rb_root timer_root; + +static int timer_key_cmp (struct list_head *list,struct timer_info *l, +struct timer_info *r) +{ +struct sort_dimension *sort; +int ret =0 ; + +BUG_ON (list_empty (list)); + +list_for_each_entry (sort, list, list) { + ret = sort->cmp (l, r); +if (ret) +return ret; + } + +return ret; +} + +static void timer_traversal (struct rb_root *root, +void (*fun)(struct timer_info *)) +{ +struct rb_node *next; + + next =rb_first (root); + +while (next) { +struct timer_info *timer; + + timer = rb_entry(next, struct timer_info, node); +fun (timer); + next =rb_next (next); + } +} + +static struct timer_info * +__timer_search (struct rb_root *root,struct timer_info *timer, +struct list_head *_sort_list) +{ +struct rb_node *node = root->rb_node; + +while (node) { +struct timer_info *timer_info; +int cmp;
tools/perf/builtin-timer.c line 286
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+ + timer_info = container_of(node, struct timer_info, node); + + cmp =timer_key_cmp (_sort_list, timer, timer_info); +if (cmp >0 ) + node = node->rb_left; +else if (cmp <0 ) + node = node->rb_right; +else +return timer_info;
This looks more than odd. You search for a timer in the list by using the compare functions which are used for sorting ?
How should that work ?
Quoted 33 lines
tools/perf/builtin-timer.c new file +33
-0+ } +return NULL ; +} + +static struct timer_info * +timer_search (struct timer_info *timer) +{ +return __timer_search (&timer_root, timer, &default_cmp); +} + +static void +__timer_insert (struct rb_root *root,struct timer_info *data, +struct list_head *_sort_list) +{ +struct rb_node **new = &(root->rb_node), *parent =NULL ; + +while (*new) { +struct timer_info *this; +int cmp; + + this = container_of(*new, struct timer_info, node); + parent = *new; + + cmp =timer_key_cmp (_sort_list, data, this); + +if (cmp >0 ) + new = &((*new)->rb_left); +else + new = &((*new)->rb_right); + } + +rb_link_node (&data->node, parent, new); +rb_insert_color (&data->node, root);
tools/perf/builtin-timer.c line 329
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+} + +static void timer_insert (struct timer_info *timer_info) +{ +dprintf ("timer %p insert...\n" , timer_info->timer); +__timer_insert (&timer_root, timer_info, &default_cmp); +} + +static struct timer_info +*__timer_find (void *timer,enum timer_type type,int itimer_type)
As I said before, that's wrong. Enumerate the itimer subtypes in the timer types and you get rid of all that extra itimer_type business.
Quoted 7 lines
tools/perf/builtin-timer.c new file +7
-0+{ +struct timer_info *find =NULL ; +struct timer_info timer_info = { + .timer = timer, + .itimer_type = itimer_type, + }; +
tools/perf/builtin-timer.c line 346
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+ find =timer_search (&timer_info); +if (find && find->type != type) { + +dprintf ("find timer[%p], but type[%s] is not we expect[%s]," +"set to initializtion state.\n" , find->timer, + timer_type_string[find->type], timer_type_string[type]); + + find->type = type; + find->bug++; + find->state =TIMER_INIT ;
Why does a timer_search fail ? And why is fixing up the type if it is not matching a good idea ?
Quoted 49 lines
tools/perf/builtin-timer.c new file +49
-0+ } + +dprintf ("find timer[%p] %s\n" , timer, find ?"success" :"fail" ); +return find; +} + +static struct timer_info * +timer_findnew (void *timer,enum timer_type type,int itimer_type) +{ +struct timer_info *find; + + find =__timer_find (timer, type, itimer_type); +if (find) +return find; + + find =zalloc (sizeof (*find)); +if (!find) +die ("No memory" ); + + find->type = type; + find->timer = timer; + find->itimer_type = itimer_type; +timer_insert (find); + +return find; +} + +static struct rb_root timer_result; + +static void sort_result (void ) +{ +struct rb_node *node; + +for (;;) { +struct timer_info *data; + node =rb_first (&timer_root); +if (!node) +break ; + +rb_erase (node, &timer_root); + data = rb_entry(node, struct timer_info, node); + data->timer_avg_lat = (double )data->timer_total_lat / + (double )data->expire_nr; + data->handler_avg_time = (double )data->handler_total_time / + (double )data->expire_nr; +if (data->expire_nr) +__timer_insert (&timer_result, data, &sort_list); + } +}
tools/perf/builtin-timer.c line 405
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+ +static int timer_check_set_state (struct timer_info *timer_info, +enum timer_state new_state) +{ +enum timer_state current_state = timer_info->state; +int ret =1 ; + +/* +* if current state is TIMER_INIT, it means it's the first time we +* catched, no need check.
Check whatfor ?
Quoted 5 lines
tools/perf/builtin-timer.c new file +5
-0+*/ +if (current_state ==TIMER_INIT ) { +dprintf ("timer[%p] initialize, set state to %s\n" , + timer_info->timer, timer_state_string[new_state]); + ret =0 ;
tools/perf/builtin-timer.c line 420
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+ } + + timer_info->state = new_state; +return ret; +} + +#define FILL_RAM_FIELD_PTR (event, field, data) \ + field = raw_field_ptr(event, #field, data) +#define FILLL_RAW_FIELD_VALUE (event, field, data) \ + field = (typeof(field))raw_field_value(event, #field, data)
I can neither figure out what FILL_RAM_FIELD means nor why FILLL_RAW_FIELD needs 3 L.
tools/perf/builtin-timer.c line 428
Quoted 7 lines
tools/perf/builtin-timer.c new file +7
-0+static void *get_timer (enum timer_type type,struct event *event,void *data) +{ +if (type ==HRTIMER ) { +void *hrtimer =NULL ; + +FILL_RAM_FIELD_PTR (event, hrtimer, data); +return hrtimer;
Shudder.
return raw_field_ptr(event, "hrtimer", data);
Quoted 54 lines
tools/perf/builtin-timer.c new file +54
-0+ }else { +void *timer =NULL ; + +FILL_RAM_FIELD_PTR (event, timer, data); +return timer; + } +} + +static void +timer_hrtimer_start (enum timer_type type,void *data,struct event *event) +{ +void *timer, *function =NULL ; +u64 expires; +struct timer_info *timer_info; + + timer =get_timer (type, event, data); + +FILL_RAM_FIELD_PTR (event, function, data); +FILLL_RAW_FIELD_VALUE (event, expires, data); + +dprintf ("timer-start: Timer[%p], type:%s, function:%p, expires:%llu\n" , + timer, timer_type_string[type], function, expires); + + timer_info =timer_findnew (timer, type,MAX_ITIMER_TYPE ); +timer_check_set_state (timer_info,TIMER_START ); + timer_info->function = function; + timer_info->timer_expire_ts = expires; +} + +static void +timer_hrtimer_expires_entry (enum timer_type type,void *data, +struct event *event,u64 timestamp, +struct thread *thread) +{ +void *timer; +int ret; +u64 now; +s64 delta; +struct timer_info *timer_info; + +FILLL_RAW_FIELD_VALUE (event, now, data); + timer =get_timer (type, event, data); + timer_info =timer_findnew (timer, type,MAX_ITIMER_TYPE ); + timer_info->handler_entry_ts = timestamp; + ret =timer_check_set_state (timer_info,TIMER_EXPIRE_ENTRY ); + +dprintf ("timer-expires-entry: Timer[%p], type:%s, now:%llu, ts:%llu\n" , + timer, timer_type_string[type], now, timestamp); +if (ret ==0 ) +return ; + + delta = (s64)(now - timer_info->timer_expire_ts); +if (delta <0 ) { + timer_info->bug++;
tools/perf/builtin-timer.c line 492
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+dprintf ("timer-expires-entry: Timer[%p], type:%s," +" timer latency < 0, bug=%d.\n" , timer, + timer_type_string[type], timer_info->bug); +return ; + } + + timer_info->expire_nr++; + timer_info->timer_last_lat = delta; + timer_info->timer_total_lat += delta; +if (timer_info->timer_max_lat <= (u64 )delta) {
Why can't we just make timer_max_lat s64 so we get rid of those silly typecasts ?
tools/perf/builtin-timer.c line 495
Quoted 3 lines
tools/perf/builtin-timer.c new file +3
-0+ timer_info->timer_max_lat = delta; + timer_info->timer_max_lat_at_ts = timestamp; + timer_info->timer_max_lat_at_thread = thread;
What does us the information in which thread context the latency happend show ? Actually nothing. All it tells us that thread X was on the CPU at that point. That's useless.
Quoted 4 lines
tools/perf/builtin-timer.c new file +4
-0+static void +itimer_state_handler (void *data,struct event *event,int this_cpu__used , +u64 timestamp__used ,struct thread *thread) +{
tools/perf/builtin-timer.c line 596
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+u64 value_sec, value_usec, expires; +struct timer_info *timer_info; +void *timer =NULL ; +int which; + +FILLL_RAW_FIELD_VALUE (event, value_sec, data); +FILLL_RAW_FIELD_VALUE (event, value_usec, data); +FILLL_RAW_FIELD_VALUE (event, expires, data); +FILLL_RAW_FIELD_VALUE (event, which, data); +FILL_RAM_FIELD_PTR (event, timer, data);
This is complete obfuscated, while
value_sec = get_value(data, event, "value_sec");
is obvious.
tools/perf/builtin-timer.c line 602
Quoted 5 lines
tools/perf/builtin-timer.c new file +5
-0+ timer_info =timer_findnew (thread,ITIMER , which); + +/* itimer canceled, we skip this event */ +if (!value_sec && !value_usec) +return ;
You throw away valuable information here about canceled timers.
Quoted 15 lines
tools/perf/builtin-timer.c new file +15
-0+dprintf ("timer-start: Timer[%p], type:%s, task:%s, expires:%llu" , + timer_info->timer,"itimer" , thread->comm, expires); + +/* itimer started */ + timer_info->timer_expire_ts = expires; +timer_check_set_state (timer_info,TIMER_START ); +if (which ==ITIMER_REAL ) { + timer =timer_findnew (timer,HRTIMER ,MAX_ITIMER_TYPE ); + timer_info->itimer_hrtimer = timer; +dprintf (" hrtimer:%p\n" , timer); + }else { +dprintf ("\n" ); + } +} +
tools/perf/builtin-timer.c line 628
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+static void +itimer_expire_handler (void *data,struct event *event,int this_cpu__used , +u64 timestamp,struct thread *thread) +{ +int which; +u64 now; +pid_t pid; +struct thread *itimer_thread; +struct timer_info *timer_info; +s64 delta =0 ;
No need to initialize that.
Review
Quoted 1 line
(quoted diff) +1
-0+ int ret;
Please group identical types
int ret, which;
Quoted 18 lines
tools/perf/builtin-timer.c new file +18
-0+struct event_handler_unit { +const char *event_name; +void (*handler) (void *data,struct event *event,int this_cpu, +u64 timestamp,struct thread *thread); +} timer_handler[] = { +/* timer */ + {"timer_start" , timer_start_handler }, + {"timer_expire_entry" , timer_expire_entry_handler }, + {"timer_expire_exit" , timer_expire_exit_handler }, + +/* hrtimer */ + {"hrtimer_start" , hrtimer_start_handler }, + {"hrtimer_expire_entry" , hrtimer_expire_entry_handler }, + {"hrtimer_expire_exit" , hrtimer_expire_exit_handler}, + +/* itimer */ + {"itimer_state" , itimer_state_handler }, + {"itimer_expire" , itimer_expire_handler },
tools/perf/builtin-timer.c line 698
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+}; + +static void +process_raw_event (event_t *raw_event__used ,struct perf_session *session__used , +void *data,int cpu,u64 timestamp,struct thread *thread) +{ +struct event *event; +int type; +int i; +int handler_item =ARRAY_SIZE (timer_handler);
See above.
tools/perf/builtin-timer.c line 765
Quoted 11 lines
tools/perf/builtin-timer.c new file +11
-0+ type =trace_parse_common_type (data); + event =trace_find_event (type); + +for (i =0 ; i < handler_item; i++) +if (!strcmp (timer_handler[i].event_name, event->name)) + timer_handler[i].handler (data, event, cpu, timestamp, thread); +} + +static int read_events (void ) +{ +int err;
New line between variables and code.
tools/perf/builtin-timer.c line 768
Quoted 3 lines
tools/perf/builtin-timer.c new file +3
-0+ timer_session =perf_session__new (input_name,O_RDONLY ,0 ,NULL ); + +if (timer_session ==NULL )
if (!timer_session)
tools/perf/builtin-timer.c line 775
Quoted 7 lines
tools/perf/builtin-timer.c new file +7
-0+return -ENOMEM ; + + err =perf_session__process_events (timer_session, &event_ops); +perf_session__delete (timer_session); +return err; +} +
Thanks,
tglx
Re: [PATCH 2/4] perf_event: fix getting point
In reply to Frederic WeisbeckerQuoted 11 lines · Frederic Weisbecker
Quoted 5 lines · Xiao Guangrong
The point that got by
raw_field_ptr()is point to @data offset not tracepoint saved.Introduce
raw_field_arry()to get arry data is tracepoint save arry to it.Could you please rename it into
raw_field_array()? I feel like...a kind of loss there :)
Ah, it's my typo, i will fix it.
Quoted 54 lines
Quoted 39 lines
Signed-off-by Xiao Guangrong <[email protected]>--- tools/perf/builtin-sched.c | 3 +-- tools/perf/util/trace-event-parse.c | 25 ++++++++++++++++++++++--- tools/perf/util/trace-event.h | 3 +++ 3 files changed, 26 insertions(+), 5 deletions(-)tools/perf/builtin-sched.c +1
-2@@ -630,8 +630,7 @@ static void test_calibrations(void)#define FILL_ARRAY (ptr, array, event, data) \ do { \ - void *__array = raw_field_ptr(event, #array, data); \ - memcpy(ptr.array, __array, sizeof(ptr.array)); \ + raw_field_arry(event, #array, data, ptr.array, sizeof(ptr.array));\ } while(0)#define FILL_COMMON_FIELDS (ptr, event, data) \tools/perf/util/trace-event-parse.c +1
-1@@ -840,7 +840,7 @@ static int event_read_fields(struct event *event, struct format_field **fields) field->flags |=FIELD_IS_ARRAY ; type =read_token (&token); -while (strcmp (token,"]" ) !=0 ) { +while (strcmp (token,"]" ) !=0 ) {if (last_type ==EVENT_ITEM && type ==EVENT_ITEM ) len =2 ;@@ -1919,13 +1919,32 @@ raw_field_value(struct event *event, const char *name, void *data)void *raw_field_ptr (struct event *event,const char *name,void *data)tools/perf/util/trace-event-parse.c raw_field_value(struct event *event, const char *name, void *data) · line 1930
Quoted 10 lines
tools/perf/util/trace-event-parse.c +9
-0{ +void *ptr; +unsigned long long value; + + value =raw_field_value (event, name, data); + +if (!value) +return NULL ; + +memcpy (&ptr, &value,sizeof (ptr));Could you perhaps just do
Quoted 2 lines
ptr = (void *)value; ?
We can't do it in x86_32:
error: cast to pointer from integer of different size
Thanks,
Xiao
Quoted 21 lines · Frederic Weisbecker
Quoted 6 lines · Xiao Guangrong
Hi,
We introduce 'perf timer' in this patchset, it can analyze timer latency and timer function handle time, the usage and result is like below:
Quoted 9 lines
# perf timer record # perf timer lat --print-lat --print-handle ------------------------------------------------------------------------------------------------------- | Timer | TYPE | Avg-latency | Max-latency | Max-latency-at-TS |Max-lat-at-Task | |0xf7ad1f5c |hrtimer |996068.500 ns|1607650 ns|10270128658526 |init | |0xf7903f04 |timer |0.625 HZ|2 HZ|10270344082394 |swapper | |0xf787a05c |hrtimer |200239.500 ns|359929 ns|10269316024808 |main | |main :[ PROF]|itimer |0.000 HZ|0 HZ|10237021270557 |main | |main :[VIRTUAL]|itimer |0.000 HZ|0 HZ|10257314773501 |main |Cool! This is really a good work and a good idea.
Quoted 8 lines
Just have some neats in mind. hrtimers and timers don't have the same latency and granularity requirements.
As you show it, timers have an HZ granularity and hrtimers are about nanoseconds, and mixing them up in the same array of latency report is too messy. They don't have the same granularity/latency scope so they should
be reported separately.
Yeah, it has different unit and looks ugly :-(
but, the problem is we can't get HZ in userspace now, i'll export HZ by proc or other way and rectify 'perf timer' output in my next work
Thanks,
Xiao
Re: [PATCH 2/4] perf_event: fix getting point
In reply to Xiao GuangrongQuoted 18 lines · Xiao Guangrong
Quoted 18 lines
tools/perf/util/trace-event-parse.c raw_field_value(struct event *event, const char *name, void *data) · line 1930
Quoted 13 lines
tools/perf/util/trace-event-parse.c +9
-0@@ -1919,13 +1919,32 @@ raw_field_value(struct event *event, const char *name, void *data)void *raw_field_ptr (struct event *event,const char *name,void *data) { +void *ptr; +unsigned long long value; + + value =raw_field_value (event, name, data); + +if (!value) +return NULL ; + +memcpy (&ptr, &value,sizeof (ptr));Could you perhaps just do
Quoted 5 lines
Quoted 2 lines
ptr = (void *)value; ?
We can't do it in
x86_32:
error: cast to pointer from integer of different size
You can just do:
ptr = (void *)(unsigned long)value;
Re: [PATCH 2/4] perf_event: fix getting point
Quoted 8 lines · Frederic Weisbecker
Quoted 2 lines
We can't do it in
x86_32:
error: cast to pointer from integer of different sizeYou can just do:
ptr = (void *)(unsigned long)value;
Yeah, i forget this, it work well both in 32/64 machine.
Thanks,
Xiao
Re: [PATCH 4/4] perf/timer: 'perf timer' core code
In reply to Thomas GleixnerHi Thomas,
Sorry for many mistakes(typos and bad ideas) in this patch, i'll cook it more and be careful later. Thanks very much.
Quoted 7 lines · Thomas Gleixner
The output is confusing in several aspects:
1) Different time units:
Please use consistent time units for everything. micro seconds are fine and we definitely do not care about nanosecond fractions.
OK, i'll change ns to ms, and for timer, the unit is HZ, do we have the way to read kernel HZ in userspace? if not, i'll export HZ by proc/debugfs or other way
Quoted 7 lines
2) Timer description:
Why do we have hex addresses and process names intermingled ? Why don't we print the process/thread name which owns the timer always ? [PROF/VIRTUAL] is not a property of the Timer, it belongs into type.
Um, but not every timer has it's owner task, for example, if we start a timer in interrupt handle function, this timer in not owns any tasks. And itimer is started by userspace task so we can get it's owner, that why i print hex address for timer/hrtimer, and print task name for itimer.
Quoted 4 lines
3) Max-lat-at-Task:
What does this field tell ?
It means that which task in running when the maximum latency occurs. but, as you noticed, this is useless, i'll remove it in next version patch.
Quoted 5 lines
4) *handle:
That should be a more descriptive name, e.g. function runtime
OK, will fix
Quoted 4 lines
5) Max-lat-at-func:
Is this the callback function which ran the longest time ? Why is it named latency ? Why is it not decoded into a symbol ?
it's my typo, i'll export it using right/better way
Quoted 4 lines
Btw, fitting the output into 80chars allows to use the tool on a non graphical terminal as well.
OK, will fix
Quoted 9 lines
Also there are other metrics of timers which are interesting and should be retrieved by such a tool:
number of activated timers
number of canceled timers
number of expired timersin the form of simple statistics.
OK. will support it
Quoted 4 lines
The canceled timers are interesting also in a list, so we can see which timers are canceled after which time and how long before the expiry.
Um, i'll cook timer tracepoints to get the time when timer canceled, and support this function.
Quoted 7 lines
tools/perf/builtin-timer.c line 59
Quoted 4 lines
tools/perf/builtin-timer.c new file +4
-0+static const char *const timer_usage[] = { +"perf timer [<options>] {record|latency}" , +NULL +};Your example above uses "perf timer lat". What's correct ?
Actually, we only compare frontal 3 characters:
strncmp(argv[0], "lat", 3)
And 'perf sched' and other commands also use this way.
Quoted 7 lines
tools/perf/builtin-timer.c line 63
Quoted 3 lines
tools/perf/builtin-timer.c new file +3
-0+static const struct option latency_options[] = { +OPT_STRING ('s' ,"sort" , &sort_order,"key[,key2...]" , +"sort by key(s): " SORT_KEY ),Do we really need a sort order ? A single sort key should be sufficient.
i think it's necessary.
for example, if we interest in timer's max latency, then we can use '-s max-timer-latency' to sort it.
And, if it has many timer with the some max latency, then we can use '-s max-timer-latency,avg-timer-latency' to sort
Quoted 4 lines
I'd prefer to have a selector instead which lets me filter timer types. If I debug hrtimers then I have no interest in itimers or timer list timers.
OK, will support this filter
Quoted 8 lines
tools/perf/builtin-timer.c line 217
Quoted 5 lines
tools/perf/builtin-timer.c new file +5
-0+static LIST_HEAD (sort_list); + +static void setup_sorting (void ) +{ +char *tmp, *tok, *str =strdup (sort_order);Please hand
sort_orderin as an argument.
Sorry for my stupid question:
'sort_order' is a global variable and setup_sorting() only called one time, why need hand sort_order in as an argument?
Quoted 23 lines
Quoted 9 lines
tools/perf/builtin-timer.c new file +9
-0+static struct timer_info * +__timer_search (struct rb_root *root,struct timer_info *timer, +struct list_head *_sort_list) +{ +struct rb_node *node = root->rb_node; + +while (node) { +struct timer_info *timer_info; +int cmp;tools/perf/builtin-timer.c line 286
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+ + timer_info = container_of(node, struct timer_info, node); + + cmp =timer_key_cmp (_sort_list, timer, timer_info); +if (cmp >0 ) + node = node->rb_left; +else if (cmp <0 ) + node = node->rb_right; +else +return timer_info;This looks more than odd. You search for a timer in the list by using the compare functions which are used for sorting ?
Quoted 2 lines
How should that work ?
We put/get timer in a rb-tree base on the specify order, for example:
we default use this order:
sort_dimension__add("timer", &default_cmp);sort_dimension__add("itimer-type", &default_cmp);
if timer_info->timer is bigger, we put it to left child, littler to right child, if the timer_info->timer is the same, then we compare timer_info->itimer_type.
Quoted 12 lines
Quoted 12 lines
tools/perf/builtin-timer.c new file +12
-0+{ +struct timer_info *find =NULL ; +struct timer_info timer_info = { + .timer = timer, + .itimer_type = itimer_type, + }; + + find =timer_search (&timer_info); +if (find && find->type != type) { + + dprintf("find timer[%p], but type[%s] is not we expect[%s]," + "set to initializtion state.\n", find->timer,
Quoted 9 lines
tools/perf/builtin-timer.c line 346
Quoted 5 lines
tools/perf/builtin-timer.c new file +5
-0+ timer_type_string[find->type], timer_type_string[type]); + + find->type = type; + find->bug++; + find->state =TIMER_INIT ;Why does a
timer_searchfail ? And why is fixing up the type if it is not matching a good idea ?
We search timer base on timer_info->timer and timer_info->itimer_type(not timer_info->type), if we find the timer's type is changed(for example, the timer is "ITIMER" before, and change to "HRTIMER" later), is should a bug. this case is hardly to happen but should catch it.
Quoted 10 lines
tools/perf/builtin-timer.c line 428
Quoted 7 lines
tools/perf/builtin-timer.c new file +7
-0+static void *get_timer (enum timer_type type,struct event *event,void *data) +{ +if (type ==HRTIMER ) { +void *hrtimer =NULL ; + +FILL_RAM_FIELD_PTR (event, hrtimer, data); +return hrtimer;Shudder.
Quoted 2 lines
return raw_field_ptr(event, "hrtimer", data);
Yeah, it's a clear way.
Quoted 17 lines
Quoted 4 lines
tools/perf/builtin-timer.c new file +4
-0+static void +itimer_state_handler (void *data,struct event *event,int this_cpu__used , +u64 timestamp__used ,struct thread *thread) +{tools/perf/builtin-timer.c line 596
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+u64 value_sec, value_usec, expires; +struct timer_info *timer_info; +void *timer =NULL ; +int which; + +FILLL_RAW_FIELD_VALUE (event, value_sec, data); +FILLL_RAW_FIELD_VALUE (event, value_usec, data); +FILLL_RAW_FIELD_VALUE (event, expires, data); +FILLL_RAW_FIELD_VALUE (event, which, data); +FILL_RAM_FIELD_PTR (event, timer, data);This is complete obfuscated, while
Quoted 4 lines
value_sec = get_value(data, event, "value_sec");is obvious.
Sorry, i cannot get this. As i understand:
#define FILL_RAW_FIELD_VALUE(event, field, data) \
field = (typeof(field))raw_field_value(event, #field, data)
After FILL_RAW_FIELD_VALUE(event, value_sec, data) expanded, it's:
value_sec = raw_field_value(event, "value_sec", data)
Why it's wrong? :-(
Quoted 8 lines
tools/perf/builtin-timer.c line 602
Quoted 5 lines
tools/perf/builtin-timer.c new file +5
-0+ timer_info =timer_findnew (thread,ITIMER , which); + +/* itimer canceled, we skip this event */ +if (!value_sec && !value_usec) +return ;You throw away valuable information here about canceled timers.
We are not catch *_cancel event in this patch, i'll catch it to support 'number of canceled timers' in the next version.
Thanks,
Xiao
Quoted 33 lines · Xiao Guangrong
Quoted 21 lines · Frederic Weisbecker
Quoted 6 lines · Xiao Guangrong
Hi,
We introduce 'perf timer' in this patchset, it can analyze timer latency and timer function handle time, the usage and result is like below:
Quoted 9 lines
# perf timer record # perf timer lat --print-lat --print-handle ------------------------------------------------------------------------------------------------------- | Timer | TYPE | Avg-latency | Max-latency | Max-latency-at-TS |Max-lat-at-Task | |0xf7ad1f5c |hrtimer |996068.500 ns|1607650 ns|10270128658526 |init | |0xf7903f04 |timer |0.625 HZ|2 HZ|10270344082394 |swapper | |0xf787a05c |hrtimer |200239.500 ns|359929 ns|10269316024808 |main | |main :[ PROF]|itimer |0.000 HZ|0 HZ|10237021270557 |main | |main :[VIRTUAL]|itimer |0.000 HZ|0 HZ|10257314773501 |main |Cool! This is really a good work and a good idea.
Quoted 8 lines
Just have some neats in mind. hrtimers and timers don't have the same latency and granularity requirements.
As you show it, timers have an HZ granularity and hrtimers are about nanoseconds, and mixing them up in the same array of latency report is too messy. They don't have the same granularity/latency scope so they should
be reported separately.
Quoted 4 lines
Yeah, it has different unit and looks ugly :-(
but, the problem is we can't get HZ in userspace now, i'll export HZ by proc or other way and rectify 'perf timer' output in my next work
We could export HZ as part of the ftrace event format file, in /debug/tracing/events/timers/ or so. Not sure what the best way would be to do it (without causing runtime overhead in event processing), but we should reduce our reliance on /proc and export such instrumentation information via a single, coherent interface.
There's a couple of other 'system settings' values that would be useful to expose in such a way - for example the number of online CPUs. (in fact that could be exposed as a set of online/offline events plus a way to read the current count as well)
Ingo
Quoted 12 lines · Ingo Molnar
Quoted 5 lines
Quoted 1 line
Yeah, it has different unit and looks ugly :-(
but, the problem is we can't get HZ in userspace now, i'll export HZ by proc or other way and rectify 'perf timer' output in my next work
We could export HZ as part of the ftrace event format file, in /debug/tracing/events/timers/ or so. Not sure what the best way would be to do it (without causing runtime overhead in event processing), but we should reduce our reliance on /proc and export such instrumentation information via a single, coherent interface.
Hi ingo,
Um, if we export this value in tracepoint format, it should add for every timer's tracepoint.
Quoted 5 lines
There's a couple of other 'system settings' values that would be useful to expose in such a way - for example the number of online CPUs. (in fact that could be exposed as a set of online/offline events plus a way to read the current count as well)
How about export those values in debugfs/tracing/*? perf tracepoint self depend on this filesystem.
Thanks,
Xiao
Quoted 16 lines · Xiao Guangrong
Quoted 11 lines · Ingo Molnar
Quoted 5 lines
Quoted 1 line
Yeah, it has different unit and looks ugly :-(
but, the problem is we can't get HZ in userspace now, i'll export HZ by proc or other way and rectify 'perf timer' output in my next work
We could export HZ as part of the ftrace event format file, in /debug/tracing/events/timers/ or so. Not sure what the best way would be to do it (without causing runtime overhead in event processing), but we should reduce our reliance on /proc and export such instrumentation information via a single, coherent interface.
Quoted 4 lines
Hi ingo,
Um, if we export this value in tracepoint format, it should add for every timer's tracepoint.
Not if it's a special timer tracepoint 'HZ' dedicated for this. (We might even get events through it on dynticks kernels, or on dynamic-sched-tick systems [in the future] when the scheduler HZ value gets modified - or so.)
Quoted 7 lines
Quoted 4 lines
There's a couple of other 'system settings' values that would be useful to expose in such a way - for example the number of online CPUs. (in fact that could be exposed as a set of online/offline events plus a way to read the current count as well)
How about export those values in debugfs/tracing/*? perf tracepoint self depend on this filesystem.
No - in the long run most of those APIs will go away, and what remains are the ftrace event format descriptors in essence. (Which we then might move to /proc/events/ or so.)
Ingo
Re: [PATCH 4/4] perf/timer: 'perf timer' core code
In reply to Xiao GuangrongXiao,
Quoted 3 lines · Xiao Guangrong
Sorry for many mistakes(typos and bad ideas) in this patch, i'll cook it more and be careful later. Thanks very much.
Nothing to be sorry about. That's why we review code.
Quoted 10 lines
Quoted 8 lines · Thomas Gleixner
The output is confusing in several aspects:
2) Timer description:
Why do we have hex addresses and process names intermingled ? Why don't we print the process/thread name which owns the timer always ? [PROF/VIRTUAL] is not a property of the Timer, it belongs into type.
Quoted 4 lines
Um, but not every timer has it's owner task, for example, if we start a timer in interrupt handle function, this timer in not owns any tasks. And itimer is started by userspace task so we can get it's owner, that why i print hex address for timer/hrtimer, and print task name for itimer.
Well, lot's of timers have an owner task. At least all user space related ones. And if the timer is rearmed in interrupt context, then this does not change the owner at all.
Quoted 9 lines
Quoted 8 lines
tools/perf/builtin-timer.c line 217
Quoted 5 lines
tools/perf/builtin-timer.c new file +5
-0+static LIST_HEAD (sort_list); + +static void setup_sorting (void ) +{ +char *tmp, *tok, *str =strdup (sort_order);Please hand
sort_orderin as an argument.
Quoted 3 lines
Sorry for my stupid question:
'sort_order' is a global variable andsetup_sorting()only called one time, why need handsort_orderin as an argument?
Fair enough.
Quoted 6 lines
Quoted 2 lines
How should that work ?
We put/get timer in a rb-tree base on the specify order, for example:
we default use this order:
Quoted 6 lines
sort_dimension__add("timer", &default_cmp);sort_dimension__add("itimer-type", &default_cmp);if
timer_info->timeris bigger, we put it to left child, littler to right child, if thetimer_info->timeris the same, then we comparetimer_info->itimer_type.
Hmm, I wonder whether a hash table would be more efficient for the recording side.
Quoted 22 lines
Quoted 12 lines
Quoted 12 lines
tools/perf/builtin-timer.c new file +12
-0+{ +struct timer_info *find =NULL ; +struct timer_info timer_info = { + .timer = timer, + .itimer_type = itimer_type, + }; + + find =timer_search (&timer_info); +if (find && find->type != type) { + + dprintf("find timer[%p], but type[%s] is not we expect[%s]," + "set to initializtion state.\n", find->timer,Quoted 9 lines
tools/perf/builtin-timer.c line 346
Quoted 5 lines
tools/perf/builtin-timer.c new file +5
-0+ timer_type_string[find->type], timer_type_string[type]); + + find->type = type; + find->bug++; + find->state =TIMER_INIT ;Why does a
timer_searchfail ? And why is fixing up the type if it is not matching a good idea ?
Quoted 5 lines
We search timer base on
timer_info->timerandtimer_info->itimer_type(not timer_info->type), if we find the timer's type ischanged(for example, the timer is "ITIMER" before, and change to "HRTIMER" later), is should a bug. this case is hardly to happen but should catch it.
No, it's not a bug at all. You _can_ have a hrtimer and a timer_list timer at the same address in a trace. There are two ways to make that happen:
1) kmalloc'ed memory contains a timer_list. timer operation is done, memory is kfreed. Now another kmalloc gets the just freed memory and has a hrtimer at the same address which was used by the timer_list before.
2) timer_list and hrtimer are also allocated on stack. There is no guarantee that they are at different addresses.
Quoted 39 lines
Quoted 10 lines
tools/perf/builtin-timer.c line 428
Quoted 7 lines
tools/perf/builtin-timer.c new file +7
-0+static void *get_timer (enum timer_type type,struct event *event,void *data) +{ +if (type ==HRTIMER ) { +void *hrtimer =NULL ; + +FILL_RAM_FIELD_PTR (event, hrtimer, data); +return hrtimer;Shudder.
Quoted 2 lines
return raw_field_ptr(event, "hrtimer", data);Yeah, it's a clear way.
Quoted 17 lines
Quoted 4 lines
tools/perf/builtin-timer.c new file +4
-0+static void +itimer_state_handler (void *data,struct event *event,int this_cpu__used , +u64 timestamp__used ,struct thread *thread) +{tools/perf/builtin-timer.c line 596
Quoted 10 lines
tools/perf/builtin-timer.c new file +10
-0+u64 value_sec, value_usec, expires; +struct timer_info *timer_info; +void *timer =NULL ; +int which; + +FILLL_RAW_FIELD_VALUE (event, value_sec, data); +FILLL_RAW_FIELD_VALUE (event, value_usec, data); +FILLL_RAW_FIELD_VALUE (event, expires, data); +FILLL_RAW_FIELD_VALUE (event, which, data); +FILL_RAM_FIELD_PTR (event, timer, data);This is complete obfuscated, while
Quoted 4 lines
value_sec = get_value(data, event, "value_sec");is obvious.
Sorry, i cannot get this. As i understand:
Quoted 7 lines
#define
FILL_RAW_FIELD_VALUE(event, field, data)\
field = (typeof(field))raw_field_value(event, #field, data)After
FILL_RAW_FIELD_VALUE(event, value_sec, data)expanded, it's:
value_sec=raw_field_value(event, "value_sec", data)Why it's wrong? :-(
Simply because the macro hides the fact that this is an assignment of a value to a variable. That makes the code harder to read.
FILLL_RAW_FIELD_VALUE(event, value_sec, data);
vs.
value_sec = get_value(data, event, "value_sec");
The latter is fast to parse and entirely clear.
Btw, you agreed above that the open coded call of raw_field_value() is clearer than the macro magic. :)
Thanks,
tglx
Re: [PATCH 4/4] perf/timer: 'perf timer' core code
Hi Thomas,
Quoted 3 lines · Thomas Gleixner
Nothing to be sorry about. That's why we review code.
Thanks.
Quoted 9 lines
Quoted 4 lines
Um, but not every timer has it's owner task, for example, if we start a timer in interrupt handle function, this timer in not owns any tasks. And itimer is started by userspace task so we can get it's owner, that why i print hex address for timer/hrtimer, and print task name for itimer.
Well, lot's of timers have an owner task. At least all user space related ones. And if the timer is rearmed in interrupt context, then this does not change the owner at all.
Sorry, i'm confused, for example, has below sequence:
Task1 running----->| (interrupt)
|------------- start timerT(start timerT in interrupt handler)
......
( After a while, schedule to another task, and interruption coming )
......
Task2 running----->| (interrupt)
|------------- start timerT again
Then, which task is the timerT owner?
Am I missed something?
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
How should that work ?
We put/get timer in a rb-tree base on the specify order, for example:
we default use this order:
sort_dimension__add("timer", &default_cmp);sort_dimension__add("itimer-type", &default_cmp);
Quoted 7 lines
Quoted 3 lines
if
timer_info->timeris bigger, we put it to left child, littler to right child, if thetimer_info->timeris the same, then we comparetimer_info->itimer_type.Hmm, I wonder whether a hash table would be more efficient for the recording side.
Um. i'll record it address your way.
Quoted 11 lines
Quoted 6 lines
We search timer base on
timer_info->timerandtimer_info->itimer_type(not timer_info->type), if we find the timer's type ischanged(for example, the timer is "ITIMER" before, and change to "HRTIMER" later), is should a bug. this case is hardly to happen but should catch it.No, it's not a bug at all. You _can_ have a hrtimer and a
timer_listtimer at the same address in a trace. There are two ways to make that happen:
Quoted 8 lines
1) kmalloc'ed memory contains a
timer_list. timer operation is done, memory is kfreed. Now another kmalloc gets the just freed memory and has a hrtimer at the same address which was used by thetimer_listbefore.2)
timer_listand hrtimer are also allocated on stack. There is no guarantee that they are at different addresses.
Yeah, my mistake.
Quoted 9 lines
Simply because the macro hides the fact that this is an assignment of a value to a variable. That makes the code harder to read.
FILLL_RAW_FIELD_VALUE(event, value_sec, data);
vs.value_sec = get_value(data, event, "value_sec");The latter is fast to parse and entirely clear.
Yeah.
Quoted 3 lines
Btw, you agreed above that the open coded call of
raw_field_value()is clearer than the macro magic. :)
Sorry, i misunderstand your mean before.
Thanks,
Xiao
v2 cover perf_event: introduce 'perf timer' to analyze timer's behavior
In reply to Frederic WeisbeckerHi,
We introduce 'perf timer' in this patchset, it can report number of activated/expired/canceled timers, timer latency and timer function runtime.
Changlog v1->v2:
- export HZ in timer's tracepoint, we can use it to get TIMER/ITIMER_VIRTUAL/ITIMER_PROF exact latency as Ingo's suggestion
- rename 'perf timer latency' to 'perf timer report', because it not only show timer latency but also show timer function runtime, timer activated/canceled/expired statistics and canceled timer list(TODO)
Below changes are all from Thomas's suggestion:
- fix a lot of typos and bad ideas
- use hash table instead of RB tree to record timer
- let output information more readable
- report number of activated/canceled/expired timers
- support to filter timer types:
'--type' can do it
TODO:
- Show canceled timer list that is suggested by Thomas
- Any suggestion are welcome.
===============================================
Usage:
- record timer events:
#perf timer record
#^C
- show timer statistics:
#perf timer report
Statistics
==========
Activated timers number:
HRTIMER: 37199 TIMER: 828 ITIMER_PROF: 2 ITIMER_VIRTUAL: 2 ITIMER_REAL: 18
Expired timers number:
HRTIMER: 34912 TIMER: 617 ITIMER_PROF: 1 ITIMER_VIRTUAL: 1 ITIMER_REAL: 1
Canceled timers number:
HRTIMER: 37200 TIMER: 829 ITIMER_PROF: 1 ITIMER_VIRTUAL: 1 ITIMER_REAL: 17
- show timer latency:
#perf timer report --print-lat
Timer Latency List
[ Abbreviations: Avg = average; lat = latency; ts = timestamp ]
-------------------------------------------------------------------------
Timer | TYPE | Avg-lat ms| Max-lat ms| Max-lat-ts s
-------------------------------------------------------------------------
smbd |HRTIMER |1.563 |1.563 |40666.379 irqbalance |HRTIMER |0.842 |2.484 |40670.231 ...... vi |HRTIMER |0.123 |0.123 |40637.065 events/1 |TIMER |168.420 |1008.000 |40677.006 events/0 |TIMER |87.759 |1008.000 |40618.009 ...... main |ITIMER_PROF |0.000 |0.000 |40661.023 main |ITIMER_VIRTUAL |0.000 |0.000 |40642.829 main |ITIMER_REAL |0.133 |0.133 |40622.731
- show timer function runtime:
#perf timer report --print-runtime
Timer Function Runtime List
[ Abbreviations: Avg = average; func = function; rt = runtime; ts = timestamp ]
-------------------------------------------------------------------------------
Timer | Type |Avg-rt ms|Max-rt ms| Max-rt-func | Max-rt-ts s
-------------------------------------------------------------------------------
smbd |HRTIMER|0.020 |0.020 |hrtimer_wakeup |40666.379 irqbalance |HRTIMER|0.012 |0.020 |hrtimer_wakeup |40670.231 ...... events/1 |TIMER |0.009 |0.017 |delayed_work_timer_fn |40646.090 events/0 |TIMER |0.009 |0.091 |delayed_work_timer_fn |40651.105
- you can use '--print-lat' and '--print-runtime' at the same time:
#perf timer report --print-runtime --print-lat
Timer Latency List
[ Abbreviations: Avg = average; lat = latency; ts = timestamp ]
-------------------------------------------------------------------------
Timer | TYPE | Avg-lat ms| Max-lat ms| Max-lat-ts s
-------------------------------------------------------------------------
smbd |HRTIMER |1.563 |1.563 |40666.379 irqbalance |HRTIMER |0.842 |2.484 |40670.231 ...... Timer Function Runtime List
[ Abbreviations: Avg = average; func = function; rt = runtime; ts = timestamp ]
-------------------------------------------------------------------------------
Timer | Type |Avg-rt ms|Max-rt ms| Max-rt-func | Max-rt-ts s
-------------------------------------------------------------------------------
smbd |HRTIMER|0.020 |0.020 |hrtimer_wakeup |40666.379 irqbalance |HRTIMER|0.012 |0.020 |hrtimer_wakeup |40670.231
- Select timer type which you are interesting:
#perf timer report --print-lat --type timer
Timer Latency List
[ Abbreviations: Avg = average; lat = latency; ts = timestamp ]
-------------------------------------------------------------------------
Timer | TYPE | Avg-lat ms| Max-lat ms| Max-lat-ts s
-------------------------------------------------------------------------
events/1 |TIMER |168.420 |1008.000 |40677.006 events/0 |TIMER |87.759 |1008.000 |40618.009 events/1 |TIMER |57.000 |153.000 |40634.571
You can specify more types, such as: '--type timer,hrtimer'
PATCH v2 1/5 perf_event: fix getting point
The point that got by raw_field_ptr() is point to @data offset not tracepoint saved.
Introduce raw_field_array() to get array data
Changlog v1->v2:
- fix the typo 'arry' -> 'array' that is pointed out by Frederic Weisbecker
- use '(void *)(unsigned long)value' instead of 'memcpy()' as Frederic Weisbecker's suggestion
--- tools/perf/builtin-sched.c | 3 +-- tools/perf/util/trace-event-parse.c | 23 ++++++++++++++++++++--- tools/perf/util/trace-event.h | 3 +++ 3 files changed, 24 insertions(+), 5 deletions(-)
tools/perf/builtin-sched.c +1 -2
@@ -630,8 +630,7 @@ static void test_calibrations(void)#define FILL_ARRAY (ptr, array, event, data) \ do { \ - void *__array = raw_field_ptr(event, #array, data); \ - memcpy(ptr.array, __array, sizeof(ptr.array)); \ + raw_field_array(event, #array, data, ptr.array, sizeof(ptr.array));\ } while(0)#define FILL_COMMON_FIELDS (ptr, event, data) \
tools/perf/util/trace-event-parse.c +20 -3
@@ -840,7 +840,7 @@ static int event_read_fields(struct event *event, struct format_field **fields) field->flags |=FIELD_IS_ARRAY ; type =read_token (&token); -while (strcmp (token,"]" ) !=0 ) { +while (strcmp (token,"]" ) !=0 ) {if (last_type ==EVENT_ITEM && type ==EVENT_ITEM ) len =2 ;
@@ -1919,13 +1919,30 @@ raw_field_value(struct event *event, const char *name, void *data)void *raw_field_ptr (struct event *event,const char *name,void *data) { +unsigned long long value; + + value =raw_field_value (event, name, data); + +if (!value) +return NULL ; + +return (void *)(unsigned long )value; +} + +unsigned long long +raw_field_array (struct event *event,const char *name,void *data, +void *array,int array_size) +{ +int len;struct format_field *field; field =find_any_field (event, name);if (!field) -return NULL ; +return 0ULL ; -return data + field->offset; + len =min (array_size, field->size); +memcpy (array, data + field->offset, len); +return len; }static int get_common_info (const char *type,int *offset,int *size)
tools/perf/util/trace-event.h +3 -0
@@ -255,6 +255,9 @@ unsigned long long read_size(void *ptr, int size);unsigned long long raw_field_value (struct event *event,const char *name,void *data);void *raw_field_ptr (struct event *event,const char *name,void *data); +unsigned long long +raw_field_array (struct event *event,const char *name,void *data, +void *array,int array_size);unsigned long long eval_flag (const char *flag);int read_tracing_data (int fd,struct perf_event_attr *pattrs,int nb_events);
Signature · 1 line
1.6.1.2
PATCH v2 2/5 : trace_event: export HZ in timer's tracepoint format
In reply to Xiao GuangrongExport HZ in timer's tracepoint, we can use it to get TIMER/ITIMER_VIRTUAL/ITIMER_PROF exact latency and it's suggested by Ingo
--- include/trace/events/timer.h | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)
include/trace/events/timer.h +4 -1
@@ -74,14 +74,17 @@ TRACE_EVENT(timer_expire_entry,TP_STRUCT__entry (__field (void *,timer ) __field( unsigned long, now ) +__field ( int, hz ) ),TP_fast_assign ( __entry->timer = timer; __entry->now = jiffies; + __entry->hz =HZ ; ), -TP_printk ("timer=%p now=%lu" , __entry->timer, __entry->now) +TP_printk ("timer=%p now=%lu HZ=%d" , __entry->timer, __entry->now, + __entry->hz) );/**
Signature · 1 line
1.6.1.2
PATCH v2 3/5 trace_event: record task' real_timer in itimer_state tracepoint
In reply to Xiao GuangrongWhen itimer is ITIMER_REAL, we should cooperate with hrtimer's tracepoit to get it's latency, this patch can let us get itimer's hrtimer easy
--- include/trace/events/timer.h | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-)
include/trace/events/timer.h +6 -2
@@ -285,6 +285,7 @@ TRACE_EVENT(itimer_state,TP_STRUCT__entry (__field (int ,which ) + __field( void *, timer )__field ( cputime_t, expires )__field ( long, value_sec ) __field( long, value_usec )
@@ -294,6 +295,8 @@ TRACE_EVENT(itimer_state,TP_fast_assign ( __entry->which = which; + __entry->timer = which ==ITIMER_REAL ? + ¤t->signal->real_timer :NULL ; __entry->expires = expires; __entry->value_sec = value->it_value.tv_sec; __entry->value_usec = value->it_value.tv_usec;
@@ -301,8 +304,9 @@ TRACE_EVENT(itimer_state, __entry->interval_usec = value->it_interval.tv_usec; ), -TP_printk ("which=%d expires=%llu it_value=%ld.%ld it_interval=%ld.%ld" , - __entry->which, (unsigned long long )__entry->expires, +TP_printk ("which=%d timer=%p expires=%llu it_value=%ld.%ld it_interval=" +"%ld.%ld" , __entry->which, __entry->timer, + (unsigned long long )__entry->expires, __entry->value_sec, __entry->value_usec, __entry->interval_sec, __entry->interval_usec) );
Signature · 1 line
1.6.1.2
PATCH v2 4/5 perf/timer: add document for 'perf timer'
Add document for 'perf timer', it let people know how to use it
--- tools/perf/Documentation/perf-timer.txt | 50 +++++++++++++++++++++++++++++++ 1 files changed, 50 insertions(+), 0 deletions(-) create mode 100644 tools/perf/Documentation/perf-timer.txt
tools/perf/Documentation/perf-timer.txt new file +50 -0
@@ -0,0 +1,50 @@ +perf-timer(1) +============== + +NAME +---- +perf-timer - Tool to trace/measure timer properties + +SYNOPSIS +-------- +[verse] +'perf timer' {record|report} [<options>] + +DESCRIPTION +----------- +There are two variants of perf timer: + + 'perf timer record <command>' to record the timer events + of an arbitrary workload. + + 'perf timer report' to report number of activated/expired/canceled + timers, timer latency and timer function runtime. + +OPTIONS +------- +-D:: +--dump-raw-trace=:: + Display verbose dump of the sched data. + +-i <file>:: +--input=<file>:: + Select the input file (default: perf.data) + +-s <key[,key2...]>:: +--sort=<key[,key2...]>:: + Sort the output (default: type, avg-lat, max-lat, + avg-runtime, max-runtime) + +--print-lat:: + Print timer latency + +--print-runtime:: + Print timer function runtime + +--type:: + Timer type selector, you can specify one or more types + (timer, hrtimer, itimer-real, itimer-virtual, itimer-prof) + +SEE ALSO +-------- +linkperf:perf-record[1]
Signature · 1 line
1.6.1.2
PATCH v2 5/5 perf/timer: add 'perf timer' core code
It's the core code of 'perf timer', it can report number of activated/expired/canceled timers, timer latency and timer function runtime
Sample output:
- Record timer event:
#perf timer record
^C
- Show timer latency:
#perf timer report --print-lat
Timer Latency List
[ Abbreviations: Avg = average; lat = latency; ts = timestamp ]
-------------------------------------------------------------------------
Timer | TYPE | Avg-lat ms| Max-lat ms| Max-lat-ts s
-------------------------------------------------------------------------
swapper |HRTIMER |0.180 |2.404 |42722.586 swapper |TIMER |2.000 |2.000 |42722.586 events/0 |TIMER |2.000 |2.000 |42723.002 events/0 |TIMER |0.000 |0.000 |42723.641 sshd |TIMER |0.000 |0.000 |42723.641
Statistics
==========
Activated timers number:
HRTIMER: 105 TIMER: 11 ITIMER_PROF: 0 ITIMER_VIRTUAL: 0 ITIMER_REAL: 0
Expired timers number:
HRTIMER: 94 TIMER: 12 ITIMER_PROF: 0 ITIMER_VIRTUAL: 0 ITIMER_REAL: 0
Canceled timers number:
HRTIMER: 105 TIMER: 12 ITIMER_PROF: 0 ITIMER_VIRTUAL: 0 ITIMER_REAL: 0
- Show timer function runtime:
#perf timer report --print-runtime
Timer Function Runtime List
[ Abbreviations: Avg = average; func = function; rt = runtime; ts = timestamp ]
-------------------------------------------------------------------------------
Timer | Type |Avg-rt ms|Max-rt ms| Max-rt-func | Max-rt-ts s
-------------------------------------------------------------------------------
swapper |HRTIMER|0.010 |0.026 |tick_sched_timer |42722.586 0xc15f2b70 |HRTIMER|0.009 |0.009 |Not - Catch |42722.000 swapper |TIMER |0.058 |0.111 |tcp_write_timer |42721.930 events/0 |TIMER |0.007 |0.010 |delayed_work_timer_fn |42723.002 swapper |TIMER |0.037 |0.037 |dev_watchdog |42723.002 events/0 |TIMER |0.028 |0.028 |delayed_work_timer_fn |42723.641 swapper |TIMER |0.023 |0.023 |pcnet32_watchdog |42722.001 events/1 |TIMER |0.014 |0.014 |Not - Catch |42723.003 events/0 |TIMER |0.010 |0.010 |Not - Catch |42722.001 sshd |TIMER |0.009 |0.009 |delayed_work_timer_fn |42723.641 swapper |TIMER |0.004 |0.004 |Not - Catch |42721.638 events/1 |TIMER |0.003 |0.003 |Not - Catch |42723.003
Statistics
==========
Activated timers number:
HRTIMER: 105 TIMER: 11 ITIMER_PROF: 0 ITIMER_VIRTUAL: 0 ITIMER_REAL: 0
Expired timers number:
HRTIMER: 94 TIMER: 12 ITIMER_PROF: 0 ITIMER_VIRTUAL: 0 ITIMER_REAL: 0
Canceled timers number:
HRTIMER: 105 TIMER: 12 ITIMER_PROF: 0 ITIMER_VIRTUAL: 0 ITIMER_REAL: 0
- Specify types which you are interesting and choose one or more of "timer, hrtimer, itimer-real, itimer-virtual, itimer-prof"
Timer Latency List
[ Abbreviations: Avg = average; lat = latency; ts = timestamp ]
-------------------------------------------------------------------------
Timer | TYPE | Avg-lat ms| Max-lat ms| Max-lat-ts s
-------------------------------------------------------------------------
swapper |TIMER |2.000 |2.000 |42722.586 events/0 |TIMER |2.000 |2.000 |42723.002 events/0 |TIMER |0.000 |0.000 |42723.641 sshd |TIMER |0.000 |0.000 |42723.641
Changlog v1->v2:
- rename 'perf timer latency' to 'perf timer report', because it not only show timer latency but also show timer function runtime, timer activated/canceled/expired statistics and canceled timer list(TODO)
Below changes are all from Thomas's suggestion:
- fix a lot of typos and bad ideas
- use hash table instead of RB tree to record timer
- let output information more readable
- report number of activated/canceled/expired timers
- support to filter timer types:
'--type' can do it
TODO:
- Show canceled timer list that is suggested by Thomas
- Any suggestion are welcome.
--- tools/perf/Makefile | 1 + tools/perf/builtin-timer.c | 1000 +++++++++++++++++++++++++++++++++++++++++++ tools/perf/builtin.h | 1 + tools/perf/command-list.txt | 1 + tools/perf/perf.c | 1 + 5 files changed, 1004 insertions(+), 0 deletions(-) create mode 100644 tools/perf/builtin-timer.c
tools/perf/Makefile +1 -0
@@ -440,6 +440,7 @@ BUILTIN_OBJS += builtin-top.o BUILTIN_OBJS += builtin-trace.o BUILTIN_OBJS += builtin-probe.o BUILTIN_OBJS += builtin-kmem.o +BUILTIN_OBJS += builtin-timer.o PERFLIBS = $(LIB_FILE)
tools/perf/builtin-timer.c new file +1000 -0
1,000 lines not on this page · raw message
tools/perf/builtin.h +1 -0
@@ -30,5 +30,6 @@ extern int cmd_trace(int argc, const char **argv, const char *prefix);extern int cmd_version (int argc,const char **argv,const char *prefix);extern int cmd_probe (int argc,const char **argv,const char *prefix);extern int cmd_kmem (int argc,const char **argv,const char *prefix); +extern int cmd_timer (int argc,const char **argv,const char *prefix);#endif
tools/perf/command-list.txt +1 -0
@@ -16,3 +16,4 @@ perf-top mainporcelain common perf-trace mainporcelain common perf-probe mainporcelain common perf-kmem mainporcelain common +perf-timer mainporcelain common
tools/perf/perf.c +1 -0
@@ -301,6 +301,7 @@ static void handle_internal_command(int argc, const char **argv) { "sched", cmd_sched, 0 }, { "probe", cmd_probe, 0 }, { "kmem", cmd_kmem, 0 }, + { "timer", cmd_timer, 0 }, };unsigned int i;static const char ext[] =STRIP_EXTENSION ;
Signature · 1 line
1.6.1.2
Re: [PATCH v2 2/5]: trace_event: export HZ in timer's tracepoint format
In reply to Xiao GuangrongAh, sorry for this title error. :-(
s/[PATCH v2 2/5]:/[PATCH v2 2/5]
Quoted 4 lines · Xiao Guangrong
Export HZ in timer's tracepoint, we can use it to get TIMER/ITIMER_VIRTUAL/ITIMER_PROF exact latency and it's suggested by Ingo
Re: [PATCH v2 2/5]: trace_event: export HZ in timer's tracepoint format
In reply to Xiao GuangrongQuoted 20 lines · Xiao Guangrong
Export HZ in timer's tracepoint, we can use it to get TIMER/ITIMER_VIRTUAL/ITIMER_PROF exact latency and it's suggested by Ingo
Signed-off-by Xiao Guangrong <[email protected]>--- include/trace/events/timer.h | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)include/trace/events/timer.h +1
-0@@ -74,14 +74,17 @@ TRACE_EVENT(timer_expire_entry,TP_STRUCT__entry (__field (void *,timer ) __field( unsigned long, now ) +__field ( int, hz ) ),
include/trace/events/timer.h TRACE_EVENT(timer_expire_entry, · line 88
Quoted 10 lines
include/trace/events/timer.h +3
-1TP_fast_assign ( __entry->timer = timer; __entry->now = jiffies; + __entry->hz =HZ ; ), -TP_printk ("timer=%p now=%lu" , __entry->timer, __entry->now) +TP_printk ("timer=%p now=%lu HZ=%d" , __entry->timer, __entry->now, + __entry->hz) );
I think we can do something slightly different and more efficient: just create a new timer event to report the value of HZ.
That way we dont clutter the timer_expire_entry record format with a repetitive HZ field. It's an extra 4 bytes overhead: that has to be written, passed along, copied and thrown away in 99.9999% of the cases - such overhead should be avoided.
If you created a special timer_params event, which would produce precisely one event when triggered via say a new perf ioctl. I.e. add something like this to perf_event.h:
#define PERF_EVENT_IOC_INJECT _IOW('$', 7, __u64)and add code to kernel/perf_event.c's perf_ioctl() function that takes that __u64 parameter as an event ID and injects an 'artificial' event.
Such a new feature would be useful for other things as well: backtesting rare events, injecting other types of 'parameter/query events', etc.
There might be more details to this, but it would be a useful scheme IMO - and it would still integrate nicely with the whole ftrace event enumeration scheme so tooling support would be easier.
What do you think?
Ingo
Re: [PATCH v2 2/5]: trace_event: export HZ in timer's tracepoint format
Quoted 18 lines · Ingo Molnar
I think we can do something slightly different and more efficient: just create a new timer event to report the value of HZ.
That way we dont clutter the
timer_expire_entryrecord format with a repetitive HZ field. It's an extra 4 bytes overhead: that has to be written, passed along, copied and thrown away in 99.9999% of the cases - such overhead should be avoided.If you created a special
timer_paramsevent, which would produce precisely one event when triggered via say a new perf ioctl. I.e. add something like this toperf_event.h:#define PERF_EVENT_IOC_INJECT _IOW('$', 7, __u64)and add code to kernel/perf_event.c's
perf_ioctl()function that takes that__u64parameter as an event ID and injects an 'artificial' event.
Quoted 9 lines
Such a new feature would be useful for other things as well: backtesting rare events, injecting other types of 'parameter/query events', etc.
There might be more details to this, but it would be a useful scheme IMO - and it would still integrate nicely with the whole ftrace event enumeration scheme so tooling support would be easier.
What do you think?
Sure, this is the better way, i'll fix it address your suggestion in the next version.
Thanks,
Xiao
v3 cover perf tools: introduce 'perf timer' to analyze timer's behavior
In reply to Ingo MolnarChangelog v2->v3:
introduce 'inject' event to get kernel HZ address Ingo's suggestion [ see http://lkml.org/lkml/2009/12/28/31 ]
PATCH v3 1/5 perf_event: introduce 'inject' event and get HZ
In reply to Ingo Molnar'inject' event is a very useful feature and it's suggested by Ingo [ See http://lkml.org/lkml/2009/12/28/31 ]
--- include/linux/perf_event.h | 13 +++++++++++ kernel/perf_event.c | 47 +++++++++++++++++++++++++++++++++++++++++++ tools/perf/builtin-record.c | 13 +++++++++++ tools/perf/util/session.c | 5 ++++ tools/perf/util/session.h | 3 +- 5 files changed, 80 insertions(+), 1 deletions(-)
include/linux/perf_event.h +13 -0
@@ -228,6 +228,7 @@ struct perf_event_attr {#define PERF_EVENT_IOC_PERIOD _IOW('$', 4, __u64)#define PERF_EVENT_IOC_SET_OUTPUT _IO ('$', 5)#define PERF_EVENT_IOC_SET_FILTER _IOW('$', 6, char *) +#define PERF_EVENT_IOC_INJECT _IO ('$', 7)enum perf_event_ioc_flags {PERF_IOC_FLAG_GROUP =1U <<0 ,
@@ -413,10 +414,22 @@ enum perf_event_type {* }; */ PERF_RECORD_SAMPLE =9 , +/* +* struct { +* struct perf_event_header header; +* u32 inject_event_id; +* u64 value; +* }; +*/ +PERF_RECORD_INJECT =10 , PERF_RECORD_MAX, /* non-ABI */ }; +enum perf_inject_event { +PERF_INJECT_HZ =0x01 , +}; +enum perf_callchain_context {PERF_CONTEXT_HV = (__u64)-32 , PERF_CONTEXT_KERNEL = (__u64)-128,
kernel/perf_event.c +47 -0
59 lines not on this page · raw message
tools/perf/builtin-record.c +13 -0
@@ -65,6 +65,8 @@ static int file_new = 1;static struct perf_session *session; +u32 inject_events; +struct mmap_data {int counter;void *base;
@@ -381,6 +383,17 @@ try_again: } } +if (inject_events) { + ret =ioctl (fd[nr_cpu][counter],PERF_EVENT_IOC_INJECT , + inject_events); +if (ret) { +error ("failed to inject event(%u) with %d (%s)\n" , + inject_events, errno,strerror (errno)); +exit (-1 ); + } + inject_events =0 ; + } +ioctl (fd[nr_cpu][counter],PERF_EVENT_IOC_ENABLE ); }
tools/perf/util/session.c +5 -0
@@ -179,6 +179,8 @@ static void perf_event_ops__fill_defaults(struct perf_event_ops *handler) handler->throttle = process_event_stub;if (handler->unthrottle ==NULL ) handler->unthrottle = process_event_stub; +if (handler->inject ==NULL ) + handler->inject = process_event_stub; }static const char *event__name[] = {
@@ -192,6 +194,7 @@ static const char *event__name[] = { [PERF_RECORD_FORK] = "FORK", [PERF_RECORD_READ] = "READ", [PERF_RECORD_SAMPLE] = "SAMPLE", + [PERF_RECORD_INJECT] = "INJECT", };unsigned long event__total[PERF_RECORD_MAX ];
@@ -239,6 +242,8 @@ static int perf_session__process_event(struct perf_session *self,return ops->throttle (event, self);case PERF_RECORD_UNTHROTTLE :return ops->unthrottle (event, self); +case PERF_RECORD_INJECT : +return ops->inject (event, self);default : self->unknown_events++;return -1 ;
tools/perf/util/session.h +2 -1
@@ -40,7 +40,8 @@ struct perf_event_ops { lost, read, throttle, - unthrottle; + unthrottle, + inject; };struct perf_session *perf_session__new (const char *filename,int mode,bool force);
Signature · 1 line
1.6.1.2
PATCH v3 2/5 trace_event: record task' real_timer in itimer_state tracepoint
In reply to Ingo MolnarWhen itimer is ITIMER_REAL, we should cooperate with hrtimer's tracepoit to get it's latency, this patch can let us get itimer's hrtimer easy
--- include/trace/events/timer.h | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-)
include/trace/events/timer.h +6 -2
@@ -285,6 +285,7 @@ TRACE_EVENT(itimer_state,TP_STRUCT__entry (__field (int ,which ) + __field( void *, timer )__field ( cputime_t, expires )__field ( long, value_sec ) __field( long, value_usec )
@@ -294,6 +295,8 @@ TRACE_EVENT(itimer_state,TP_fast_assign ( __entry->which = which; + __entry->timer = which ==ITIMER_REAL ? + ¤t->signal->real_timer :NULL ; __entry->expires = expires; __entry->value_sec = value->it_value.tv_sec; __entry->value_usec = value->it_value.tv_usec;
@@ -301,8 +304,9 @@ TRACE_EVENT(itimer_state, __entry->interval_usec = value->it_interval.tv_usec; ), -TP_printk ("which=%d expires=%llu it_value=%ld.%ld it_interval=%ld.%ld" , - __entry->which, (unsigned long long )__entry->expires, +TP_printk ("which=%d timer=%p expires=%llu it_value=%ld.%ld it_interval=" +"%ld.%ld" , __entry->which, __entry->timer, + (unsigned long long )__entry->expires, __entry->value_sec, __entry->value_usec, __entry->interval_sec, __entry->interval_usec) );
Signature · 1 line
1.6.1.2
PATCH v3 3/5 perf tools: fix getting point
In reply to Ingo MolnarThe point that got by raw_field_ptr() is point to @data offset not tracepoint saved.
Introduce raw_field_array() to get array data
Changlog v1->v2:
- fix the typo 'arry' -> 'array' that is pointed out by Frederic Weisbecker
- use '(void *)(unsigned long)value' instead of 'memcpy()' as Frederic Weisbecker's suggestion
--- tools/perf/builtin-sched.c | 3 +-- tools/perf/util/trace-event-parse.c | 23 ++++++++++++++++++++--- tools/perf/util/trace-event.h | 3 +++ 3 files changed, 24 insertions(+), 5 deletions(-)
tools/perf/builtin-sched.c +1 -2
@@ -630,8 +630,7 @@ static void test_calibrations(void)#define FILL_ARRAY (ptr, array, event, data) \ do { \ - void *__array = raw_field_ptr(event, #array, data); \ - memcpy(ptr.array, __array, sizeof(ptr.array)); \ + raw_field_array(event, #array, data, ptr.array, sizeof(ptr.array));\ } while(0)#define FILL_COMMON_FIELDS (ptr, event, data) \
tools/perf/util/trace-event-parse.c +20 -3
@@ -840,7 +840,7 @@ static int event_read_fields(struct event *event, struct format_field **fields) field->flags |=FIELD_IS_ARRAY ; type =read_token (&token); -while (strcmp (token,"]" ) !=0 ) { +while (strcmp (token,"]" ) !=0 ) {if (last_type ==EVENT_ITEM && type ==EVENT_ITEM ) len =2 ;
@@ -1919,13 +1919,30 @@ raw_field_value(struct event *event, const char *name, void *data)void *raw_field_ptr (struct event *event,const char *name,void *data) { +unsigned long long value; + + value =raw_field_value (event, name, data); + +if (!value) +return NULL ; + +return (void *)(unsigned long )value; +} + +unsigned long long +raw_field_array (struct event *event,const char *name,void *data, +void *array,int array_size) +{ +int len;struct format_field *field; field =find_any_field (event, name);if (!field) -return NULL ; +return 0ULL ; -return data + field->offset; + len =min (array_size, field->size); +memcpy (array, data + field->offset, len); +return len; }static int get_common_info (const char *type,int *offset,int *size)
tools/perf/util/trace-event.h +3 -0
@@ -255,6 +255,9 @@ unsigned long long read_size(void *ptr, int size);unsigned long long raw_field_value (struct event *event,const char *name,void *data);void *raw_field_ptr (struct event *event,const char *name,void *data); +unsigned long long +raw_field_array (struct event *event,const char *name,void *data, +void *array,int array_size);unsigned long long eval_flag (const char *flag);int read_tracing_data (int fd,struct perf_event_attr *pattrs,int nb_events);
Signature · 1 line
1.6.1.2
PATCH v3 4/5 perf timer: add document for 'perf timer'
In reply to Ingo MolnarAdd document for 'perf timer', it let people know how to use it
--- tools/perf/Documentation/perf-timer.txt | 50 +++++++++++++++++++++++++++++++ 1 files changed, 50 insertions(+), 0 deletions(-) create mode 100644 tools/perf/Documentation/perf-timer.txt
tools/perf/Documentation/perf-timer.txt new file +50 -0
@@ -0,0 +1,50 @@ +perf-timer(1) +============== + +NAME +---- +perf-timer - Tool to trace/measure timer properties + +SYNOPSIS +-------- +[verse] +'perf timer' {record|report} [<options>] + +DESCRIPTION +----------- +There are two variants of perf timer: + + 'perf timer record <command>' to record the timer events + of an arbitrary workload. + + 'perf timer report' to report number of activated/expired/canceled + timers, timer latency and timer function runtime. + +OPTIONS +------- +-D:: +--dump-raw-trace=:: + Display verbose dump of the sched data. + +-i <file>:: +--input=<file>:: + Select the input file (default: perf.data) + +-s <key[,key2...]>:: +--sort=<key[,key2...]>:: + Sort the output (default: type, avg-lat, max-lat, + avg-runtime, max-runtime) + +--print-lat:: + Print timer latency + +--print-runtime:: + Print timer function runtime + +--type:: + Timer type selector, you can specify one or more types + (timer, hrtimer, itimer-real, itimer-virtual, itimer-prof) + +SEE ALSO +-------- +linkperf:perf-record[1]
Signature · 1 line
1.6.1.2
PATCH v3 5/5 perf timer: add 'perf timer' core code
In reply to Ingo MolnarIt's the core code of 'perf timer', it can report number of activated/expired/canceled timers, timer latency and timer function runtime
Sample output:
- Record timer event:
#perf timer record
^C
- Show timer latency:
#perf timer report --print-lat
Timer Latency List
[ Abbreviations: Avg = average; lat = latency; ts = timestamp ]
-------------------------------------------------------------------------
Timer | TYPE | Avg-lat ms| Max-lat ms| Max-lat-ts s
-------------------------------------------------------------------------
swapper |HRTIMER |0.180 |2.404 |42722.586 swapper |TIMER |2.000 |2.000 |42722.586 events/0 |TIMER |2.000 |2.000 |42723.002 events/0 |TIMER |0.000 |0.000 |42723.641 sshd |TIMER |0.000 |0.000 |42723.641
Statistics
==========
Activated timers number:
HRTIMER: 105 TIMER: 11 ITIMER_PROF: 0 ITIMER_VIRTUAL: 0 ITIMER_REAL: 0
Expired timers number:
HRTIMER: 94 TIMER: 12 ITIMER_PROF: 0 ITIMER_VIRTUAL: 0 ITIMER_REAL: 0
Canceled timers number:
HRTIMER: 105 TIMER: 12 ITIMER_PROF: 0 ITIMER_VIRTUAL: 0 ITIMER_REAL: 0
- Show timer function runtime:
#perf timer report --print-runtime
Timer Function Runtime List
[ Abbreviations: Avg = average; func = function; rt = runtime; ts = timestamp ]
-------------------------------------------------------------------------------
Timer | Type |Avg-rt ms|Max-rt ms| Max-rt-func | Max-rt-ts s
-------------------------------------------------------------------------------
swapper |HRTIMER|0.010 |0.026 |tick_sched_timer |42722.586 0xc15f2b70 |HRTIMER|0.009 |0.009 |Not - Catch |42722.000 swapper |TIMER |0.058 |0.111 |tcp_write_timer |42721.930 events/0 |TIMER |0.007 |0.010 |delayed_work_timer_fn |42723.002 swapper |TIMER |0.037 |0.037 |dev_watchdog |42723.002 events/0 |TIMER |0.028 |0.028 |delayed_work_timer_fn |42723.641 swapper |TIMER |0.023 |0.023 |pcnet32_watchdog |42722.001 events/1 |TIMER |0.014 |0.014 |Not - Catch |42723.003 events/0 |TIMER |0.010 |0.010 |Not - Catch |42722.001 sshd |TIMER |0.009 |0.009 |delayed_work_timer_fn |42723.641 swapper |TIMER |0.004 |0.004 |Not - Catch |42721.638 events/1 |TIMER |0.003 |0.003 |Not - Catch |42723.003
Statistics
==========
Activated timers number:
HRTIMER: 105 TIMER: 11 ITIMER_PROF: 0 ITIMER_VIRTUAL: 0 ITIMER_REAL: 0
Expired timers number:
HRTIMER: 94 TIMER: 12 ITIMER_PROF: 0 ITIMER_VIRTUAL: 0 ITIMER_REAL: 0
Canceled timers number:
HRTIMER: 105 TIMER: 12 ITIMER_PROF: 0 ITIMER_VIRTUAL: 0 ITIMER_REAL: 0
- Specify types which you are interesting and choose one or more of "timer, hrtimer, itimer-real, itimer-virtual, itimer-prof"
#perf timer report --print-lat --type timer
Timer Latency List
[ Abbreviations: Avg = average; lat = latency; ts = timestamp ]
-------------------------------------------------------------------------
Timer | TYPE | Avg-lat ms| Max-lat ms| Max-lat-ts s
-------------------------------------------------------------------------
swapper |TIMER |2.000 |2.000 |42722.586 events/0 |TIMER |2.000 |2.000 |42723.002 events/0 |TIMER |0.000 |0.000 |42723.641 sshd |TIMER |0.000 |0.000 |42723.641
Changlog v1->v2:
- rename 'perf timer latency' to 'perf timer report', because it not only show timer latency but also show timer function runtime, timer activated/canceled/expired statistics and canceled timer list(TODO)
Below changes are all from Thomas's suggestion:
- fix a lot of typos and bad ideas
- use hash table instead of RB tree to record timer
- let output information more readable
- report number of activated/canceled/expired timers
- support to filter timer types:
'--type' can do it
TODO:
- Show canceled timer list that is suggested by Thomas
- Any suggestion are welcome.
--- tools/perf/Makefile | 1 + tools/perf/builtin-timer.c | 996 +++++++++++++++++++++++++++++++++++++++++++ tools/perf/builtin.h | 1 + tools/perf/command-list.txt | 1 + tools/perf/perf.c | 1 + tools/perf/util/event.h | 7 + 6 files changed, 1007 insertions(+), 0 deletions(-) create mode 100644 tools/perf/builtin-timer.c
tools/perf/Makefile +1 -0
@@ -449,6 +449,7 @@ BUILTIN_OBJS += builtin-top.o BUILTIN_OBJS += builtin-trace.o BUILTIN_OBJS += builtin-probe.o BUILTIN_OBJS += builtin-kmem.o +BUILTIN_OBJS += builtin-timer.o PERFLIBS = $(LIB_FILE)
tools/perf/builtin-timer.c new file +996 -0
996 lines not on this page · raw message
tools/perf/builtin.h +1 -0
@@ -30,5 +30,6 @@ extern int cmd_trace(int argc, const char **argv, const char *prefix);extern int cmd_version (int argc,const char **argv,const char *prefix);extern int cmd_probe (int argc,const char **argv,const char *prefix);extern int cmd_kmem (int argc,const char **argv,const char *prefix); +extern int cmd_timer (int argc,const char **argv,const char *prefix);#endif
tools/perf/command-list.txt +1 -0
@@ -16,3 +16,4 @@ perf-top mainporcelain common perf-trace mainporcelain common perf-probe mainporcelain common perf-kmem mainporcelain common +perf-timer mainporcelain common
tools/perf/perf.c +1 -0
@@ -301,6 +301,7 @@ static void handle_internal_command(int argc, const char **argv) { "sched", cmd_sched, 0 }, { "probe", cmd_probe, 0 }, { "kmem", cmd_kmem, 0 }, + { "timer", cmd_timer, 0 }, };unsigned int i;static const char ext[] =STRIP_EXTENSION ;
tools/perf/util/event.h +7 -0
@@ -75,6 +75,12 @@ struct sample_data {void *raw_data; }; +struct inject_event { +struct perf_event_header header; +u32 inject_event_id; +u64 value; +}; +#define BUILD_ID_SIZE 20struct build_id_event {
@@ -92,6 +98,7 @@ typedef union event_union {struct lost_event lost;struct read_event read;struct sample_event sample; +struct inject_event inject; } event_t;struct events_stats {
Signature · 1 line
1.6.1.2
Peter Zijlstra
maintainerRe: [PATCH v3 1/5] perf_event: introduce 'inject' event and get HZ
In reply to Xiao GuangrongQuoted 2 lines · Xiao Guangrong
'inject' event is a very useful feature and it's suggested by Ingo [ See http://lkml.org/lkml/2009/12/28/31 ]
I really dislike the name, event injection to me would be like a write() interface where you provide the actual event data to be stuffed in the output stream.
This just seems like a very weird way of getting data out. A kind of like sysconf() but done very strange.
Re: [PATCH v3 1/5] perf_event: introduce 'inject' event and get HZ
Quoted 10 lines · Peter Zijlstra
Quoted 2 lines · Xiao Guangrong
'inject' event is a very useful feature and it's suggested by Ingo [ See http://lkml.org/lkml/2009/12/28/31 ]
I really dislike the name, event injection to me would be like a
write()interface where you provide the actual event data to be stuffed in the output stream.This just seems like a very weird way of getting data out. A kind of like
sysconf()but done very strange.
What kind of API would you suggest?
Ingo
Peter Zijlstra
maintainerRe: [PATCH v3 1/5] perf_event: introduce 'inject' event and get HZ
Quoted 6 lines · Ingo Molnar
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Xiao Guangrong
'inject' event is a very useful feature and it's suggested by Ingo [ See http://lkml.org/lkml/2009/12/28/31 ]
Quoted 8 lines
Quoted 6 lines
I really dislike the name, event injection to me would be like a
write()interface where you provide the actual event data to be stuffed in the output stream.This just seems like a very weird way of getting data out. A kind of like
sysconf()but done very strange.What kind of API would you suggest?
sysconf() seems ideal for getting single, mostly constant variables out of the kernel, we already have non POSIX (read Linux specific) names in there.
_SC_KERNEL_RELOCATION_OFFSET_SC_KERNEL_HZ
or somethings like that comes to mind.
Re: [PATCH v3 1/5] perf_event: introduce 'inject' event and get HZ
In reply to Peter ZijlstraQuoted 8 lines · Peter Zijlstra
Quoted 2 lines · Xiao Guangrong
'inject' event is a very useful feature and it's suggested by Ingo [ See http://lkml.org/lkml/2009/12/28/31 ]
I really dislike the name, event injection to me would be like a
write()interface where you provide the actual event data to be stuffed in the output stream.
Yes, it like write and it writes something form kernel to perf. And, i think this name is suitable for it's doing that injects an 'artificial' event, it's well if you have other name for it :-)
Quoted 3 lines
This just seems like a very weird way of getting data out. A kind of like
sysconf()but done very strange.
It's since some parameter is only used by perf, suce as HZ in this patch and 'relocation offset' in my other patchset, it also well if has better way to do it :-)
Thanks,
Xiao
Re: [PATCH v3 1/5] perf_event: introduce 'inject' event and get HZ
In reply to Peter ZijlstraQuoted 16 lines · Peter Zijlstra
Quoted 6 lines · Ingo Molnar
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Xiao Guangrong
'inject' event is a very useful feature and it's suggested by Ingo [ See http://lkml.org/lkml/2009/12/28/31 ]
Quoted 8 lines
Quoted 6 lines
I really dislike the name, event injection to me would be like a
write()interface where you provide the actual event data to be stuffed in the output stream.This just seems like a very weird way of getting data out. A kind of like
sysconf()but done very strange.What kind of API would you suggest?
Quoted 8 lines
sysconf()seems ideal for getting single, mostly constant variables out of the kernel, we already have non POSIX (read Linux specific) names in there.
_SC_KERNEL_RELOCATION_OFFSET_SC_KERNEL_HZor somethings like that comes to mind.
Nah. HZ might be static here but we'll have more dynamic parameters so the sysconf API is rather inflexible there.
I was thinking of something that fits into existing perf API schemes, to keep it simple and self-sufficient, and to allow for extensions/etc.
Ingo
Peter Zijlstra
maintainerRe: [PATCH v3 1/5] perf_event: introduce 'inject' event and get HZ
In reply to Xiao GuangrongQuoted 8 lines · Xiao Guangrong
Quoted 4 lines
I really dislike the name, event injection to me would be like a
write()interface where you provide the actual event data to be stuffed in the output stream.Yes, it like write and it writes something form kernel to perf. And, i think this name is suitable for it's doing that injects an 'artificial' event, it's well if you have other name for it :-)
No, what I means is that you cannot actually inject an arbitrary event. Sure it injects something, but that's pretty limited.
Quoted 7 lines
Quoted 3 lines
This just seems like a very weird way of getting data out. A kind of like
sysconf()but done very strange.It's since some parameter is only used by perf, suce as HZ in this patch and 'relocation offset' in my other patchset, it also well if has better way to do it :-)
The thing is, this proposed interface is very much tied to perf and makes it basically impossible for !perf to use. Whereas I can imagine esp. the relocation offset to be interesting for other things. Basically everything needing to resolve a kernel symbol.
Peter Zijlstra
maintainerRe: [PATCH v3 1/5] perf_event: introduce 'inject' event and get HZ
In reply to Peter ZijlstraQuoted 16 lines · Peter Zijlstra
Quoted 6 lines · Ingo Molnar
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Xiao Guangrong
'inject' event is a very useful feature and it's suggested by Ingo [ See http://lkml.org/lkml/2009/12/28/31 ]
Quoted 8 lines
Quoted 6 lines
I really dislike the name, event injection to me would be like a
write()interface where you provide the actual event data to be stuffed in the output stream.This just seems like a very weird way of getting data out. A kind of like
sysconf()but done very strange.What kind of API would you suggest?
Quoted 8 lines
sysconf()seems ideal for getting single, mostly constant variables out of the kernel, we already have non POSIX (read Linux specific) names in there.
_SC_KERNEL_RELOCATION_OFFSET_SC_KERNEL_HZor somethings like that comes to mind.
OK so there is no sysconf() syscall and its all implemented in glibc, which is utter suckage.. will have to come up with something saner then.
Re: [PATCH v3 1/5] perf_event: introduce 'inject' event and get HZ
Quoted 2 lines · Peter Zijlstra
OK so there is no
sysconf()syscall and its all implemented in glibc, which is utter suckage.. will have to come up with something saner then.
I think integrating it into the perf stream of events would be the most desirable approach - tooling could make use of it easily.
For data record (and hierarchy) description /debug/tracing/events/ would be useful.
We dont want to actually 'activate' them in an 'event' fashion - but we could use event IDs and the record format and the transport to describe and recover the values.
For transport we could use the mmap ring-buffer and it would also be nice to add (or resurrect) some 'quick & easy' read()/write() based transport. It a bit like a value read-out from a counter - just here it's not a regular counter but some sort of kernel-internal value that we want to provide.
Hm?
Ingo