'perf sched': Utility to capture, measure and analyze scheduler latencies and behavior
Subsystems (2)
The recent discussions on lkml about scheduler interactivity have again demonstrated a particular weakness in Linux instrumentation: the lack of a turn-key solution that gives us both high level summaries and fine low level details about CPU scheduler behavior.
This makes it needlessly hard to talk about interactivity with an objective basis, desktop users often have to resort to post subjective impressions how smooth the desktop 'feels' - instead of being able to report hard numbers. We've got LatencyTop with a great GUI and latency maximums but that does not give us finegrained traces that scheduler developers could act upon. We've got ftrace for finegrained traces but that does not give us high-level information and use and analysis is difficult.
We are pleased to announce the first experimental version of a utility that tries to meet this ambitious goal: the new 'perf sched' family of tools that uses performance events to objective characterise arbitrary workloads from a scheduling and latency point of view.
In the past few days we've also used this new tool to re-tune the scheduler for interactivity and scalability. While it's still early code i can already observe that this tool surpassed our expectations and we can now analyze most scheduler properties from within 'perf sched' alone.
'perf sched' has five sub-commands currently:
perf sched record # low-overhead recording of arbitrary workloads perf sched latency # output per task latency metrics perf sched map # show summary/map of context-switching perf sched trace # output finegrained trace perf sched replay # replay a captured workload using simlated threads
Desktop users would generally use 'perf sched record' to capture a trace (which creates perf.data), and 'perf sched latency' tool to check latencies (which analyzes the trace in perf.data). The other tools too use an already recorded perf.data.
'scehd latency' example usage:
perf sched record sleep 10 # record full system activity for 10 seconds
perf sched latency --sort max # report latencies sorted by max
-----------------------------------------------------------------------------------------
Task | Runtime ms | Switches | Average delay ms | Maximum delay ms |
-----------------------------------------------------------------------------------------
:14086:14086 | 0.095 ms | 2 | avg: 3.445 ms | max: 6.891 ms | gnome-session:13792 | 31.713 ms | 102 | avg: 0.160 ms | max: 5.992 ms | metacity:14038 | 49.220 ms | 637 | avg: 0.066 ms | max: 5.942 ms | gconfd-2:13971 | 48.587 ms | 777 | avg: 0.047 ms | max: 5.793 ms | gnome-power-man:14050 | 140.601 ms | 434 | avg: 0.097 ms | max: 5.367 ms | python:14049 | 114.694 ms | 125 | avg: 0.120 ms | max: 5.343 ms | kblockd/1:236 | 3.458 ms | 498 | avg: 0.179 ms | max: 5.271 ms | Xorg:3122 | 1073.107 ms | 2920 | avg: 0.030 ms | max: 5.265 ms | dbus-daemon:2063 | 64.593 ms | 665 | avg: 0.103 ms | max: 4.730 ms | :14040:14040 | 30.786 ms | 255 | avg: 0.095 ms | max: 4.155 ms | events/1:8 | 0.105 ms | 13 | avg: 0.598 ms | max: 3.775 ms | console-kit-dae:2080 | 14.867 ms | 152 | avg: 0.142 ms | max: 3.760 ms | gnome-settings-:14023 | 572.653 ms | 979 | avg: 0.056 ms | max: 3.627 ms |
...
-----------------------------------------------------------------------------------------
TOTAL: | 3144.817 ms | 11654 |
---------------------------------------------------
this particular capture showed latencies that occured during a Gnome startup session. The following output shows firefox latencies during a 'make -j4' kernel build on a dual-core laptop:
europe:~> perf sched record sleep 10
[ perf record: Woken up 15 times to write data ]
[ perf record: Captured and wrote 7.408 MB perf.data (~323639 samples) ]
-----------------------------------------------------------------------------------------
Task | Runtime ms | Switches | Average delay ms | Maximum delay ms |
-----------------------------------------------------------------------------------------
...
firefox:14329 | 845.724 ms | 1411 | avg: 0.529 ms | max: 14.982 ms |
...
-----------------------------------------------------------------------------------------
TOTAL: | 20196.186 ms | 11011 |
---------------------------------------------------
Another sub-tool that interprets and analyzes the events in perf.data is 'perf sched map', which prints an abbreviated text map of scheduling events. For example here is a snipped from the life of a 16-CPU box:
$ perf sched map ...
N1 O1 . . . S1 . . . B0 . *I0 C1 . M1 . 23002.773423 secs N1 O1 . *Q0 . S1 . . . B0 . I0 C1 . M1 . 23002.773423 secs N1 O1 . Q0 . S1 . . . B0 . *R1 C1 . M1 . 23002.773485 secs N1 O1 . Q0 . S1 . *S0 . B0 . R1 C1 . M1 . 23002.773478 secs *L0 O1 . Q0 . S1 . S0 . B0 . R1 C1 . M1 . 23002.773523 secs L0 O1 . *. . S1 . S0 . B0 . R1 C1 . M1 . 23002.773531 secs L0 O1 . . . S1 . S0 . B0 . R1 C1 *T1 M1 . 23002.773547 secs T1 => irqbalance:2089 L0 O1 . . . S1 . S0 . *P0 . R1 C1 T1 M1 . 23002.773549 secs *N1 O1 . . . S1 . S0 . P0 . R1 C1 T1 M1 . 23002.773566 secs N1 O1 . . . *J0 . S0 . P0 . R1 C1 T1 M1 . 23002.773571 secs N1 O1 . . . J0 . S0 *B0 P0 . R1 C1 T1 M1 . 23002.773592 secs N1 O1 . . . J0 . *U0 B0 P0 . R1 C1 T1 M1 . 23002.773582 secs N1 O1 . . . *S1 . U0 B0 P0 . R1 C1 T1 M1 . 23002.773604 secs N1 O1 . . . S1 . U0 B0 *. . R1 C1 T1 M1 . 23002.773615 secs N1 O1 . . . S1 . U0 B0 . . *K0 C1 T1 M1 . 23002.773631 secs N1 O1 . *M0 . S1 . U0 B0 . . K0 C1 T1 M1 . 23002.773624 secs N1 O1 . M0 . S1 . U0 *. . . K0 C1 T1 M1 . 23002.773644 secs N1 O1 . M0 . S1 . U0 . . . *R1 C1 T1 M1 . 23002.773662 secs N1 O1 . M0 . S1 . *. . . . R1 C1 T1 M1 . 23002.773648 secs N1 O1 . *. . S1 . . . . . R1 C1 T1 M1 . 23002.773680 secs N1 O1 . . . *L0 . . . . . R1 C1 T1 M1 . 23002.773717 secs *N0 O1 . . . L0 . . . . . R1 C1 T1 M1 . 23002.773709 secs *N1 O1 . . . L0 . . . . . R1 C1 T1 M1 . 23002.773747 secs
Columns stand for individual CPUs, from CPU0 to CPU15, and the two-letter shortcuts stand for tasks that are running on a CPU.
'*' denotes the CPU that had the event.
A dot signals an idle CPU.
New tasks are assigned new two-letter shortcuts - when they occur first they are printed. In the above example 'T1' stood for irqbalance:
T1 => irqbalance:2089
The map can be used to characterise the 'health' of a workload at a glance, without having to go through hundreds of scheduling events. Problems in CPU distribution, bad scheduling patterns and incorrect migrations can be seen from this map.
'perf sched trace' is another sub-tool, it shows the trace as it happened, with all its fine details:
pipe-test-100k-13520 [001] 1254.354513808: sched_stat_wait: task: pipe-test-100k:13521 wait: 5362 [ns]
pipe-test-100k-13520 [001] 1254.354514876: sched_switch: task pipe-test-100k:13520 [120] (S) ==> pipe-test-100k:13521 [120]
:13521-13521 [001] 1254.354517927: sched_stat_runtime: task: pipe-test-100k:13521 runtime: 5092 [ns], vruntime: 133967391150 [ns]
:13521-13521 [001] 1254.354518984: sched_stat_sleep: task: pipe-test-100k:13520 sleep: 5092 [ns]
:13521-13521 [001] 1254.354520011: sched_wakeup: task pipe-test-100k:13520 [120] success=1 [001]
:13521-13521 [001] 1254.354521668: sched_stat_runtime: task: pipe-test-100k:13521 runtime: 3951 [ns], vruntime: 133967395101 [ns]
:13521-13521 [001] 1254.354522773: sched_stat_wait: task: pipe-test-100k:13520 wait: 3951 [ns]
:13521-13521 [001] 1254.354523860: sched_switch: task pipe-test-100k:13521 [120] (R) ==> pipe-test-100k:13520 [120]
pipe-test-100k-13520 [001] 1254.354527060: sched_stat_runtime: task: pipe-test-100k:13520 runtime: 5375 [ns], vruntime: 133962396525 [ns]
pipe-test-100k-13520 [001] 1254.354528225: sched_stat_wait: task: pipe-test-100k:13521 wait: 5375 [ns]
pipe-test-100k-13520 [001] 1254.354529294: sched_switch: task pipe-test-100k:13520 [120] (S) ==> pipe-test-100k:13521 [120]
:13521-13521 [001] 1254.354532345: sched_stat_runtime: task: pipe-test-100k:13521 runtime: 5092 [ns], vruntime: 133967400193 [ns]Some problems can only be seen on this level.
There's also the 'perf sched replay' sub-tool - an (experimental) scheduling workload 'simulator'. This tool takes recorded events and turns them into sleep/run patterns of user-space threads which are then executed as if they real workload was running:
$ perf sched replay run measurement overhead: 3771 nsecs sleep measurement overhead: 66617 nsecs the run test took 999708 nsecs the sleep test took 1097207 nsecs nr_run_events: 200221 nr_sleep_events: 200235 nr_wakeup_events: 100130 task 0 ( perf: 13519), nr_events: 148 task 1 ( perf: 13520), nr_events: 200037 task 2 ( pipe-test-100k: 13521), nr_events: 300090 task 3 ( ksoftirqd/0: 4), nr_events: 8 task 4 ( swapper: 0), nr_events: 170 task 5 ( gnome-power-man: 3192), nr_events: 3 task 6 ( gdm-simple-gree: 3234), nr_events: 3 task 7 ( Xorg: 3122), nr_events: 5 task 8 ( hald-addon-stor: 2234), nr_events: 27 task 9 ( ata/0: 321), nr_events: 29 task 10 ( scsi_eh_4: 704), nr_events: 37 task 11 ( events/1: 8), nr_events: 3 task 12 ( events/0: 7), nr_events: 6 task 13 ( flush-8:0: 6980), nr_events: 20 ------------------------------------------------------------ #1 : 2038.157, ravg: 2038.16, cpu: 0.09 / 0.09 #2 : 2042.153, ravg: 2038.56, cpu: 0.11 / 0.09 ^C
'top' showed the following while the replay was running:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 13559 mingo 20 0 39108 29m 872 R 73.7 1.5 0:04.94 :pipe-test-100k 13561 mingo 20 0 39108 29m 872 R 62.7 1.5 0:04.46 :swapper 13558 mingo 20 0 39108 29m 872 S 42.2 1.5 0:02.80 :perf 13557 mingo 20 0 39108 29m 872 S 12.6 1.5 0:00.83 :perf 13570 mingo 20 0 39108 29m 872 S 0.3 1.5 0:00.01 :flush-8:0
The ':' prefix is artificial so that the simulated tasks are not confused with real tasks. Note that these are fully simulated tasks - there's no :pipe-test-100k binary executed by perf sched replay - the workload simulator renames its worker threads to the original command name.
The idea behind 'perf sched replay' is to allow the replication of difficult to set up workloads and their measurement, and to allow the tweaking of the scheduler without having to re-create the original scenario. It's a scheduler sandbox and workloads can be examined via their perf.data recordings alone.
All in one, 'perf sched' is extensible and modular and can take new sub-commands in the future as well - we thought latencies to be the first and most important mode of analysis to be added.
Also note that alternative schedulers such as BFS can easily add support for perf sched as well, by adding the following three tracepoints:
sched:sched_switch
sched:sched_stat_runtime
sched:sched_wakeup
It's around 3 lines of code added. This would be highly desirable, to allow apples-to-apples comparison of schedulers, using the same tooling.
The latest version of 'perf sched', plus the re-tuned scheduler can be found in the -tip tree:
http://people.redhat.com/mingo/tip.git/README
Or 'perf sched' alone can be found in the following standalone Git tree as well:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git perfcounters/core
Plus a combo patch against Linus's latest tree is attached below as well, for your testing and review pleasure.
Thanks,
Ingo
------------------>
Linus,
Please pull the latest perfcounters/core git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git perfcounters/core
Thanks,
Ingo
------------------>
Frederic Weisbecker (11):
perf tools: Allow the specification of all tracepoints at once perf sched: Fix bad event alignment
perf sched: Make it easier to plug in new sub profilers
perf sched: Add sched latency profiling
perf sched: Rename struct lat_snapshot to struct work atoms
perf sched: Add involuntarily sleeping task in work atoms
perf sched: Export the total, max latency and total runtime to thread atoms list perf sched: Fix nsec to msec conversion
perf sched: Implement multidimensional sorting
perf tools: Add an option to multiplex counters in a single channel perf tools: Fix processing of randomly serialized sched traces
Ingo Molnar (22):
perf: Add 'perf sched' tool
perf sched: Import schedbench.c
perf sched: Implement the scheduling workload replay engine perf sched: Tighten up the code
perf sched: Clean up latency and replay sub-commands perf sched: Display time in milliseconds, reorganize output perf sched: Add runtime stats
perf sched: Output runtime and context switch totals perf sched: Add 'perf sched latency' and 'perf sched replay' perf sched: Finish latency => atom rename and misc cleanups perf sched: Clean up PID sorting logic
perf sched: Implement the 'perf sched record' subcommand
perf_counter: Allow mmap if paranoid checks are turned off perf sched: Add 'perf sched trace', improve documentation perf_counter, sched: Add sched_stat_runtime tracepoint perf tools: Implement counter output multiplexing perf sched: Fix 'perf sched latency' output on 32-bit systems perf sched: Account for lost events, increase default buffering perf sched: Sanity check context switch events
perf sched: Make idle thread and comm/pid names more consistent perf sched: Add 'perf sched map' scheduling event map printout perf sched: Determine the number of CPUs automatically
Peter Zijlstra (3):
perf_counter: Do not throttle single swcounter events
perf_counter: Allow for a wakeup watermark
perf record: Disable profiling before draining the buffer
mingo (2):
perf sched: Print PIDs too
perf sched: Add support for sched:sched_stat_runtime events
include/linux/perf_counter.h | 10 +- include/trace/events/sched.h | 33 + kernel/perf_counter.c | 52 +- kernel/sched_fair.c | 1 + tools/perf/Documentation/perf-sched.txt | 41 + tools/perf/Documentation/perf-trace.txt | 25 + tools/perf/Makefile | 1 + tools/perf/builtin-record.c | 56 +- tools/perf/builtin-sched.c | 2002 +++++++++++++++++++++++++++++++ tools/perf/builtin.h | 5 +- tools/perf/command-list.txt | 2 + tools/perf/perf.c | 1 + tools/perf/util/event.h | 2 +- tools/perf/util/parse-events.c | 207 +++- tools/perf/util/thread.c | 4 +- tools/perf/util/thread.h | 9 +- tools/perf/util/trace-event-parse.c | 45 +- tools/perf/util/trace-event-read.c | 6 +- tools/perf/util/trace-event.h | 5 + 19 files changed, 2400 insertions(+), 107 deletions(-) create mode 100644 tools/perf/Documentation/perf-sched.txt create mode 100644 tools/perf/Documentation/perf-trace.txt create mode 100644 tools/perf/builtin-sched.c
include/linux/perf_counter.h +8 -2
@@ -199,10 +199,14 @@ struct perf_counter_attr { inherit_stat : 1, /* per task counts */ enable_on_exec : 1, /* next exec enables */ task : 1, /* trace fork/exit */ + watermark : 1, /* wakeup_watermark */ - __reserved_1 : 50; + __reserved_1 : 49; -__u32 wakeup_events;/* wakeup every n events */ +union { +__u32 wakeup_events;/* wakeup every n events */ +__u32 wakeup_watermark;/* bytes before wakeup */ + };__u32 __reserved_2;__u64 __reserved_3;
@@ -521,6 +525,8 @@ struct perf_mmap_data {atomic_t wakeup;/* needs a wakeup */ atomic_t lost;/* nr records lost */ +long watermark;/* wakeup watermark */ +struct perf_counter_mmap_page *user_page;void *data_pages[0 ]; };
include/trace/events/sched.h +33 -0
@@ -380,6 +380,39 @@ TRACE_EVENT(sched_stat_wait, );/* +* Tracepoint for accounting runtime (time the task is executing +* on a CPU). +*/ +TRACE_EVENT (sched_stat_runtime, + + TP_PROTO(struct task_struct *tsk, u64 runtime, u64 vruntime), + +TP_ARGS (tsk, runtime, vruntime), + +TP_STRUCT__entry ( + __array( char, comm, TASK_COMM_LEN ) + __field( pid_t, pid ) + __field( u64, runtime ) +__field ( u64, vruntime ) + ), + + TP_fast_assign( + memcpy(__entry->comm, tsk->comm, TASK_COMM_LEN); + __entry->pid = tsk->pid; + __entry->runtime = runtime; + __entry->vruntime = vruntime; + ) +TP_perf_assign ( + __perf_count(runtime); + ), + +TP_printk ("task: %s:%d runtime: %Lu [ns], vruntime: %Lu [ns]" , + __entry->comm, __entry->pid, + (unsigned long long )__entry->runtime, + (unsigned long long )__entry->vruntime) +); + +/* * Tracepoint for accounting sleep time (time the task is not runnable, * including iowait, see below). */
kernel/perf_counter.c +34 -18
132 lines not on this page · raw message
kernel/sched_fair.c +1 -0
@@ -513,6 +513,7 @@ static void update_curr(struct cfs_rq *cfs_rq)if (entity_is_task (curr)) {struct task_struct *curtask =task_of (curr); +trace_sched_stat_runtime (curtask, delta_exec, curr->vruntime);cpuacct_charge (curtask, delta_exec);account_group_exec_runtime (curtask, delta_exec); }
tools/perf/Documentation/perf-sched.txt new file +41 -0
41 lines not on this page · raw message
tools/perf/Documentation/perf-trace.txt new file +25 -0
@@ -0,0 +1,25 @@ +perf-trace(1) +============== + +NAME +---- +perf-trace - Read perf.data (created by perf record) and display trace output + +SYNOPSIS +-------- +[verse] +'perf trace' [-i <file> | --input=file] symbol_name + +DESCRIPTION +----------- +This command reads the input file and displays the trace recorded. + +OPTIONS +------- +-D:: +--dump-raw-trace=:: + Display verbose dump of the trace data. + +SEE ALSO +-------- +linkperf:perf-record[1]
tools/perf/Makefile +1 -0
@@ -376,6 +376,7 @@ LIB_OBJS += util/trace-event-info.o BUILTIN_OBJS += builtin-annotate.o BUILTIN_OBJS += builtin-help.o +BUILTIN_OBJS += builtin-sched.o BUILTIN_OBJS += builtin-list.o BUILTIN_OBJS += builtin-record.o BUILTIN_OBJS += builtin-report.o
tools/perf/builtin-record.c +41 -15
95 lines not on this page · raw message
tools/perf/builtin-sched.c new file +2002 -0
2,002 lines not on this page · raw message
tools/perf/builtin.h +3 -2
@@ -16,12 +16,13 @@ extern int check_pager_config(const char *cmd);extern int cmd_annotate (int argc,const char **argv,const char *prefix);extern int cmd_help (int argc,const char **argv,const char *prefix); +extern int cmd_sched (int argc,const char **argv,const char *prefix); +extern int cmd_list (int argc,const char **argv,const char *prefix);extern int cmd_record (int argc,const char **argv,const char *prefix);extern int cmd_report (int argc,const char **argv,const char *prefix);extern int cmd_stat (int argc,const char **argv,const char *prefix);extern int cmd_top (int argc,const char **argv,const char *prefix); -extern int cmd_version (int argc,const char **argv,const char *prefix); -extern int cmd_list (int argc,const char **argv,const char *prefix);extern int cmd_trace (int argc,const char **argv,const char *prefix); +extern int cmd_version (int argc,const char **argv,const char *prefix);#endif
tools/perf/command-list.txt +2 -0
@@ -4,7 +4,9 @@ # perf-annotate mainporcelain common perf-list mainporcelain common +perf-sched mainporcelain common perf-record mainporcelain common perf-report mainporcelain common perf-stat mainporcelain common perf-top mainporcelain common +perf-trace mainporcelain common
tools/perf/perf.c +1 -0
@@ -293,6 +293,7 @@ static void handle_internal_command(int argc, const char **argv) { "annotate", cmd_annotate, 0 }, { "version", cmd_version, 0 }, { "trace", cmd_trace, 0 }, + { "sched", cmd_sched, 0 }, };unsigned int i;static const char ext[] =STRIP_EXTENSION ;
tools/perf/util/event.h +1 -1
@@ -52,7 +52,7 @@ struct lost_event {*/ struct read_event {struct perf_event_header header; -u32 pid,tid; +u32 pid, tid;u64 value;u64 time_enabled;u64 time_running;
tools/perf/util/parse-events.c +156 -51
323 lines not on this page · raw message
tools/perf/util/thread.c +2 -2
@@ -8,7 +8,7 @@static struct thread *thread__new (pid_t pid) { -struct thread *self =malloc (sizeof (*self)); +struct thread *self =calloc (1 ,sizeof (*self));if (self !=NULL ) { self->pid = pid;
@@ -85,7 +85,7 @@ register_idle_thread(struct rb_root *threads, struct thread **last_match) {struct thread *thread =threads__findnew (0 , threads, last_match); -if (!thread ||thread__set_comm (thread,"[init]" )) { +if (!thread ||thread__set_comm (thread,"swapper" )) {fprintf (stderr,"problem inserting idle task.\n" );exit (-1 ); }
tools/perf/util/thread.h +5 -4
@@ -4,10 +4,11 @@#include "symbol.h" struct thread { -struct rb_node rb_node; -struct list_head maps; -pid_t pid; -char *comm; +struct rb_node rb_node; +struct list_head maps; +pid_t pid; +char shortname[3 ]; +char *comm; };int thread__set_comm (struct thread *self,const char *comm);
tools/perf/util/trace-event-parse.c +35 -10
82 lines not on this page · raw message
tools/perf/util/trace-event-read.c +4 -2
@@ -458,12 +458,13 @@ struct record *trace_read_data(int cpu)return data; } -void trace_report (void ) +void trace_report (void ) {const char *input_file ="trace.info" ;char buf[BUFSIZ ];char test[] = {23 ,8 ,68 };char *version; +int show_version =0 ;int show_funcs =0 ;int show_printk =0 ;
@@ -480,7 +481,8 @@ void trace_report (void)die ("not a trace file (missing tracing)" ); version =read_string (); -printf ("version = %s\n" , version); +if (show_version) +printf ("version = %s\n" , version);free (version);read_or_die (buf,1 );
tools/perf/util/trace-event.h +5 -0
@@ -234,6 +234,11 @@ extern int header_page_data_offset;extern int header_page_data_size;int parse_header_page (char *buf,unsigned long size); +int trace_parse_common_type (void *data); +struct event *trace_find_event (int id); +unsigned long long +raw_field_value (struct event *event,const char *name,void *data); +void *raw_field_ptr (struct event *event,const char *name,void *data);void read_tracing_data (struct perf_counter_attr *pattrs,int nb_counters);