perf sched latency: Refine outputs, unit scaling, and histogram support
v8 cover perf sched latency: Refine outputs, unit scaling, and histogram support
Hi Namhyung, Ian, Arnaldo,
This patch series improves 'perf sched latency' by suppressing misleading empty table output, extending pipe mode stream processing, introducing dynamic unit auto-scaling for latency and runtime statistics, and adding latency histogram visualisation alongside time-span filtering.
Patch 1 addresses an issue where 'perf sched latency' fell through and returned success (0) when perf_session__has_traces() failed due to missing tracepoint events in a perf.data file. This caused empty header tables and zeroed summary statistics to be rendered. In addition, thread__get_runtime() in map_switch_event() is guarded against potential NULL pointer dereferences under memory allocation failures.
Patch 2 extends pipe mode stream support. Because event attributes in pipe mode are received dynamically during event processing, session->evlist is not populated prior to event processing.
To handle pipe input correctly:
- Register the missing .attr, .tracing_data, .build_id, and .feature callbacks in cmd_sched()
- Promote the handlers array to file-scope (latency_handlers[]) and dynamically assign matching tracepoint handlers (or process_sched_ignore) inside perf_sched__process_tracepoint_sample() when evsel->handler is NULL; replace process_sched_wakeup_ignore() with process_sched_ignore()
- Perform the trace check post-processing when handling pipe data
Patch 3 introduces dynamic auto-scaling for latency and runtime display columns (Runtime, Avg delay, Max delay). Previously, all values were unconditionally formatted in milliseconds (ms), making microsecond- or second-scale latencies difficult to read. Columns are now dynamically scaled to the most appropriate unit (ns, us, ms, s), column headers are updated, and format specifiers are aligned character-for-character with table headers.
Patch 4 adds three new command-line options to 'perf sched latency':
--histogram (-H):
Displays an ASCII bar chart of CPU wait latencies between snapshots
--hist-mode:
Configures the bucketing scheme to either logarithmic (log) or 100 us equal-width linear (linear) mode
--time:
Filters trace event processing to a specified [start,stop] time span
Changes since v7:
- Added a guard clause in perf_sched__process_tracepoint_sample() to reduce loop indentation (Namhyung Kim)
- Converted dummy process_sched_wakeup_ignore() references to process_sched_ignore and eliminated redundant code (Namhyung Kim)
- Replaced string comparisons with faster integer checks (i.e., thread__tid(...) != 0) for swapper (idle thread) handling (Namhyung Kim)
- Added automated test cases for --histogram, --hist-mode, and --time into tools/perf/tests/shell/sched.sh (Namhyung Kim)
- Link to v7: https://lore.kernel.org/lkml/[email protected]/
Changes since v6:
- Fixed O(N) hot-path traversal overhead in perf_sched__process_tracepoint_sample() for unhandled tracepoints by assigning a dummy ignore handler (process_sched_ignore)
- Replaced evlist__set_tracepoints_handlers() in sample processing with a per-evsel lookup to prevent -EEXIST early aborts and avoid dropping late-arriving pipe events
- Updated commit message for the pipe mode patch to reflect the single-pass handler assignment logic
- Link to v6: https://lore.kernel.org/lkml/[email protected]/
Changes since v5:
- Split pipe mode trace sample handling from Patch 1 into a standalone patch (Namhyung Kim)
- Added a Fixes: tag to the pipe mode patch referencing commit 27295592c22e ("perf session: Share the common trace sample_check routine as perf_session__has_traces")
- Updated commit log with before and after illustrations of table header formatting
- Link to v5: https://lore.kernel.org/lkml/[email protected]/
Changes since v4:
- Added the missing .feature callback to sched.tool in cmd_sched()
- Fixed memory leak by reusing uncompleted work atoms when sched_in events are skipped or lost
- Promoted the handlers array to file-scope (i.e., latency_handlers[]) and added a dynamic lookup via evlist__set_tracepoints_handlers() inside perf_sched__process_tracepoint_sample() whenever a sample arrives with an unattached handler (i.e., evsel->handler == NULL)
- Prevented double-counting wakeups (i.e., ignore sched:sched_wakeup) in pipe mode and non-pipe modes
- Added missing trailing pipe in output_lat_thread's new auto-scaling format string
- Stripped redundant prefix strings from each row's format string to produce a clean, tabular output
- Link to v4: https://lore.kernel.org/lkml/[email protected]/
Changes since v3:
- Registered Missing Callbacks. In perf_tool__init configuration inside cmd_sched(), added the .attr, .tracing_data, and .build_id callbacks. Without these callbacks, pipe mode drops header attributes entirely, preventing tracepoints from being populated in session->evlist
- Introduced an explicit post-processing pipe check. This ensures that pipe mode aborts correctly and does not produce superfluous empty latency tables when no trace samples are available
- Prevented potential NULL pointer dereference. Added a NULL check for thread__get_runtime() in map_switch_event() under memory allocation failures
- Fixed column alignment. Modified format specifiers to align character-for-character with header column widths across all rows
- Adjusted header label padding. Updated header label padding and column width delimiters in perf_sched__lat() to match the exact field format specifiers printed in output_lat_thread()
- Fixed swapper histogram inclusion. Excluded "swapper" (i.e., CPU-specific idle thread) from global_hist buckets
- Preserved task state machine across --time bounds. Moved time-window filtering inside add_sched_in_event() to gate latency statistics recording without breaking wakeup state tracking across time interval boundaries
- Link to v3: https://lore.kernel.org/lkml/[email protected]/
Changes since v2:
- Ensured vertical pipe separators align across all latency table columns
- Excluded "swapper" (idle thread) latency samples from global_hist
- Preserved task state machine transitions across --time boundaries
- Suppressed empty table headers, total lines, and histogram graphs when no matching trace samples exist (e.g., when --time, --CPU, or --pids exclude all samples), outputting "No matching trace samples found." instead
- Linked to v2: https://lore.kernel.org/lkml/[email protected]/
Changes since v1:
- Fixed integer overflow in latency_bucket() (Ian Rogers)
- Linked to v1: https://lore.kernel.org/lkml/[email protected]/
Aaron Tomlin (4):
perf sched: Suppress latency table output when trace samples are
missing
perf sched: Handle missing trace samples in pipe mode perf sched latency: Auto-scale latency and runtime display units perf sched latency: Add histogram and time interval options
tools/perf/Documentation/perf-sched.txt | 6 + tools/perf/builtin-sched.c | 347 ++++++++++++++++++++---- tools/perf/tests/shell/sched.sh | 29 ++ 3 files changed, 332 insertions(+), 50 deletions(-)
Signature · 1 line
2.55.0
PATCH v8 1/4 perf sched: Suppress latency table output when trace samples are missing
When 'perf sched latency' is executed on a perf.data file that lacks tracepoint samples (i.e., a file recorded without the -R flag or containing only non-tracepoint events), perf_session__has_traces() correctly outputs an error message. However, perf_sched__read_events() subsequently falls through and returns 0 (success).
Consequently, caller functions such as perf_sched__lat() assume event processing succeeded and proceed to render empty latency header tables and total summary statistics.
Fix this behaviour by ensuring perf_sched__read_events() aborts early and returns a suitable error code when perf_session__has_traces() evaluates to false.
Additionally, validate thread__get_runtime() against NULL in map_switch_event() to prevent potential null-pointer dereferences.
--- tools/perf/builtin-sched.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)
tools/perf/builtin-sched.c +12 -11
@@ -1833,7 +1833,7 @@ static int map_switch_event(struct perf_sched *sched, struct perf_sample *sampl sched_out:if (sched->map.task_name) { tr =thread__get_runtime (sched->curr_out_thread[this_cpu.cpu]); -if (strcmp (tr->shortname,"" ) ==0 ) +if (tr ==NULL ||strcmp (tr->shortname,"" ) ==0 ) goto out;if (proceed ==1 )
@@ -2001,7 +2001,7 @@ static int perf_sched__read_events(struct perf_sched *sched) .mode = PERF_DATA_MODE_READ, .force = sched->force, }; -int rc =-1 ; +int rc =-1 , err; session =perf_session__new (&data, &sched->tool);if (IS_ERR (session)) {
@@ -2018,18 +2018,19 @@ static int perf_sched__read_events(struct perf_sched *sched)if (perf_session__set_tracepoints_handlers (session, handlers)) goto out_delete; -if (perf_session__has_traces (session,"record -R" )) { -int err =perf_session__process_events (session); -if (err) { -pr_err ("Failed to process events, error %d" , err); - goto out_delete; - } +if (!perf_session__has_traces (session,"record -R" )) + goto out_delete; - sched->nr_events = session->evlist->stats.nr_events[0 ]; - sched->nr_lost_events = session->evlist->stats.total_lost; - sched->nr_lost_chunks = session->evlist->stats.nr_events[PERF_RECORD_LOST ]; + err =perf_session__process_events (session); +if (err) { +pr_err ("Failed to process events, error %d" , err); + goto out_delete; } + sched->nr_events = session->evlist->stats.nr_events[0 ]; + sched->nr_lost_events = session->evlist->stats.total_lost; + sched->nr_lost_chunks = session->evlist->stats.nr_events[PERF_RECORD_LOST ]; + rc =0 ; out_delete:perf_session__delete (session);
Signature · 1 line
2.55.0
PATCH v8 2/4 perf sched: Handle missing trace samples in pipe mode
In reply to Aaron TomlinFor pipe mode streams, event attributes are received dynamically during event processing, meaning session->evlist is not populated prior to perf_session__process_events(). To handle pipe input correctly:
- Register the missing .attr, .tracing_data, .build_id, and .feature callbacks in cmd_sched()
- Promote the handlers array to file-scope (latency_handlers[]) and dynamically assign matching tracepoint handlers
(or process_sched_ignore) inside perf_sched__process_tracepoint_sample() when evsel->handler is NULL; replace process_sched_wakeup_ignore() with process_sched_ignore()
- Perform the trace check post-processing when handling pipe data
--- tools/perf/builtin-sched.c | 72 +++++++++++++++++++++++++++----------- 1 file changed, 51 insertions(+), 21 deletions(-)
tools/perf/builtin-sched.c +51 -21
@@ -1581,12 +1581,6 @@ static int process_sched_wakeup_event(const struct perf_tool *tool,return 0 ; } -static int process_sched_wakeup_ignore (const struct perf_tool *tool__maybe_unused , -struct perf_sample *sample__maybe_unused , -struct machine *machine__maybe_unused ) -{ -return 0 ; -}static bool thread__has_color (struct thread *thread) {
@@ -1938,6 +1932,22 @@ typedef int (*tracepoint_handler)(const struct perf_tool *tool,struct perf_sample *sample, struct machine *machine); +static struct evsel_str_handler latency_handlers[] = { + {"sched:sched_switch" , process_sched_switch_event, }, + {"sched:sched_stat_runtime" , process_sched_runtime_event, }, + {"sched:sched_wakeup" , process_sched_wakeup_event, }, + {"sched:sched_waking" , process_sched_wakeup_event, }, + {"sched:sched_wakeup_new" , process_sched_wakeup_event, }, + {"sched:sched_migrate_task" , process_sched_migrate_task_event, }, +}; + +static int process_sched_ignore (const struct perf_tool *tool__maybe_unused , +struct perf_sample *sample__maybe_unused , +struct machine *machine__maybe_unused ) +{ +return 0 ; +} + static int perf_sched__process_tracepoint_sample(const struct perf_tool *tool __maybe_unused, union perf_event *event __maybe_unused, struct perf_sample *sample,
@@ -1946,7 +1956,23 @@ static int perf_sched__process_tracepoint_sample(const struct perf_tool *tool __struct evsel *evsel = sample->evsel;int err =0 ; -if (evsel->handler !=NULL ) { +if (evsel->handler ==NULL ) { + evsel->handler = process_sched_ignore; +for (size_t i =0 ; i <ARRAY_SIZE (latency_handlers); i++) { +if (!evsel__name_is (evsel, latency_handlers[i].name)) +continue ; + +if (!strcmp (latency_handlers[i].name,"sched:sched_wakeup" ) && + sample->evsel->evlist && +evlist__find_tracepoint_by_name (sample->evsel->evlist,"sched:sched_waking" )) +break ; + + evsel->handler = latency_handlers[i].handler; +break ; + } + } + +if (evsel->handler != process_sched_ignore) {tracepoint_handler f = evsel->handler; err =f (tool, sample, machine); }
@@ -1987,14 +2013,6 @@ static int perf_sched__process_comm(const struct perf_tool *tool __maybe_unused,static int perf_sched__read_events (struct perf_sched *sched) { -struct evsel_str_handler handlers[] = { - {"sched:sched_switch" , process_sched_switch_event, }, - {"sched:sched_stat_runtime" , process_sched_runtime_event, }, - {"sched:sched_wakeup" , process_sched_wakeup_event, }, - {"sched:sched_waking" , process_sched_wakeup_event, }, - {"sched:sched_wakeup_new" , process_sched_wakeup_event, }, - {"sched:sched_migrate_task" , process_sched_migrate_task_event, }, - };struct perf_session *session;struct perf_data data = { .path = input_name,
@@ -2011,14 +2029,17 @@ static int perf_sched__read_events(struct perf_sched *sched)symbol__init (perf_session__env (session)); -/* prefer sched_waking if it is captured */ -if (evlist__find_tracepoint_by_name (session->evlist,"sched:sched_waking" )) - handlers[2 ].handler = process_sched_wakeup_ignore; +if (!perf_data__is_pipe (session->data)) { +/* prefer sched_waking if it is captured */ +if (evlist__find_tracepoint_by_name (session->evlist,"sched:sched_waking" )) + latency_handlers[2 ].handler = process_sched_ignore; -if (perf_session__set_tracepoints_handlers (session, handlers)) - goto out_delete; +if (perf_session__set_tracepoints_handlers (session, latency_handlers)) + goto out_delete; + } -if (!perf_session__has_traces (session,"record -R" )) +if (!perf_data__is_pipe (session->data) && + !perf_session__has_traces (session,"record -R" )) goto out_delete; err =perf_session__process_events (session);
@@ -2027,6 +2048,11 @@ static int perf_sched__read_events(struct perf_sched *sched) goto out_delete; } +if (perf_data__is_pipe (session->data) && + !perf_session__has_traces (session,"record -R" )) { + goto out_delete; + } + sched->nr_events = session->evlist->stats.nr_events[0 ]; sched->nr_lost_events = session->evlist->stats.total_lost; sched->nr_lost_chunks = session->evlist->stats.nr_events[PERF_RECORD_LOST ];
@@ -5168,6 +5194,10 @@ int cmd_sched(int argc, const char **argv) sched.tool.namespaces = perf_event__process_namespaces; sched.tool.lost = perf_event__process_lost; sched.tool.fork = perf_sched__process_fork_event; + sched.tool.attr = perf_event__process_attr; + sched.tool.tracing_data = perf_event__process_tracing_data; + sched.tool.build_id = perf_event__process_build_id; + sched.tool.feature = perf_event__process_feature; argc =parse_options_subcommand (argc, argv, sched_options, sched_subcommands, sched_usage,PARSE_OPT_STOP_AT_NON_OPTION );
Signature · 1 line
2.55.0
PATCH v8 3/4 perf sched latency: Auto-scale latency and runtime display units
In reply to Aaron TomlinCurrently, 'perf sched latency' displays task runtime and delay values exclusively in milliseconds (ms). This can be hard to read when latencies are very small (in the microsecond or nanosecond range) or unusually large (seconds).
Introduce auto-scaling for latency and runtime display columns. Values are dynamically scaled and output with the most appropriate unit:
nanoseconds (ns), microseconds (us), milliseconds (ms), or seconds (s).
Additionally, rename column headers from "Runtime ms", "Avg delay ms", and "Max delay ms" to "Runtime", "Avg delay", and "Max delay" respectively, adjust spacing to maintain column alignment and stripe redundant prefix strings from each row's format string to produce a clean, tabular output.
For illustrative purposes, a comparison of the latency table header before and after this change is shown below:
Before:
-------------------------------------------------------------------------------------------------------------------------------------------
Task | Runtime ms | Count | Avg delay ms | Max delay ms | Max delay start | Max delay end |
-------------------------------------------------------------------------------------------------------------------------------------------
kworker/2:2-mm_:154757 | 0.033 ms | 1 | avg: 0.829 ms | max: 0.829 ms | max start: 169486.543205 s | max end: 169486.544034 s
After:
------------------------------------------------------------------------------------------------------------------------------------------
Task | Runtime | Count | Avg delay | Max delay | Max delay start | Max delay end |
------------------------------------------------------------------------------------------------------------------------------------------
kworker/2:2-mm_:154757 | 32.873 us | 1 | 829.347 us | 829.347 us | 169486.543205 s | 169486.544034 s |
--- tools/perf/builtin-sched.c | 46 ++++++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 14 deletions(-)
tools/perf/builtin-sched.c +32 -14
@@ -246,6 +246,17 @@ struct perf_sched {struct perf_data *data; }; +static int scnprintf_latency_unit (char *buf,size_t size,u64 nsecs) +{ +if (nsecs <1000 ) +return scnprintf (buf, size,"%6" PRIu64" ns" , nsecs); +if (nsecs <NSEC_PER_MSEC ) +return scnprintf (buf, size,"%6.3f us" , (double )nsecs /NSEC_PER_USEC ); +if (nsecs <NSEC_PER_SEC ) +return scnprintf (buf, size,"%6.3f ms" , (double )nsecs /NSEC_PER_MSEC ); +return scnprintf (buf, size,"%6.3f s " , (double )nsecs /NSEC_PER_SEC ); +} +/* per thread run time data */ struct thread_runtime {u64 last_time;/* time of previous sched in/out event */
@@ -1405,6 +1416,8 @@ static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_int i;int ret;u64 avg; +char runtime_lat[32 ]; +char avg_lat[32 ], max_lat[32 ];char max_lat_start[32 ], max_lat_end[32 ];if (!work_list->nb_atoms)
@@ -1419,10 +1432,10 @@ static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_ sched->all_count += work_list->nb_atoms;if (work_list->num_merged >1 ) { - ret =printf (" %s:(%d) " ,thread__comm_str (work_list->thread), + ret =printf (" %s:(%d)" ,thread__comm_str (work_list->thread), work_list->num_merged); }else { - ret =printf (" %s:%d " ,thread__comm_str (work_list->thread), + ret =printf (" %s:%d" ,thread__comm_str (work_list->thread),thread__tid (work_list->thread)); }
@@ -1430,14 +1443,17 @@ static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_printf (" " ); avg = work_list->total_lat / work_list->nb_atoms; +scnprintf_latency_unit (runtime_lat,sizeof (runtime_lat), work_list->total_runtime); +scnprintf_latency_unit (avg_lat,sizeof (avg_lat), avg); +scnprintf_latency_unit (max_lat,sizeof (max_lat), work_list->max_lat);timestamp__scnprintf_usec (work_list->max_lat_start, max_lat_start,sizeof (max_lat_start));timestamp__scnprintf_usec (work_list->max_lat_end, max_lat_end,sizeof (max_lat_end)); -printf ("|%11.3f ms |%9" PRIu64" | avg:%8.3f ms | max:%8.3f ms | max start: %12s s | max end: %12s s\n" , - (double )work_list->total_runtime /NSEC_PER_MSEC , - work_list->nb_atoms, (double )avg /NSEC_PER_MSEC , - (double )work_list->max_lat /NSEC_PER_MSEC , - max_lat_start, max_lat_end); +printf (" |%15s |%9" PRIu64" |%16s |%16s |%20s s |%20s s |\n" , + runtime_lat, + work_list->nb_atoms, avg_lat, max_lat, + max_lat_start, max_lat_end); + }static int pid_cmp (struct work_atoms *l,struct work_atoms *r)
@@ -3625,6 +3641,7 @@ static int perf_sched__lat(struct perf_sched *sched) {int rc =-1 ;struct rb_node *next; +char total_runtime_str[32 ];setup_pager ();
@@ -3637,9 +3654,9 @@ static int perf_sched__lat(struct perf_sched *sched)perf_sched__merge_lat (sched);perf_sched__sort_lat (sched); -printf ("\n -------------------------------------------------------------------------------------------------------------------------------------------\n" ); -printf (" Task | Runtime ms | Count | Avg delay ms | Max delay ms | Max delay start | Max delay end |\n" ); -printf (" -------------------------------------------------------------------------------------------------------------------------------------------\n" ); +printf ("\n ------------------------------------------------------------------------------------------------------------------------------------------\n" ); +printf (" Task | Runtime | Count | Avg delay | Max delay | Max delay start | Max delay end |\n" ); +printf (" ------------------------------------------------------------------------------------------------------------------------------------------\n" ); next =rb_first_cached (&sched->sorted_atom_root);
@@ -3651,11 +3668,12 @@ static int perf_sched__lat(struct perf_sched *sched) next =rb_next (next); } -printf (" -----------------------------------------------------------------------------------------------------------------\n" ); -printf (" TOTAL: |%11.3f ms |%9" PRIu64" |\n" , - (double )sched->all_runtime /NSEC_PER_MSEC , sched->all_count); +printf (" ------------------------------------------------------------------------------------------------------------------------------------------\n" ); +scnprintf_latency_unit (total_runtime_str,sizeof (total_runtime_str), sched->all_runtime); +printf (" TOTAL: |%15s |%9" PRIu64" |\n" , + total_runtime_str, sched->all_count); -printf (" ---------------------------------------------------\n" ); +printf (" ------------------------------------------------------\n" );print_bad_events (sched);printf ("\n" );
Signature · 1 line
2.55.0
PATCH v8 4/4 perf sched latency: Add histogram and time interval options
In reply to Aaron TomlinWhile 'perf sched latency' reports task runtime and delay statistics (average and maximum delay), it does not provide a visual representation of how task wait times are distributed across latency ranges between snapshots (start and finish of the analysis window).
The --histogram option collects CPU wait latencies (time between when a task becomes runnable and when it gets scheduled onto a CPU) into 22 latency buckets, displaying an ASCII bar chart distribution.
The --hist-mode option configures the bucketing scheme:
- log (default). Logarithmic latency buckets ranging from sub-microsecond (< 1 us) up to >= 1.05 seconds
- linear. Equal-width linear latency buckets
(i.e., 100 us steps up to >= 2.1 ms)
The --time option allows filtering trace event processing to a specific time interval [start,stop].
Example histogram output excerpt:
❯ sudo perf sched latency --histogram --CPU 0
CPU Wait Latency Distribution Histogram (between snapshots) (total samples: 36114)
-------------------------------------------------------------------
Latency Range | Count | Pct | Histogram Graph
-------------------------------------------------------------------
< 1 us | 17 | 0.0% | #
2 - 4 us | 673 | 1.9% | #
4 - 8 us | 6237 | 17.3% | ######
8 - 16 us | 3224 | 8.9% | ###
16 - 32 us | 1388 | 3.8% | #
32 - 64 us | 709 | 2.0% | #
64 - 128 us | 690 | 1.9% | #
128 - 256 us | 789 | 2.2% | #
256 - 512 us | 541 | 1.5% | #
512 - 1024 us | 2256 | 6.2% | ##
1 - 2 ms | 3577 | 9.9% | ###
2 - 4 ms | 13259 | 36.7% | ##############
4 - 8 ms | 2523 | 7.0% | ##
8 - 16 ms | 222 | 0.6% | #
16 - 32 ms | 10 | 0.0% | #
>= 1.05 s | 3 | 0.0% | #
---------------------------------------------------------------------- tools/perf/Documentation/perf-sched.txt | 6 + tools/perf/builtin-sched.c | 210 +++++++++++++++++++++++- tools/perf/tests/shell/sched.sh | 29 ++++ 3 files changed, 239 insertions(+), 6 deletions(-)
tools/perf/Documentation/perf-sched.txt +6 -0
@@ -40,6 +40,12 @@ There are several variants of 'perf sched': Tasks with the same command name are merged and the merge count is given within (), However if -p option is used, pid is mentioned. + If -H or --histogram option is passed, a CPU wait latency distribution + histogram is displayed illustrating how long tasks waited for CPU + runtime across latency buckets between snapshots. The --time + option (start,stop) limits analysis to a specific snapshot time interval. + The --hist-mode option (log or linear) configures the latency bucketing scheme. + 'perf sched script' to see a detailed trace of the workload that was recorded (aliased to 'perf script' for now).
tools/perf/builtin-sched.c +204 -6
334 lines not on this page · raw message
tools/perf/tests/shell/sched.sh +29 -0
@@ -76,6 +76,32 @@ test_sched_latency() { fi } +test_sched_latency_histogram() { + echo "Sched latency histogram" + + if ! perf sched latency -H -i "${perfdata}" | grep -q "Latency Distribution Histogram" + then + echo "Sched latency histogram [Failed missing log histogram]" + err=1 + fi + + if ! perf sched latency --histogram --hist-mode linear -i "${perfdata}" | grep -q "Latency Distribution Histogram" + then + echo "Sched latency histogram [Failed missing linear histogram]" + err=1 + fi +} + +test_sched_latency_time() { + echo "Sched latency time filter" + + if ! perf sched latency --time 0, -i "${perfdata}" | grep -q perf-noploop + then + echo "Sched latency time filter [Failed missing output]" + err=1 + fi +} + test_sched_script() { echo "Sched script"
@@ -108,9 +134,12 @@ test_sched_timehist() { test_sched_record test_sched_latency +test_sched_latency_histogram +test_sched_latency_time test_sched_script test_sched_map test_sched_timehist cleanup exit $err +
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH v8 2/4] perf sched: Handle missing trace samples in pipe mode
In reply to Aaron TomlinThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Missing explicit inclusion of `<string.h>` for `strcmp`. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Missing explicit inclusion of <string.h> for strcmp.
--
--- Patch [2]: [PATCH v8 2/4] perf sched: Handle missing trace samples in pipe mode --- commit 4cf6347760b0dc1a50946397641b493e66cea7c0
Author: Aaron Tomlin <[email protected]>
perf sched: Handle missing trace samples in pipe mode
This commit updates perf sched to properly handle pipe mode streams by registering missing attributes and dynamically assigning tracepoint handlers when the evsel handler is uninitialized.
Quoted 4 lines
tools/perf/builtin-sched.c +0
-0
[ ... ]
tools/perf/builtin-sched.c static int perf_sched__process_tracepoint_sample(const struct perf_tool *tool __ · line 1965
Quoted 12 lines
tools/perf/builtin-sched.c +7
-1@@ -1946,7 +1956,23 @@ static int perf_sched__process_tracepoint_sample(const struct perf_tool *tool __struct evsel *evsel = sample->evsel;int err =0 ; -if (evsel->handler !=NULL ) { +if (evsel->handler ==NULL ) { + evsel->handler = process_sched_ignore; +for (size_t i =0 ; i <ARRAY_SIZE (latency_handlers); i++) { +if (!evsel__name_is (evsel, latency_handlers[i].name)) +continue ; + + if (!strcmp(latency_handlers[i].name, "sched:sched_wakeup") &&
[Severity: Low]
This isn't a bug, but are we missing an explicit include for <string.h>?
Even if the file previously omitted it, the perf subsystem guidelines require files using POSIX libc functions like strcmp() to directly include their respective headers. Relying on implicit inclusions can lead to compilation failures under musl libc.
Quoted 13 lines
tools/perf/builtin-sched.c +10
-0+ sample->evsel->evlist && + evlist__find_tracepoint_by_name(sample->evsel->evlist, "sched:sched_waking")) + break; + + evsel->handler = latency_handlers[i].handler; +break ; + } + } + +if (evsel->handler != process_sched_ignore) {tracepoint_handler f = evsel->handler; err =f (tool, sample, machine); }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Quoted 38 lines · Aaron Tomlin
Hi Namhyung, Ian, Arnaldo,
This patch series improves 'perf sched latency' by suppressing misleading empty table output, extending pipe mode stream processing, introducing dynamic unit auto-scaling for latency and runtime statistics, and adding latency histogram visualisation alongside time-span filtering.
Patch 1 addresses an issue where 'perf sched latency' fell through and returned success (0) when
perf_session__has_traces()failed due to missing tracepoint events in a perf.data file. This caused empty header tables and zeroed summary statistics to be rendered. In addition,thread__get_runtime()inmap_switch_event()is guarded against potential NULL pointer dereferences under memory allocation failures.Patch 2 extends pipe mode stream support. Because event attributes in pipe mode are received dynamically during event processing,
session->evlistis not populated prior to event processing.
To handle pipe input correctly:
- Register the missing .attr, .tracing_data, .build_id, and .feature callbacks incmd_sched()- Promote the handlers array to file-scope (latency_handlers[]) and dynamically assign matching tracepoint handlers (or
process_sched_ignore) insideperf_sched__process_tracepoint_sample()whenevsel->handleris NULL; replaceprocess_sched_wakeup_ignore()withprocess_sched_ignore()- Perform the trace check post-processing when handling pipe data
Patch 3 introduces dynamic auto-scaling for latency and runtime display columns (Runtime, Avg delay, Max delay). Previously, all values were unconditionally formatted in milliseconds (ms), making microsecond- or second-scale latencies difficult to read. Columns are now dynamically scaled to the most appropriate unit (ns, us, ms, s), column headers are updated, and format specifiers are aligned character-for-character with table headers.
Quoted 10 lines
Patch 4 adds three new command-line options to 'perf sched latency':
--histogram (-H):
Displays an ASCII bar chart of CPU wait latencies between snapshots
--hist-mode:
Configures the bucketing scheme to either logarithmic (log) or 100 us equal-width linear (linear) mode
--time:
Filters trace event processing to a specified [start,stop] time span
Thanks for these improvements and the tests for coverage! For the series:
Thanks,
Ian
Quoted 136 lines
Changes since v7:
- Added a guard clause in
perf_sched__process_tracepoint_sample()to reduce loop indentation (Namhyung Kim)- Converted dummy
process_sched_wakeup_ignore()references toprocess_sched_ignoreand eliminated redundant code (Namhyung Kim)- Replaced string comparisons with faster integer checks (i.e.,
thread__tid(...)!= 0) for swapper (idle thread) handling (Namhyung Kim)- Added automated test cases for --histogram, --hist-mode, and --time into
tools/perf/tests/shell/sched.sh(Namhyung Kim)- Link to v7: https://lore.kernel.org/lkml/[email protected]/
Changes since v6:
- Fixed O(N) hot-path traversal overhead in
perf_sched__process_tracepoint_sample()for unhandled tracepoints by assigning a dummy ignore handler (process_sched_ignore)- Replaced
evlist__set_tracepoints_handlers()in sample processing with a per-evsel lookup to prevent-EEXISTearly aborts and avoid dropping late-arriving pipe events- Updated commit message for the pipe mode patch to reflect the single-pass handler assignment logic
- Link to v6: https://lore.kernel.org/lkml/[email protected]/
Changes since v5:
- Split pipe mode trace sample handling from Patch 1 into a standalone patch (Namhyung Kim)
- Added a Fixes: tag to the pipe mode patch referencing commit
27295592c22e("perf session: Share the common tracesample_checkroutine asperf_session__has_traces")- Updated commit log with before and after illustrations of table header formatting
- Link to v5: https://lore.kernel.org/lkml/[email protected]/
Changes since v4:
- Added the missing .feature callback to sched.tool in
cmd_sched()- Fixed memory leak by reusing uncompleted work atoms when
sched_inevents are skipped or lost- Promoted the handlers array to file-scope (i.e., latency_handlers[]) and added a dynamic lookup via
evlist__set_tracepoints_handlers()insideperf_sched__process_tracepoint_sample()whenever a sample arrives with an unattached handler (i.e.,evsel->handler== NULL)- Prevented double-counting wakeups (i.e., ignore sched:sched_wakeup) in pipe mode and non-pipe modes
- Added missing trailing pipe in output_lat_thread's new auto-scaling format string
- Stripped redundant prefix strings from each row's format string to produce a clean, tabular output
- Link to v4: https://lore.kernel.org/lkml/[email protected]/
Changes since v3:
- Registered Missing Callbacks. In
perf_tool__initconfiguration insidecmd_sched(), added the .attr, .tracing_data, and .build_id callbacks. Without these callbacks, pipe mode drops header attributes entirely, preventing tracepoints from being populated insession->evlist- Introduced an explicit post-processing pipe check. This ensures that pipe mode aborts correctly and does not produce superfluous empty latency tables when no trace samples are available
- Prevented potential NULL pointer dereference. Added a NULL check for
thread__get_runtime()inmap_switch_event()under memory allocation failures- Fixed column alignment. Modified format specifiers to align character-for-character with header column widths across all rows
- Adjusted header label padding. Updated header label padding and column width delimiters in
perf_sched__lat()to match the exact field format specifiers printed inoutput_lat_thread()- Fixed swapper histogram inclusion. Excluded "swapper" (i.e., CPU-specific idle thread) from
global_histbuckets- Preserved task state machine across --time bounds. Moved time-window filtering inside
add_sched_in_event()to gate latency statistics recording without breaking wakeup state tracking across time interval boundaries- Link to v3: https://lore.kernel.org/lkml/[email protected]/
Changes since v2:
- Ensured vertical pipe separators align across all latency table columns
- Excluded "swapper" (idle thread) latency samples from
global_hist- Preserved task state machine transitions across --time boundaries
- Suppressed empty table headers, total lines, and histogram graphs when no matching trace samples exist (e.g., when --time, --CPU, or --pids exclude all samples), outputting "No matching trace samples found." instead
- Linked to v2: https://lore.kernel.org/lkml/[email protected]/
Changes since v1:
- Fixed integer overflow in
latency_bucket()(Ian Rogers)- Linked to v1: https://lore.kernel.org/lkml/[email protected]/
Aaron Tomlin (4):
perf sched: Suppress latency table output when trace samples are
missing
perf sched: Handle missing trace samples in pipe mode perf sched latency: Auto-scale latency and runtime display units perf sched latency: Add histogram and time interval optionstools/perf/Documentation/perf-sched.txt | 6 + tools/perf/builtin-sched.c | 347 ++++++++++++++++++++---- tools/perf/tests/shell/sched.sh | 29 ++ 3 files changed, 332 insertions(+), 50 deletions(-)--
2.55.0
Namhyung Kim
maintainerHello,
Quoted 6 lines · Aaron Tomlin
Hi Namhyung, Ian, Arnaldo,
This patch series improves 'perf sched latency' by suppressing misleading empty table output, extending pipe mode stream processing, introducing dynamic unit auto-scaling for latency and runtime statistics, and adding latency histogram visualisation alongside time-span filtering.
Thanks for working on this. But it doesn't apply unfortunately. Can you please rebase on to the current perf-tools-next?
Thanks,
Namhyung
Quoted 50 lines · Ian Rogers
Quoted 38 lines · Aaron Tomlin
Hi Namhyung, Ian, Arnaldo,
This patch series improves 'perf sched latency' by suppressing misleading empty table output, extending pipe mode stream processing, introducing dynamic unit auto-scaling for latency and runtime statistics, and adding latency histogram visualisation alongside time-span filtering.
Patch 1 addresses an issue where 'perf sched latency' fell through and returned success (0) when
perf_session__has_traces()failed due to missing tracepoint events in a perf.data file. This caused empty header tables and zeroed summary statistics to be rendered. In addition,thread__get_runtime()inmap_switch_event()is guarded against potential NULL pointer dereferences under memory allocation failures.Patch 2 extends pipe mode stream support. Because event attributes in pipe mode are received dynamically during event processing,
session->evlistis not populated prior to event processing.
To handle pipe input correctly:
- Register the missing .attr, .tracing_data, .build_id, and .feature callbacks incmd_sched()- Promote the handlers array to file-scope (latency_handlers[]) and dynamically assign matching tracepoint handlers (or
process_sched_ignore) insideperf_sched__process_tracepoint_sample()whenevsel->handleris NULL; replaceprocess_sched_wakeup_ignore()withprocess_sched_ignore()- Perform the trace check post-processing when handling pipe data
Patch 3 introduces dynamic auto-scaling for latency and runtime display columns (Runtime, Avg delay, Max delay). Previously, all values were unconditionally formatted in milliseconds (ms), making microsecond- or second-scale latencies difficult to read. Columns are now dynamically scaled to the most appropriate unit (ns, us, ms, s), column headers are updated, and format specifiers are aligned character-for-character with table headers.
Quoted 10 lines
Patch 4 adds three new command-line options to 'perf sched latency':
--histogram (-H):
Displays an ASCII bar chart of CPU wait latencies between snapshots
--hist-mode:
Configures the bucketing scheme to either logarithmic (log) or 100 us equal-width linear (linear) mode
--time:
Filters trace event processing to a specified [start,stop] time span
Quoted 6 lines
Thanks for these improvements and the tests for coverage! For the series:
Reviewed-by Ian Rogers <[email protected]>Thanks,
Ian
Thank you Ian!
Signature · 1 line
Aaron Tomlin