perf sched latency: Refine outputs, unit scaling, and histogram support
v4 cover perf sched latency: Refine outputs, unit scaling, and histogram support
This patch series improves 'perf sched latency' by fixing misleading empty table output, introducing dynamic unit auto-scaling for latency and runtime statistics, and adding latency histogram visualisation along with time-span filtering.
Patch 1 fixes an issue where 'perf sched latency' would fall through and return success (0) when perf_session__has_traces() failed due to missing tracepoint events in perf.data. This resulted in empty header tables and zeroed summary statistics being printed. To support pipe mode (i.e., 'perf sched latency -i -'), missing callbacks (i.e., .attr, .tracing_data, .build_id) are registered in cmd_sched(), and trace checks are performed post-processing when handling pipe streams. Additionally, map_switch_event() is guarded against potential NULL pointer dereferences under memory allocation failures.
Patch 2 introduces dynamic auto-scaling for latency and runtime display columns (Runtime, Avg delay, Max delay). Previously, all values were unconditionally formatted in milliseconds (ms), making microsecond or second-scale latencies hard to read. Columns are now scaled dynamically to the most appropriate unit (ns, us, ms, s), column headers are updated, and format specifiers (%14s for Runtime, %11s for Avg/Max delay) are aligned character-for-character with the table headers.
Patch 3 adds three new command-line options to 'perf sched latency':
--histogram (-H):
Displays an ASCII bar chart of CPU wait latencies between snapshots
--hist-mode:
Configures the bucketing scheme to either logarithmic (log) or 100 us equal-width linear (linear) mode
--time:
Filters event processing to a specified [start,stop] time span
Changes since v3:
- Registered Missing Callbacks. In perf_tool__init configuration inside cmd_sched(), added the .attr, .tracing_data, and .build_id callbacks. Without these callbacks, pipe mode drops header attributes entirely, preventing tracepoints from being populated in session->evlist
- Introduced an explicit post-processing pipe check. This ensures that pipe mode aborts correctly and does not produce superfluous empty latency tables when no trace samples are available
- Prevented potential NULL pointer dereference. Added a NULL check for thread__get_runtime() in map_switch_event() under memory allocation failures
- Fixed column alignment. Modified format specifiers to align character-for-character with header column widths across all rows
- Adjusted header label padding. Updated header label padding and column width delimiters in perf_sched__lat() to match the exact field format specifiers printed in output_lat_thread()
- Fixed swapper histogram inclusion. Excluded "swapper" (i.e., CPU-specific idle thread) from global_hist buckets
- Preserved task state machine across --time bounds. Moved time-window filtering inside add_sched_in_event() to gate latency statistics recording without breaking wakeup state tracking across time interval boundaries
- Link to v3: https://lore.kernel.org/lkml/[email protected]/
Changes since v2:
- Ensured vertical pipe separators align across all latency table columns
- Excluded "swapper" (idle thread) latency samples from global_hist
- Preserved task state machine transitions across --time boundaries
- Suppressed empty table headers, total lines, and histogram graphs when no matching trace samples exist (e.g., when --time, --CPU, or --pids exclude all samples), outputting "No matching trace samples found." instead
- Linked to v2: https://lore.kernel.org/lkml/[email protected]/
Changes since v1:
- Fixed integer overflow in latency_bucket() (Ian Rogers)
- Linked to v1: https://lore.kernel.org/lkml/[email protected]/
Aaron Tomlin (3):
perf sched: Suppress latency table output when trace samples are
missing
perf sched latency: Auto-scale latency and runtime display units perf sched latency: Add histogram and time interval options
tools/perf/Documentation/perf-sched.txt | 6 + tools/perf/builtin-sched.c | 263 +++++++++++++++++++++--- 2 files changed, 244 insertions(+), 25 deletions(-)
Signature · 1 line
2.55.0
PATCH v4 1/3 perf sched: Suppress latency table output when trace samples are missing
When 'perf sched latency' is executed on a perf.data file that lacks tracepoint samples (i.e., a file recorded without the -R flag or containing only non-tracepoint events), perf_session__has_traces() correctly outputs an error message. However, perf_sched__read_events() subsequently falls through and returns 0 (success).
Consequently, caller functions such as perf_sched__lat() assume event processing succeeded and proceed to render empty latency header tables and total summary statistics.
Fix this behavior by ensuring perf_sched__read_events() aborts early and returns a suitable error code when perf_session__has_traces() evaluates to false.
For pipe mode streams, event attributes are received dynamically during event processing, meaning session->evlist is not populated prior to perf_session__process_events(). To handle pipe input correctly, register the missing .attr, .tracing_data, and .build_id callbacks in cmd_sched(), and perform the trace check after event processing when handling pipe data.
Additionally, validate thread__get_runtime() against NULL in map_switch_event() to prevent potential null-pointer dereferences.
--- tools/perf/builtin-sched.c | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-)
tools/perf/builtin-sched.c +21 -11
@@ -1833,7 +1833,7 @@ static int map_switch_event(struct perf_sched *sched, struct perf_sample *sampl sched_out:if (sched->map.task_name) { tr =thread__get_runtime (sched->curr_out_thread[this_cpu.cpu]); -if (strcmp (tr->shortname,"" ) ==0 ) +if (tr ==NULL ||strcmp (tr->shortname,"" ) ==0 ) goto out;if (proceed ==1 )
@@ -2001,7 +2001,7 @@ static int perf_sched__read_events(struct perf_sched *sched) .mode = PERF_DATA_MODE_READ, .force = sched->force, }; -int rc =-1 ; +int rc =-1 , err; session =perf_session__new (&data, &sched->tool);if (IS_ERR (session)) {
@@ -2018,18 +2018,25 @@ static int perf_sched__read_events(struct perf_sched *sched)if (perf_session__set_tracepoints_handlers (session, handlers)) goto out_delete; -if (perf_session__has_traces (session,"record -R" )) { -int err =perf_session__process_events (session); -if (err) { -pr_err ("Failed to process events, error %d" , err); - goto out_delete; - } +if (!perf_data__is_pipe (session->data) && + !perf_session__has_traces (session,"record -R" )) + goto out_delete; - sched->nr_events = session->evlist->stats.nr_events[0 ]; - sched->nr_lost_events = session->evlist->stats.total_lost; - sched->nr_lost_chunks = session->evlist->stats.nr_events[PERF_RECORD_LOST ]; + err =perf_session__process_events (session); +if (err) { +pr_err ("Failed to process events, error %d" , err); + goto out_delete; } +if (perf_data__is_pipe (session->data) && + !perf_session__has_traces (session,"record -R" )) { + goto out_delete; + } + + sched->nr_events = session->evlist->stats.nr_events[0 ]; + sched->nr_lost_events = session->evlist->stats.total_lost; + sched->nr_lost_chunks = session->evlist->stats.nr_events[PERF_RECORD_LOST ]; + rc =0 ; out_delete:perf_session__delete (session);
@@ -5167,6 +5174,9 @@ int cmd_sched(int argc, const char **argv) sched.tool.namespaces = perf_event__process_namespaces; sched.tool.lost = perf_event__process_lost; sched.tool.fork = perf_sched__process_fork_event; + sched.tool.attr = perf_event__process_attr; + sched.tool.tracing_data = perf_event__process_tracing_data; + sched.tool.build_id = perf_event__process_build_id; argc =parse_options_subcommand (argc, argv, sched_options, sched_subcommands, sched_usage,PARSE_OPT_STOP_AT_NON_OPTION );
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH v4 1/3] perf sched: Suppress latency table output when trace samples are missing
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] Tracepoint handlers are not set for events read from a pipe, causing all trace samples to be…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] Tracepoint handlers are not set for events read from a pipe, causing all trace samples to be silently dropped.
- [Medium] The patch adds dynamic callbacks for pipe mode but omits the .feature callback, leaving perf_env unpopulated. --
--- Patch [1]: [PATCH v4 1/3] perf sched: Suppress latency table output when trace samples are missing --- commit e9be5c25be0368a795779cd7fdb10479f3b3c099
Author: Aaron Tomlin <[email protected]>
perf sched: Suppress latency table output when trace samples are missing
This commit updates perf_sched__read_events() to abort early and return an error when trace samples are missing. It also adds callbacks in cmd_sched() to support pipe mode streams and adds a null check in map_switch_event().
Quoted 4 lines
tools/perf/builtin-sched.c +0
-0
[ ... ]
tools/perf/builtin-sched.c static int perf_sched__read_events(struct perf_sched *sched) · line 2019
Quoted 12 lines
tools/perf/builtin-sched.c +1
-1@@ -2001,7 +2001,7 @@ static int perf_sched__read_events(struct perf_sched *sched) .mode = PERF_DATA_MODE_READ, .force = sched->force, }; -int rc =-1 ; +int rc =-1 , err; session =perf_session__new (&data, &sched->tool);if (IS_ERR (session)) {@@ -2018,18 +2018,25 @@ static int perf_sched__read_events(struct perf_sched *sched)if (perf_session__set_tracepoints_handlers (session, handlers)) goto out_delete;
[Severity: Medium]
Does setting the tracepoint handlers before processing events work for pipe mode?
Since the evlist is empty in pipe mode until events are processed, it appears perf_session__set_tracepoints_handlers() won't set up the handlers.
When samples later arrive in perf_sched__process_tracepoint_sample():
if (evsel->handler != NULL) {
tracepoint_handler f = evsel->handler;
err = f(tool, sample, machine);
}Will they be silently dropped because evsel->handler is NULL?
Quoted 14 lines
tools/perf/builtin-sched.c +4
-9-if (perf_session__has_traces (session,"record -R" )) { -int err =perf_session__process_events (session); -if (err) { -pr_err ("Failed to process events, error %d" , err); - goto out_delete; - } +if (!perf_data__is_pipe (session->data) && + !perf_session__has_traces (session,"record -R" )) + goto out_delete; - sched->nr_events = session->evlist->stats.nr_events[0 ]; - sched->nr_lost_events = session->evlist->stats.total_lost; - sched->nr_lost_chunks = session->evlist->stats.nr_events[PERF_RECORD_LOST ]; + err =perf_session__process_events (session);
[ ... ]
tools/perf/builtin-sched.c int cmd_sched(int argc, const char **argv) · line 5179
Quoted 7 lines
tools/perf/builtin-sched.c +3
-0@@ -5167,6 +5174,9 @@ int cmd_sched(int argc, const char **argv) sched.tool.namespaces = perf_event__process_namespaces; sched.tool.lost = perf_event__process_lost; sched.tool.fork = perf_sched__process_fork_event; + sched.tool.attr = perf_event__process_attr; + sched.tool.tracing_data = perf_event__process_tracing_data; + sched.tool.build_id = perf_event__process_build_id;
[Severity: Medium]
Does this code omit the feature callback required for pipe mode?
Without registering perf_event__process_feature to tool.feature, it seems feature events from the pipe stream will be silently dropped, leaving struct perf_env unpopulated.
Quoted 3 lines
tools/perf/builtin-sched.c +0
-0argc =parse_options_subcommand (argc, argv, sched_options, sched_subcommands, sched_usage,PARSE_OPT_STOP_AT_NON_OPTION );
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [PATCH v4 1/3] perf sched: Suppress latency table output when trace samples are missing
Quoted 74 lines · On Wed, Jul 29, 2026 at 03:00:38PM +0000, [email protected] wrote:
--- Patch [1]: [PATCH v4 1/3] perf sched: Suppress latency table output when trace samples are missing --- commit
e9be5c25be0368a795779cd7fdb10479f3b3c099
Author: Aaron Tomlin <[email protected]>perf sched: Suppress latency table output when trace samples are missing
This commit updates
perf_sched__read_events()to abort early and return an error when trace samples are missing. It also adds callbacks incmd_sched()to support pipe mode streams and adds a null check inmap_switch_event().Quoted 4 lines
tools/perf/builtin-sched.c +0
-0[ ... ]
tools/perf/builtin-sched.c static int perf_sched__read_events(struct perf_sched *sched) · line 2019
Quoted 12 lines
tools/perf/builtin-sched.c +1
-1@@ -2001,7 +2001,7 @@ static int perf_sched__read_events(struct perf_sched *sched) .mode = PERF_DATA_MODE_READ, .force = sched->force, }; -int rc =-1 ; +int rc =-1 , err; session =perf_session__new (&data, &sched->tool);if (IS_ERR (session)) {@@ -2018,18 +2018,25 @@ static int perf_sched__read_events(struct perf_sched *sched)if (perf_session__set_tracepoints_handlers (session, handlers)) goto out_delete;[Severity: Medium]
Does setting the tracepoint handlers before processing events work for pipe mode?Since the evlist is empty in pipe mode until events are processed, it appears
perf_session__set_tracepoints_handlers()won't set up the handlers.When samples later arrive in
perf_sched__process_tracepoint_sample():if (evsel->handler != NULL) { tracepoint_handler f = evsel->handler; err = f(tool, sample, machine); }Will they be silently dropped because
evsel->handleris NULL?Quoted 14 lines
tools/perf/builtin-sched.c +4
-9-if (perf_session__has_traces (session,"record -R" )) { -int err =perf_session__process_events (session); -if (err) { -pr_err ("Failed to process events, error %d" , err); - goto out_delete; - } +if (!perf_data__is_pipe (session->data) && + !perf_session__has_traces (session,"record -R" )) + goto out_delete; - sched->nr_events = session->evlist->stats.nr_events[0 ]; - sched->nr_lost_events = session->evlist->stats.total_lost; - sched->nr_lost_chunks = session->evlist->stats.nr_events[PERF_RECORD_LOST ]; + err =perf_session__process_events (session);[ ... ]
tools/perf/builtin-sched.c int cmd_sched(int argc, const char **argv) · line 5179
Quoted 7 lines
tools/perf/builtin-sched.c +3
-0@@ -5167,6 +5174,9 @@ int cmd_sched(int argc, const char **argv) sched.tool.namespaces = perf_event__process_namespaces; sched.tool.lost = perf_event__process_lost; sched.tool.fork = perf_sched__process_fork_event; + sched.tool.attr = perf_event__process_attr; + sched.tool.tracing_data = perf_event__process_tracing_data; + sched.tool.build_id = perf_event__process_build_id;[Severity: Medium]
Does this code omit the feature callback required for pipe mode?
Quoted 7 lines
Without registering
perf_event__process_featureto tool.feature, it seems feature events from the pipe stream will be silently dropped, leaving structperf_envunpopulated.Quoted 3 lines
tools/perf/builtin-sched.c +0
-0argc =parse_options_subcommand (argc, argv, sched_options, sched_subcommands, sched_usage,PARSE_OPT_STOP_AT_NON_OPTION );
Hi Namhyung, Arnaldo, Ian,
This feedback is valid.
In pipe mode, header attributes arrive dynamically during event processing, meaning session->evlist starts empty prior to perf_session__process_events(). Calling perf_session__set_tracepoints_handlers() upfront resulted in zero handlers being attached, causing incoming tracepoint samples to be silently dropped (i.e., evsel->handler == NULL).
I will resolve this in the next iteration.
Kind regards,
Signature · 1 line
Aaron Tomlin
PATCH v4 2/3 perf sched latency: Auto-scale latency and runtime display units
In reply to Aaron TomlinCurrently, 'perf sched latency' displays task runtime and delay values exclusively in milliseconds (ms). This can be hard to read when latencies are very small (in the microsecond or nanosecond range) or unusually large (seconds).
Introduce auto-scaling for latency and runtime display columns. Values are dynamically scaled and output with the most appropriate unit:
nanoseconds (ns), microseconds (us), milliseconds (ms), or seconds (s).
Additionally, rename column headers from "Runtime ms", "Avg delay ms", and "Max delay ms" to "Runtime", "Avg delay", and "Max delay" respectively, and adjust spacing to maintain column alignment.
--- tools/perf/builtin-sched.c | 40 +++++++++++++++++++++++++++----------- 1 file changed, 29 insertions(+), 11 deletions(-)
tools/perf/builtin-sched.c +29 -11
@@ -246,6 +246,17 @@ struct perf_sched {struct perf_data *data; }; +static int scnprintf_latency_unit (char *buf,size_t size,u64 nsecs) +{ +if (nsecs <1000 ) +return scnprintf (buf, size,"%6" PRIu64" ns" , nsecs); +if (nsecs <NSEC_PER_MSEC ) +return scnprintf (buf, size,"%6.3f us" , (double )nsecs /NSEC_PER_USEC ); +if (nsecs <NSEC_PER_SEC ) +return scnprintf (buf, size,"%6.3f ms" , (double )nsecs /NSEC_PER_MSEC ); +return scnprintf (buf, size,"%6.3f s " , (double )nsecs /NSEC_PER_SEC ); +} +/* per thread run time data */ struct thread_runtime {u64 last_time;/* time of previous sched in/out event */
@@ -1405,6 +1416,8 @@ static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_int i;int ret;u64 avg; +char runtime_lat[32 ]; +char avg_lat[32 ], max_lat[32 ];char max_lat_start[32 ], max_lat_end[32 ];if (!work_list->nb_atoms)
@@ -1430,14 +1443,17 @@ static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_printf (" " ); avg = work_list->total_lat / work_list->nb_atoms; +scnprintf_latency_unit (runtime_lat,sizeof (runtime_lat), work_list->total_runtime); +scnprintf_latency_unit (avg_lat,sizeof (avg_lat), avg); +scnprintf_latency_unit (max_lat,sizeof (max_lat), work_list->max_lat);timestamp__scnprintf_usec (work_list->max_lat_start, max_lat_start,sizeof (max_lat_start));timestamp__scnprintf_usec (work_list->max_lat_end, max_lat_end,sizeof (max_lat_end)); -printf ("|%11.3f ms |%9" PRIu64" | avg:%8.3f ms | max:%8.3f ms | max start: %12s s | max end: %12s s\n" , - (double )work_list->total_runtime /NSEC_PER_MSEC , - work_list->nb_atoms, (double )avg /NSEC_PER_MSEC , - (double )work_list->max_lat /NSEC_PER_MSEC , - max_lat_start, max_lat_end); +printf ("|%14s |%9" PRIu64" | avg:%11s | max:%11s | max start: %12s s | max end: %12s s\n" , + runtime_lat, + work_list->nb_atoms, avg_lat, max_lat, + max_lat_start, max_lat_end); + }static int pid_cmp (struct work_atoms *l,struct work_atoms *r)
@@ -3605,6 +3621,7 @@ static int perf_sched__lat(struct perf_sched *sched) {int rc =-1 ;struct rb_node *next; +char total_runtime_str[32 ];setup_pager ();
@@ -3617,9 +3634,9 @@ static int perf_sched__lat(struct perf_sched *sched)perf_sched__merge_lat (sched);perf_sched__sort_lat (sched); -printf ("\n -------------------------------------------------------------------------------------------------------------------------------------------\n" ); -printf (" Task | Runtime ms | Count | Avg delay ms | Max delay ms | Max delay start | Max delay end |\n" ); -printf (" -------------------------------------------------------------------------------------------------------------------------------------------\n" ); +printf ("\n ------------------------------------------------------------------------------------------------------------------------------------------------\n" ); +printf (" Task | Runtime | Count | Avg delay | Max delay | Max delay start | Max delay end |\n" ); +printf (" ------------------------------------------------------------------------------------------------------------------------------------------------\n" ); next =rb_first_cached (&sched->sorted_atom_root);
@@ -3631,9 +3648,10 @@ static int perf_sched__lat(struct perf_sched *sched) next =rb_next (next); } -printf (" -----------------------------------------------------------------------------------------------------------------\n" ); -printf (" TOTAL: |%11.3f ms |%9" PRIu64" |\n" , - (double )sched->all_runtime /NSEC_PER_MSEC , sched->all_count); +printf (" ------------------------------------------------------------------------------------------------------------------------------------------------\n" ); +scnprintf_latency_unit (total_runtime_str,sizeof (total_runtime_str), sched->all_runtime); +printf (" TOTAL: |%14s |%9" PRIu64" |\n" , + total_runtime_str, sched->all_count);printf (" ---------------------------------------------------\n" );
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH v4 2/3] perf sched latency: Auto-scale latency and runtime display units
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] The patch breaks the tabular alignment of the `perf sched latency` output due to mismatched column…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The patch breaks the tabular alignment of the perf sched latency output due to mismatched column widths and overextended separator lines. --
--- Patch [2]: [PATCH v4 2/3] perf sched latency: Auto-scale latency and runtime display units --- commit 652fc7d49ad81ba1ea0ce14f520241722f7c1054
Author: Aaron Tomlin <[email protected]>
perf sched latency: Auto-scale latency and runtime display units
This commit introduces auto-scaling for latency and runtime display columns in the perf sched latency output. Values are dynamically scaled and output with the most appropriate unit (ns, us, ms, or s). It also renames column headers and adjusts spacing to accommodate these changes.
Quoted 4 lines
tools/perf/builtin-sched.c +0
-0
[ ... ]
Quoted 11 lines
tools/perf/builtin-sched.c +3
-1@@ -1430,14 +1443,17 @@ static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_printf (" " ); avg = work_list->total_lat / work_list->nb_atoms; +scnprintf_latency_unit (runtime_lat,sizeof (runtime_lat), work_list->total_runtime); +scnprintf_latency_unit (avg_lat,sizeof (avg_lat), avg); +scnprintf_latency_unit (max_lat,sizeof (max_lat), work_list->max_lat);timestamp__scnprintf_usec (work_list->max_lat_start, max_lat_start,sizeof (max_lat_start));timestamp__scnprintf_usec (work_list->max_lat_end, max_lat_end,sizeof (max_lat_end)); - printf("|%11.3f ms |%9" PRIu64 " | avg:%8.3f ms | max:%8.3f ms | max start: %12s s | max end: %12s s\n",
tools/perf/builtin-sched.c static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_ · line 1457
Quoted 10 lines
tools/perf/builtin-sched.c +5
-4- (double )work_list->total_runtime /NSEC_PER_MSEC , - work_list->nb_atoms, (double )avg /NSEC_PER_MSEC , - (double )work_list->max_lat /NSEC_PER_MSEC , - max_lat_start, max_lat_end); +printf ("|%14s |%9" PRIu64" | avg:%11s | max:%11s | max start: %12s s | max end: %12s s\n" , + runtime_lat, + work_list->nb_atoms, avg_lat, max_lat, + max_lat_start, max_lat_end); + }
[Severity: Low]
Does this format string in output_lat_thread() shift the subsequent columns out of alignment with the header?
The data row format string here allocates 15 characters (14 padded plus 1 trailing space) for the Runtime column.
However, looking at the header defined in perf_sched__lat(), the header
allocates 16 characters for that same column (" Runtime "). This
1-character discrepancy appears to shift the data rows to the left relative to the header.
[ ... ]
tools/perf/builtin-sched.c static int perf_sched__lat(struct perf_sched *sched) · line 3639
Quoted 10 lines
tools/perf/builtin-sched.c +3
-3@@ -3617,9 +3634,9 @@ static int perf_sched__lat(struct perf_sched *sched)perf_sched__merge_lat (sched);perf_sched__sort_lat (sched); -printf ("\n -------------------------------------------------------------------------------------------------------------------------------------------\n" ); -printf (" Task | Runtime ms | Count | Avg delay ms | Max delay ms | Max delay start | Max delay end |\n" ); -printf (" -------------------------------------------------------------------------------------------------------------------------------------------\n" ); +printf ("\n ------------------------------------------------------------------------------------------------------------------------------------------------\n" ); +printf (" Task | Runtime | Count | Avg delay | Max delay | Max delay start | Max delay end |\n" ); +printf (" ------------------------------------------------------------------------------------------------------------------------------------------------\n" );
[Severity: Low]
Are these horizontal separator lines slightly too long for the table?
It looks like the separator line was lengthened to 144 hyphens, which overshoots the right edge of the table columns by several characters, breaking the visual alignment of the output.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH v4 3/3 perf sched latency: Add histogram and time interval options
In reply to Aaron TomlinWhile 'perf sched latency' reports task runtime and delay statistics (average and maximum delay), it does not provide a visual representation of how task wait times are distributed across latency ranges between snapshots (start and finish of the analysis window).
The --histogram option collects CPU wait latencies (time between when a task becomes runnable and when it gets scheduled onto a CPU) into 22 latency buckets, displaying an ASCII bar chart distribution.
The --hist-mode option configures the bucketing scheme:
- log (default). Logarithmic latency buckets ranging from sub-microsecond (< 1 us) up to >= 1.05 seconds
- linear. Equal-width linear latency buckets
(i.e., 100 us steps up to >= 2.1 ms)
The --time option allows filtering trace event processing to a specific time interval [start,stop].
Example histogram output excerpt:
❯ sudo perf sched latency --histogram --CPU 0
CPU Wait Latency Distribution Histogram (between snapshots) (total samples: 36114)
-------------------------------------------------------------------
Latency Range | Count | Pct | Histogram Graph
-------------------------------------------------------------------
< 1 us | 17 | 0.0% | #
2 - 4 us | 673 | 1.9% | #
4 - 8 us | 6237 | 17.3% | ######
8 - 16 us | 3224 | 8.9% | ###
16 - 32 us | 1388 | 3.8% | #
32 - 64 us | 709 | 2.0% | #
64 - 128 us | 690 | 1.9% | #
128 - 256 us | 789 | 2.2% | #
256 - 512 us | 541 | 1.5% | #
512 - 1024 us | 2256 | 6.2% | ##
1 - 2 ms | 3577 | 9.9% | ###
2 - 4 ms | 13259 | 36.7% | ##############
4 - 8 ms | 2523 | 7.0% | ##
8 - 16 ms | 222 | 0.6% | #
16 - 32 ms | 10 | 0.0% | #
>= 1.05 s | 3 | 0.0% | #
---------------------------------------------------------------------- tools/perf/Documentation/perf-sched.txt | 6 + tools/perf/builtin-sched.c | 191 +++++++++++++++++++++++- 2 files changed, 194 insertions(+), 3 deletions(-)
tools/perf/Documentation/perf-sched.txt +6 -0
@@ -40,6 +40,12 @@ There are several variants of 'perf sched': Tasks with the same command name are merged and the merge count is given within (), However if -p option is used, pid is mentioned. + If -H or --histogram option is passed, a CPU wait latency distribution + histogram is displayed illustrating how long tasks waited for CPU + runtime across latency buckets between snapshots. The --time + option (start,stop) limits analysis to a specific snapshot time interval. + The --hist-mode option (log or linear) configures the latency bucketing scheme. + 'perf sched script' to see a detailed trace of the workload that was recorded (aliased to 'perf script' for now).
tools/perf/builtin-sched.c +188 -3
291 lines not on this page · raw message
Signature · 1 line
2.55.0