perf ftrace: Support inlined functions and display enhancements in function graph tracer
v2 cover perf ftrace: Support inlined functions and display enhancements in function graph tracer
The Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call-graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining (such as the scheduler, memory management, locking primitives, and RCU).
When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment (e.g. /* <-wake_up_new_task+0x1d1/0x3e0 */). By interrogating DWARF debug information from the kernel image (vmlinux) utilising libdw, perf ftrace resolves this return address to its inlined callchain and synthesises the intermediate inlined frames directly into the streamed call-graph.
Synthesised inlined functions are rendered with an explicit /* (inline) */ annotation at both entry and exit:
# CPU DURATION FUNCTION CALLS
# | | | | | | |
1) 0.856 us | mutex_unlock(lock=0xffffffffbc28bfa0);
0) | wake_up_new_task(p=0xffff8b1f183bac80) {
0) | rb_irq_work_queue() { /* (inline) */
0) | rb_wakeups() { /* (inline) */
0) 1.374 us | housekeeping_any_cpu(type=3);
0) | arch_irq_work_raise() {
0) | apic_wait_icr_idle() { /* (inline) */
0) | x2apic_send_IPI_self(vector=246) {
0) | instr_sysvec_irq_work() { /* (inline) */
0) | irq_enter_rcu() {
0) | instr_sysvec_irq_work() { /* (inline) */
0) 0.673 us | irqtime_account_irq();
0) | } /* instr_sysvec_irq_work (inline) */
0) 1.697 us | }
This series decomposes the feature into a modular, bisectable progression of four patches:
1. Stack frame optimisation in __cmd_ftrace()
Replaces the 4096-byte stack array in __cmd_ftrace() with dynamic heap allocation alongside struct strbuf streaming, reducing the stack frame footprint by over 96% (down to ~160 bytes) while ensuring complete line-delimited records and robust EOF handling.
2. Return address comment filtering
Introduces ftrace_parse_retaddr() and ftrace_filter_retaddr() with --filter-retaddr and --graph-opts filter-retaddr to strip caller comments from trace output, accompanied by an initial unit test suite in tools/perf/tests/ftrace.c.
3. Inlined function call-graph reconstruction
Introduces --inline and -k or --vmlinux, dynamically synthesising inlined frames marked with /* (inline) */. Correctly matches bare closing braces ('}') when funcgraph-tail is disabled, clamps stack depth to prevent indentation drift, and expands the automated test suite to verify inline resolution, comment filtering, and stack saturation recovery.
4. Execution duration suppression
Introduces --graph-opts noduration (and duration=[0|1]) to toggle the kernel's funcgraph-duration tracing option, omitting the duration column when the user is focused purely on call-graph topology and inlined hierarchy.
Changes since v1:
- Modularised the implementation into a 4-patch bisectable series rather than a monolithic commit
- Resolved potential infinite loop on poll() upon trace_pipe EOF
- Fixed stuck call stack frames when funcgraph-tail is disabled by supporting bare closing delimiters matching LIFO stack frames
- Prevented unbounded indentation growth on deep call trees by clamping inline expansion and synchronising cs->inlined_depth to frames successfully pushed onto cs->stack
- Added dedicated unit tests covering bare closing braces and stack saturation recovery
- Refined option documentation in perf-ftrace.txt
- Link to v1: https://lore.kernel.org/lkml/[email protected]/
Aaron Tomlin (4):
perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming perf ftrace: Support filtering return address comments perf ftrace: Support display of inlined functions in function graph
tracer
perf ftrace: Support omitting execution duration in function graph
tracer
tools/perf/Documentation/perf-ftrace.txt | 21 + tools/perf/builtin-ftrace.c | 126 +++++- tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/ftrace.c | 240 ++++++++++ tools/perf/tests/tests.h | 1 + tools/perf/util/Build | 1 + tools/perf/util/ftrace.c | 537 +++++++++++++++++++++++ tools/perf/util/ftrace.h | 17 + 9 files changed, 933 insertions(+), 12 deletions(-) create mode 100644 tools/perf/tests/ftrace.c create mode 100644 tools/perf/util/ftrace.c
Signature · 1 line
2.55.0
PATCH v2 1/4 perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming
In __cmd_ftrace(), incoming trace data from trace_pipe was historically buffered in a fixed 4096-byte array allocated directly on the stack frame. Furthermore, reads were written directly to stdout via fwrite() in arbitrary chunk sizes, rather than as discrete, well-delimited trace lines.
Refactor __cmd_ftrace() to employ dynamic heap allocation for the read buffer alongside a dynamic struct strbuf to accumulate streamed characters into whole lines. Pre-allocating the strbuf ensures that line processing incurs zero heap reallocations during steady-state polling, while the stack frame of __cmd_ftrace() is reduced by over 96% (down to ~160 bytes).
--- tools/perf/builtin-ftrace.c | 52 ++++++++++++++++++++++++++++++------- 1 file changed, 43 insertions(+), 9 deletions(-)
tools/perf/builtin-ftrace.c +43 -9
@@ -41,7 +41,10 @@#include "util/units.h" #include "util/parse-sublevel-options.h" +#include "util/strbuf.h" +#define DEFAULT_TRACER "function_graph" +#define TRACE_BUF_SIZE 4096static volatile sig_atomic_t workload_exec_errno;static volatile sig_atomic_t done;
@@ -738,7 +741,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace) {char *trace_file;int trace_fd; -char buf[4096 ]; +char *buf =NULL ; +struct strbuf linebuf =STRBUF_INIT ;struct pollfd pollfd = { .events =POLLIN , };
@@ -767,6 +771,13 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)setup_pager (); + buf =malloc (TRACE_BUF_SIZE ); +if (!buf) { +pr_err ("failed to allocate trace buffer\n" ); + goto out_reset; + } +strbuf_init (&linebuf,512 ); + trace_file =get_tracing_instance_file ("trace_pipe" );if (!trace_file) {pr_err ("failed to open trace_pipe\n" );
@@ -810,11 +821,19 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)break ;if (pollfd.revents &POLLIN ) { -int n =read (trace_fd, buf,sizeof (buf)); -if (n <0 ) -break ; -if (fwrite (buf, n,1 , stdout) !=1 ) +int n =read (trace_fd, buf,TRACE_BUF_SIZE ); + +if (n <=0 )break ; +for (int i =0 ; i < n; i++) { +if (buf[i] =='\n' ) { +fprintf (stdout,"%s\n" , linebuf.buf); +strbuf_setlen (&linebuf,0 ); + }else { +if (strbuf_addch (&linebuf, buf[i]) <0 ) + goto out_close_fd; + } + }/* flush output since stdout is in full buffering mode due to pager */ fflush (stdout); }
@@ -823,7 +842,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)write_tracing_file ("tracing_on" ,"0" );if (workload_exec_errno) { -const char *emsg =str_error_r (workload_exec_errno, buf,sizeof (buf)); +const char *emsg =str_error_r (workload_exec_errno, buf,TRACE_BUF_SIZE );/* flush stdout first so below error msg appears at the end. */ fflush (stdout);pr_err ("workload failed: %s\n" , emsg);
@@ -832,14 +851,29 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)/* read remaining buffer contents */ while (true) { -int n =read (trace_fd, buf,sizeof (buf)); +int n =read (trace_fd, buf,TRACE_BUF_SIZE ); +if (n <=0 )break ; -if (fwrite (buf, n,1 , stdout) !=1 ) -break ; +for (int i =0 ; i < n; i++) { +if (buf[i] =='\n' ) { +fprintf (stdout,"%s\n" , linebuf.buf); +strbuf_setlen (&linebuf,0 ); + }else { +if (strbuf_addch (&linebuf, buf[i]) <0 ) + goto out_close_fd; + } + } + } + +if (linebuf.len >0 ) { +fprintf (stdout,"%s\n" , linebuf.buf); +strbuf_setlen (&linebuf,0 ); } out_close_fd: +free (buf); +strbuf_release (&linebuf);close (trace_fd); out_reset:exit_tracing_instance ();
Signature · 1 line
2.55.0
PATCH v2 2/4 perf ftrace: Support filtering return address comments
In reply to Aaron TomlinWhen tracing with the Linux kernel's function graph tracer, enabling funcgraph-retaddr records the return address of each function entry as a comment in the trace stream (e.g. /* <-load_elf_phdrs+0x6c/0xb0 */). While valuable for tracing execution origins, these trailing comments can add considerable visual clutter to the call graph, particularly when examining deeply nested call trees or when synthesising higher-level abstractions.
Introduce support for filtering out return address comments from function graph trace output. Provide both a standalone --filter-retaddr flag and a --graph-opts filter-retaddr sub-option.
Key aspects of this implementation:
1. Return address parsing and filtering
Implement ftrace_parse_retaddr() and ftrace_filter_retaddr() in tools/perf/util/ftrace.c to robustly match return address comments across diverse output styles (standard format, return values, and bracketed addresses) and strip them in-place with memmove(3).
2. Command-line interface
Expose --filter-retaddr and --graph-opts filter-retaddr in tools/perf/builtin-ftrace.c, updating the documentation in tools/perf/Documentation/perf-ftrace.txt accordingly.
3. Automated verification
Add a dedicated unit test suite ("Ftrace return address processing") in tools/perf/tests/ftrace.c to validate comment parsing and string filtering across all supported formats.
--- tools/perf/Documentation/perf-ftrace.txt | 5 ++ tools/perf/builtin-ftrace.c | 17 ++++- tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/ftrace.c | 80 ++++++++++++++++++++++++ tools/perf/tests/tests.h | 1 + tools/perf/util/Build | 1 + tools/perf/util/ftrace.c | 68 ++++++++++++++++++++ tools/perf/util/ftrace.h | 5 ++ 9 files changed, 178 insertions(+), 1 deletion(-) create mode 100644 tools/perf/tests/ftrace.c create mode 100644 tools/perf/util/ftrace.c
tools/perf/Documentation/perf-ftrace.txt +5 -0
@@ -127,6 +127,7 @@ OPTIONS for 'perf ftrace trace' - retval - Show function return value. - retval-hex - Show function return value in hexadecimal format. - retaddr - Show function return address. + - filter-retaddr - Filter out function return address comments from output. - nosleep-time - Measure on-CPU time only for function_graph tracer. - noirqs - Ignore functions that happen inside interrupt. - verbose - Show process names, PIDs, timestamps, etc.
@@ -134,6 +135,10 @@ OPTIONS for 'perf ftrace trace' - depth=<n> - Set max depth for function graph tracer to follow. - tail - Print function name at the end. +--filter-retaddr:: + Filter out function return address comments (`/* <-caller+offset */`) + from function_graph output. + OPTIONS for 'perf ftrace latency' ---------------------------------
tools/perf/builtin-ftrace.c +16 -1
@@ -827,6 +827,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)break ;for (int i =0 ; i < n; i++) {if (buf[i] =='\n' ) { +if (ftrace->filter_retaddr) +ftrace_filter_retaddr (linebuf.buf);fprintf (stdout,"%s\n" , linebuf.buf);strbuf_setlen (&linebuf,0 ); }else {
@@ -857,6 +859,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)break ;for (int i =0 ; i < n; i++) {if (buf[i] =='\n' ) { +if (ftrace->filter_retaddr) +ftrace_filter_retaddr (linebuf.buf);fprintf (stdout,"%s\n" , linebuf.buf);strbuf_setlen (&linebuf,0 ); }else {
@@ -867,6 +871,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace) }if (linebuf.len >0 ) { +if (ftrace->filter_retaddr) +ftrace_filter_retaddr (linebuf.buf);fprintf (stdout,"%s\n" , linebuf.buf);strbuf_setlen (&linebuf,0 ); }
@@ -1729,6 +1735,7 @@ static int parse_graph_tracer_opts(const struct option *opt, {int ret;struct perf_ftrace *ftrace = (struct perf_ftrace *) opt->value; +int filter_retaddr =-1 ;struct sublevel_option graph_tracer_opts[] = { { .name ="args" , .value_ptr = &ftrace->graph_args }, { .name ="retval" , .value_ptr = &ftrace->graph_retval },
@@ -1740,6 +1747,7 @@ static int parse_graph_tracer_opts(const struct option *opt, { .name = "thresh", .value_ptr = &ftrace->graph_thresh }, { .name = "depth", .value_ptr = &ftrace->graph_depth }, { .name = "tail", .value_ptr = &ftrace->graph_tail }, + { .name = "filter-retaddr", .value_ptr = &filter_retaddr }, { .name = NULL, } };
@@ -1750,6 +1758,11 @@ static int parse_graph_tracer_opts(const struct option *opt,if (ret)return ret; +if (filter_retaddr !=-1 ) { + ftrace->filter_retaddr = (filter_retaddr !=0 ); + ftrace->filter_retaddr_set = true; + } +return 0 ; }
@@ -1825,12 +1838,14 @@ int cmd_ftrace(int argc, const char **argv)OPT_CALLBACK ('g' ,"nograph-funcs" , &ftrace.nograph_funcs,"func" ,"Set nograph filter on given functions" , parse_filter_func),OPT_CALLBACK (0 ,"graph-opts" , &ftrace,"options" , -"Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,verbose,thresh=<n>,depth=<n>" , +"Graph tracer options, available options: args,retval,retval-hex,retaddr,filter-retaddr,nosleep-time,noirqs,verbose,thresh=<n>,depth=<n>" , parse_graph_tracer_opts),OPT_CALLBACK ('m' ,"buffer-size" , &ftrace.percpu_buffer_size,"size" ,"Size of per cpu buffer, needs to use a B, K, M or G suffix." , parse_buffer_size),OPT_BOOLEAN (0 ,"inherit" , &ftrace.inherit,"Trace children processes" ), +OPT_BOOLEAN_SET (0 ,"filter-retaddr" , &ftrace.filter_retaddr, &ftrace.filter_retaddr_set, +"Filter out funcgraph return address comments" ),OPT_INTEGER ('D' ,"delay" , &ftrace.target.initial_delay,"Number of milliseconds to wait before starting tracing after program start" ), OPT_PARENT(common_options),
tools/perf/tests/Build +1 -0
@@ -72,6 +72,7 @@ perf-test-y += hwmon_pmu.o perf-test-y += tool_pmu.o perf-test-y += subcmd-help.o perf-test-y += kallsyms-split.o +perf-test-y += ftrace.o ifeq ($(SRCARCH),$(filter $(SRCARCH),x86 arm arm64 powerpc riscv)) perf-test-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
tools/perf/tests/builtin-test.c +1 -0
@@ -137,6 +137,7 @@ static struct test_suite *generic_tests[] = { &suite__perf_hooks, &suite__unit_number__scnprint, &suite__mem2node, + &suite__ftrace, &suite__time_utils, &suite__jit_write_elf, &suite__pfm,
tools/perf/tests/ftrace.c new file +80 -0
80 lines not on this page · raw message
tools/perf/tests/tests.h +1 -0
@@ -163,6 +163,7 @@ DECLARE_SUITE(perf_hooks);DECLARE_SUITE (unit_number__scnprint);DECLARE_SUITE (mem2node);DECLARE_SUITE (maps); +DECLARE_SUITE (ftrace);DECLARE_SUITE (time_utils);DECLARE_SUITE (jit_write_elf);DECLARE_SUITE (api_io);
tools/perf/util/Build +1 -0
@@ -169,6 +169,7 @@ perf-util-y += list_sort.o perf-util-y += mutex.o perf-util-y += sharded_mutex.o perf-util-y += intel-tpebs.o +perf-util-y += ftrace.o perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_counter.o perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_counter_cgroup.o
tools/perf/util/ftrace.c new file +68 -0
68 lines not on this page · raw message
tools/perf/util/ftrace.h +5 -0
@@ -39,6 +39,8 @@ struct perf_ftrace {int graph_verbose;int graph_thresh;int graph_tail; +bool filter_retaddr; +bool filter_retaddr_set; };struct filter_entry {
@@ -93,4 +95,7 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)#endif /* HAVE_BPF_SKEL */ +bool ftrace_parse_retaddr (const char *str,char *sym_name,size_t sym_len,u64 *offset); +void ftrace_filter_retaddr (char *str); +#endif /* __PERF_FTRACE_H__ */
Signature · 1 line
2.55.0
PATCH v2 3/4 perf ftrace: Support display of inlined functions in function graph tracer
In reply to Aaron TomlinThe Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining.
Address this by introducing the --inline option to perf ftrace for the function_graph tracer.
When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment (e.g. /* <-irq_work_queue_on+0x81/0xa0 */). By interrogating DWARF debug information from the kernel image (vmlinux) utilising libdw, perf ftrace resolves this return address to its inlined callchain and synthesises the intermediate inlined frames directly into the streamed call graph.
Key aspects of this implementation:
1. Inlined frame presentation
Synthesised inlined functions are rendered with an explicit /* (inline) */ annotation at both entry and exit:
# CPU DURATION FUNCTION CALLS
# | | | | | | |
1) 0.856 us | mutex_unlock(lock=0xffffffffbc28bfa0);
0) | wake_up_new_task(p=0xffff8b1f183bac80) {
0) | rb_irq_work_queue() { /* (inline) */
0) | rb_wakeups() { /* (inline) */
0) 1.374 us | housekeeping_any_cpu(type=3 [HK_TYPE_KERNEL_NOISE]);
0) | arch_irq_work_raise() {
0) | apic_wait_icr_idle() { /* (inline) */
0) | x2apic_send_IPI_self(vector=246) {
0) | instr_sysvec_irq_work() { /* (inline) */
0) | irq_enter_rcu() {
0) | instr_sysvec_irq_work() { /* (inline) */
0) 0.673 us | irqtime_account_irq(curr=0xffff8b1f086a2c80, offset=0x1000000);
0) | } /* instr_sysvec_irq_work (inline) */
0) 1.697 us | }
This affords immediate visual clarity to the reader upon entering an inlined scope, without necessitating a search for the closing delimiter.
2. Call graph reconstruction and dynamic stack tracking
Maintains a per-CPU stack tracking real and inlined frames, synthesising entry and exit delimiters to accurately reconstruct the execution hierarchy. Bare closing braces without trailing function names (the default when funcgraph-tail is disabled) are correctly matched against preceding frames on the stack. Furthermore, stack depth saturation guards ensure that inlined frame depth tracking remains clamped and indentation cannot drift even under deep call trees exceeding FTRACE_MAX_STACK.
3. Option handling and diagnostics
Adds -k or --vmlinux to designate an explicit debug image, and provides clear diagnostic guidance should --inline be invoked on a kernel lacking CONFIG_FUNCTION_GRAPH_RETADDR. Automatically enables return address comment filtering unless explicitly requested.
4. Automated verification
Expands the unit test suite in tools/perf/tests/ftrace.c to validate DWARF inline symbol resolution, comment filtering toggles, bare closing braces handling, and stack saturation recovery.
--- tools/perf/Documentation/perf-ftrace.txt | 19 +- tools/perf/builtin-ftrace.c | 56 ++- tools/perf/tests/ftrace.c | 162 +++++++- tools/perf/util/ftrace.c | 475 ++++++++++++++++++++++- tools/perf/util/ftrace.h | 11 + 5 files changed, 704 insertions(+), 19 deletions(-)
tools/perf/Documentation/perf-ftrace.txt +17 -2
@@ -127,7 +127,7 @@ OPTIONS for 'perf ftrace trace' - retval - Show function return value. - retval-hex - Show function return value in hexadecimal format. - retaddr - Show function return address. - - filter-retaddr - Filter out function return address comments from output. + - filter-retaddr - Filter out function return address comments (enabled by default with --inline). - nosleep-time - Measure on-CPU time only for function_graph tracer. - noirqs - Ignore functions that happen inside interrupt. - verbose - Show process names, PIDs, timestamps, etc.
@@ -135,9 +135,24 @@ OPTIONS for 'perf ftrace trace' - depth=<n> - Set max depth for function graph tracer to follow. - tail - Print function name at the end. +--inline:: + Show inlined functions in function_graph tracer. This requires + kernel support for the funcgraph-retaddr option and a vmlinux with + debug symbols. Inlined functions are displayed in the call graph + hierarchy, and their inlined status is indicated with `/* (inline) */` + on entry and exit, e.g. `func() { /* (inline) */` and `} /* func (inline) */`. + By default, raw return address comments (`/* <-caller+offset */`) + are hidden in the output; specify `--graph-opts retaddr` to show them. + --filter-retaddr:: Filter out function return address comments (`/* <-caller+offset */`) - from function_graph output. + from function_graph output. This is enabled by default when `--inline` + is used; use `--no-filter-retaddr` to disable it. + +-k:: +--vmlinux=:: + Path to the vmlinux file containing debug symbols for resolving + inlined functions. OPTIONS for 'perf ftrace latency'
tools/perf/builtin-ftrace.c +43 -13
105 lines not on this page · raw message
tools/perf/tests/ftrace.c +161 -1
180 lines not on this page · raw message
tools/perf/util/ftrace.c +472 -3
488 lines not on this page · raw message
tools/perf/util/ftrace.h +11 -0
@@ -41,6 +41,8 @@ struct perf_ftrace {int graph_tail;bool filter_retaddr;bool filter_retaddr_set; +bool use_inline; +struct machine *machine; };struct filter_entry {
@@ -95,7 +97,16 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)#endif /* HAVE_BPF_SKEL */ +struct machine ; +bool ftrace_parse_retaddr (const char *str,char *sym_name,size_t sym_len,u64 *offset);void ftrace_filter_retaddr (char *str); +int ftrace_resolve_inlines (struct machine *machine, +const char *sym_name,u64 offset, +const char **inlined_names,int max_inlines); +int ftrace_process_fgraph_line (struct perf_ftrace *ftrace,const char *line,FILE *out); +void ftrace_cpu_states__clear (void ); +int perf_ftrace__setup_inlines (struct perf_ftrace *ftrace); +void perf_ftrace__cleanup_inlines (struct perf_ftrace *ftrace);#endif /* __PERF_FTRACE_H__ */
Signature · 1 line
2.55.0
PATCH v2 4/4 perf ftrace: Support omitting execution duration in function graph tracer
In reply to Aaron TomlinThe Linux kernel's function graph tracer displays the execution duration of completed functions by default via the funcgraph-duration trace option. Whilst informative, in scenarios where the user is solely interested in the call graph topology, call flow sequence, or inlined hierarchy, the duration column can add visual clutter and widen each trace line.
Introduce support for omitting function duration by exposing the noduration sub-option under --graph-opts (and conversely duration=[0|1]). When specified, perf ftrace configures the kernel's tracing instance option funcgraph-duration to 0, and restores it to 1 upon resetting tracing options.
--- tools/perf/Documentation/perf-ftrace.txt | 1 + tools/perf/builtin-ftrace.c | 25 +++++++++++++++++++++++- tools/perf/util/ftrace.h | 1 + 3 files changed, 26 insertions(+), 1 deletion(-)
tools/perf/Documentation/perf-ftrace.txt +1 -0
@@ -130,6 +130,7 @@ OPTIONS for 'perf ftrace trace' - filter-retaddr - Filter out function return address comments (enabled by default with --inline). - nosleep-time - Measure on-CPU time only for function_graph tracer. - noirqs - Ignore functions that happen inside interrupt. + - noduration - Do not print function duration. - verbose - Show process names, PIDs, timestamps, etc. - thresh=<n> - Setup trace duration threshold in microseconds. - depth=<n> - Set max depth for function graph tracer to follow.
tools/perf/builtin-ftrace.c +24 -1
@@ -299,6 +299,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)write_tracing_option_file ("func_stack_trace" ,"0" );write_tracing_option_file ("sleep-time" ,"1" );write_tracing_option_file ("funcgraph-irqs" ,"1" ); +write_tracing_option_file ("funcgraph-duration" ,"1" );write_tracing_option_file ("funcgraph-proc" ,"0" );write_tracing_option_file ("funcgraph-abstime" ,"0" );write_tracing_option_file ("funcgraph-tail" ,"0" );
@@ -598,6 +599,17 @@ static int set_tracing_funcgraph_irqs(struct perf_ftrace *ftrace)return 0 ; } +static int set_tracing_funcgraph_duration (struct perf_ftrace *ftrace) +{ +if (!ftrace->graph_noduration) +return 0 ; + +if (write_tracing_option_file ("funcgraph-duration" ,"0" ) <0 ) +return -1 ; + +return 0 ; +} +static int set_tracing_funcgraph_verbose (struct perf_ftrace *ftrace) {if (!ftrace->graph_verbose)
@@ -707,6 +719,11 @@ static int set_tracing_options(struct perf_ftrace *ftrace)return -1 ; } +if (set_tracing_funcgraph_duration (ftrace) <0 ) { +pr_err ("failed to set tracing option funcgraph-duration\n" ); +return -1 ; + } +if (set_tracing_funcgraph_verbose (ftrace) <0 ) {pr_err ("failed to set tracing option funcgraph-proc/funcgraph-abstime\n" );return -1 ;
@@ -1747,6 +1764,7 @@ static int parse_graph_tracer_opts(const struct option *opt,int ret;struct perf_ftrace *ftrace = (struct perf_ftrace *) opt->value;int filter_retaddr =-1 ; +int duration =-1 ;struct sublevel_option graph_tracer_opts[] = { { .name ="args" , .value_ptr = &ftrace->graph_args }, { .name ="retval" , .value_ptr = &ftrace->graph_retval },
@@ -1754,6 +1772,8 @@ static int parse_graph_tracer_opts(const struct option *opt, { .name = "retaddr", .value_ptr = &ftrace->graph_retaddr }, { .name = "nosleep-time", .value_ptr = &ftrace->graph_nosleep_time }, { .name = "noirqs", .value_ptr = &ftrace->graph_noirqs }, + { .name = "noduration", .value_ptr = &ftrace->graph_noduration }, + { .name = "duration", .value_ptr = &duration }, { .name = "verbose", .value_ptr = &ftrace->graph_verbose }, { .name = "thresh", .value_ptr = &ftrace->graph_thresh }, { .name = "depth", .value_ptr = &ftrace->graph_depth },
@@ -1769,6 +1789,9 @@ static int parse_graph_tracer_opts(const struct option *opt,if (ret)return ret; +if (duration !=-1 ) + ftrace->graph_noduration = !duration; +if (filter_retaddr !=-1 ) { ftrace->filter_retaddr = (filter_retaddr !=0 ); ftrace->filter_retaddr_set = true;
@@ -1849,7 +1872,7 @@ int cmd_ftrace(int argc, const char **argv)OPT_CALLBACK ('g' ,"nograph-funcs" , &ftrace.nograph_funcs,"func" ,"Set nograph filter on given functions" , parse_filter_func),OPT_CALLBACK (0 ,"graph-opts" , &ftrace,"options" , -"Graph tracer options, available options: args,retval,retval-hex,retaddr,filter-retaddr,nosleep-time,noirqs,verbose,thresh=<n>,depth=<n>" , +"Graph tracer options, available options: args,retval,retval-hex,retaddr,filter-retaddr,nosleep-time,noirqs,noduration,verbose,thresh=<n>,depth=<n>,tail" , parse_graph_tracer_opts),OPT_CALLBACK ('m' ,"buffer-size" , &ftrace.percpu_buffer_size,"size" , "Size of per cpu buffer, needs to use a B, K, M or G suffix.", parse_buffer_size),
tools/perf/util/ftrace.h +1 -0
@@ -36,6 +36,7 @@ struct perf_ftrace {int graph_retaddr;int graph_nosleep_time;int graph_noirqs; +int graph_noduration;int graph_verbose;int graph_thresh;int graph_tail;
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH perf-tools-next v2 4/4] perf ftrace: Support omitting execution duration in function graph tracer
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf ftrace: --graph-opts duration silently clobbers noduration --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf ftrace: --graph-opts duration silently clobbers noduration --
commit e23672676ea60d48741ccc720b34c8778668c668
Author: Aaron Tomlin <[email protected]>
perf ftrace: Support omitting execution duration in function graph tracer
The Linux kernel's function graph tracer displays the execution duration of completed functions by default via the funcgraph-duration trace option. Introduce support for omitting function duration by exposing the noduration sub-option under --graph-opts (and conversely duration=[0|1]).
Quoted 4 lines
tools/perf/builtin-ftrace.c +0
-0
[ ... ]
Quoted 10 lines
tools/perf/builtin-ftrace.c +2
-0@@ -1754,6 +1772,8 @@ static int parse_graph_tracer_opts(const struct option *opt, { .name = "retaddr", .value_ptr = &ftrace->graph_retaddr }, { .name = "nosleep-time", .value_ptr = &ftrace->graph_nosleep_time }, { .name = "noirqs", .value_ptr = &ftrace->graph_noirqs }, + { .name = "noduration", .value_ptr = &ftrace->graph_noduration }, + { .name = "duration", .value_ptr = &duration }, { .name = "verbose", .value_ptr = &ftrace->graph_verbose }, { .name = "thresh", .value_ptr = &ftrace->graph_thresh }, { .name = "depth", .value_ptr = &ftrace->graph_depth },
tools/perf/builtin-ftrace.c static int parse_graph_tracer_opts(const struct option *opt, · line 1794
Quoted 6 lines
tools/perf/builtin-ftrace.c +3
-0@@ -1769,6 +1789,9 @@ static int parse_graph_tracer_opts(const struct option *opt,if (ret)return ret; +if (duration !=-1 ) + ftrace->graph_noduration = !duration; +
[Severity: Medium]
Does this logic unconditionally overwrite the noduration parameter?
If a user executes perf ftrace with --graph-opts duration=1,noduration, the token parsing loop in perf_parse_sublevel_options() correctly processes them sequentially. It sets duration to 1 and ftrace->graph_noduration to 1.
However, since duration != -1 is evaluated after the loop, it overwrites ftrace->graph_noduration with !1 (which is 0). This causes the duration parameter to always clobber the noduration parameter, regardless of their order in the comma-separated list.
Would it be possible to parse both directives strictly in the order they are specified without unconditionally overriding one after the loop?
Quoted 3 lines
tools/perf/builtin-ftrace.c +0
-0if (filter_retaddr !=-1 ) { ftrace->filter_retaddr = (filter_retaddr !=0 ); ftrace->filter_retaddr_set = true;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
[email protected]
AI reviewRe: [PATCH perf-tools-next v2 1/4] perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming
In reply to Aaron TomlinThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf: Memory leak of buf and linebuf on early ftrace setup error paths --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf: Memory leak of buf and linebuf on early ftrace setup error paths --
commit e5886f5c4e0cc584a5a7ac05c7c069ab1b80f135
Author: Aaron Tomlin <[email protected]>
perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming
This commit refactors __cmd_ftrace() to use a dynamically allocated buffer and a string buffer for streaming trace data instead of a fixed stack array.
Quoted 4 lines
tools/perf/builtin-ftrace.c +0
-0
[ ... ]
Quoted 4 lines
tools/perf/builtin-ftrace.c +0
-0@@ -767,6 +771,13 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)setup_pager ();
tools/perf/builtin-ftrace.c static int __cmd_ftrace(struct perf_ftrace *ftrace) · line 783
Quoted 10 lines
tools/perf/builtin-ftrace.c +7
-0+ buf =malloc (TRACE_BUF_SIZE ); +if (!buf) { +pr_err ("failed to allocate trace buffer\n" ); + goto out_reset; + } +strbuf_init (&linebuf,512 ); + trace_file =get_tracing_instance_file ("trace_pipe" );if (!trace_file) {pr_err ("failed to open trace_pipe\n" );
[Severity: Medium]
If get_tracing_instance_file("trace_pipe") returns NULL, or if the subsequent open(trace_file, O_RDONLY) fails within __cmd_ftrace() in tools/perf/builtin-ftrace.c, the error path executes goto out_reset.
Because the out_reset label completely skips the resource teardown label out_close_fd:
out_close_fd:
free(buf); strbuf_release(&linebuf); close(trace_fd); out_reset: exit_tracing_instance();
Does this permanently leak the heap-allocated buf (TRACE_BUF_SIZE bytes) and the internal buffer of linebuf on these early ftrace setup error paths?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewRe: [PATCH perf-tools-next v2 2/4] perf ftrace: Support filtering return address comments
In reply to Aaron TomlinThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf/ftrace: memory leak of trace buffer on error paths in __cmd_ftrace() --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/ftrace: memory leak of trace buffer on error paths in __cmd_ftrace() --
commit 473e334ba892f3b7d5261168be5e86956a4de438
Author: Aaron Tomlin <[email protected]>
perf ftrace: Support filtering return address comments
This commit introduces a feature to filter out return address comments from the function graph tracer output in perf ftrace. It implements parsing and filtering logic, adds a --filter-retaddr command-line option, and includes unit tests for validation.
tools/perf/builtin-ftrace.c static int __cmd_ftrace(struct perf_ftrace *ftrace) · line 833
Quoted 11 lines
tools/perf/builtin-ftrace.c +2
-0@@ -827,6 +827,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)break ;for (int i =0 ; i < n; i++) {if (buf[i] =='\n' ) { +if (ftrace->filter_retaddr) +ftrace_filter_retaddr (linebuf.buf);fprintf (stdout,"%s\n" , linebuf.buf);strbuf_setlen (&linebuf,0 );
[Severity: Medium]
Does __cmd_ftrace() in tools/perf/builtin-ftrace.c leak the allocated buf and linebuf on error paths?
Looking at the initialization sequence, buf is dynamically allocated and linebuf is initialized:
buf = malloc(TRACE_BUF_SIZE);
if (!buf) {
pr_err("failed to allocate trace buffer\n");
goto out_reset;
}
strbuf_init(&linebuf, 512);However, if get_tracing_instance_file() fails or trace_fd fails to open, the code jumps to out_reset:
if (!trace_file) {
pr_err("failed to open trace_pipe\n");
goto out_reset;
}The out_reset label is located after the cleanup for buf and linebuf:
out_close_fd:
free(buf); strbuf_release(&linebuf); close(trace_fd); out_reset: exit_tracing_instance();
Will this bypass the free(buf) and strbuf_release(&linebuf) calls, causing a regression where these resources are leaked when early error paths are taken?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[email protected]
AI reviewRe: [PATCH perf-tools-next v2 3/4] perf ftrace: Support display of inlined functions in function graph tracer
In reply to Aaron TomlinThank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider: - [Medium] perf: fgraph inline tracking fails to close frames if num_inlines == 0 - [Medium] perf: fgraph…
Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [Medium] perf: fgraph inline tracking fails to close frames if num_inlines == 0
- [Medium] perf: fgraph stack unwinding fails permanently on dropped trace events
- [Low] perf: unmatched exit events erroneously pop active inlined frames
- [Low] perf: memory leak of trace buffers on open failure
- [Low] perf tests: memory leak on inline processing test failure
--
commit d46c8024964e46cbda045d2aa35a4276291e5d25
Author: Aaron Tomlin <[email protected]>
perf ftrace: Support display of inlined functions in function graph tracer
This commit introduces the --inline option to perf ftrace for the function_graph tracer. It resolves the return address to its inlined callchain using DWARF debug information and synthesizes intermediate inlined frames directly into the streamed call graph.
Quoted 2 lines
tools/perf/builtin-ftrace.c +0
-0
[ ... ]
Quoted 11 lines
tools/perf/builtin-ftrace.c +3
-0@@ -769,6 +774,9 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace) goto out_reset; } +if (perf_ftrace__setup_inlines (ftrace) <0 ) + goto out_reset; +setup_pager (); buf =malloc (TRACE_BUF_SIZE );@@ -827,9 +835,10 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
[ ... ]
Quoted 4 lines
tools/perf/builtin-ftrace.c +0
-0trace_file =get_tracing_instance_file ("trace_pipe" );if (!trace_file) {pr_err ("failed to open trace_pipe\n" ); goto out_reset;
tools/perf/builtin-ftrace.c
Quoted 10 lines
tools/perf/builtin-ftrace.c +0
-0} trace_fd =open (trace_file,O_RDONLY );put_tracing_file (trace_file);if (trace_fd <0 ) {pr_err ("failed to open trace_pipe\n" ); goto out_reset; }
[Severity: Low]
Does this error path leak the trace buffers?
In __cmd_ftrace(), buf is allocated and linebuf is initialized just before opening the trace pipe. If get_tracing_instance_file() or open() fails, the code jumps directly to out_reset, which bypasses the out_close_fd block where free(buf) and strbuf_release(&linebuf) are called.
Quoted 2 lines
tools/perf/tests/ftrace.c +0
-0
[ ... ]
Quoted 4 lines
tools/perf/tests/ftrace.c +4
-0+/* Feed simulated function graph trace lines */ +ftrace_process_fgraph_line (&ftrace, line_entry, out); +ftrace_process_fgraph_line (&ftrace, line_enqueue, out); +ftrace_process_fgraph_line (&ftrace, line_exit_enqueue, out);
tools/perf/tests/ftrace.c static int test_filter_retaddr(void) · line 116
Quoted 10 lines
tools/perf/tests/ftrace.c +10
-0+ftrace_process_fgraph_line (&ftrace, line_exit_task, out); + +fclose (out); +ftrace_cpu_states__clear (); + +pr_debug ("Processed output:\n%s\n" , out_buf); + +/* Verify activate_task entry with inline hint */ +TEST_ASSERT_VAL ("contains activate_task entry with inline hint" , +strstr (out_buf,"activate_task() { /* (inline) */" ) !=NULL );
[Severity: Low]
Can a failure here leak memory and machine resources during tests?
If TEST_ASSERT_VAL fails, it immediately returns TEST_FAIL. Since test_inline_processing() allocates ftrace.machine and out_buf, returning early bypasses free(out_buf) and perf_ftrace__cleanup_inlines(&ftrace).
Quoted 2 lines
tools/perf/util/ftrace.c +0
-0
[ ... ]
tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 379
Quoted 11 lines
tools/perf/util/ftrace.c +11
-0+/* Determine which inlined frames are currently open above the caller */ + caller_idx =-1 ; +if (num_inlines >0 ) { +for (i = cs->depth -1 ; i >=0 ; i--) { +if (!cs->stack[i].inlined && cs->stack[i].name && +strcmp (cs->stack[i].name, caller_sym) ==0 ) { + caller_idx = i; +break ; + } + } + }
[Severity: Medium]
Does guarding this loop with if (num_inlines > 0) prevent finding the caller when a non-inlined function executes?
If a non-inlined function runs, ftrace_resolve_inlines() returns num_inlines == 0. Because of this check, the stack search is skipped and caller_idx remains -1.
As a result, active_inlines evaluates to 0, which bypasses the loop meant to close stale inlined frames. This causes the new non-inlined function to be incorrectly pushed to the stack with the extra indentation of the unclosed inlined function, disrupting the call graph hierarchy.
[ ... ]
tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 327
Quoted 2 lines
tools/perf/util/ftrace.c +2
-0+/* Close all inlined frames above match_idx */ +while (cs->depth >0 && (match_idx <0 || cs->depth -1 > match_idx)) {
[Severity: Low]
Can spurious exit events erroneously pop active inlined frames?
If a trace output contains an unmatched or spurious exit event, the stack search finds no matching real frame, leaving match_idx as -1. The condition (match_idx < 0 || cs->depth - 1 > match_idx) will evaluate to true, which indiscriminately pops and closes currently active inlined frames from the top of the stack until it hits a real frame.
Quoted 4 lines
tools/perf/util/ftrace.c +4
-0+if (cs->stack[cs->depth -1 ].inlined) { +struct ftrace_stack_frame *top = &cs->stack[cs->depth -1 ]; +int exit_indent = top->indent; +
tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 341
Quoted 10 lines
tools/perf/util/ftrace.c +10
-0+if (cs->inlined_depth >0 ) + cs->inlined_depth--; +fprintf (out,"%s%*s} /* %s (inline) */\n" , + blank_prefix, exit_indent,"" , top->name); +free (top->name); + cs->depth--; + }else { +break ; + } + }
[Severity: Medium]
Does this loop permanently trap frames if it encounters a dropped trace event?
If an exit event for a non-inlined function is dropped (e.g., from a ring buffer overrun), an orphaned non-inlined frame remains on the stack. When this unwinding loop hits that orphaned frame (!cs->stack[...].inlined), it calls break instead of popping it.
Because cs->depth doesn't decrease past the orphaned frame, the subsequent check for the correctly matched real frame fails, leaving both the matched frame and orphaned frames permanently stuck on the stack until depth is exhausted.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Re: [PATCH perf-tools-next v2 1/4] perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming
In reply to [email protected]Quoted 43 lines · On Tue, Oct 06, 2026 at 11:35:44PM +0000, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf: Memory leak of buf and linebuf on early ftrace setup error paths --commit
e5886f5c4e0cc584a5a7ac05c7c069ab1b80f135
Author: Aaron Tomlin <[email protected]>perf ftrace: Optimise
__cmd_ftrace()stack frame with strbuf streamingThis commit refactors
__cmd_ftrace()to use a dynamically allocated buffer and a string buffer for streaming trace data instead of a fixed stack array.Quoted 4 lines
tools/perf/builtin-ftrace.c +0
-0[ ... ]
Quoted 4 lines
tools/perf/builtin-ftrace.c +0
-0@@ -767,6 +771,13 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)setup_pager ();tools/perf/builtin-ftrace.c static int __cmd_ftrace(struct perf_ftrace *ftrace) · line 783
Quoted 10 lines
tools/perf/builtin-ftrace.c +7
-0+ buf =malloc (TRACE_BUF_SIZE ); +if (!buf) { +pr_err ("failed to allocate trace buffer\n" ); + goto out_reset; + } +strbuf_init (&linebuf,512 ); + trace_file =get_tracing_instance_file ("trace_pipe" );if (!trace_file) {pr_err ("failed to open trace_pipe\n" );[Severity: Medium]
Ifget_tracing_instance_file("trace_pipe")returns NULL, or if the subsequentopen(trace_file, O_RDONLY)fails within__cmd_ftrace()intools/perf/builtin-ftrace.c, the error path executes gotoout_reset.Because the
out_resetlabel completely skips the resource teardown labelout_close_fd:
out_close_fd:
Quoted 8 lines
free(buf); strbuf_release(&linebuf); close(trace_fd); out_reset: exit_tracing_instance();Does this permanently leak the heap-allocated buf (
TRACE_BUF_SIZEbytes) and the internal buffer of linebuf on these early ftrace setup error paths?
Yes. I will address this in the next iteration. Since buf and linebuf are allocated before attempting to resolve and open trace_pipe, taking 'goto out_reset' bypassed the resource teardown and leaked both heap allocations.
Quoted 3 lines
Signature · 2 lines
-- Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Kind regards,
Signature · 1 line
Aaron Tomlin
Re: [PATCH perf-tools-next v2 2/4] perf ftrace: Support filtering return address comments
In reply to [email protected]Quoted 57 lines · On Tue, Oct 06, 2026 at 11:37:04PM +0000, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf/ftrace: memory leak of trace buffer on error paths in__cmd_ftrace()--commit
473e334ba892f3b7d5261168be5e86956a4de438
Author: Aaron Tomlin <[email protected]>perf ftrace: Support filtering return address comments
This commit introduces a feature to filter out return address comments from the function graph tracer output in perf ftrace. It implements parsing and filtering logic, adds a --filter-retaddr command-line option, and includes unit tests for validation.
tools/perf/builtin-ftrace.c static int __cmd_ftrace(struct perf_ftrace *ftrace) · line 833
Quoted 11 lines
tools/perf/builtin-ftrace.c +2
-0@@ -827,6 +827,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)break ;for (int i =0 ; i < n; i++) {if (buf[i] =='\n' ) { +if (ftrace->filter_retaddr) +ftrace_filter_retaddr (linebuf.buf);fprintf (stdout,"%s\n" , linebuf.buf);strbuf_setlen (&linebuf,0 );[Severity: Medium]
Does__cmd_ftrace()intools/perf/builtin-ftrace.cleak the allocated buf and linebuf on error paths?Looking at the initialization sequence, buf is dynamically allocated and linebuf is initialized:
buf = malloc(TRACE_BUF_SIZE); if (!buf) { pr_err("failed to allocate trace buffer\n"); goto out_reset; } strbuf_init(&linebuf, 512);However, if
get_tracing_instance_file()fails ortrace_fdfails to open, the code jumps toout_reset:if (!trace_file) { pr_err("failed to open trace_pipe\n"); goto out_reset; }The
out_resetlabel is located after the cleanup for buf and linebuf:
out_close_fd:free(buf); strbuf_release(&linebuf); close(trace_fd); out_reset: exit_tracing_instance();
Quoted 3 lines
Will this bypass the free(buf) and
strbuf_release(&linebuf)calls, causing a regression where these resources are leaked when early error paths are taken?
This will be addressed in patch 1/4.
Quoted 2 lines
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Signature · 1 line
Aaron Tomlin
Namhyung Kim
maintainerHello,
Interesting work!
Quoted 21 lines · Aaron Tomlin
The Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call-graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining (such as the scheduler, memory management, locking primitives, and RCU).
When the kernel is configured with
CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment (e.g. /* <-wake_up_new_task+0x1d1/0x3e0 */). By interrogating DWARF debug information from the kernel image (vmlinux) utilising libdw, perf ftrace resolves this return address to its inlined callchain and synthesises the intermediate inlined frames directly into the streamed call-graph.Synthesised inlined functions are rendered with an explicit /* (inline) */ annotation at both entry and exit:
Quoted 16 lines
# CPU DURATION FUNCTION CALLS
# | | | | | | |
1) 0.856 us |mutex_unlock(lock=0xffffffffbc28bfa0);
0) |wake_up_new_task(p=0xffff8b1f183bac80){
0) |rb_irq_work_queue(){ /* (inline) */
0) |rb_wakeups(){ /* (inline) */
0) 1.374 us |housekeeping_any_cpu(type=3);
0) |arch_irq_work_raise(){
0) |apic_wait_icr_idle(){ /* (inline) */
0) |x2apic_send_IPI_self(vector=246){
0) |instr_sysvec_irq_work(){ /* (inline) */
0) |irq_enter_rcu(){
0) |instr_sysvec_irq_work(){ /* (inline) */
0) 0.673 us |irqtime_account_irq();
0) | } /*instr_sysvec_irq_work(inline) */
0) 1.697 us | }
Let me read this output..
It seems housekeeping_any_cpu() is called from the inlined chain:
wake_up_new_task()
-> rb_irq_work_queue()
-> rb_wakeups()
-> housekeeping_any_cpu()But I think the actual call chain would seem like:
wake_up_new_task()
-> rb_wakeups()
-> rb_irq_work_queue()
-> housekeeping_any_cpu()And I don't know how wake_up_new_task() called rb_wakeups() directly.
Also how do you know if arch_irq_work_raise() is called within the same inlined call-chain? It seems rb_irq_work_queue() indeed calls the function after housekeeping_any_cpu():
rb_irq_work_queue()
-> irq_work_queue_on()
-> __irq_work_queue_local()
-> irq_work_raise()I guess you compare the callchain with the previous one and merge the call if they match. Then it would support partial match and close some parents correctly like when other function is called from rb_wakeups(), for example.
Quoted 11 lines
This series decomposes the feature into a modular, bisectable progression of four patches:
1. Stack frame optimisation in
__cmd_ftrace()Replaces the 4096-byte stack array in
__cmd_ftrace()with dynamic heap allocation alongside struct strbuf streaming, reducing the stack frame footprint by over 96% (down to ~160 bytes) while ensuring complete line-delimited records and robust EOF handling.
Quoted 6 lines
2. Return address comment filtering
Introduces
ftrace_parse_retaddr()andftrace_filter_retaddr()with --filter-retaddr and --graph-opts filter-retaddr to strip caller comments from trace output, accompanied by an initial unit test suite intools/perf/tests/ftrace.c.
I'm curious why do we need options as we already have 'retaddr' graph-option to enable it. I think it should work the same as 'retaddr' is not given, no? Otherwise, we could add 'noretaddr' instead.
Also I think it's only useful with the --inline option. If so, we could handle that automatically when it's given (and 'retaddr' is not given).
Thanks,
Namhyung
Quoted 62 lines
3. Inlined function call-graph reconstruction
Introduces --inline and -k or --vmlinux, dynamically synthesising inlined frames marked with /* (inline) */. Correctly matches bare closing braces ('}') when funcgraph-tail is disabled, clamps stack depth to prevent indentation drift, and expands the automated test suite to verify inline resolution, comment filtering, and stack saturation recovery.
4. Execution duration suppression
Introduces --graph-opts noduration (and duration=[0|1]) to toggle the kernel's funcgraph-duration tracing option, omitting the duration column when the user is focused purely on call-graph topology and inlined hierarchy.
Changes since v1:
- Modularised the implementation into a 4-patch bisectable series rather than a monolithic commit
- Resolved potential infinite loop on
poll()upontrace_pipeEOF- Fixed stuck call stack frames when funcgraph-tail is disabled by supporting bare closing delimiters matching LIFO stack frames
- Prevented unbounded indentation growth on deep call trees by clamping inline expansion and synchronising
cs->inlined_depthto frames successfully pushed ontocs->stack- Added dedicated unit tests covering bare closing braces and stack saturation recovery
- Refined option documentation in perf-ftrace.txt
- Link to v1: https://lore.kernel.org/lkml/[email protected]/
Aaron Tomlin (4):
perf ftrace: Optimise__cmd_ftrace()stack frame with strbuf streaming perf ftrace: Support filtering return address comments perf ftrace: Support display of inlined functions in function graph
tracer
perf ftrace: Support omitting execution duration in function graph
tracertools/perf/Documentation/perf-ftrace.txt | 21 + tools/perf/builtin-ftrace.c | 126 +++++- tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/ftrace.c | 240 ++++++++++ tools/perf/tests/tests.h | 1 + tools/perf/util/Build | 1 + tools/perf/util/ftrace.c | 537 +++++++++++++++++++++++ tools/perf/util/ftrace.h | 17 + 9 files changed, 933 insertions(+), 12 deletions(-) create mode 100644 tools/perf/tests/ftrace.c create mode 100644 tools/perf/util/ftrace.cSignature · 1 line
2.55.0
Quoted 3 lines · Namhyung Kim
Hello,
Interesting work!
Thank you Namhyung.
Quoted 48 lines · Aaron Tomlin
Quoted 21 lines
The Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call-graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining (such as the scheduler, memory management, locking primitives, and RCU).
When the kernel is configured with
CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment (e.g. /* <-wake_up_new_task+0x1d1/0x3e0 */). By interrogating DWARF debug information from the kernel image (vmlinux) utilising libdw, perf ftrace resolves this return address to its inlined callchain and synthesises the intermediate inlined frames directly into the streamed call-graph.Synthesised inlined functions are rendered with an explicit /* (inline) */ annotation at both entry and exit:
Quoted 16 lines
# CPU DURATION FUNCTION CALLS
# | | | | | | |
1) 0.856 us |mutex_unlock(lock=0xffffffffbc28bfa0);
0) |wake_up_new_task(p=0xffff8b1f183bac80){
0) |rb_irq_work_queue(){ /* (inline) */
0) |rb_wakeups(){ /* (inline) */
0) 1.374 us |housekeeping_any_cpu(type=3);
0) |arch_irq_work_raise(){
0) |apic_wait_icr_idle(){ /* (inline) */
0) |x2apic_send_IPI_self(vector=246){
0) |instr_sysvec_irq_work(){ /* (inline) */
0) |irq_enter_rcu(){
0) |instr_sysvec_irq_work(){ /* (inline) */
0) 0.673 us |irqtime_account_irq();
0) | } /*instr_sysvec_irq_work(inline) */
0) 1.697 us | }Let me read this output..
It seems
housekeeping_any_cpu()is called from the inlined chain:
wake_up_new_task()-> rb_irq_work_queue() -> rb_wakeups() -> housekeeping_any_cpu()
Quoted 8 lines
But I think the actual call chain would seem like:
wake_up_new_task()-> rb_wakeups() -> rb_irq_work_queue() -> housekeeping_any_cpu()And I don't know how
wake_up_new_task()calledrb_wakeups()directly.
That resulted in the inverted nesting of rb_irq_work_queue() and rb_wakeups(), as well as the unnatural parentage under wake_up_new_task().
In the actual implementation, DWARF inline resolution via libdw traverses function DIEs from parent to child (using cu_walk_functions_at()), so synthesised inlined frames are always ordered strictly root-to-leaf. In this scenario, rb_wakeups() correctly precedes rb_irq_work_queue().
In the next iteration, I will update the commit log in with a genuine, reproducible trace from the kernel (e.g. wake_up_new_task() -> activate_task() [inline] -> enqueue_task(), as exercised in the test suite) to avoid any confusion.
Quoted 5 lines
Also how do you know if
arch_irq_work_raise()is called within the same inlined call-chain? It seemsrb_irq_work_queue()indeed calls the function afterhousekeeping_any_cpu():
rb_irq_work_queue()
Quoted 8 lines
-> irq_work_queue_on() -> __irq_work_queue_local() -> irq_work_raise()I guess you compare the callchain with the previous one and merge the call if they match. Then it would support partial match and close some parents correctly like when other function is called from
rb_wakeups(), for example.
Yes. Each CPU maintains a call-stack tracking both real and synthesised inlined frames. When a function record arrives with a return address comment:
1. We resolve the inlined call chain from DWARF information and locate the enclosing real caller on the stack
2. We determine the longest common prefix between the currently active inlined frames above that caller and the newly resolved inlined chain
3. Any active inlined frames beyond the common prefix are closed and popped by emitting synthesised closing delimiters: "} /* <func> (inline) */"
4. Any new inlined frames from the common prefix index onwards are opened and pushed onto the stack
Consequently, when sibling calls are made within the same inlined parent (or when a subsequent call branches off an intermediate inlined ancestor like rb_wakeups()), shared ancestors are preserved while only the divergent frames are unwound and replaced. When the enclosing real function exits, any remaining open inlined frames above it are closed as well.
This partial matching and stack unwinding behaviour will also validated by the unit tests in tools/perf/tests/ftrace.c.
Quoted 17 lines
Quoted 10 lines
This series decomposes the feature into a modular, bisectable progression of four patches:
1. Stack frame optimisation in
__cmd_ftrace()Replaces the 4096-byte stack array in
__cmd_ftrace()with dynamic heap allocation alongside struct strbuf streaming, reducing the stack frame footprint by over 96% (down to ~160 bytes) while ensuring complete line-delimited records and robust EOF handling.Quoted 6 lines
2. Return address comment filtering
Introduces
ftrace_parse_retaddr()andftrace_filter_retaddr()with --filter-retaddr and --graph-opts filter-retaddr to strip caller comments from trace output, accompanied by an initial unit test suite intools/perf/tests/ftrace.c.
Quoted 6 lines
I'm curious why do we need options as we already have 'retaddr' graph-option to enable it. I think it should work the same as 'retaddr' is not given, no? Otherwise, we could add 'noretaddr' instead.
Also I think it's only useful with the --inline option. If so, we could handle that automatically when it's given (and 'retaddr' is not given).
That is a very good point. The only reason return address comment filtering is required in the first place is because --inline needs the kernel's funcgraph-retaddr option enabled behind the scenes in tracefs. Without the caller's return address and instruction offset, DWARF cannot resolve the inlined call chain. However, leaving those raw comments (/* <-caller+offset */) in the stream creates considerable visual clutter when the user is primarily interested in the reconstructed inlined hierarchy.
Handling this automatically is much cleaner:
1. When --inline is specified
perf ftrace enables funcgraph-retaddr in the kernel for symbol resolution, while automatically filtering the comments from the rendered output
2. If the user explicitly passes --graph-opts retaddr alongside --inline
We retain the return address comments in the output as requested
3. When --inline is not specified
The kernel's funcgraph-retaddr option remains disabled by default unless --graph-opts retaddr is supplied, in which case the comments are shown without filtering
For the next iteration, I will drop --graph-opts filter-retaddr and --filter-retaddr, folding the filtering logic directly into the --inline implementation so that it behaves automatically based on whether --graph-opts retaddr is present.
Kind regards,
Signature · 1 line
Aaron Tomlin