perf ftrace: Support display of inlined functions in function graph tracer
PATCH 1/1 perf ftrace: Support display of inlined functions in function graph tracer
The Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining.
Address this by introducing the --inline option to perf ftrace for the function_graph tracer.
When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment (e.g. /* <-irq_work_queue_on+0x81/0xa0 */). By interrogating DWARF debug information from the kernel image (vmlinux) utilising libdw, perf ftrace resolves this return address to its inlined callchain and synthesises the intermediate inlined frames directly into the streamed call graph.
Key aspects of this implementation:
1. Inlined Frame Presentation
Synthesised inlined functions are rendered with an explicit /* (inline) */ annotation at both entry and exit:
# CPU DURATION FUNCTION CALLS
# | | | | | | |
1) 0.856 us | mutex_unlock(lock=0xffffffffbc28bfa0);
0) | wake_up_new_task(p=0xffff8b1f183bac80) {
0) | rb_irq_work_queue() { /* (inline) */
0) | rb_wakeups() { /* (inline) */
0) 1.374 us | housekeeping_any_cpu(type=3 [HK_TYPE_KERNEL_NOISE]);
0) | arch_irq_work_raise() {
0) | apic_wait_icr_idle() { /* (inline) */
0) | x2apic_send_IPI_self(vector=246) {
0) | instr_sysvec_irq_work() { /* (inline) */
0) | irq_enter_rcu() {
0) | instr_sysvec_irq_work() { /* (inline) */
0) 0.673 us | irqtime_account_irq(curr=0xffff8b1f086a2c80, offset=0x1000000);
0) | } /* instr_sysvec_irq_work (inline) */
0) 1.697 us | }
This affords immediate visual clarity to the reader upon entering an inlined scope, without necessitating a search for the closing delimiter.
2. Return Address Filtering & Clutter Reduction
The raw return address comment is suppressed by default whenever --inline is enabled, avoiding redundant commentary that could otherwise mislead the reader regarding caller provenance. Should the raw return address be desired, it may be explicitly requested via --graph-opts retaddr or --no-filter-retaddr.
3. Stack Frame Optimisation in __cmd_ftrace()
The previous implementation accumulated stream data across read boundaries using large fixed stack arrays. This has been refactored to employ a single heap-allocated buffer alongside a dynamic struct strbuf. Pre-allocating the strbuf ensures that line processing incurs zero heap adjustments in the hot polling loop, while the stack frame footprint of __cmd_ftrace() is reduced by 98.7% (to 160 bytes).
4. Option Handling & Diagnostics
Adds -k or --vmlinux to designate an explicit debug image, and provides clear diagnostic guidance should --inline be invoked on a kernel lacking CONFIG_FUNCTION_GRAPH_RETADDR.
5. Automated Verification
Introduces a dedicated unit test suite (i.e. Ftrace inline processing) in tools/perf/tests/ftrace.c covering return address extraction, comment filtering, and DWARF inline symbol resolution.
--- tools/perf/Documentation/perf-ftrace.txt | 20 + tools/perf/builtin-ftrace.c | 104 ++++- tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/ftrace.c | 165 +++++++ tools/perf/tests/tests.h | 1 + tools/perf/util/Build | 1 + tools/perf/util/ftrace.c | 528 +++++++++++++++++++++++ tools/perf/util/ftrace.h | 16 + 9 files changed, 826 insertions(+), 11 deletions(-) create mode 100644 tools/perf/tests/ftrace.c create mode 100644 tools/perf/util/ftrace.c
tools/perf/Documentation/perf-ftrace.txt +20 -0
@@ -127,6 +127,7 @@ OPTIONS for 'perf ftrace trace' - retval - Show function return value. - retval-hex - Show function return value in hexadecimal format. - retaddr - Show function return address. + - filter-retaddr - Filter out function return address comments (enabled by default with --inline). - nosleep-time - Measure on-CPU time only for function_graph tracer. - noirqs - Ignore functions that happen inside interrupt. - verbose - Show process names, PIDs, timestamps, etc.
@@ -134,6 +135,25 @@ OPTIONS for 'perf ftrace trace' - depth=<n> - Set max depth for function graph tracer to follow. - tail - Print function name at the end. +--inline:: + Show inlined functions in function_graph tracer. This requires + kernel support for the funcgraph-retaddr option and a vmlinux with + debug symbols. Inlined functions are displayed in the call graph + hierarchy, and their inlined status is indicated with `/* (inline) */` + on entry and exit, e.g. `func() { /* (inline) */` and `} /* func (inline) */`. + By default, raw return address comments (`/* <-caller+offset */`) + are hidden in the output; specify `--graph-opts retaddr` to show them. + +--filter-retaddr:: + Filter out function return address comments (`/* <-caller+offset */`) + from function_graph output. This is enabled by default when `--inline` + is used; use `--no-filter-retaddr` to disable it. + +-k:: +--vmlinux=:: + Path to the vmlinux file containing debug symbols for resolving + inlined functions. + OPTIONS for 'perf ftrace latency' ---------------------------------
tools/perf/builtin-ftrace.c +93 -11
195 lines not on this page · raw message
tools/perf/tests/Build +1 -0
@@ -72,6 +72,7 @@ perf-test-y += hwmon_pmu.o perf-test-y += tool_pmu.o perf-test-y += subcmd-help.o perf-test-y += kallsyms-split.o +perf-test-y += ftrace.o ifeq ($(SRCARCH),$(filter $(SRCARCH),x86 arm arm64 powerpc riscv)) perf-test-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
tools/perf/tests/builtin-test.c +1 -0
@@ -142,6 +142,7 @@ static struct test_suite *generic_tests[] = { &suite__pfm, &suite__api_io, &suite__maps, + &suite__ftrace_inline, &suite__demangle_java, &suite__demangle_ocaml, &suite__demangle_rust,
tools/perf/tests/ftrace.c new file +165 -0
165 lines not on this page · raw message
tools/perf/tests/tests.h +1 -0
@@ -163,6 +163,7 @@ DECLARE_SUITE(perf_hooks);DECLARE_SUITE (unit_number__scnprint);DECLARE_SUITE (mem2node);DECLARE_SUITE (maps); +DECLARE_SUITE (ftrace_inline);DECLARE_SUITE (time_utils);DECLARE_SUITE (jit_write_elf);DECLARE_SUITE (api_io);
tools/perf/util/Build +1 -0
@@ -169,6 +169,7 @@ perf-util-y += list_sort.o perf-util-y += mutex.o perf-util-y += sharded_mutex.o perf-util-y += intel-tpebs.o +perf-util-y += ftrace.o perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_counter.o perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_counter_cgroup.o
tools/perf/util/ftrace.c new file +528 -0
528 lines not on this page · raw message
tools/perf/util/ftrace.h +16 -0
@@ -39,6 +39,10 @@ struct perf_ftrace {int graph_verbose;int graph_thresh;int graph_tail; +bool use_inline; +bool filter_retaddr; +bool filter_retaddr_set; +struct machine *machine; };struct filter_entry {
@@ -93,4 +97,16 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)#endif /* HAVE_BPF_SKEL */ +struct machine ; + +bool ftrace_parse_retaddr (const char *str,char *sym_name,size_t sym_len,u64 *offset); +void ftrace_filter_retaddr (char *str); +int ftrace_resolve_inlines (struct machine *machine, +const char *sym_name,u64 offset, +const char **inlined_names,int max_inlines); +int ftrace_process_fgraph_line (struct perf_ftrace *ftrace,const char *line,FILE *out); +void ftrace_cpu_states__clear (void ); +int perf_ftrace__setup_inlines (struct perf_ftrace *ftrace); +void perf_ftrace__cleanup_inlines (struct perf_ftrace *ftrace); +#endif /* __PERF_FTRACE_H__ */
Signature · 1 line
2.55.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] perf: infinite spinloop on trace EOF in ftrace - [Medium] perf: infinite parser state growth…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] perf: infinite spinloop on trace EOF in ftrace
- [Medium] perf: infinite parser state growth without funcgraph-tail
--
commit 8997c170e283406995c2dcd46938bbd3757d6735
Author: Aaron Tomlin <[email protected]>
perf ftrace: Support display of inlined functions in function graph tracer
This commit introduces the --inline option to perf ftrace for the function_graph tracer. It resolves return addresses to their inlined callchains and synthesizes intermediate inlined frames directly into the streamed call graph.
Quoted 6 lines
tools/perf/builtin-ftrace.c +0
-0@@ -810,11 +829,24 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)break ;
tools/perf/builtin-ftrace.c static int __cmd_ftrace(struct perf_ftrace *ftrace) · line 836
Quoted 10 lines
tools/perf/builtin-ftrace.c +3
-3if (pollfd.revents &POLLIN ) { -int n =read (trace_fd, buf,sizeof (buf)); +int n =read (trace_fd, buf,TRACE_BUF_SIZE );if (n <0 )break ; -if (fwrite (buf, n,1 , stdout) !=1 ) -break ; +if (ftrace->use_inline || ftrace->filter_retaddr) { +for (int i =0 ; i < n; i++) {
[Severity: Medium]
When the trace_pipe reaches EOF, read() will return 0. The old code handled this via fwrite() != 1 which broke the poll loop.
With the new logic under --inline or --filter-retaddr, if n == 0, the for loop is skipped and the outer loop doesn't break. Does this cause an infinite spinloop on poll() when ftrace reaches EOF?
Quoted 37 lines
tools/perf/builtin-ftrace.c +13
-0+if (buf[i] =='\n' ) { +if (ftrace_process_fgraph_line (ftrace, linebuf.buf, stdout) <0 ) + goto out_close_fd; +strbuf_setlen (&linebuf,0 ); + }else { +if (strbuf_addch (&linebuf, buf[i]) <0 ) + goto out_close_fd; + } + } + }else { +if (fwrite (buf, n,1 , stdout) !=1 ) +break ; + }/* flush output since stdout is in full buffering mode due to pager */ fflush (stdout); }tools/perf/util/ftrace.c new file +13
-0+int ftrace_process_fgraph_line (struct perf_ftrace *ftrace,const char *line,FILE *out) +{ +if (is_exit) { +char exit_name[128 ]; + +extract_exit_func_name (func_start, exit_name,sizeof (exit_name)); + +/* Close all inlined frames above match_idx */ +while (cs->depth >0 && (match_idx <0 || cs->depth -1 > match_idx)) { +if (cs->stack[cs->depth -1 ].inlined) { + cs->inlined_depth--; + }else { +break ;
tools/perf/util/ftrace.c line 343
Quoted 10 lines
tools/perf/util/ftrace.c new file +10
-0+ } + } + +/* Pop the matched real frame */ +if (cs->depth >0 && !cs->stack[cs->depth -1 ].inlined && + cs->stack[cs->depth -1 ].name && +strcmp (cs->stack[cs->depth -1 ].name, exit_name) ==0 ) { +free (cs->stack[cs->depth -1 ].name); + cs->depth--; + }
[Severity: Medium]
If funcgraph-tail is disabled (the default setting), function exit events lack the trailing comment containing the function name. This means extract_exit_func_name() will yield an empty string.
Because exit_name is empty, will strcmp() against the stack frame fail, preventing the real frame from ever being popped?
Quoted 16 lines
[ ... ]
tools/perf/util/ftrace.c new file +15
-0+/* Open new inlined frames */ +for (i = common; i < num_inlines; i++) { +int entry_indent = leading_spaces + cs->inlined_depth *2 ; + +if (!inlined_names[i]) { +pr_err ("Not enough memory\n" ); +for (int j =0 ; j < num_inlines; j++) +free ((void *)inlined_names[j]); +return -ENOMEM ; + } +fprintf (out,"%s%*s%s() { /* (inline) */\n" , blank_prefix, entry_indent,"" , inlined_names[i]); +if (cs->depth <FTRACE_MAX_STACK ) { + cs->stack[cs->depth].name =strdup (inlined_names[i]); +if (!cs->stack[cs->depth].name) { +pr_err ("Not enough memory\n" );
tools/perf/util/ftrace.c line 437
Quoted 10 lines
tools/perf/util/ftrace.c new file +10
-0+for (int j =0 ; j < num_inlines; j++) +free ((void *)inlined_names[j]); +return -ENOMEM ; + } + cs->stack[cs->depth].inlined = true; + cs->stack[cs->depth].indent = entry_indent; + cs->depth++; + } + cs->inlined_depth++; + }
[Severity: Medium]
If real frames are never popped, FTRACE_MAX_STACK will eventually saturate. Once cs->depth reaches FTRACE_MAX_STACK, new frames are no longer pushed to the stack, but cs->inlined_depth continues to unconditionally increment here.
Does this cause infinite indentation growth, since cs->inlined_depth will never decrement on exit due to the stuck real frames blocking the popping loop above?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 21 lines · On Mon, Oct 05, 2026 at 09:27:07PM +0000, [email protected] wrote:
Quoted 6 lines
tools/perf/builtin-ftrace.c +0
-0@@ -810,11 +829,24 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)break ;tools/perf/builtin-ftrace.c static int __cmd_ftrace(struct perf_ftrace *ftrace) · line 836
Quoted 10 lines
tools/perf/builtin-ftrace.c +3
-3if (pollfd.revents &POLLIN ) { -int n =read (trace_fd, buf,sizeof (buf)); +int n =read (trace_fd, buf,TRACE_BUF_SIZE );if (n <0 )break ; -if (fwrite (buf, n,1 , stdout) !=1 ) -break ; +if (ftrace->use_inline || ftrace->filter_retaddr) { +for (int i =0 ; i < n; i++) {[Severity: Medium]
When thetrace_pipereaches EOF,read()will return 0. The old code handled this viafwrite()!= 1 which broke the poll loop.
Quoted 3 lines
With the new logic under --inline or --filter-retaddr, if n == 0, the for loop is skipped and the outer loop doesn't break. Does this cause an infinite spinloop on
poll()when ftrace reaches EOF?
When read() returns 0 on EOF, skipping the line-buffering loop without breaking would cause poll() to immediately wake again and spin. This has been resolved by explicitly checking for EOF (n <= 0) upon read() across both the polling loop and the post-tracing drain loop:
int n = read(trace_fd, buf, TRACE_BUF_SIZE);
if (n <= 0)
break;Quoted 54 lines
Quoted 38 lines
tools/perf/builtin-ftrace.c +13
-0+if (buf[i] =='\n' ) { +if (ftrace_process_fgraph_line (ftrace, linebuf.buf, stdout) <0 ) + goto out_close_fd; +strbuf_setlen (&linebuf,0 ); + }else { +if (strbuf_addch (&linebuf, buf[i]) <0 ) + goto out_close_fd; + } + } + }else { +if (fwrite (buf, n,1 , stdout) !=1 ) +break ; + }/* flush output since stdout is in full buffering mode due to pager */ fflush (stdout); }tools/perf/util/ftrace.c new file +13
-0+int ftrace_process_fgraph_line (struct perf_ftrace *ftrace,const char *line,FILE *out) +{ +if (is_exit) { +char exit_name[128 ]; + +extract_exit_func_name (func_start, exit_name,sizeof (exit_name)); + +/* Close all inlined frames above match_idx */ +while (cs->depth >0 && (match_idx <0 || cs->depth -1 > match_idx)) { +if (cs->stack[cs->depth -1 ].inlined) { + cs->inlined_depth--; + }else { +break ;tools/perf/util/ftrace.c line 343
Quoted 10 lines
tools/perf/util/ftrace.c new file +10
-0+ } + } + +/* Pop the matched real frame */ +if (cs->depth >0 && !cs->stack[cs->depth -1 ].inlined && + cs->stack[cs->depth -1 ].name && +strcmp (cs->stack[cs->depth -1 ].name, exit_name) ==0 ) { +free (cs->stack[cs->depth -1 ].name); + cs->depth--; + }[Severity: Medium]
If funcgraph-tail is disabled (the default setting), function exit events lack the trailing comment containing the function name. This meansextract_exit_func_name()will yield an empty string.
Quoted 2 lines
Because
exit_nameis empty, willstrcmp()against the stack frame fail, preventing the real frame from ever being popped?
In the kernel's function_graph tracer, funcgraph-tail is indeed disabled by default, emitting bare closing braces '}' without a trailing function name comment. Because extract_exit_func_name() yielded an empty string, strcmp() failed against real frames, causing them to remain permanently on the call stack and preventing inlined frames underneath from closing.
To address this, frame matching in ftrace_process_fgraph_line() now supports both formats:
1. When funcgraph-tail is enabled (or comments exist), it matches the innermost real frame by name
2. When funcgraph-tail is disabled (exit_name[0] == '\0'), it adheres to standard LIFO call stack semantics and pops the innermost non-inlined frame (match_idx = cs->depth - 1)
Quoted 33 lines
Quoted 17 lines
[ ... ]
tools/perf/util/ftrace.c new file +15
-0+/* Open new inlined frames */ +for (i = common; i < num_inlines; i++) { +int entry_indent = leading_spaces + cs->inlined_depth *2 ; + +if (!inlined_names[i]) { +pr_err ("Not enough memory\n" ); +for (int j =0 ; j < num_inlines; j++) +free ((void *)inlined_names[j]); +return -ENOMEM ; + } +fprintf (out,"%s%*s%s() { /* (inline) */\n" , blank_prefix, entry_indent,"" , inlined_names[i]); +if (cs->depth <FTRACE_MAX_STACK ) { + cs->stack[cs->depth].name =strdup (inlined_names[i]); +if (!cs->stack[cs->depth].name) { +pr_err ("Not enough memory\n" );tools/perf/util/ftrace.c line 437
Quoted 10 lines
tools/perf/util/ftrace.c new file +10
-0+for (int j =0 ; j < num_inlines; j++) +free ((void *)inlined_names[j]); +return -ENOMEM ; + } + cs->stack[cs->depth].inlined = true; + cs->stack[cs->depth].indent = entry_indent; + cs->depth++; + } + cs->inlined_depth++; + }[Severity: Medium]
If real frames are never popped,FTRACE_MAX_STACKwill eventually saturate. Oncecs->depthreachesFTRACE_MAX_STACK, new frames are no longer pushed to the stack, butcs->inlined_depthcontinues to unconditionally increment here.
Quoted 5 lines
Does this cause infinite indentation growth, since
cs->inlined_depthwill never decrement on exit due to the stuck real frames blocking the popping loop above?Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Indeed. Once the call stack reached FTRACE_MAX_STACK, new frames were dropped while cs->inlined_depth continued to increment unconditionally, leading to unbounded indentation growth. This has been addressed with three safeguards:
1. Clamping inline frame generation
If cs->depth reaches FTRACE_MAX_STACK, inline expansion breaks
early, preventing synthesis of frames that cannot be tracked.
2. Synchronised depth tracking
cs->inlined_depth is now incremented only when a frame is
successfully allocated and pushed onto cs->stack.
3. Underflow protection
cs->inlined_depth decrements on exit are guarded with
(cs->inlined_depth > 0).Signature · 1 line
Aaron Tomlin