Skip to contentSkip to conversation

perf ftrace: Support display of inlined functions in function graph tracer

PATCH Superseded Aaron Tomlin 2 participants 3 messages Oct 5 – Oct 6 perf events

v1 of 13 Superseded Latest is v13 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 perf ftrace: Support display of inlined functions in function graph tracer

The Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining.

Address this by introducing the --inline option to perf ftrace for the function_graph tracer.

When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment (e.g. /* <-irq_work_queue_on+0x81/0xa0 */). By interrogating DWARF debug information from the kernel image (vmlinux) utilising libdw, perf ftrace resolves this return address to its inlined callchain and synthesises the intermediate inlined frames directly into the streamed call graph.

Key aspects of this implementation:

1. Inlined Frame Presentation

Synthesised inlined functions are rendered with an explicit /* (inline) */ annotation at both entry and exit:

# CPU DURATION FUNCTION CALLS
# | | | | | | |
1) 0.856 us | mutex_unlock(lock=0xffffffffbc28bfa0);
0) | wake_up_new_task(p=0xffff8b1f183bac80) {
0) | rb_irq_work_queue() { /* (inline) */
0) | rb_wakeups() { /* (inline) */
0) 1.374 us | housekeeping_any_cpu(type=3 [HK_TYPE_KERNEL_NOISE]);
0) | arch_irq_work_raise() {
0) | apic_wait_icr_idle() { /* (inline) */
0) | x2apic_send_IPI_self(vector=246) {
0) | instr_sysvec_irq_work() { /* (inline) */
0) | irq_enter_rcu() {
0) | instr_sysvec_irq_work() { /* (inline) */
0) 0.673 us | irqtime_account_irq(curr=0xffff8b1f086a2c80, offset=0x1000000);
0) | } /* instr_sysvec_irq_work (inline) */
0) 1.697 us | }

This affords immediate visual clarity to the reader upon entering an inlined scope, without necessitating a search for the closing delimiter.

2. Return Address Filtering & Clutter Reduction

The raw return address comment is suppressed by default whenever --inline is enabled, avoiding redundant commentary that could otherwise mislead the reader regarding caller provenance. Should the raw return address be desired, it may be explicitly requested via --graph-opts retaddr or --no-filter-retaddr.

3. Stack Frame Optimisation in __cmd_ftrace()

The previous implementation accumulated stream data across read boundaries using large fixed stack arrays. This has been refactored to employ a single heap-allocated buffer alongside a dynamic struct strbuf. Pre-allocating the strbuf ensures that line processing incurs zero heap adjustments in the hot polling loop, while the stack frame footprint of __cmd_ftrace() is reduced by 98.7% (to 160 bytes).

4. Option Handling & Diagnostics

Adds -k or --vmlinux to designate an explicit debug image, and provides clear diagnostic guidance should --inline be invoked on a kernel lacking CONFIG_FUNCTION_GRAPH_RETADDR.

5. Automated Verification

Introduces a dedicated unit test suite (i.e. Ftrace inline processing) in tools/perf/tests/ftrace.c covering return address extraction, comment filtering, and DWARF inline symbol resolution.

---
 tools/perf/Documentation/perf-ftrace.txt |  20 +
 tools/perf/builtin-ftrace.c              | 104 ++++-
 tools/perf/tests/Build                   |   1 +
 tools/perf/tests/builtin-test.c          |   1 +
 tools/perf/tests/ftrace.c                | 165 +++++++
 tools/perf/tests/tests.h                 |   1 +
 tools/perf/util/Build                    |   1 +
 tools/perf/util/ftrace.c                 | 528 +++++++++++++++++++++++
 tools/perf/util/ftrace.h                 |  16 +
 9 files changed, 826 insertions(+), 11 deletions(-)
 create mode 100644 tools/perf/tests/ftrace.c
 create mode 100644 tools/perf/util/ftrace.c
tools/perf/Documentation/perf-ftrace.txt +20 -0
index 3f3808e513fe..ca0f1d523dbc 100644
@@ -127,6 +127,7 @@ OPTIONS for 'perf ftrace trace'
 	  - retval       - Show function return value.
 	  - retval-hex   - Show function return value in hexadecimal format.
 	  - retaddr      - Show function return address.
+	  - filter-retaddr - Filter out function return address comments (enabled by default with --inline).
 	  - nosleep-time - Measure on-CPU time only for function_graph tracer.
 	  - noirqs       - Ignore functions that happen inside interrupt.
 	  - verbose      - Show process names, PIDs, timestamps, etc.
@@ -134,6 +135,25 @@ OPTIONS for 'perf ftrace trace'
 	  - depth=<n>    - Set max depth for function graph tracer to follow.
 	  - tail         - Print function name at the end.
 
+--inline::
+	Show inlined functions in function_graph tracer. This requires
+	kernel support for the funcgraph-retaddr option and a vmlinux with
+	debug symbols. Inlined functions are displayed in the call graph
+	hierarchy, and their inlined status is indicated with `/* (inline) */`
+	on entry and exit, e.g. `func() { /* (inline) */` and `} /* func (inline) */`.
+	By default, raw return address comments (`/* <-caller+offset */`)
+	are hidden in the output; specify `--graph-opts retaddr` to show them.
+
+--filter-retaddr::
+	Filter out function return address comments (`/* <-caller+offset */`)
+	from function_graph output. This is enabled by default when `--inline`
+	is used; use `--no-filter-retaddr` to disable it.
+
+-k::
+--vmlinux=::
+	Path to the vmlinux file containing debug symbols for resolving
+	inlined functions.
+
 
 OPTIONS for 'perf ftrace latency'
 ---------------------------------
tools/perf/builtin-ftrace.c +93 -11

195 lines not on this page · raw message

tools/perf/tests/Build +1 -0
index 395f2866094a..136138932fcb 100644
@@ -72,6 +72,7 @@ perf-test-y += hwmon_pmu.o
 perf-test-y += tool_pmu.o
 perf-test-y += subcmd-help.o
 perf-test-y += kallsyms-split.o
+perf-test-y += ftrace.o
 
 ifeq ($(SRCARCH),$(filter $(SRCARCH),x86 arm arm64 powerpc riscv))
 perf-test-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
tools/perf/tests/builtin-test.c +1 -0
index d2f594921e25..865f79353b21 100644
@@ -142,6 +142,7 @@ static struct test_suite *generic_tests[] = {
 	&suite__pfm,
 	&suite__api_io,
 	&suite__maps,
+	&suite__ftrace_inline,
 	&suite__demangle_java,
 	&suite__demangle_ocaml,
 	&suite__demangle_rust,
tools/perf/tests/ftrace.c new file +165 -0

165 lines not on this page · raw message

tools/perf/tests/tests.h +1 -0
index 9c96f33483d1..dd489210426f 100644
@@ -163,6 +163,7 @@ DECLARE_SUITE(perf_hooks);
 DECLARE_SUITE(unit_number__scnprint);
 DECLARE_SUITE(mem2node);
 DECLARE_SUITE(maps);
+DECLARE_SUITE(ftrace_inline);
 DECLARE_SUITE(time_utils);
 DECLARE_SUITE(jit_write_elf);
 DECLARE_SUITE(api_io);
tools/perf/util/Build +1 -0
index 2c1f880c4c47..5b082d09006d 100644
@@ -169,6 +169,7 @@ perf-util-y += list_sort.o
 perf-util-y += mutex.o
 perf-util-y += sharded_mutex.o
 perf-util-y += intel-tpebs.o
+perf-util-y += ftrace.o
 
 perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_counter.o
 perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_counter_cgroup.o
tools/perf/util/ftrace.c new file +528 -0

528 lines not on this page · raw message

tools/perf/util/ftrace.h +16 -0
index 950f2efafad2..65514ca970b4 100644
@@ -39,6 +39,10 @@ struct perf_ftrace {
 	int			graph_verbose;
 	int			graph_thresh;
 	int			graph_tail;
+	bool			use_inline;
+	bool			filter_retaddr;
+	bool			filter_retaddr_set;
+	struct machine		*machine;
 };
 
 struct filter_entry {
@@ -93,4 +97,16 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)
 
 #endif  /* HAVE_BPF_SKEL */
 
+struct machine;
+
+bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset);
+void ftrace_filter_retaddr(char *str);
+int ftrace_resolve_inlines(struct machine *machine,
+			   const char *sym_name, u64 offset,
+			   const char **inlined_names, int max_inlines);
+int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out);
+void ftrace_cpu_states__clear(void);
+int perf_ftrace__setup_inlines(struct perf_ftrace *ftrace);
+void perf_ftrace__cleanup_inlines(struct perf_ftrace *ftrace);
+
 #endif  /* __PERF_FTRACE_H__ */
Signature · 1 line
2.55.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] perf: infinite spinloop on trace EOF in ftrace - [Medium] perf: infinite parser state growth…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] perf: infinite spinloop on trace EOF in ftrace
- [Medium] perf: infinite parser state growth without funcgraph-tail
--

commit 8997c170e283406995c2dcd46938bbd3757d6735
Author: Aaron Tomlin <[email protected]>
perf ftrace: Support display of inlined functions in function graph tracer

This commit introduces the --inline option to perf ftrace for the function_graph tracer. It resolves return addresses to their inlined callchains and synthesizes intermediate inlined frames directly into the streamed call graph.

Quoted 6 lines
tools/perf/builtin-ftrace.c +0 -0
[ ... ]
@@ -810,11 +829,24 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 			break;

tools/perf/builtin-ftrace.c static int __cmd_ftrace(struct perf_ftrace *ftrace) · line 836

Quoted 10 lines
tools/perf/builtin-ftrace.c +3 -3
 
 		if (pollfd.revents & POLLIN) {
-			int n = read(trace_fd, buf, sizeof(buf));
+			int n = read(trace_fd, buf, TRACE_BUF_SIZE);
 			if (n < 0)
 				break;
-			if (fwrite(buf, n, 1, stdout) != 1)
-				break;
+			if (ftrace->use_inline || ftrace->filter_retaddr) {
+				for (int i = 0; i < n; i++) {

[Severity: Medium]
When the trace_pipe reaches EOF, read() will return 0. The old code handled this via fwrite() != 1 which broke the poll loop.

With the new logic under --inline or --filter-retaddr, if n == 0, the for loop is skipped and the outer loop doesn't break. Does this cause an infinite spinloop on poll() when ftrace reaches EOF?

Quoted 37 lines
tools/perf/builtin-ftrace.c +13 -0
+					if (buf[i] == '\n') {
+						if (ftrace_process_fgraph_line(ftrace, linebuf.buf, stdout) < 0)
+							goto out_close_fd;
+						strbuf_setlen(&linebuf, 0);
+					} else {
+						if (strbuf_addch(&linebuf, buf[i]) < 0)
+							goto out_close_fd;
+					}
+				}
+			} else {
+				if (fwrite(buf, n, 1, stdout) != 1)
+					break;
+			}
 			/* flush output since stdout is in full buffering mode due to pager */
 			fflush(stdout);
 		}
tools/perf/util/ftrace.c new file +13 -0
[ ... ]
+int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out)
+{
[ ... ]
+	if (is_exit) {
+		char exit_name[128];
+
+		extract_exit_func_name(func_start, exit_name, sizeof(exit_name));
+
[ ... ]
+		/* Close all inlined frames above match_idx */
+		while (cs->depth > 0 && (match_idx < 0 || cs->depth - 1 > match_idx)) {
+			if (cs->stack[cs->depth - 1].inlined) {
[ ... ]
+				cs->inlined_depth--;
[ ... ]
+			} else {
+				break;

tools/perf/util/ftrace.c line 343

Quoted 10 lines
tools/perf/util/ftrace.c new file +10 -0
+			}
+		}
+
+		/* Pop the matched real frame */
+		if (cs->depth > 0 && !cs->stack[cs->depth - 1].inlined &&
+		    cs->stack[cs->depth - 1].name &&
+		    strcmp(cs->stack[cs->depth - 1].name, exit_name) == 0) {
+			free(cs->stack[cs->depth - 1].name);
+			cs->depth--;
+		}

[Severity: Medium]
If funcgraph-tail is disabled (the default setting), function exit events lack the trailing comment containing the function name. This means extract_exit_func_name() will yield an empty string.

Because exit_name is empty, will strcmp() against the stack frame fail, preventing the real frame from ever being popped?

Quoted 16 lines

[ ... ]

tools/perf/util/ftrace.c new file +15 -0
+	/* Open new inlined frames */
+	for (i = common; i < num_inlines; i++) {
+		int entry_indent = leading_spaces + cs->inlined_depth * 2;
+
+		if (!inlined_names[i]) {
+			pr_err("Not enough memory\n");
+			for (int j = 0; j < num_inlines; j++)
+				free((void *)inlined_names[j]);
+			return -ENOMEM;
+		}
+		fprintf(out, "%s%*s%s() { /* (inline) */\n", blank_prefix, entry_indent, "", inlined_names[i]);
+		if (cs->depth < FTRACE_MAX_STACK) {
+			cs->stack[cs->depth].name = strdup(inlined_names[i]);
+			if (!cs->stack[cs->depth].name) {
+				pr_err("Not enough memory\n");

tools/perf/util/ftrace.c line 437

Quoted 10 lines
tools/perf/util/ftrace.c new file +10 -0
+				for (int j = 0; j < num_inlines; j++)
+					free((void *)inlined_names[j]);
+				return -ENOMEM;
+			}
+			cs->stack[cs->depth].inlined = true;
+			cs->stack[cs->depth].indent = entry_indent;
+			cs->depth++;
+		}
+		cs->inlined_depth++;
+	}

[Severity: Medium]
If real frames are never popped, FTRACE_MAX_STACK will eventually saturate. Once cs->depth reaches FTRACE_MAX_STACK, new frames are no longer pushed to the stack, but cs->inlined_depth continues to unconditionally increment here.

Does this cause infinite indentation growth, since cs->inlined_depth will never decrement on exit due to the stuck real frames blocking the popping loop above?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
Quoted 21 lines · On Mon, Oct 05, 2026 at 09:27:07PM +0000, [email protected] wrote:
Quoted 6 lines
tools/perf/builtin-ftrace.c +0 -0
[ ... ]
@@ -810,11 +829,24 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 			break;

tools/perf/builtin-ftrace.c static int __cmd_ftrace(struct perf_ftrace *ftrace) · line 836

Quoted 10 lines
tools/perf/builtin-ftrace.c +3 -3
 
 		if (pollfd.revents & POLLIN) {
-			int n = read(trace_fd, buf, sizeof(buf));
+			int n = read(trace_fd, buf, TRACE_BUF_SIZE);
 			if (n < 0)
 				break;
-			if (fwrite(buf, n, 1, stdout) != 1)
-				break;
+			if (ftrace->use_inline || ftrace->filter_retaddr) {
+				for (int i = 0; i < n; i++) {

[Severity: Medium]
When the trace_pipe reaches EOF, read() will return 0. The old code handled this via fwrite() != 1 which broke the poll loop.

Quoted 3 lines

With the new logic under --inline or --filter-retaddr, if n == 0, the for loop is skipped and the outer loop doesn't break. Does this cause an infinite spinloop on poll() when ftrace reaches EOF?

When read() returns 0 on EOF, skipping the line-buffering loop without breaking would cause poll() to immediately wake again and spin. This has been resolved by explicitly checking for EOF (n <= 0) upon read() across both the polling loop and the post-tracing drain loop:

        int n = read(trace_fd, buf, TRACE_BUF_SIZE);
        if (n <= 0)
            break;
Quoted 54 lines
Quoted 38 lines
tools/perf/builtin-ftrace.c +13 -0
+					if (buf[i] == '\n') {
+						if (ftrace_process_fgraph_line(ftrace, linebuf.buf, stdout) < 0)
+							goto out_close_fd;
+						strbuf_setlen(&linebuf, 0);
+					} else {
+						if (strbuf_addch(&linebuf, buf[i]) < 0)
+							goto out_close_fd;
+					}
+				}
+			} else {
+				if (fwrite(buf, n, 1, stdout) != 1)
+					break;
+			}
 			/* flush output since stdout is in full buffering mode due to pager */
 			fflush(stdout);
 		}
tools/perf/util/ftrace.c new file +13 -0
[ ... ]
+int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out)
+{
[ ... ]
+	if (is_exit) {
+		char exit_name[128];
+
+		extract_exit_func_name(func_start, exit_name, sizeof(exit_name));
+
[ ... ]
+		/* Close all inlined frames above match_idx */
+		while (cs->depth > 0 && (match_idx < 0 || cs->depth - 1 > match_idx)) {
+			if (cs->stack[cs->depth - 1].inlined) {
[ ... ]
+				cs->inlined_depth--;
[ ... ]
+			} else {
+				break;

tools/perf/util/ftrace.c line 343

Quoted 10 lines
tools/perf/util/ftrace.c new file +10 -0
+			}
+		}
+
+		/* Pop the matched real frame */
+		if (cs->depth > 0 && !cs->stack[cs->depth - 1].inlined &&
+		    cs->stack[cs->depth - 1].name &&
+		    strcmp(cs->stack[cs->depth - 1].name, exit_name) == 0) {
+			free(cs->stack[cs->depth - 1].name);
+			cs->depth--;
+		}

[Severity: Medium]
If funcgraph-tail is disabled (the default setting), function exit events lack the trailing comment containing the function name. This means extract_exit_func_name() will yield an empty string.

Quoted 2 lines

Because exit_name is empty, will strcmp() against the stack frame fail, preventing the real frame from ever being popped?

In the kernel's function_graph tracer, funcgraph-tail is indeed disabled by default, emitting bare closing braces '}' without a trailing function name comment. Because extract_exit_func_name() yielded an empty string, strcmp() failed against real frames, causing them to remain permanently on the call stack and preventing inlined frames underneath from closing.

To address this, frame matching in ftrace_process_fgraph_line() now supports both formats:

1. When funcgraph-tail is enabled (or comments exist), it matches the innermost real frame by name

2. When funcgraph-tail is disabled (exit_name[0] == '\0'), it adheres to standard LIFO call stack semantics and pops the innermost non-inlined frame (match_idx = cs->depth - 1)

Quoted 33 lines
Quoted 17 lines

[ ... ]

tools/perf/util/ftrace.c new file +15 -0
+	/* Open new inlined frames */
+	for (i = common; i < num_inlines; i++) {
+		int entry_indent = leading_spaces + cs->inlined_depth * 2;
+
+		if (!inlined_names[i]) {
+			pr_err("Not enough memory\n");
+			for (int j = 0; j < num_inlines; j++)
+				free((void *)inlined_names[j]);
+			return -ENOMEM;
+		}
+		fprintf(out, "%s%*s%s() { /* (inline) */\n", blank_prefix, entry_indent, "", inlined_names[i]);
+		if (cs->depth < FTRACE_MAX_STACK) {
+			cs->stack[cs->depth].name = strdup(inlined_names[i]);
+			if (!cs->stack[cs->depth].name) {
+				pr_err("Not enough memory\n");

tools/perf/util/ftrace.c line 437

Quoted 10 lines
tools/perf/util/ftrace.c new file +10 -0
+				for (int j = 0; j < num_inlines; j++)
+					free((void *)inlined_names[j]);
+				return -ENOMEM;
+			}
+			cs->stack[cs->depth].inlined = true;
+			cs->stack[cs->depth].indent = entry_indent;
+			cs->depth++;
+		}
+		cs->inlined_depth++;
+	}

[Severity: Medium]
If real frames are never popped, FTRACE_MAX_STACK will eventually saturate. Once cs->depth reaches FTRACE_MAX_STACK, new frames are no longer pushed to the stack, but cs->inlined_depth continues to unconditionally increment here.

Quoted 5 lines

Does this cause infinite indentation growth, since cs->inlined_depth will never decrement on exit due to the stuck real frames blocking the popping loop above?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1

Indeed. Once the call stack reached FTRACE_MAX_STACK, new frames were dropped while cs->inlined_depth continued to increment unconditionally, leading to unbounded indentation growth. This has been addressed with three safeguards:

    1.  Clamping inline frame generation

        If cs->depth reaches FTRACE_MAX_STACK, inline expansion breaks
        early, preventing synthesis of frames that cannot be tracked.

    2.  Synchronised depth tracking

        cs->inlined_depth is now incremented only when a frame is
        successfully allocated and pushed onto cs->stack.

    3.  Underflow protection

        cs->inlined_depth decrements on exit are guarded with
        (cs->inlined_depth > 0).
Signature · 1 line
Aaron Tomlin