Skip to contentSkip to conversation

perf ftrace: Support inlined functions and display enhancements in function graph tracer

PATCH Superseded Aaron Tomlin 2 participants 13 messages Oct 15 – Oct 16 perf events

v5 of 13 Superseded Latest is v13 →

lore
Share & tools
lore
#0 lore raw

v5 cover perf ftrace: Support inlined functions and display enhancements in function graph tracer

The Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining (such as the scheduler, memory management, locking primitives, and RCU).

When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment (e.g. /* <-wake_up_new_task+0x1d1/0x3e0 */). By interrogating DWARF debug information from the kernel image (vmlinux) utilising libdw, perf ftrace resolves this return address to its inlined callchain and synthesises the intermediate inlined frames directly into the streamed call graph.

This patch series addresses these challenges through five focused commits:

1. Stack frame optimisation in __cmd_ftrace()

Replaces the 4096-byte stack array in __cmd_ftrace() with dynamic heap allocation alongside struct strbuf streaming, reducing the stack frame footprint by over 96% (down to ~160 bytes) whilst ensuring complete line-delimited records, explicit NUL termination, and robust error unwind and EOF handling.

2. Return address comment filtering

Introduces ftrace_parse_retaddr() and ftrace_filter_retaddr() with --filter-retaddr and --graph-opts filter-retaddr to strip caller comments from trace output, accompanied by an initial unit test suite in tools/perf/tests/ftrace.c.

3. Inlined function call graph reconstruction

Introduces --inline and -k/--vmlinux, dynamically synthesising inlined frames marked with /* (inline) */. Correctly matches bare closing braces when funcgraph-tail is disabled, clamps stack depth to prevent indentation drift, and expands the automated test suite to verify inline resolution, comment filtering, and stack saturation recovery.

4. Execution duration suppression

Introduces --graph-opts noduration (accepting noduration=[0|1]) to toggle the kernel's funcgraph-duration tracing option, omitting the duration column when the user is focused purely on call graph topology and inlined hierarchy. Rejects noduration when invoked under 'perf ftrace profile'.

5. Extraneous function filtering in function graph tracer

Introduces userspace call graph filtering when targeted tracing (-G or --graph-funcs) is active. Suppresses transient unrelated entries arriving before filter activation as well as detached callee trees (such as ring-buffer wakeups) originating outside the target hierarchy.

Synthesised inlined functions are rendered with an explicit /* (inline) */ annotation at both entry and exit:

# tracer: function_graph
#
# CPU FUNCTION CALLS
# | | | | |
3) activate_task() {
3) enqueue_task() {
3) psi_enqueue() { /* (inline) */
3) arch_static_branch_jump() { /* (inline) */
3) enqueue_task_fair() {
3) enqueue_hierarchy() { /* (inline) */
3) update_curr();
3) enqueue_entity() {
3) account_entity_enqueue() { /* (inline) */
3) account_llc_enqueue();
3) } /* account_entity_enqueue (inline) */
3) }
3) } /* enqueue_hierarchy (inline) */
3) place_entity() {
3) avg_vruntime();
3) }
3) __enqueue_entity();
3) dl_server_start();
3) }
3) } /* arch_static_branch_jump (inline) */
3) } /* psi_enqueue (inline) */
3) sched_info_enqueue() { /* (inline) */
3) psi_task_change() {
3) psi_write_begin() { /* (inline) */
3) psi_flags_change();
3) } /* psi_write_begin (inline) */
3) psi_group_change() {
3) record_times();
3) }
3) }
3) } /* sched_info_enqueue (inline) */
3) }
3) }

Changes since v4:

- Handled non-fatal EAGAIN, EWOULDBLOCK, and EINTR in __cmd_ftrace() polling and draining loops to prevent premature session termination on non-blocking reads or signal interruptions

- Eliminated an unused local variable endptr in ftrace_parse_retaddr() by passing NULL to strtoull()

- Prevented stack buffer overflows in make_blank_prefix() by clamping the blanking loop bounds and end pointer to the destination buffer length when processing trace records with exceptionally long prefixes

- Tracked call stack saturation with overflow_depth to prevent untracked function exits with bare closing braces from prematurely popping tracked frames from the parser stack on deep call chains exceeding FTRACE_MAX_STACK

- Ensured stale non-inlined callee frames resulting from dropped function exits do not block inactive inlined frames from being closed upon subsequent entries by unwinding the stack down to caller_idx + 1 + common

- Fixed filter state not resetting for targeted leaf calls when --inline is active: synthesised inline frames inflated cs->depth beyond cs->target_root_depth, causing subsequent sibling functions to be falsely classified as detached orphans and bypassing inline unwinding; resolved by tracking target root entries (is_target_root), explicitly unwinding all synthesised inlines, and resetting filter state upon leaf completion

- Expanded unit test coverage to verify recovery across dropped exit events and bare closing braces, and to verify that targeted leaf calls with synthesised inlined frames properly close inlines and reset filter state

- Link to v4: https://lore.kernel.org/lkml/[email protected]/

Changes since v3:

- Explicitly rejected the noduration option when invoked under the profile subcommand, since profiling inherently requires function duration to compute timing statistics

- Fixed make_blank_prefix() clobbering the comm-pid column in verbose mode by searching backwards for the preceding "|" delimiter, and returning early when graph_noduration is active

- Fixed filter state not resetting for targeted leaf calls at non-zero call stack depths by checking cs->depth <= cs->target_root_depth and clearing cs->target_root_depth, preventing subsequent non-targeted sibling functions within the parent frame from leaking into trace output

- Expanded unit test coverage to verify comm-pid preservation under verbose and verbose-noduration inline modes, and to verify that targeted leaf calls at depth > 0 properly reset the target filter state.

- Link to v3: https://lore.kernel.org/lkml/[email protected]/

Changes since v2:

- Expanded the series from 4 to 5 patches by adding a dedicated commit ("perf ftrace: Filter extraneous functions in function graph tracer") to enforce strict target callgraph boundaries when -G or --graph-funcs is specified

- Suppressed unrelated top-level functions that leak through the transient tracefs filter initialisation window before RCU synchronisation completes

- Filtered detached/orphan calls from appearing as nested children within the target graph by validating caller return addresses against the active target stack

- Tracked nested orphan scope depth (cs->orphan_depth) to cleanly suppress child calls and closing delimiters of detached subtrees without desynchronising the target stack

- Added automated unit test coverage in tools/perf/tests/ftrace.c (test_graph_target_filter()) validating top-level suppression (both with and without --inline) and detached orphan wakeup suppression

- Link to v2: https://lore.kernel.org/lkml/[email protected]/

Changes since v1:

- Modularised the implementation into a 4-patch bisectable series rather than a monolithic commit

- Resolved potential infinite loop on poll() upon trace_pipe EOF

- Fixed stuck call stack frames when funcgraph-tail is disabled by supporting bare closing delimiters matching LIFO stack frames

- Prevented unbounded indentation growth on deep call trees by clamping inline expansion and synchronising cs->inlined_depth to frames successfully pushed onto cs->stack

- Added dedicated unit tests covering bare closing braces and stack saturation recovery

- Refined option documentation in perf-ftrace.txt

- Link to v1: https://lore.kernel.org/lkml/[email protected]/

Aaron Tomlin (5):
perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming perf ftrace: Support filtering return address comments perf ftrace: Support display of inlined functions in function graph
tracer
perf ftrace: Support omitting execution duration in function graph
tracer
perf ftrace: Filter extraneous functions in function graph tracer

 tools/perf/Documentation/perf-ftrace.txt |  20 +
 tools/perf/builtin-ftrace.c              | 156 +++-
 tools/perf/tests/Build                   |   1 +
 tools/perf/tests/builtin-test.c          |   1 +
 tools/perf/tests/ftrace.c                | 837 ++++++++++++++++++++
 tools/perf/tests/tests.h                 |   1 +
 tools/perf/util/Build                    |   1 +
 tools/perf/util/ftrace.c                 | 930 +++++++++++++++++++++++
 tools/perf/util/ftrace.h                 |  17 +
 9 files changed, 1948 insertions(+), 16 deletions(-)
 create mode 100644 tools/perf/tests/ftrace.c
 create mode 100644 tools/perf/util/ftrace.c
Signature · 1 line
2.55.0
#1 lore raw

PATCH v5 1/5 perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming

In __cmd_ftrace(), incoming trace data from trace_pipe was historically buffered in a fixed 4096-byte array allocated directly on the stack frame. Furthermore, reads were written directly to stdout via fwrite() in arbitrary chunk sizes, rather than as discrete, well-delimited trace lines.

Refactor __cmd_ftrace() to employ dynamic heap allocation for the read buffer alongside a dynamic struct strbuf to accumulate streamed characters into whole lines. Pre-allocating the strbuf ensures that line processing incurs zero heap reallocations during steady-state polling, while the stack frame of __cmd_ftrace() is reduced by over 96% (down to ~160 bytes). Robust error and EOF detection on stdout and the trace stream ensures polling terminates cleanly upon pipeline or pager exit, while retrying on non-fatal EAGAIN and EINTR conditions prevents premature termination on non-blocking reads or signal interruptions.

---
 tools/perf/builtin-ftrace.c | 78 ++++++++++++++++++++++++++++++-------
 1 file changed, 65 insertions(+), 13 deletions(-)
tools/perf/builtin-ftrace.c +65 -13
index 6017493ff179..6ee72ae90f67 100644
@@ -41,7 +41,10 @@
 #include "util/units.h"
 #include "util/parse-sublevel-options.h"
 
+#include "util/strbuf.h"
+
 #define DEFAULT_TRACER  "function_graph"
+#define TRACE_BUF_SIZE  4096
 
 static volatile sig_atomic_t workload_exec_errno;
 static volatile sig_atomic_t done;
@@ -738,7 +741,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 {
 	char *trace_file;
 	int trace_fd;
-	char buf[4096];
+	char *buf = NULL;
+	struct strbuf linebuf = STRBUF_INIT;
 	struct pollfd pollfd = {
 		.events = POLLIN,
 	};
@@ -767,10 +771,18 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 
 	setup_pager();
 
+	buf = malloc(TRACE_BUF_SIZE);
+	if (!buf) {
+		pr_err("failed to allocate trace buffer\n");
+		goto out_reset;
+	}
+	strbuf_init(&linebuf, 512);
+	strbuf_setlen(&linebuf, 0);
+
 	trace_file = get_tracing_instance_file("trace_pipe");
 	if (!trace_file) {
 		pr_err("failed to open trace_pipe\n");
-		goto out_reset;
+		goto out_free;
 	}
 
 	trace_fd = open(trace_file, O_RDONLY);
@@ -779,7 +791,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 
 	if (trace_fd < 0) {
 		pr_err("failed to open trace_pipe\n");
-		goto out_reset;
+		goto out_free;
 	}
 
 	fcntl(trace_fd, F_SETFL, O_NONBLOCK);
@@ -806,24 +818,42 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	}
 
 	while (!done) {
-		if (poll(&pollfd, 1, -1) < 0)
+		if (poll(&pollfd, 1, -1) < 0) {
+			if (errno == EINTR)
+				continue;
 			break;
+		}
 
 		if (pollfd.revents & POLLIN) {
-			int n = read(trace_fd, buf, sizeof(buf));
-			if (n < 0)
+			int n = read(trace_fd, buf, TRACE_BUF_SIZE);
+
+			if (n < 0) {
+				if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
+					continue;
 				break;
-			if (fwrite(buf, n, 1, stdout) != 1)
+			}
+			if (n == 0)
 				break;
+			for (int i = 0; i < n; i++) {
+				if (buf[i] == '\n') {
+					if (fprintf(stdout, "%s\n", linebuf.buf) < 0)
+						break;
+					strbuf_setlen(&linebuf, 0);
+				} else {
+					if (strbuf_addch(&linebuf, buf[i]) < 0)
+						goto out_close_fd;
+				}
+			}
 			/* flush output since stdout is in full buffering mode due to pager */
-			fflush(stdout);
+			if (fflush(stdout) < 0 || ferror(stdout))
+				break;
 		}
 	}
 
 	write_tracing_file("tracing_on", "0");
 
 	if (workload_exec_errno) {
-		const char *emsg = str_error_r(workload_exec_errno, buf, sizeof(buf));
+		const char *emsg = str_error_r(workload_exec_errno, buf, TRACE_BUF_SIZE);
 		/* flush stdout first so below error msg appears at the end. */
 		fflush(stdout);
 		pr_err("workload failed: %s\n", emsg);
@@ -831,16 +861,38 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	}
 
 	/* read remaining buffer contents */
-	while (true) {
-		int n = read(trace_fd, buf, sizeof(buf));
-		if (n <= 0)
+	while (!ferror(stdout)) {
+		int n = read(trace_fd, buf, TRACE_BUF_SIZE);
+
+		if (n < 0) {
+			if (errno == EINTR)
+				continue;
 			break;
-		if (fwrite(buf, n, 1, stdout) != 1)
+		}
+		if (n == 0)
 			break;
+		for (int i = 0; i < n; i++) {
+			if (buf[i] == '\n') {
+				if (fprintf(stdout, "%s\n", linebuf.buf) < 0)
+					break;
+				strbuf_setlen(&linebuf, 0);
+			} else {
+				if (strbuf_addch(&linebuf, buf[i]) < 0)
+					goto out_close_fd;
+			}
+		}
+	}
+
+	if (!ferror(stdout) && linebuf.len > 0) {
+		fprintf(stdout, "%s\n", linebuf.buf);
+		strbuf_setlen(&linebuf, 0);
 	}
 
 out_close_fd:
 	close(trace_fd);
+out_free:
+	free(buf);
+	strbuf_release(&linebuf);
 out_reset:
 	exit_tracing_instance();
 out:
Signature · 1 line
2.55.0
#4 lore raw

PATCH v5 2/5 perf ftrace: Support filtering return address comments

In reply to Aaron Tomlin

When tracing with the Linux kernel's function graph tracer, enabling funcgraph-retaddr records the return address of each function entry as a comment in the trace stream (e.g. /* <-load_elf_phdrs+0x6c/0xb0 */). While valuable for tracing execution origins, these trailing comments can add considerable visual clutter to the call graph, particularly when examining deeply nested call trees or when synthesising higher-level abstractions.

Introduce support for filtering out return address comments from function graph trace output. Provide a standalone --filter-retaddr flag.

Key aspects of this implementation:

1. Return address parsing and filtering

Implement ftrace_filter_retaddr() in tools/perf/util/ftrace.c to locate return address comments and strip them in-place with memmove(3).

Additionally, introduce ftrace_parse_retaddr() to parse caller symbol names and offsets across diverse output styles (standard format, return values, and bracketed addresses), establishing the parsing foundation for subsequent inlined callchain resolution.

2. Command-line interface

Expose --filter-retaddr in tools/perf/builtin-ftrace.c, updating the documentation in tools/perf/Documentation/perf-ftrace.txt accordingly.

3. Automated verification

Add a dedicated unit test suite ("Ftrace return address processing") in tools/perf/tests/ftrace.c to validate comment parsing and string filtering across all supported formats.

Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/Documentation/perf-ftrace.txt |  4 ++
 tools/perf/builtin-ftrace.c              |  8 +++
 tools/perf/tests/Build                   |  1 +
 tools/perf/tests/builtin-test.c          |  1 +
 tools/perf/tests/ftrace.c                | 80 ++++++++++++++++++++++
 tools/perf/tests/tests.h                 |  1 +
 tools/perf/util/Build                    |  1 +
 tools/perf/util/ftrace.c                 | 86 ++++++++++++++++++++++++
 tools/perf/util/ftrace.h                 |  5 ++
 9 files changed, 187 insertions(+)
 create mode 100644 tools/perf/tests/ftrace.c
 create mode 100644 tools/perf/util/ftrace.c
tools/perf/Documentation/perf-ftrace.txt +4 -0
index 3f3808e513fe..e8105083e67c 100644
@@ -134,6 +134,10 @@ OPTIONS for 'perf ftrace trace'
 	  - depth=<n>    - Set max depth for function graph tracer to follow.
 	  - tail         - Print function name at the end.
 
+--filter-retaddr::
+	Filter out function return address comments (`/* <-caller+offset */`)
+	from function_graph output.
+
 
 OPTIONS for 'perf ftrace latency'
 ---------------------------------
tools/perf/builtin-ftrace.c +8 -0
index 6ee72ae90f67..802e31122770 100644
@@ -836,6 +836,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 				break;
 			for (int i = 0; i < n; i++) {
 				if (buf[i] == '\n') {
+					if (ftrace->filter_retaddr)
+						ftrace_filter_retaddr(linebuf.buf);
 					if (fprintf(stdout, "%s\n", linebuf.buf) < 0)
 						break;
 					strbuf_setlen(&linebuf, 0);
@@ -873,6 +875,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 			break;
 		for (int i = 0; i < n; i++) {
 			if (buf[i] == '\n') {
+				if (ftrace->filter_retaddr)
+					ftrace_filter_retaddr(linebuf.buf);
 				if (fprintf(stdout, "%s\n", linebuf.buf) < 0)
 					break;
 				strbuf_setlen(&linebuf, 0);
@@ -884,6 +888,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	}
 
 	if (!ferror(stdout) && linebuf.len > 0) {
+		if (ftrace->filter_retaddr)
+			ftrace_filter_retaddr(linebuf.buf);
 		fprintf(stdout, "%s\n", linebuf.buf);
 		strbuf_setlen(&linebuf, 0);
 	}
@@ -1849,6 +1855,8 @@ int cmd_ftrace(int argc, const char **argv)
 		     "Size of per cpu buffer, needs to use a B, K, M or G suffix.", parse_buffer_size),
 	OPT_BOOLEAN(0, "inherit", &ftrace.inherit,
 		    "Trace children processes"),
+	OPT_BOOLEAN_SET(0, "filter-retaddr", &ftrace.filter_retaddr, &ftrace.filter_retaddr_set,
+			"Filter out funcgraph return address comments"),
 	OPT_INTEGER('D', "delay", &ftrace.target.initial_delay,
 		    "Number of milliseconds to wait before starting tracing after program start"),
 	OPT_PARENT(common_options),
tools/perf/tests/Build +1 -0
index 395f2866094a..136138932fcb 100644
@@ -72,6 +72,7 @@ perf-test-y += hwmon_pmu.o
 perf-test-y += tool_pmu.o
 perf-test-y += subcmd-help.o
 perf-test-y += kallsyms-split.o
+perf-test-y += ftrace.o
 
 ifeq ($(SRCARCH),$(filter $(SRCARCH),x86 arm arm64 powerpc riscv))
 perf-test-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
tools/perf/tests/builtin-test.c +1 -0
index d2f594921e25..9663f48afb92 100644
@@ -137,6 +137,7 @@ static struct test_suite *generic_tests[] = {
 	&suite__perf_hooks,
 	&suite__unit_number__scnprint,
 	&suite__mem2node,
+	&suite__ftrace,
 	&suite__time_utils,
 	&suite__jit_write_elf,
 	&suite__pfm,
tools/perf/tests/ftrace.c new file +80 -0

80 lines not on this page · raw message

tools/perf/tests/tests.h +1 -0
index 9c96f33483d1..1aec32e094c8 100644
@@ -163,6 +163,7 @@ DECLARE_SUITE(perf_hooks);
 DECLARE_SUITE(unit_number__scnprint);
 DECLARE_SUITE(mem2node);
 DECLARE_SUITE(maps);
+DECLARE_SUITE(ftrace);
 DECLARE_SUITE(time_utils);
 DECLARE_SUITE(jit_write_elf);
 DECLARE_SUITE(api_io);
tools/perf/util/Build +1 -0
index 2c1f880c4c47..5b082d09006d 100644
@@ -169,6 +169,7 @@ perf-util-y += list_sort.o
 perf-util-y += mutex.o
 perf-util-y += sharded_mutex.o
 perf-util-y += intel-tpebs.o
+perf-util-y += ftrace.o
 
 perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_counter.o
 perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_counter_cgroup.o
tools/perf/util/ftrace.c new file +86 -0

86 lines not on this page · raw message

tools/perf/util/ftrace.h +5 -0
index 950f2efafad2..50216d3fef64 100644
@@ -39,6 +39,8 @@ struct perf_ftrace {
 	int			graph_verbose;
 	int			graph_thresh;
 	int			graph_tail;
+	bool			filter_retaddr;
+	bool			filter_retaddr_set;
 };
 
 struct filter_entry {
@@ -93,4 +95,7 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)
 
 #endif  /* HAVE_BPF_SKEL */
 
+bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset);
+void ftrace_filter_retaddr(char *str);
+
 #endif  /* __PERF_FTRACE_H__ */
Signature · 1 line
2.55.0
#6 lore raw

PATCH v5 3/5 perf ftrace: Support display of inlined functions in function graph tracer

In reply to Aaron Tomlin

The Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining.

Address this by introducing the --inline option to perf ftrace for the function_graph tracer.

When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment (e.g. /* <-irq_work_queue_on+0x81/0xa0 */). By interrogating DWARF debug information from the kernel image (vmlinux) utilising libdw, perf ftrace resolves this return address to its inlined callchain and synthesises the intermediate inlined frames directly into the streamed call graph.

Key aspects of this implementation:

1. Inlined frame presentation

Synthesised inlined functions are rendered with an explicit /* (inline) */ annotation at both entry and exit:

# CPU DURATION FUNCTION CALLS
# | | | | | | |
0) | wake_up_new_task() {
0) | activate_task() { /* (inline) */
0) 1.234 us | enqueue_task();
0) | } /* activate_task (inline) */
0) 2.567 us | }

This affords immediate visual clarity to the reader upon entering an inlined scope, without necessitating a search for the closing delimiter.

2. Call graph reconstruction and dynamic stack tracking

Maintains a per-CPU stack tracking real and inlined frames, synthesising entry and exit delimiters to accurately reconstruct the execution hierarchy. Inlined frames are structured strictly root-to-leaf and validated against the active stack to prevent un-traced callers from attaching orphan inlines. Bare closing braces without trailing function names (the default when funcgraph-tail is disabled) are correctly matched against preceding frames on the stack. Context column alignment, including task comm-pid delimiters in verbose mode, is preserved when generating prefixes for synthesised inline frames. Furthermore, stack depth saturation guards ensure that inlined frame depth tracking remains clamped and indentation cannot drift even under deep call trees exceeding FTRACE_MAX_STACK.

3. Option handling and diagnostics

Adds -k or --vmlinux to designate an explicit debug image, and provides clear diagnostic guidance should --inline be invoked on a kernel lacking CONFIG_FUNCTION_GRAPH_RETADDR. Automatically enables return address comment filtering unless explicitly requested.

4. Automated verification

Expands the unit test suite in tools/perf/tests/ftrace.c to validate DWARF inline symbol resolution, comment filtering toggles, bare closing braces handling, and stack saturation recovery.

Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/Documentation/perf-ftrace.txt |  17 +-
 tools/perf/builtin-ftrace.c              |  58 +-
 tools/perf/tests/ftrace.c                | 380 ++++++++++++-
 tools/perf/util/ftrace.c                 | 649 ++++++++++++++++++++++-
 tools/perf/util/ftrace.h                 |  11 +
 5 files changed, 1097 insertions(+), 18 deletions(-)
tools/perf/Documentation/perf-ftrace.txt +16 -1
index e8105083e67c..445c8ccc22d2 100644
@@ -134,9 +134,24 @@ OPTIONS for 'perf ftrace trace'
 	  - depth=<n>    - Set max depth for function graph tracer to follow.
 	  - tail         - Print function name at the end.
 
+--inline::
+	Show inlined functions in function_graph tracer. This requires
+	kernel support for the funcgraph-retaddr option and a vmlinux with
+	debug symbols. Inlined functions are displayed in the call graph
+	hierarchy, and their inlined status is indicated with `/* (inline) */`
+	on entry and exit, e.g. `func() { /* (inline) */` and `} /* func (inline) */`.
+	By default, raw return address comments (`/* <-caller+offset */`)
+	are hidden in the output; specify `--graph-opts retaddr` to show them.
+
 --filter-retaddr::
 	Filter out function return address comments (`/* <-caller+offset */`)
-	from function_graph output.
+	from function_graph output. This is enabled by default when `--inline`
+	is used; use `--no-filter-retaddr` to disable it.
+
+-k::
+--vmlinux=::
+	Path to the vmlinux file containing debug symbols for resolving
+	inlined functions.
 
 
 OPTIONS for 'perf ftrace latency'
tools/perf/builtin-ftrace.c +45 -13

107 lines not on this page · raw message

tools/perf/tests/ftrace.c +379 -1

398 lines not on this page · raw message

tools/perf/util/ftrace.c +646 -3

662 lines not on this page · raw message

tools/perf/util/ftrace.h +11 -0
index 50216d3fef64..4140902d9de6 100644
@@ -41,6 +41,8 @@ struct perf_ftrace {
 	int			graph_tail;
 	bool			filter_retaddr;
 	bool			filter_retaddr_set;
+	bool			use_inline;
+	struct machine		*machine;
 };
 
 struct filter_entry {
@@ -95,7 +97,16 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)
 
 #endif  /* HAVE_BPF_SKEL */
 
+struct machine;
+
 bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset);
 void ftrace_filter_retaddr(char *str);
+int ftrace_resolve_inlines(struct machine *machine,
+			   const char *sym_name, u64 offset,
+			   const char **inlined_names, int max_inlines);
+int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out);
+void ftrace_cpu_states__clear(void);
+int perf_ftrace__setup_inlines(struct perf_ftrace *ftrace);
+void perf_ftrace__cleanup_inlines(struct perf_ftrace *ftrace);
 
 #endif  /* __PERF_FTRACE_H__ */
Signature · 1 line
2.55.0
#9 lore raw

PATCH v5 4/5 perf ftrace: Support omitting execution duration in function graph tracer

In reply to Aaron Tomlin

The Linux kernel's function graph tracer displays the execution duration of completed functions by default via the funcgraph-duration trace option. Whilst informative, in scenarios where the user is solely interested in the call graph topology, call flow sequence, or inlined hierarchy, the duration column can add visual clutter and widen each trace line.

Introduce support for omitting function duration by exposing the noduration sub-option under --graph-opts. Consistent with the existing nosleep-time and noirqs sub-options, it accepts an optional value, so noduration=0 explicitly retains the duration column; when specified more than once, the last occurrence takes effect. When enabled, perf ftrace configures the kernel's tracing instance option funcgraph-duration to 0, and restores it to 1 upon resetting tracing options.

Because the profile subcommand fundamentally requires function execution duration to compute timing statistics and call counts, combining noduration with perf ftrace profile is rejected with an explicit error.

---
 tools/perf/Documentation/perf-ftrace.txt |  1 +
 tools/perf/builtin-ftrace.c              | 34 ++++++++++++-
 tools/perf/tests/ftrace.c                | 64 ++++++++++++++++++++++--
 tools/perf/util/ftrace.c                 | 18 +++++--
 tools/perf/util/ftrace.h                 |  1 +
 5 files changed, 109 insertions(+), 9 deletions(-)
tools/perf/Documentation/perf-ftrace.txt +1 -0
index 445c8ccc22d2..a50f3bae86b2 100644
@@ -129,6 +129,7 @@ OPTIONS for 'perf ftrace trace'
 	  - retaddr      - Show function return address.
 	  - nosleep-time - Measure on-CPU time only for function_graph tracer.
 	  - noirqs       - Ignore functions that happen inside interrupt.
+	  - noduration   - Do not print function duration.
 	  - verbose      - Show process names, PIDs, timestamps, etc.
 	  - thresh=<n>   - Setup trace duration threshold in microseconds.
 	  - depth=<n>    - Set max depth for function graph tracer to follow.
tools/perf/builtin-ftrace.c +33 -1
index 018f7e04b739..fb835546e995 100644
@@ -299,6 +299,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("func_stack_trace", "0");
 	write_tracing_option_file("sleep-time", "1");
 	write_tracing_option_file("funcgraph-irqs", "1");
+	write_tracing_option_file("funcgraph-duration", "1");
 	write_tracing_option_file("funcgraph-proc", "0");
 	write_tracing_option_file("funcgraph-abstime", "0");
 	write_tracing_option_file("funcgraph-tail", "0");
@@ -598,6 +599,26 @@ static int set_tracing_funcgraph_irqs(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+/**
+ * set_tracing_funcgraph_duration - Set funcgraph-duration tracing option
+ * @ftrace: Perf ftrace context
+ *
+ * Configure the kernel's funcgraph-duration tracing option based on the
+ * ftrace->graph_noduration flag.
+ *
+ * Return: 0 on success, or a negative error code on failure.
+ */
+static int set_tracing_funcgraph_duration(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->graph_noduration)
+		return 0;
+
+	if (write_tracing_option_file("funcgraph-duration", "0") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int set_tracing_funcgraph_verbose(struct perf_ftrace *ftrace)
 {
 	if (!ftrace->graph_verbose)
@@ -707,6 +728,11 @@ static int set_tracing_options(struct perf_ftrace *ftrace)
 		return -1;
 	}
 
+	if (set_tracing_funcgraph_duration(ftrace) < 0) {
+		pr_err("failed to set tracing option funcgraph-duration\n");
+		return -1;
+	}
+
 	if (set_tracing_funcgraph_verbose(ftrace) < 0) {
 		pr_err("failed to set tracing option funcgraph-proc/funcgraph-abstime\n");
 		return -1;
@@ -1773,6 +1799,7 @@ static int parse_graph_tracer_opts(const struct option *opt,
 		{ .name = "retaddr",		.value_ptr = &ftrace->graph_retaddr },
 		{ .name = "nosleep-time",	.value_ptr = &ftrace->graph_nosleep_time },
 		{ .name = "noirqs",		.value_ptr = &ftrace->graph_noirqs },
+		{ .name = "noduration",		.value_ptr = &ftrace->graph_noduration },
 		{ .name = "verbose",		.value_ptr = &ftrace->graph_verbose },
 		{ .name = "thresh",		.value_ptr = &ftrace->graph_thresh },
 		{ .name = "depth",		.value_ptr = &ftrace->graph_depth },
@@ -1862,7 +1889,7 @@ int cmd_ftrace(int argc, const char **argv)
 	OPT_CALLBACK('g', "nograph-funcs", &ftrace.nograph_funcs, "func",
 		     "Set nograph filter on given functions", parse_filter_func),
 	OPT_CALLBACK(0, "graph-opts", &ftrace, "options",
-		     "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,verbose,thresh=<n>,depth=<n>",
+		     "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,noduration,verbose,thresh=<n>,depth=<n>,tail",
 		     parse_graph_tracer_opts),
 	OPT_CALLBACK('m', "buffer-size", &ftrace.percpu_buffer_size, "size",
 		     "Size of per cpu buffer, needs to use a B, K, M or G suffix.", parse_buffer_size),
@@ -2069,6 +2096,11 @@ int cmd_ftrace(int argc, const char **argv)
 		cmd_func = __cmd_latency;
 		break;
 	case PERF_FTRACE_PROFILE:
+		if (ftrace.graph_noduration) {
+			pr_err("Error: 'noduration' option cannot be used with profile\n");
+			ret = -EINVAL;
+			goto out_delete_filters;
+		}
 		cmd_func = __cmd_profile;
 		break;
 	case PERF_FTRACE_NONE:
tools/perf/tests/ftrace.c +60 -4

99 lines not on this page · raw message

tools/perf/util/ftrace.c +14 -4
index 09b2fd30a01b..71f00d393349 100644
@@ -295,6 +295,9 @@ static void make_blank_prefix(const struct perf_ftrace *ftrace, const char *line
 	memcpy(buf, line, len);
 	buf[len] = '\0';
 
+	if (ftrace->graph_noduration)
+		return;
+
 	if (*pipe_char == '|') {
 		if (ftrace->graph_verbose) {
 			/* In verbose mode, duration starts after the comm-pid '|' */
@@ -333,10 +336,12 @@ static void make_blank_prefix(const struct perf_ftrace *ftrace, const char *line
  *   [ abstime |]  CPU)  [ comm-pid |]  [ duration |]  func
  *
  * The abstime and comm-pid columns are only present with --graph-opts
- * verbose.
+ * verbose, whereas the duration column, and thus its '|' delimiter, is
+ * omitted with --graph-opts noduration.
  *
  * Return: A pointer to the last character of the context columns (the final
- * '|' delimiter). Returns NULL if @line is not a trace record.
+ * '|' delimiter, or the CPU ')' when no delimiter follows it). Returns NULL
+ * if @line is not a trace record (e.g. a header or a context switch marker).
  */
 static const char *find_func_column(const struct perf_ftrace *ftrace,
 				    const char *line, int *cpu)
@@ -364,9 +369,14 @@ static const char *find_func_column(const struct perf_ftrace *ftrace,
 			return NULL;
 	}
 
-	p = strchr(p + 1, '|');
-	if (!p)
+	if (!ftrace->graph_noduration) {
+		p = strchr(p + 1, '|');
+		if (!p)
+			return NULL;
+	} else if (strstr(p, " => ")) {
+		/* Context switch marker: " CPU) comm-pid => comm-pid" */
 		return NULL;
+	}
 
 	*cpu = (int)val;
 	return p;
tools/perf/util/ftrace.h +1 -0
index 4140902d9de6..d202a09d112f 100644
@@ -36,6 +36,7 @@ struct perf_ftrace {
 	int			graph_retaddr;
 	int			graph_nosleep_time;
 	int			graph_noirqs;
+	int			graph_noduration;
 	int			graph_verbose;
 	int			graph_thresh;
 	int			graph_tail;
Signature · 1 line
2.55.0
#11 lore raw

PATCH v5 5/5 perf ftrace: Filter extraneous functions in function graph tracer

In reply to Aaron Tomlin

When tracing with the function graph tracer using a targeted filter (-G or --graph-funcs), users occasionally observe unrelated functions in the trace output:

1. Top-level functions (e.g. most notably finish_task_switch.isra.0()) appearing before or alongside the requested target function. This occurs because clearing and rewriting set_graph_function creates a transient window where remote CPUs still observe an empty filter (tracing all functions) before RCU synchronisation completes, coupled with perf ftrace sleeping in poll() on trace_pipe right as tracing starts.

2. Detached/orphan calls inside the target callgraph (e.g. housekeeping_any_cpu() and arch_irq_work_raise()) appearing as child frames under the target function. These routines are invoked by rb_wakeups() to notify perf's poll() on trace_pipe whenever an event is committed, and execute while the target function is on the CPU.

Resolve both issues deterministically in userspace:

1. Track active target callgraph state per CPU in tools/perf/util/ftrace.c. Any entry not matching the requested -G targets is suppressed along with its nested children until its scope exits. When a target function is matched (at root or nested depths), record its root depth; for leaf target calls, immediately reset the filter state upon completion to ensure subsequent sibling functions remain suppressed.

2. Within an active target callgraph, validate that child entries descend from an active frame on the target stack using their return addresses (CONFIG_FUNCTION_GRAPH_RETADDR). Any detached calls originating outside the target hierarchy (such as ring-buffer wakeups) are suppressed along with their subtrees.

For example:

# tracer: function_graph
#
# CPU FUNCTION CALLS
# | | | | |
3) activate_task() {
3) enqueue_task() {
3) psi_enqueue() { /* (inline) */
3) arch_static_branch_jump() { /* (inline) */
3) enqueue_task_fair() {
3) enqueue_hierarchy() { /* (inline) */
3) update_curr();
3) enqueue_entity() {
3) account_entity_enqueue() { /* (inline) */
3) account_llc_enqueue();
3) } /* account_entity_enqueue (inline) */
3) }
3) } /* enqueue_hierarchy (inline) */
3) place_entity() {
3) avg_vruntime();
3) }
3) __enqueue_entity();
3) dl_server_start();
3) }
3) } /* arch_static_branch_jump (inline) */
3) } /* psi_enqueue (inline) */
3) sched_info_enqueue() { /* (inline) */
3) psi_task_change() {
3) psi_write_begin() { /* (inline) */
3) psi_flags_change();
3) } /* psi_write_begin (inline) */
3) psi_group_change() {
3) record_times();
3) }
3) }
3) } /* sched_info_enqueue (inline) */
3) }
3) }

---
 tools/perf/tests/ftrace.c | 323 ++++++++++++++++++++++++++++++++++++++
 tools/perf/util/ftrace.c  | 215 +++++++++++++++++++++++--
 2 files changed, 526 insertions(+), 12 deletions(-)
tools/perf/tests/ftrace.c +323 -0
index a1262c92ad9f..5954706dafe6 100644
@@ -484,6 +484,325 @@ static int test_column_layouts(void)
 	return TEST_OK;
 }
 
+static int test_graph_target_filter(void)
+{
+	static const char line_idle[] = " 7)   3.000 us    |  sched_idle_set_state();";
+	static const char line_sw_entry[] = " 6)               |  finish_task_switch.isra.0() {";
+	static const char line_sw_child[] = " 6)   1.000 us    |    _raw_spin_unlock();";
+	static const char line_sw_exit[] = " 6)   2.000 us    |  }";
+	static const char line_tgt_entry[] = " 2)               |  activate_task() {";
+	static const char line_tgt_child[] = " 2)   1.500 us    |    enqueue_task();";
+	static const char line_tgt_exit[] = " 2)   2.500 us    |  }";
+	struct perf_ftrace ftrace;
+	struct filter_entry *entry;
+	struct filter_entry *pos, *tmp;
+	char *out_buf = NULL;
+	size_t out_len = 0;
+	FILE *out;
+	int ret = TEST_OK;
+
+	memset(&ftrace, 0, sizeof(ftrace));
+	INIT_LIST_HEAD(&ftrace.graph_funcs);
+
+	entry = malloc(sizeof(*entry) + strlen("activate_task") + 1);
+	if (!entry)
+		return TEST_FAIL;
+	strcpy(entry->name, "activate_task");
+	list_add_tail(&entry->list, &ftrace.graph_funcs);
+
+	/* 1. Test without --inline */
+	ftrace.use_inline = false;
+	out = open_memstream(&out_buf, &out_len);
+	if (!out) {
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	ftrace_process_fgraph_line(&ftrace, line_idle, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_entry, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_child, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_exit, out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_entry, out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_child, out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_exit, out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	if (strstr(out_buf, "sched_idle_set_state")) {
+		pr_debug("FAILED: sched_idle_set_state not suppressed\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (strstr(out_buf, "finish_task_switch")) {
+		pr_debug("FAILED: finish_task_switch not suppressed\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (strstr(out_buf, "_raw_spin_unlock")) {
+		pr_debug("FAILED: _raw_spin_unlock not suppressed\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (!strstr(out_buf, "activate_task() {")) {
+		pr_debug("FAILED: activate_task entry missing\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (!strstr(out_buf, "enqueue_task();")) {
+		pr_debug("FAILED: enqueue_task child missing\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	free(out_buf);
+	out_buf = NULL;
+
+	/* 2. Test with --inline */
+	ftrace.use_inline = true;
+	out = open_memstream(&out_buf, &out_len);
+	if (!out) {
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	ftrace_process_fgraph_line(&ftrace, line_idle, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_entry, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_child, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_exit, out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_entry, out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_child, out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_exit, out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	pr_debug("Target filter output with --inline:\n%s\n", out_buf);
+
+	if (strstr(out_buf, "finish_task_switch")) {
+		pr_debug("FAILED: finish_task_switch not suppressed with --inline\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (!strstr(out_buf, "activate_task() {")) {
+		pr_debug("FAILED: activate_task entry missing with --inline\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	free(out_buf);
+	out_buf = NULL;
+
+	/* 3. Test suppressing detached orphan calls within target graph */
+	ftrace.use_inline = true;
+	ftrace.filter_retaddr = 1;
+	out = open_memstream(&out_buf, &out_len);
+	if (!out) {
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	ftrace_process_fgraph_line(&ftrace, line_tgt_entry, out);
+	/* Leaf orphan call from ring_buffer_unlock_commit */
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)               |    housekeeping_any_cpu(); /* <-ring_buffer_unlock_commit+0x1a0/0x1f0 */",
+		out);
+	/* Non-leaf orphan call from irq_work_queue_on */
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)               |    arch_irq_work_raise() { /* <-irq_work_queue_on+0x81/0xa0 */",
+		out);
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)               |      apic_wait_icr_idle() { /* (inline) */",
+		out);
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)               |        x2apic_send_IPI_self();",
+		out);
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)               |      } /* apic_wait_icr_idle (inline) */",
+		out);
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)   1.000 us    |    }",
+		out);
+	/* Genuine child call from activate_task */
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)               |    enqueue_task() { /* <-activate_task+0x1d1/0x3e0 */",
+		out);
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)   1.500 us    |    }",
+		out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_exit, out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	pr_debug("Orphan call filter output:\n%s\n", out_buf);
+
+	if (strstr(out_buf, "housekeeping_any_cpu")) {
+		pr_debug("FAILED: housekeeping_any_cpu orphan not suppressed\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (strstr(out_buf, "arch_irq_work_raise")) {
+		pr_debug("FAILED: arch_irq_work_raise orphan not suppressed\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (strstr(out_buf, "x2apic_send_IPI_self")) {
+		pr_debug("FAILED: x2apic_send_IPI_self not suppressed\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (!strstr(out_buf, "enqueue_task() {")) {
+		pr_debug("FAILED: genuine enqueue_task missing\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	free(out_buf);
+	out_buf = NULL;
+
+	/* 4. Test targeted leaf call at depth > 0 resetting filter */
+	ftrace.use_inline = false;
+	list_for_each_entry_safe(pos, tmp, &ftrace.graph_funcs, list) {
+		list_del(&pos->list);
+		free(pos);
+	}
+	entry = malloc(sizeof(*entry) + strlen("enqueue_task") + 1);
+	if (!entry) {
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	strcpy(entry->name, "enqueue_task");
+	list_add_tail(&entry->list, &ftrace.graph_funcs);
+
+	out = open_memstream(&out_buf, &out_len);
+	if (!out) {
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	ftrace_process_fgraph_line(&ftrace, line_sw_entry, out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_child, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_child, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_exit, out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	if (strstr(out_buf, "finish_task_switch")) {
+		pr_debug("FAILED: finish_task_switch not suppressed\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (!strstr(out_buf, "enqueue_task();")) {
+		pr_debug("FAILED: targeted leaf enqueue_task missing\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (strstr(out_buf, "_raw_spin_unlock")) {
+		pr_debug("FAILED: sibling _raw_spin_unlock not suppressed after leaf target\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	free(out_buf);
+	out_buf = NULL;
+
+	/* 5. Test targeted leaf call at depth > 0 with inlines resetting filter */
+	ftrace.use_inline = true;
+	out = open_memstream(&out_buf, &out_len);
+	if (!out) {
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	ftrace_process_fgraph_line(&ftrace, line_sw_entry, out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_child, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_child, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_exit, out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	if (strstr(out_buf, "finish_task_switch")) {
+		pr_debug("FAILED: finish_task_switch not suppressed with inlines\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (!strstr(out_buf, "enqueue_task();")) {
+		pr_debug("FAILED: targeted leaf enqueue_task missing with inlines\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (strstr(out_buf, "_raw_spin_unlock")) {
+		pr_debug("FAILED: sibling _raw_spin_unlock leaked with inlines\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	free(out_buf);
+	out_buf = NULL;
+
+	/* 6. Test targeted leaf call with synthesized DWARF inlines closing and resetting filter */
+	if (access("vmlinux", R_OK) == 0) {
+		symbol_conf.vmlinux_name = "vmlinux";
+		symbol_conf.ignore_vmlinux_buildid = true;
+		if (perf_ftrace__setup_inlines(&ftrace) == 0 && ftrace.machine) {
+			out = open_memstream(&out_buf, &out_len);
+			if (!out) {
+				perf_ftrace__cleanup_inlines(&ftrace);
+				ret = TEST_FAIL;
+				goto out_free_filter;
+			}
+
+			ftrace_process_fgraph_line(&ftrace,
+				" 0)               |  wake_up_new_task() {", out);
+			/* Targeted leaf with caller having inline activate_task */
+			ftrace_process_fgraph_line(&ftrace,
+				" 0)   1.500 us    |    enqueue_task(); /* <-wake_up_new_task+0x1d1/0x3e0 */",
+				out);
+			/* Sibling call after targeted leaf */
+			ftrace_process_fgraph_line(&ftrace,
+				" 0)   1.000 us    |    check_preempt_curr(); /* <-wake_up_new_task+0x0/0x3e0 */",
+				out);
+			ftrace_process_fgraph_line(&ftrace,
+				" 0)   2.500 us    |  } /* wake_up_new_task */", out);
+
+			fclose(out);
+			ftrace_cpu_states__clear();
+			perf_ftrace__cleanup_inlines(&ftrace);
+
+			if (!strstr(out_buf, "enqueue_task();")) {
+				pr_debug("FAILED: targeted leaf enqueue_task missing\n");
+				ret = TEST_FAIL;
+				goto out_free_filter;
+			}
+			if (!strstr(out_buf, "} /* activate_task (inline) */")) {
+				pr_debug("FAILED: inline activate_task not closed after leaf target\n");
+				ret = TEST_FAIL;
+				goto out_free_filter;
+			}
+			if (strstr(out_buf, "check_preempt_curr")) {
+				pr_debug("FAILED: sibling check_preempt_curr leaked after leaf target\n");
+				ret = TEST_FAIL;
+				goto out_free_filter;
+			}
+
+			free(out_buf);
+			out_buf = NULL;
+		}
+	}
+
+out_free_filter:
+	if (out_buf)
+		free(out_buf);
+	list_for_each_entry_safe(pos, tmp, &ftrace.graph_funcs, list) {
+		list_del(&pos->list);
+		free(pos);
+	}
+	return ret;
+}
+
 static int test__ftrace(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
 {
 	int ret;
@@ -504,6 +823,10 @@ static int test__ftrace(struct test_suite *test __maybe_unused, int subtest __ma
 	if (ret != TEST_OK)
 		return ret;
 
+	ret = test_graph_target_filter();
+	if (ret != TEST_OK)
+		return ret;
+
 	ret = test_inline_processing();
 	if (ret != TEST_OK)
 		return ret;
tools/perf/util/ftrace.c +203 -12

278 lines not on this page · raw message

Signature · 1 line
2.55.0
#10 lore raw

Re: [PATCH perf-tools-next v5 4/5] perf ftrace: Support omitting execution duration in function graph tracer

In reply to Aaron Tomlin
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 16 lines

The Linux kernel's function graph tracer displays the execution duration of completed functions by default via the funcgraph-duration trace option. Whilst informative, in scenarios where the user is solely interested in the call graph topology, call flow sequence, or inlined hierarchy, the duration column can add visual clutter and widen each trace line.

Introduce support for omitting function duration by exposing the noduration sub-option under --graph-opts. Consistent with the existing nosleep-time and noirqs sub-options, it accepts an optional value, so noduration=0 explicitly retains the duration column; when specified more than once, the last occurrence takes effect. When enabled, perf ftrace configures the kernel's tracing instance option funcgraph-duration to 0, and restores it to 1 upon resetting tracing options.

Quoted 5 lines

Because the profile subcommand fundamentally requires function execution duration to compute timing statistics and call counts, combining noduration with perf ftrace profile is rejected with an explicit error.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
#2 lore raw

Re: [PATCH perf-tools-next v5 1/5] perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming

In reply to Aaron Tomlin
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] perf: missing explicit inclusion of <stdio.h> for POSIX stdio streams --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] perf: missing explicit inclusion of <stdio.h> for POSIX stdio streams --

commit 84dafb5e4aabd0b60e918d9f85e26ec8e27d5c88
Author: Aaron Tomlin <[email protected]>

perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming

This commit optimizes __cmd_ftrace() by replacing a fixed-size stack buffer with heap allocation and a dynamic strbuf for trace data accumulation. It also introduces error checking and stream flushing to improve robustness during steady-state polling.

tools/perf/builtin-ftrace.c line 45

Quoted 10 lines
tools/perf/builtin-ftrace.c +2 -0
index 6017493ff1797..6ee72ae90f67a 100644
@@ -41,7 +41,10 @@
 #include "util/units.h"
 #include "util/parse-sublevel-options.h"
 
+#include "util/strbuf.h"
+

[Severity: Low]
Does this file explicitly include <stdio.h>?

This isn't a runtime bug, but relying on implicit inclusion via other headers for POSIX stdio streams and functions (like stdout, fprintf, fflush, and ferror) violates the perf subsystem guidelines for musl compatibility.

Explicitly including <stdio.h> ensures the code doesn't cause build breakages on libc implementations that strictly separate declarations, like musl.

[ ... ]

Quoted 40 lines
tools/perf/builtin-ftrace.c +23 -5
@@ -806,24 +818,42 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	}
 
 	while (!done) {
-		if (poll(&pollfd, 1, -1) < 0)
+		if (poll(&pollfd, 1, -1) < 0) {
+			if (errno == EINTR)
+				continue;
 			break;
+		}
 
 		if (pollfd.revents & POLLIN) {
-			int n = read(trace_fd, buf, sizeof(buf));
-			if (n < 0)
+			int n = read(trace_fd, buf, TRACE_BUF_SIZE);
+
+			if (n < 0) {
+				if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
+					continue;
 				break;
-			if (fwrite(buf, n, 1, stdout) != 1)
+			}
+			if (n == 0)
 				break;
+			for (int i = 0; i < n; i++) {
+				if (buf[i] == '\n') {
+					if (fprintf(stdout, "%s\n", linebuf.buf) < 0)
+						break;
+					strbuf_setlen(&linebuf, 0);
+				} else {
+					if (strbuf_addch(&linebuf, buf[i]) < 0)
+						goto out_close_fd;
+				}
+			}
 			/* flush output since stdout is in full buffering mode due to pager */
-			fflush(stdout);
+			if (fflush(stdout) < 0 || ferror(stdout))
+				break;
 		}
 	}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#5 lore raw

Re: [PATCH perf-tools-next v5 2/5] perf ftrace: Support filtering return address comments

In reply to Aaron Tomlin
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 14 lines

When tracing with the Linux kernel's function graph tracer, enabling funcgraph-retaddr records the return address of each function entry as a comment in the trace stream (e.g. /* <-load_elf_phdrs+0x6c/0xb0 */). While valuable for tracing execution origins, these trailing comments can add considerable visual clutter to the call graph, particularly when examining deeply nested call trees or when synthesising higher-level abstractions.

Introduce support for filtering out return address comments from function graph trace output. Provide a standalone --filter-retaddr flag.

Key aspects of this implementation:

Quoted 7 lines

1. Return address parsing and filtering
[ ... ]
processing") in tools/perf/tests/ftrace.c to validate comment parsing and string filtering across all supported formats.

Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#7 lore raw

Re: [PATCH perf-tools-next v5 3/5] perf ftrace: Support display of inlined functions in function graph tracer

In reply to Aaron Tomlin
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] perf ftrace: implicit declaration of zalloc due to missing <linux/zalloc.h> - [Low] perf tests…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] perf ftrace: implicit declaration of zalloc due to missing <linux/zalloc.h>
- [Low] perf tests: memory leak of out_buf on test assertion failure
--

commit e3c3c2f995806f10402b965ab958305c3e8ca004
Author: Aaron Tomlin <[email protected]>

perf ftrace: Support display of inlined functions in function graph tracer

This patch introduces the --inline option to perf ftrace for the function_graph tracer. It resolves return addresses to their inlined callchains using DWARF debug information and synthesises the intermediate inlined frames directly into the streamed call graph.

Quoted 4 lines
tools/perf/tests/ftrace.c +0 -0
index 596cecee844e3..3d0f5748252b5 100644

[ ... ]

Quoted 7 lines
tools/perf/tests/ftrace.c +7 -0
+static int test_stack_saturation(void)
+{
+	struct perf_ftrace ftrace;
+	char *out_buf = NULL;
+	size_t out_len = 0;
+	FILE *out;
+	int i;

[ ... ]

tools/perf/tests/ftrace.c static int test_filter_retaddr(void) · line 377

Quoted 7 lines
tools/perf/tests/ftrace.c +7 -0
+	/* Verify base_func starts at standard 2 spaces indentation without drift */
+	TEST_ASSERT_VAL("no indentation drift after stack saturation",
+			strstr(out_buf, "|  base_func() {") != NULL);
+
+	free(out_buf);
+	return TEST_OK;
+}

[Severity: Low]
Does this code leak out_buf if the assertion fails?

The TEST_ASSERT_VAL macro appears to return early upon failure, which would cause the execution to bypass the free(out_buf) call at the end of the function.

Quoted 4 lines
tools/perf/util/ftrace.c +0 -0
index aaf108369c365..09b2fd30a01b7 100644

[ ... ]

tools/perf/util/ftrace.c line 76

Quoted 7 lines
tools/perf/util/ftrace.c +7 -0
+static struct ftrace_cpu_state *get_cpu_state(int cpu)
+{
+	if (cpu < 0 || cpu >= FTRACE_MAX_CPUS)
+		return NULL;
+
+	if (!ftrace_cpu_states[cpu])
+		ftrace_cpu_states[cpu] = zalloc(sizeof(struct ftrace_cpu_state));

[Severity: Medium]
Will this cause a build failure or pointer truncation due to a missing declaration for zalloc()?

Since the appropriate header providing the prototype for zalloc() does not seem to be explicitly included, the compiler might implicitly assume it returns an int.

On 64-bit architectures, this could truncate the returned pointer. Alternatively, modern compilers configured to warn about implicit function declarations may fail the build entirely.

Quoted 3 lines
tools/perf/util/ftrace.c +3 -0
+
+	return ftrace_cpu_states[cpu];
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#12 lore raw

Re: [PATCH perf-tools-next v5 5/5] perf ftrace: Filter extraneous functions in function graph tracer

In reply to Aaron Tomlin
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Medium] perf ftrace: inaccurate target_root_depth capture causes premature filter termination - [Medium]…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Medium] perf ftrace: inaccurate target_root_depth capture causes premature filter termination
- [Medium] perf ftrace: deep target subtrees suppressed as orphans on stack overflow
- [Medium] perf ftrace: permanent stack desynchronization on dropped orphan exit --

commit bc1ceece5244de5f3fd5ae63daec2de2ddd1eee0
Author: Aaron Tomlin <[email protected]>

perf ftrace: Filter extraneous functions in function graph tracer

When tracing with the function graph tracer using a targeted filter (-G or --graph-funcs), users occasionally observe unrelated functions in the trace output:

[ ... ]

Quoted 4 lines
tools/perf/util/ftrace.c +0 -0
index 71f00d393349e..90d0e3c9f8b5b 100644

[ ... ]

tools/perf/util/ftrace.c int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL · line 520

Quoted 8 lines
tools/perf/util/ftrace.c +4 -0
@@ -476,12 +514,23 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 	if (is_exit) {
 		char exit_name[128];
 
+		if (cs->orphan_depth > 0) {
+			cs->orphan_depth--;
+			return 0;
+		}

[Severity: Medium]
If an orphan call executes and increments cs->orphan_depth, but its corresponding exit event is dropped by the kernel ring buffer, cs->orphan_depth will remain > 0.

Will the next valid exit event (belonging to the target callgraph) hit this block and be blindly consumed without popping its frame from cs->stack?

Could this lead to permanent desynchronization of the tracked call stack?

[ ... ]

Quoted 8 lines
tools/perf/util/ftrace.c +4 -0
@@ -539,6 +601,98 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 	has_open_brace = (strchr(func_start, '{') != NULL);
 	has_semicolon = (strchr(func_start, ';') != NULL);
 
+	if (cs->orphan_depth > 0) {
+		if (has_open_brace && !has_semicolon)
+			cs->orphan_depth++;
+		return 0;

tools/perf/util/ftrace.c int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL · line 617

Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+	}
+
+	if (has_filter) {
+		if (cs->depth == 0 || !cs->in_target) {
+			char entry_name[128];
+
+			extract_entry_func_name(func_start, entry_name, sizeof(entry_name));
+			if (ftrace_is_graph_func(ftrace, entry_name)) {
+				cs->in_target = true;
+				cs->target_root_depth = cs->depth;

[Severity: Medium]
Since cs->target_root_depth is captured here as the current cs->depth, what happens if the --inline option is used?

Can subsequent inline resolution logic pop stale inline frames from previous sibling calls, decrementing cs->depth before the target root itself is pushed?

If so, could the target root end up being pushed at a depth lower than the captured target_root_depth, causing the target filter to prematurely terminate when a child of the target root exits (since cs->depth <= cs->target_root_depth would evaluate to true)?

Quoted 7 lines
tools/perf/util/ftrace.c +7 -0
+				cs->orphan_depth = 0;
+				is_target_root = true;
+			} else if (cs->depth == 0) {
+				cs->in_target = false;
+				cs->orphan_depth = 0;
+			}
+		}

[ ... ]

Quoted 7 lines
tools/perf/util/ftrace.c +7 -0
+	if (!ftrace->use_inline) {
+		if (has_filter && cs->in_target && cs->depth > cs->target_root_depth) {
+			if (ftrace_parse_retaddr(func_start, caller_sym,
+						 sizeof(caller_sym), &caller_offset)) {
+				caller_idx = -1;
+				for (i = cs->depth - 1; i >= 0; i--) {
+					if (!cs->stack[i].inlined && cs->stack[i].name &&

tools/perf/util/ftrace.c int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL · line 665

Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+					    strcmp(cs->stack[i].name, caller_sym) == 0) {
+						caller_idx = i;
+						break;
+					}
+				}
+				if (caller_idx < cs->target_root_depth) {
+					if (has_open_brace && !has_semicolon)
+						cs->orphan_depth = 1;
+					return 0;
+				}

[Severity: Medium]
If the call stack depth exceeds FTRACE_MAX_STACK, cs->overflow_depth is incremented instead of pushing new frames to cs->stack.

When a subsequent legitimate child call is processed, its caller won't be found in the stack, leaving caller_idx as -1.

Will this cause caller_idx < cs->target_root_depth to evaluate to true, incorrectly flagging the valid child as an orphan and suppressing it and all its subtrees?

[ ... ]

Quoted 8 lines
tools/perf/util/ftrace.c +4 -0
@@ -565,6 +719,21 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 		}
 	}
 
+	/*
+	 * When inside a target graph filter, child functions must descend
+	 * from an active frame on the target stack. Suppress detached
+	 * calls originating outside the target hierarchy.

tools/perf/util/ftrace.c int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL · line 735

Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+	 */
+	if (has_filter && cs->in_target && cs->depth > cs->target_root_depth) {
+		if (has_retaddr && caller_idx < cs->target_root_depth) {
+			if (has_open_brace && !has_semicolon)
+				cs->orphan_depth = 1;
+			for (i = 0; i < num_inlines; i++)
+				free((void *)inlined_names[i]);
+			return 0;
+		}
+	}

[Severity: Medium]
Similar to the non-inline case, if the stack has overflowed FTRACE_MAX_STACK, caller_idx will be -1 because the parent frame was never pushed to the stack.

Does this mean caller_idx < cs->target_root_depth will incorrectly evaluate to true, wrongly classifying deep legitimate subtrees as orphans and dropping them?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
#3 lore raw

Re: [PATCH perf-tools-next v5 1/5] perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming

In reply to [email protected]
Quoted 9 lines · On Thu, Oct 15, 2026 at 01:20:15AM +0000, [email protected] wrote:

[Severity: Low]
Does this file explicitly include <stdio.h>?

This isn't a runtime bug, but relying on implicit inclusion via other headers for POSIX stdio streams and functions (like stdout, fprintf, fflush, and ferror) violates the perf subsystem guidelines for musl compatibility.

Explicitly including <stdio.h> ensures the code doesn't cause build breakages on libc implementations that strictly separate declarations, like musl.

Acknowledged. This will be resolved in the next iteration.

Signature · 1 line
Aaron Tomlin
#8 lore raw

Re: [PATCH perf-tools-next v5 3/5] perf ftrace: Support display of inlined functions in function graph tracer

In reply to [email protected]
Quoted 6 lines · On Thu, Oct 15, 2026 at 01:24:43AM +0000, [email protected] wrote:

[Severity: Low]
Does this code leak out_buf if the assertion fails?

The TEST_ASSERT_VAL macro appears to return early upon failure, which would cause the execution to bypass the free(out_buf) call at the end of the function.

Acknowledged. I will resolve this as follows in the next iteration:

	if (!strstr(out_buf, "|  base_func() {")) {
		pr_debug("FAILED %s:%d no indentation drift after stack saturation\n",
			 __FILE__, __LINE__);
		free(out_buf);
		return TEST_FAIL;
	}
Quoted 11 lines

[Severity: Medium]
Will this cause a build failure or pointer truncation due to a missing declaration for zalloc()?

Since the appropriate header providing the prototype for zalloc() does not seem to be explicitly included, the compiler might implicitly assume it returns an int.

On 64-bit architectures, this could truncate the returned pointer. Alternatively, modern compilers configured to warn about implicit function declarations may fail the build entirely.

Acknowledged. Moving forward, I will include: #include <linux/zalloc.h>.

Signature · 1 line
Aaron Tomlin