Skip to contentSkip to conversation

perf ftrace: Support inlined functions and display enhancements in function graph tracer

PATCH Superseded Aaron Tomlin 2 participants 16 messages Oct 12 – Oct 14 perf events

v3 of 13 Superseded Latest is v13 →

lore
Share & tools
lore
#0 lore raw

v3 cover perf ftrace: Support inlined functions and display enhancements in function graph tracer

The Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call-graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining (such as the scheduler, memory management, locking primitives, and RCU).

When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment (e.g. /* <-wake_up_new_task+0x1d1/0x3e0 */). By interrogating DWARF debug information from the kernel image (vmlinux) utilising libdw, perf ftrace resolves this return address to its inlined callchain and synthesises the intermediate inlined frames directly into the streamed call-graph.

This patch series addresses these challenges through five focused commits:

1. Stack frame optimisation in __cmd_ftrace()

Replaces the 4096-byte stack array in __cmd_ftrace() with dynamic heap allocation alongside struct strbuf streaming, reducing the stack frame footprint by over 96% (down to ~160 bytes) while ensuring complete line-delimited records and robust EOF handling.

2. Return address comment filtering

Introduces --filter-retaddr to strip trailing /* <-caller+offset */ comments in-place across diverse tracer formats (standard, bracketed, and return values).

3. Display of inlined functions in function graph tracer

Introduces the --inline option for function_graph tracing. Leverages CONFIG_FUNCTION_GRAPH_RETADDR comments and DWARF debug info from vmlinux (via libdw) to resolve callers to their inlined callchains. Maintains a per-CPU tracking stack to reconstruct and synthesise inlined scopes (/* (inline) */) with correct indentation and bare closing brace matching.

4. Execution duration suppression

Exposes the noduration sub-option under --graph-opts to omit the duration column, enabling concise topological views

5. Filter extraneous functions in function graph tracer

Enforces strict target callgraph boundaries in userspace. Suppresses non-matching top-level entries, and dynamically validates child entries against the target call stack using their return addresses to eliminate detached tracer wakeup side effects.

Synthesised inlined functions are rendered with an explicit /* (inline) */ annotation at both entry and exit:

# tracer: function_graph
#
# CPU FUNCTION CALLS
# | | | | |
3) activate_task() {
3) enqueue_task() {
3) psi_enqueue() { /* (inline) */
3) arch_static_branch_jump() { /* (inline) */
3) enqueue_task_fair() {
3) enqueue_hierarchy() { /* (inline) */
3) update_curr();
3) enqueue_entity() {
3) account_entity_enqueue() { /* (inline) */
3) account_llc_enqueue();
3) } /* account_entity_enqueue (inline) */
3) }
3) } /* enqueue_hierarchy (inline) */
3) place_entity() {
3) avg_vruntime();
3) }
3) __enqueue_entity();
3) dl_server_start();
3) }
3) } /* arch_static_branch_jump (inline) */
3) } /* psi_enqueue (inline) */
3) sched_info_enqueue() { /* (inline) */
3) psi_task_change() {
3) psi_write_begin() { /* (inline) */
3) psi_flags_change();
3) } /* psi_write_begin (inline) */
3) psi_group_change() {
3) record_times();
3) }
3) }
3) } /* sched_info_enqueue (inline) */
3) }
3) }

Changes since v2:

- Expanded the series from 4 to 5 patches by adding a dedicated commit ("perf ftrace: Filter extraneous functions in function graph tracer") to enforce strict target callgraph boundaries when -G or --graph-funcs is specified

- Suppressed unrelated top-level functions that leak through the transient tracefs filter initialisation window before RCU synchronisation completes

- Filtered detached/orphan calls from appearing as nested children within the target graph by validating caller return addresses against the active target stack

- Tracked nested orphan scope depth (cs->orphan_depth) to cleanly suppress child calls and closing delimiters of detached subtrees without desynchronising the target stack

- Added automated unit test coverage in tools/perf/tests/ftrace.c (test_graph_target_filter()) validating top-level suppression (both with and without --inline) and detached orphan wakeup suppression

- Link to v2: https://lore.kernel.org/lkml/[email protected]/

Changes since v1:

- Modularised the implementation into a 4-patch bisectable series rather than a monolithic commit

- Resolved potential infinite loop on poll() upon trace_pipe EOF

- Fixed stuck call stack frames when funcgraph-tail is disabled by supporting bare closing delimiters matching LIFO stack frames

- Prevented unbounded indentation growth on deep call trees by clamping inline expansion and synchronising cs->inlined_depth to frames successfully pushed onto cs->stack

- Added dedicated unit tests covering bare closing braces and stack saturation recovery

- Refined option documentation in perf-ftrace.txt

- Link to v1: https://lore.kernel.org/lkml/[email protected]/

Aaron Tomlin (5):
perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming perf ftrace: Support filtering return address comments perf ftrace: Support display of inlined functions in function graph
tracer
perf ftrace: Support omitting execution duration in function graph
tracer
perf ftrace: Filter extraneous functions in function graph tracer

 tools/perf/Documentation/perf-ftrace.txt |  21 +-
 tools/perf/builtin-ftrace.c              | 128 +++-
 tools/perf/tests/Build                   |   1 +
 tools/perf/tests/builtin-test.c          |   1 +
 tools/perf/tests/ftrace.c                | 589 +++++++++++++++
 tools/perf/tests/tests.h                 |   1 +
 tools/perf/util/Build                    |   1 +
 tools/perf/util/ftrace.c                 | 871 +++++++++++++++++++++++
 tools/perf/util/ftrace.h                 |  17 +
 9 files changed, 1615 insertions(+), 15 deletions(-)
 create mode 100644 tools/perf/tests/ftrace.c
 create mode 100644 tools/perf/util/ftrace.c
Signature · 1 line
2.55.0
#1 lore raw

PATCH v3 1/5 perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming

In __cmd_ftrace(), incoming trace data from trace_pipe was historically buffered in a fixed 4096-byte array allocated directly on the stack frame. Furthermore, reads were written directly to stdout via fwrite() in arbitrary chunk sizes, rather than as discrete, well-delimited trace lines.

Refactor __cmd_ftrace() to employ dynamic heap allocation for the read buffer alongside a dynamic struct strbuf to accumulate streamed characters into whole lines. Pre-allocating the strbuf ensures that line processing incurs zero heap reallocations during steady-state polling, while the stack frame of __cmd_ftrace() is reduced by over 96% (down to ~160 bytes).

---
 tools/perf/builtin-ftrace.c | 57 ++++++++++++++++++++++++++++++-------
 1 file changed, 46 insertions(+), 11 deletions(-)
tools/perf/builtin-ftrace.c +46 -11
index 6017493ff179..d6604be593a3 100644
@@ -41,7 +41,10 @@
 #include "util/units.h"
 #include "util/parse-sublevel-options.h"
 
+#include "util/strbuf.h"
+
 #define DEFAULT_TRACER  "function_graph"
+#define TRACE_BUF_SIZE  4096
 
 static volatile sig_atomic_t workload_exec_errno;
 static volatile sig_atomic_t done;
@@ -738,7 +741,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 {
 	char *trace_file;
 	int trace_fd;
-	char buf[4096];
+	char *buf = NULL;
+	struct strbuf linebuf = STRBUF_INIT;
 	struct pollfd pollfd = {
 		.events = POLLIN,
 	};
@@ -767,10 +771,17 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 
 	setup_pager();
 
+	buf = malloc(TRACE_BUF_SIZE);
+	if (!buf) {
+		pr_err("failed to allocate trace buffer\n");
+		goto out_reset;
+	}
+	strbuf_init(&linebuf, 512);
+
 	trace_file = get_tracing_instance_file("trace_pipe");
 	if (!trace_file) {
 		pr_err("failed to open trace_pipe\n");
-		goto out_reset;
+		goto out_free;
 	}
 
 	trace_fd = open(trace_file, O_RDONLY);
@@ -779,7 +790,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 
 	if (trace_fd < 0) {
 		pr_err("failed to open trace_pipe\n");
-		goto out_reset;
+		goto out_free;
 	}
 
 	fcntl(trace_fd, F_SETFL, O_NONBLOCK);
@@ -810,11 +821,19 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 			break;
 
 		if (pollfd.revents & POLLIN) {
-			int n = read(trace_fd, buf, sizeof(buf));
-			if (n < 0)
-				break;
-			if (fwrite(buf, n, 1, stdout) != 1)
+			int n = read(trace_fd, buf, TRACE_BUF_SIZE);
+
+			if (n <= 0)
 				break;
+			for (int i = 0; i < n; i++) {
+				if (buf[i] == '\n') {
+					fprintf(stdout, "%s\n", linebuf.buf);
+					strbuf_setlen(&linebuf, 0);
+				} else {
+					if (strbuf_addch(&linebuf, buf[i]) < 0)
+						goto out_close_fd;
+				}
+			}
 			/* flush output since stdout is in full buffering mode due to pager */
 			fflush(stdout);
 		}
@@ -823,7 +842,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	write_tracing_file("tracing_on", "0");
 
 	if (workload_exec_errno) {
-		const char *emsg = str_error_r(workload_exec_errno, buf, sizeof(buf));
+		const char *emsg = str_error_r(workload_exec_errno, buf, TRACE_BUF_SIZE);
 		/* flush stdout first so below error msg appears at the end. */
 		fflush(stdout);
 		pr_err("workload failed: %s\n", emsg);
@@ -832,15 +851,31 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 
 	/* read remaining buffer contents */
 	while (true) {
-		int n = read(trace_fd, buf, sizeof(buf));
+		int n = read(trace_fd, buf, TRACE_BUF_SIZE);
+
 		if (n <= 0)
 			break;
-		if (fwrite(buf, n, 1, stdout) != 1)
-			break;
+		for (int i = 0; i < n; i++) {
+			if (buf[i] == '\n') {
+				fprintf(stdout, "%s\n", linebuf.buf);
+				strbuf_setlen(&linebuf, 0);
+			} else {
+				if (strbuf_addch(&linebuf, buf[i]) < 0)
+					goto out_close_fd;
+			}
+		}
+	}
+
+	if (linebuf.len > 0) {
+		fprintf(stdout, "%s\n", linebuf.buf);
+		strbuf_setlen(&linebuf, 0);
 	}
 
 out_close_fd:
 	close(trace_fd);
+out_free:
+	free(buf);
+	strbuf_release(&linebuf);
 out_reset:
 	exit_tracing_instance();
 out:
Signature · 1 line
2.55.0
#4 lore raw

PATCH v3 2/5 perf ftrace: Support filtering return address comments

In reply to Aaron Tomlin

When tracing with the Linux kernel's function graph tracer, enabling funcgraph-retaddr records the return address of each function entry as a comment in the trace stream (e.g. /* <-load_elf_phdrs+0x6c/0xb0 */). While valuable for tracing execution origins, these trailing comments can add considerable visual clutter to the call graph, particularly when examining deeply nested call trees or when synthesising higher-level abstractions.

Introduce support for filtering out return address comments from function graph trace output. Provide a standalone --filter-retaddr flag.

Key aspects of this implementation:

1. Return address parsing and filtering

Implement ftrace_parse_retaddr() and ftrace_filter_retaddr() in tools/perf/util/ftrace.c to robustly match return address comments across diverse output styles (standard format, return values, and bracketed addresses) and strip them in-place with memmove(3).

2. Command-line interface

Expose --filter-retaddr in tools/perf/builtin-ftrace.c, updating the documentation in tools/perf/Documentation/perf-ftrace.txt accordingly.

3. Automated verification

Add a dedicated unit test suite ("Ftrace return address processing") in tools/perf/tests/ftrace.c to validate comment parsing and string filtering across all supported formats.

Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/Documentation/perf-ftrace.txt |  5 +-
 tools/perf/builtin-ftrace.c              |  9 +++
 tools/perf/tests/Build                   |  1 +
 tools/perf/tests/builtin-test.c          |  1 +
 tools/perf/tests/ftrace.c                | 80 ++++++++++++++++++++++
 tools/perf/tests/tests.h                 |  1 +
 tools/perf/util/Build                    |  1 +
 tools/perf/util/ftrace.c                 | 87 ++++++++++++++++++++++++
 tools/perf/util/ftrace.h                 |  5 ++
 9 files changed, 189 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/tests/ftrace.c
 create mode 100644 tools/perf/util/ftrace.c
tools/perf/Documentation/perf-ftrace.txt +4 -1
index 3f3808e513fe..3dbb7e8e783e 100644
@@ -1,5 +1,4 @@
 perf-ftrace(1)
-==============
 
 NAME
 ----
@@ -134,6 +133,10 @@ OPTIONS for 'perf ftrace trace'
 	  - depth=<n>    - Set max depth for function graph tracer to follow.
 	  - tail         - Print function name at the end.
 
+--filter-retaddr::
+	Filter out function return address comments (`/* <-caller+offset */`)
+	from function_graph output.
+
 
 OPTIONS for 'perf ftrace latency'
 ---------------------------------
tools/perf/builtin-ftrace.c +9 -0
index d6604be593a3..f3221089a479 100644
@@ -827,6 +827,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 				break;
 			for (int i = 0; i < n; i++) {
 				if (buf[i] == '\n') {
+					if (ftrace->filter_retaddr)
+						ftrace_filter_retaddr(linebuf.buf);
 					fprintf(stdout, "%s\n", linebuf.buf);
 					strbuf_setlen(&linebuf, 0);
 				} else {
@@ -857,6 +859,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 			break;
 		for (int i = 0; i < n; i++) {
 			if (buf[i] == '\n') {
+				if (ftrace->filter_retaddr)
+					ftrace_filter_retaddr(linebuf.buf);
 				fprintf(stdout, "%s\n", linebuf.buf);
 				strbuf_setlen(&linebuf, 0);
 			} else {
@@ -867,6 +871,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	}
 
 	if (linebuf.len > 0) {
+		if (ftrace->filter_retaddr)
+			ftrace_filter_retaddr(linebuf.buf);
 		fprintf(stdout, "%s\n", linebuf.buf);
 		strbuf_setlen(&linebuf, 0);
 	}
@@ -1827,11 +1833,14 @@ int cmd_ftrace(int argc, const char **argv)
 		     "Set nograph filter on given functions", parse_filter_func),
 	OPT_CALLBACK(0, "graph-opts", &ftrace, "options",
 		     "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,verbose,thresh=<n>,depth=<n>",
+		     "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,noduration,verbose,thresh=<n>,depth=<n>,tail",
 		     parse_graph_tracer_opts),
 	OPT_CALLBACK('m', "buffer-size", &ftrace.percpu_buffer_size, "size",
 		     "Size of per cpu buffer, needs to use a B, K, M or G suffix.", parse_buffer_size),
 	OPT_BOOLEAN(0, "inherit", &ftrace.inherit,
 		    "Trace children processes"),
+	OPT_BOOLEAN_SET(0, "filter-retaddr", &ftrace.filter_retaddr, &ftrace.filter_retaddr_set,
+			"Filter out funcgraph return address comments"),
 	OPT_INTEGER('D', "delay", &ftrace.target.initial_delay,
 		    "Number of milliseconds to wait before starting tracing after program start"),
 	OPT_PARENT(common_options),
tools/perf/tests/Build +1 -0
index 395f2866094a..136138932fcb 100644
@@ -72,6 +72,7 @@ perf-test-y += hwmon_pmu.o
 perf-test-y += tool_pmu.o
 perf-test-y += subcmd-help.o
 perf-test-y += kallsyms-split.o
+perf-test-y += ftrace.o
 
 ifeq ($(SRCARCH),$(filter $(SRCARCH),x86 arm arm64 powerpc riscv))
 perf-test-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
tools/perf/tests/builtin-test.c +1 -0
index d2f594921e25..9663f48afb92 100644
@@ -137,6 +137,7 @@ static struct test_suite *generic_tests[] = {
 	&suite__perf_hooks,
 	&suite__unit_number__scnprint,
 	&suite__mem2node,
+	&suite__ftrace,
 	&suite__time_utils,
 	&suite__jit_write_elf,
 	&suite__pfm,
tools/perf/tests/ftrace.c new file +80 -0

80 lines not on this page · raw message

tools/perf/tests/tests.h +1 -0
index 9c96f33483d1..1aec32e094c8 100644
@@ -163,6 +163,7 @@ DECLARE_SUITE(perf_hooks);
 DECLARE_SUITE(unit_number__scnprint);
 DECLARE_SUITE(mem2node);
 DECLARE_SUITE(maps);
+DECLARE_SUITE(ftrace);
 DECLARE_SUITE(time_utils);
 DECLARE_SUITE(jit_write_elf);
 DECLARE_SUITE(api_io);
tools/perf/util/Build +1 -0
index 2c1f880c4c47..5b082d09006d 100644
@@ -169,6 +169,7 @@ perf-util-y += list_sort.o
 perf-util-y += mutex.o
 perf-util-y += sharded_mutex.o
 perf-util-y += intel-tpebs.o
+perf-util-y += ftrace.o
 
 perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_counter.o
 perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_counter_cgroup.o
tools/perf/util/ftrace.c new file +87 -0

87 lines not on this page · raw message

tools/perf/util/ftrace.h +5 -0
index 950f2efafad2..50216d3fef64 100644
@@ -39,6 +39,8 @@ struct perf_ftrace {
 	int			graph_verbose;
 	int			graph_thresh;
 	int			graph_tail;
+	bool			filter_retaddr;
+	bool			filter_retaddr_set;
 };
 
 struct filter_entry {
@@ -93,4 +95,7 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)
 
 #endif  /* HAVE_BPF_SKEL */
 
+bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset);
+void ftrace_filter_retaddr(char *str);
+
 #endif  /* __PERF_FTRACE_H__ */
Signature · 1 line
2.55.0
#7 lore raw

PATCH v3 3/5 perf ftrace: Support display of inlined functions in function graph tracer

In reply to Aaron Tomlin

The Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining.

Address this by introducing the --inline option to perf ftrace for the function_graph tracer.

When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment (e.g. /* <-irq_work_queue_on+0x81/0xa0 */). By interrogating DWARF debug information from the kernel image (vmlinux) utilising libdw, perf ftrace resolves this return address to its inlined callchain and synthesises the intermediate inlined frames directly into the streamed call graph.

Key aspects of this implementation:

1. Inlined frame presentation

Synthesised inlined functions are rendered with an explicit /* (inline) */ annotation at both entry and exit:

# CPU DURATION FUNCTION CALLS
# | | | | | | |
0) | wake_up_new_task() {
0) | activate_task() { /* (inline) */
0) 1.234 us | enqueue_task();
0) | } /* activate_task (inline) */
0) 2.567 us | }

This affords immediate visual clarity to the reader upon entering an inlined scope, without necessitating a search for the closing delimiter.

2. Call graph reconstruction and dynamic stack tracking

Maintains a per-CPU stack tracking real and inlined frames, synthesising entry and exit delimiters to accurately reconstruct the execution hierarchy. Inlined frames are structured strictly root-to-leaf and validated against the active stack to prevent un-traced callers from attaching orphan inlines. Bare closing braces without trailing function names (the default when funcgraph-tail is disabled) are correctly matched against preceding frames on the stack. Furthermore, stack depth saturation guards ensure that inlined frame depth tracking remains clamped and indentation cannot drift even under deep call trees exceeding FTRACE_MAX_STACK.

3. Option handling and diagnostics

Adds -k or --vmlinux to designate an explicit debug image, and provides clear diagnostic guidance should --inline be invoked on a kernel lacking CONFIG_FUNCTION_GRAPH_RETADDR. Automatically enables return address comment filtering unless explicitly requested.

4. Automated verification

Expands the unit test suite in tools/perf/tests/ftrace.c to validate DWARF inline symbol resolution, comment filtering toggles, bare closing braces handling, and stack saturation recovery.

Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/Documentation/perf-ftrace.txt |  17 +-
 tools/perf/builtin-ftrace.c              |  56 +-
 tools/perf/tests/ftrace.c                | 303 ++++++++++-
 tools/perf/util/ftrace.c                 | 621 ++++++++++++++++++++++-
 tools/perf/util/ftrace.h                 |  11 +
 5 files changed, 990 insertions(+), 18 deletions(-)
tools/perf/Documentation/perf-ftrace.txt +16 -1
index 3dbb7e8e783e..86306caf569b 100644
@@ -133,9 +133,24 @@ OPTIONS for 'perf ftrace trace'
 	  - depth=<n>    - Set max depth for function graph tracer to follow.
 	  - tail         - Print function name at the end.
 
+--inline::
+	Show inlined functions in function_graph tracer. This requires
+	kernel support for the funcgraph-retaddr option and a vmlinux with
+	debug symbols. Inlined functions are displayed in the call graph
+	hierarchy, and their inlined status is indicated with `/* (inline) */`
+	on entry and exit, e.g. `func() { /* (inline) */` and `} /* func (inline) */`.
+	By default, raw return address comments (`/* <-caller+offset */`)
+	are hidden in the output; specify `--graph-opts retaddr` to show them.
+
 --filter-retaddr::
 	Filter out function return address comments (`/* <-caller+offset */`)
-	from function_graph output.
+	from function_graph output. This is enabled by default when `--inline`
+	is used; use `--no-filter-retaddr` to disable it.
+
+-k::
+--vmlinux=::
+	Path to the vmlinux file containing debug symbols for resolving
+	inlined functions.
 
 
 OPTIONS for 'perf ftrace latency'
tools/perf/builtin-ftrace.c +43 -13

105 lines not on this page · raw message

tools/perf/tests/ftrace.c +302 -1

321 lines not on this page · raw message

tools/perf/util/ftrace.c +618 -3

634 lines not on this page · raw message

tools/perf/util/ftrace.h +11 -0
index 50216d3fef64..4140902d9de6 100644
@@ -41,6 +41,8 @@ struct perf_ftrace {
 	int			graph_tail;
 	bool			filter_retaddr;
 	bool			filter_retaddr_set;
+	bool			use_inline;
+	struct machine		*machine;
 };
 
 struct filter_entry {
@@ -95,7 +97,16 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)
 
 #endif  /* HAVE_BPF_SKEL */
 
+struct machine;
+
 bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset);
 void ftrace_filter_retaddr(char *str);
+int ftrace_resolve_inlines(struct machine *machine,
+			   const char *sym_name, u64 offset,
+			   const char **inlined_names, int max_inlines);
+int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out);
+void ftrace_cpu_states__clear(void);
+int perf_ftrace__setup_inlines(struct perf_ftrace *ftrace);
+void perf_ftrace__cleanup_inlines(struct perf_ftrace *ftrace);
 
 #endif  /* __PERF_FTRACE_H__ */
Signature · 1 line
2.55.0
#10 lore raw

PATCH v3 4/5 perf ftrace: Support omitting execution duration in function graph tracer

In reply to Aaron Tomlin

The Linux kernel's function graph tracer displays the execution duration of completed functions by default via the funcgraph-duration trace option. Whilst informative, in scenarios where the user is solely interested in the call graph topology, call flow sequence, or inlined hierarchy, the duration column can add visual clutter and widen each trace line.

Introduce support for omitting function duration by exposing the noduration sub-option under --graph-opts. Consistent with the existing nosleep-time and noirqs sub-options, it accepts an optional value, so noduration=0 explicitly retains the duration column; when specified more than once, the last occurrence takes effect. When enabled, perf ftrace configures the kernel's tracing instance option funcgraph-duration to 0, and restores it to 1 upon resetting tracing options.

---
 tools/perf/Documentation/perf-ftrace.txt |  1 +
 tools/perf/builtin-ftrace.c              | 28 +++++++++++++++++++++++-
 tools/perf/tests/ftrace.c                | 25 +++++++++++++++++----
 tools/perf/util/ftrace.c                 | 15 +++++++++----
 tools/perf/util/ftrace.h                 |  1 +
 5 files changed, 61 insertions(+), 9 deletions(-)
tools/perf/Documentation/perf-ftrace.txt +1 -0
index 86306caf569b..a584583936bb 100644
@@ -128,6 +128,7 @@ OPTIONS for 'perf ftrace trace'
 	  - retaddr      - Show function return address.
 	  - nosleep-time - Measure on-CPU time only for function_graph tracer.
 	  - noirqs       - Ignore functions that happen inside interrupt.
+	  - noduration   - Do not print function duration.
 	  - verbose      - Show process names, PIDs, timestamps, etc.
 	  - thresh=<n>   - Setup trace duration threshold in microseconds.
 	  - depth=<n>    - Set max depth for function graph tracer to follow.
tools/perf/builtin-ftrace.c +27 -1
index 504007064e24..926726c5d9a8 100644
@@ -299,6 +299,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("func_stack_trace", "0");
 	write_tracing_option_file("sleep-time", "1");
 	write_tracing_option_file("funcgraph-irqs", "1");
+	write_tracing_option_file("funcgraph-duration", "1");
 	write_tracing_option_file("funcgraph-proc", "0");
 	write_tracing_option_file("funcgraph-abstime", "0");
 	write_tracing_option_file("funcgraph-tail", "0");
@@ -598,6 +599,26 @@ static int set_tracing_funcgraph_irqs(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+/**
+ * set_tracing_funcgraph_duration - Set funcgraph-duration tracing option
+ * @ftrace: Perf ftrace context
+ *
+ * Configure the kernel's funcgraph-duration tracing option based on the
+ * ftrace->graph_noduration flag.
+ *
+ * Return: 0 on success, or a negative error code on failure.
+ */
+static int set_tracing_funcgraph_duration(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->graph_noduration)
+		return 0;
+
+	if (write_tracing_option_file("funcgraph-duration", "0") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int set_tracing_funcgraph_verbose(struct perf_ftrace *ftrace)
 {
 	if (!ftrace->graph_verbose)
@@ -707,6 +728,11 @@ static int set_tracing_options(struct perf_ftrace *ftrace)
 		return -1;
 	}
 
+	if (set_tracing_funcgraph_duration(ftrace) < 0) {
+		pr_err("failed to set tracing option funcgraph-duration\n");
+		return -1;
+	}
+
 	if (set_tracing_funcgraph_verbose(ftrace) < 0) {
 		pr_err("failed to set tracing option funcgraph-proc/funcgraph-abstime\n");
 		return -1;
@@ -1754,6 +1780,7 @@ static int parse_graph_tracer_opts(const struct option *opt,
 		{ .name = "retaddr",		.value_ptr = &ftrace->graph_retaddr },
 		{ .name = "nosleep-time",	.value_ptr = &ftrace->graph_nosleep_time },
 		{ .name = "noirqs",		.value_ptr = &ftrace->graph_noirqs },
+		{ .name = "noduration",		.value_ptr = &ftrace->graph_noduration },
 		{ .name = "verbose",		.value_ptr = &ftrace->graph_verbose },
 		{ .name = "thresh",		.value_ptr = &ftrace->graph_thresh },
 		{ .name = "depth",		.value_ptr = &ftrace->graph_depth },
@@ -1843,7 +1870,6 @@ int cmd_ftrace(int argc, const char **argv)
 	OPT_CALLBACK('g', "nograph-funcs", &ftrace.nograph_funcs, "func",
 		     "Set nograph filter on given functions", parse_filter_func),
 	OPT_CALLBACK(0, "graph-opts", &ftrace, "options",
-		     "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,verbose,thresh=<n>,depth=<n>",
 		     "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,noduration,verbose,thresh=<n>,depth=<n>,tail",
 		     parse_graph_tracer_opts),
 	OPT_CALLBACK('m', "buffer-size", &ftrace.percpu_buffer_size, "size",
tools/perf/tests/ftrace.c +21 -4
index e856bed00700..cb212339e666 100644
@@ -303,20 +303,36 @@ static int test_column_layouts(void)
 {
 	static const struct {
 		const char *desc;
-				bool verbose;
+		bool noduration;
+		bool verbose;
 		const char *in;
 		const char *expected;
 	} cases[] = {
 		{
-			"default layout", false,
+			"default layout", false, false,
 			" 7)               |  wake_up_new_task() { /* <-kernel_clone+0x160/0x510 */",
 			" 7)               |  wake_up_new_task() {\n",
 		},
 		{
-			"verbose layout", true,
+			"noduration layout", true, false,
+			" 7)   housekeeping_any_cpu(); /* <-ring_buffer_unlock_commit+0x1a0/0x1f0 */",
+			" 7)   housekeeping_any_cpu();\n",
+		},
+		{
+			"verbose layout", false, true,
 			" 1234.567890 |   7)   bash-1234    |               |  wake_up_new_task() { /* <-kernel_clone+0x160/0x510 */",
 			" 1234.567890 |   7)   bash-1234    |               |  wake_up_new_task() {\n",
 		},
+		{
+			"verbose noduration layout", true, true,
+			" 1234.567890 |   7)   bash-1234    |  wake_up_new_task() { /* <-kernel_clone+0x160/0x510 */",
+			" 1234.567890 |   7)   bash-1234    |  wake_up_new_task() {\n",
+		},
+		{
+			"noduration context switch marker", true, false,
+			"   7)  bash-1234    =>  sh-5678",
+			"   7)  bash-1234    =>  sh-5678\n",
+		},
 	};
 	struct perf_ftrace ftrace;
 	char *out_buf = NULL;
@@ -328,7 +344,8 @@ static int test_column_layouts(void)
 		memset(&ftrace, 0, sizeof(ftrace));
 		ftrace.use_inline = true;
 		ftrace.filter_retaddr = true;
-				ftrace.graph_verbose = cases[i].verbose;
+		ftrace.graph_noduration = cases[i].noduration;
+		ftrace.graph_verbose = cases[i].verbose;
 
 		out = open_memstream(&out_buf, &out_len);
 		if (!out)
tools/perf/util/ftrace.c +11 -4
index 8737bb52921e..5cae01a28529 100644
@@ -315,10 +315,12 @@ static void make_blank_prefix(const char *line, const char *pipe_char, char *buf
  *   [ abstime |]  CPU)  [ comm-pid |]  [ duration |]  func
  *
  * The abstime and comm-pid columns are only present with --graph-opts
- * verbose.
+ * verbose, whereas the duration column, and thus its '|' delimiter, is
+ * omitted with --graph-opts noduration.
  *
  * Return: A pointer to the last character of the context columns (the final
- * '|' delimiter). Returns NULL if @line is not a trace record.
+ * '|' delimiter, or the CPU ')' when no delimiter follows it). Returns NULL
+ * if @line is not a trace record (e.g. a header or a context switch marker).
  */
 static const char *find_func_column(const struct perf_ftrace *ftrace,
 				    const char *line, int *cpu)
@@ -346,9 +348,14 @@ static const char *find_func_column(const struct perf_ftrace *ftrace,
 			return NULL;
 	}
 
-	p = strchr(p + 1, '|');
-	if (!p)
+	if (!ftrace->graph_noduration) {
+		p = strchr(p + 1, '|');
+		if (!p)
+			return NULL;
+	} else if (strstr(p, " => ")) {
+		/* Context switch marker: " CPU) comm-pid => comm-pid" */
 		return NULL;
+	}
 
 	*cpu = (int)val;
 	return p;
tools/perf/util/ftrace.h +1 -0
index 4140902d9de6..d202a09d112f 100644
@@ -36,6 +36,7 @@ struct perf_ftrace {
 	int			graph_retaddr;
 	int			graph_nosleep_time;
 	int			graph_noirqs;
+	int			graph_noduration;
 	int			graph_verbose;
 	int			graph_thresh;
 	int			graph_tail;
Signature · 1 line
2.55.0
#13 lore raw

PATCH v3 5/5 perf ftrace: Filter extraneous functions in function graph tracer

In reply to Aaron Tomlin

When tracing with the function graph tracer using a targeted filter (-G or --graph-funcs), users occasionally observe unrelated functions in the trace output:

1. Top-level functions (e.g. most notably finish_task_switch.isra.0()) appearing before or alongside the requested target function. This occurs because clearing and rewriting set_graph_function creates a transient window where remote CPUs still observe an empty filter (tracing all functions) before RCU synchronisation completes, coupled with perf ftrace sleeping in poll() on trace_pipe right as tracing starts.

2. Detached/orphan calls inside the target callgraph (e.g. housekeeping_any_cpu() and arch_irq_work_raise()) appearing as child frames under the target function. These routines are invoked by rb_wakeups() to notify perf's poll() on trace_pipe whenever an event is committed, and execute while the target function is on the CPU.

Resolve both issues deterministically in userspace:

1. Track active target callgraph state per CPU in tools/perf/util/ftrace.c. Any top-level entry not matching the requested -G targets is suppressed along with its nested children until its scope exits.

2. Within an active target callgraph, validate that child entries descend from an active frame on the target stack using their return addresses (CONFIG_FUNCTION_GRAPH_RETADDR). Any detached calls originating outside the target hierarchy (such as ring-buffer wakeups) are suppressed along with their subtrees.

For example:

# tracer: function_graph
#
# CPU FUNCTION CALLS
# | | | | |
3) activate_task() {
3) enqueue_task() {
3) psi_enqueue() { /* (inline) */
3) arch_static_branch_jump() { /* (inline) */
3) enqueue_task_fair() {
3) enqueue_hierarchy() { /* (inline) */
3) update_curr();
3) enqueue_entity() {
3) account_entity_enqueue() { /* (inline) */
3) account_llc_enqueue();
3) } /* account_entity_enqueue (inline) */
3) }
3) } /* enqueue_hierarchy (inline) */
3) place_entity() {
3) avg_vruntime();
3) }
3) __enqueue_entity();
3) dl_server_start();
3) }
3) } /* arch_static_branch_jump (inline) */
3) } /* psi_enqueue (inline) */
3) sched_info_enqueue() { /* (inline) */
3) psi_task_change() {
3) psi_write_begin() { /* (inline) */
3) psi_flags_change();
3) } /* psi_write_begin (inline) */
3) psi_group_change() {
3) record_times();
3) }
3) }
3) } /* sched_info_enqueue (inline) */
3) }
3) }

---
 tools/perf/tests/ftrace.c | 191 ++++++++++++++++++++++++++++++++++++++
 tools/perf/util/ftrace.c  | 174 ++++++++++++++++++++++++++++++++--
 2 files changed, 359 insertions(+), 6 deletions(-)
tools/perf/tests/ftrace.c +191 -0
index cb212339e666..0b88feb97a1e 100644
@@ -368,6 +368,193 @@ static int test_column_layouts(void)
 	return TEST_OK;
 }
 
+static int test_graph_target_filter(void)
+{
+	static const char line_idle[] = " 7)   3.000 us    |  sched_idle_set_state();";
+	static const char line_sw_entry[] = " 6)               |  finish_task_switch.isra.0() {";
+	static const char line_sw_child[] = " 6)   1.000 us    |    _raw_spin_unlock();";
+	static const char line_sw_exit[] = " 6)   2.000 us    |  }";
+	static const char line_tgt_entry[] = " 2)               |  activate_task() {";
+	static const char line_tgt_child[] = " 2)   1.500 us    |    enqueue_task();";
+	static const char line_tgt_exit[] = " 2)   2.500 us    |  }";
+	struct perf_ftrace ftrace;
+	struct filter_entry *entry;
+	struct filter_entry *pos, *tmp;
+	char *out_buf = NULL;
+	size_t out_len = 0;
+	FILE *out;
+	int ret = TEST_OK;
+
+	memset(&ftrace, 0, sizeof(ftrace));
+	INIT_LIST_HEAD(&ftrace.graph_funcs);
+
+	entry = malloc(sizeof(*entry) + strlen("activate_task") + 1);
+	if (!entry)
+		return TEST_FAIL;
+	strcpy(entry->name, "activate_task");
+	list_add_tail(&entry->list, &ftrace.graph_funcs);
+
+	/* 1. Test without --inline */
+	ftrace.use_inline = false;
+	out = open_memstream(&out_buf, &out_len);
+	if (!out) {
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	ftrace_process_fgraph_line(&ftrace, line_idle, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_entry, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_child, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_exit, out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_entry, out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_child, out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_exit, out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	if (strstr(out_buf, "sched_idle_set_state")) {
+		pr_debug("FAILED: sched_idle_set_state not suppressed\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (strstr(out_buf, "finish_task_switch")) {
+		pr_debug("FAILED: finish_task_switch not suppressed\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (strstr(out_buf, "_raw_spin_unlock")) {
+		pr_debug("FAILED: _raw_spin_unlock not suppressed\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (!strstr(out_buf, "activate_task() {")) {
+		pr_debug("FAILED: activate_task entry missing\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (!strstr(out_buf, "enqueue_task();")) {
+		pr_debug("FAILED: enqueue_task child missing\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	free(out_buf);
+	out_buf = NULL;
+
+	/* 2. Test with --inline */
+	ftrace.use_inline = true;
+	out = open_memstream(&out_buf, &out_len);
+	if (!out) {
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	ftrace_process_fgraph_line(&ftrace, line_idle, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_entry, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_child, out);
+	ftrace_process_fgraph_line(&ftrace, line_sw_exit, out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_entry, out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_child, out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_exit, out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	pr_debug("Target filter output with --inline:\n%s\n", out_buf);
+
+	if (strstr(out_buf, "finish_task_switch")) {
+		pr_debug("FAILED: finish_task_switch not suppressed with --inline\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (!strstr(out_buf, "activate_task() {")) {
+		pr_debug("FAILED: activate_task entry missing with --inline\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	free(out_buf);
+	out_buf = NULL;
+
+	/* 3. Test suppressing detached orphan calls within target graph */
+	ftrace.use_inline = true;
+	ftrace.filter_retaddr = 1;
+	out = open_memstream(&out_buf, &out_len);
+	if (!out) {
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	ftrace_process_fgraph_line(&ftrace, line_tgt_entry, out);
+	/* Leaf orphan call from ring_buffer_unlock_commit */
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)               |    housekeeping_any_cpu(); /* <-ring_buffer_unlock_commit+0x1a0/0x1f0 */",
+		out);
+	/* Non-leaf orphan call from irq_work_queue_on */
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)               |    arch_irq_work_raise() { /* <-irq_work_queue_on+0x81/0xa0 */",
+		out);
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)               |      apic_wait_icr_idle() { /* (inline) */",
+		out);
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)               |        x2apic_send_IPI_self();",
+		out);
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)               |      } /* apic_wait_icr_idle (inline) */",
+		out);
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)   1.000 us    |    }",
+		out);
+	/* Genuine child call from activate_task */
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)               |    enqueue_task() { /* <-activate_task+0x1d1/0x3e0 */",
+		out);
+	ftrace_process_fgraph_line(&ftrace,
+		" 2)   1.500 us    |    }",
+		out);
+	ftrace_process_fgraph_line(&ftrace, line_tgt_exit, out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	pr_debug("Orphan call filter output:\n%s\n", out_buf);
+
+	if (strstr(out_buf, "housekeeping_any_cpu")) {
+		pr_debug("FAILED: housekeeping_any_cpu orphan not suppressed\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (strstr(out_buf, "arch_irq_work_raise")) {
+		pr_debug("FAILED: arch_irq_work_raise orphan not suppressed\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (strstr(out_buf, "x2apic_send_IPI_self")) {
+		pr_debug("FAILED: x2apic_send_IPI_self not suppressed\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+	if (!strstr(out_buf, "enqueue_task() {")) {
+		pr_debug("FAILED: genuine enqueue_task missing\n");
+		ret = TEST_FAIL;
+		goto out_free_filter;
+	}
+
+	free(out_buf);
+	out_buf = NULL;
+
+out_free_filter:
+	if (out_buf)
+		free(out_buf);
+	list_for_each_entry_safe(pos, tmp, &ftrace.graph_funcs, list) {
+		list_del(&pos->list);
+		free(pos);
+	}
+	return ret;
+}
+
 static int test__ftrace(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
 {
 	int ret;
@@ -388,6 +575,10 @@ static int test__ftrace(struct test_suite *test __maybe_unused, int subtest __ma
 	if (ret != TEST_OK)
 		return ret;
 
+	ret = test_graph_target_filter();
+	if (ret != TEST_OK)
+		return ret;
+
 	ret = test_inline_processing();
 	if (ret != TEST_OK)
 		return ret;
tools/perf/util/ftrace.c +168 -6

229 lines not on this page · raw message

Signature · 1 line
2.55.0
#14 lore raw

Re: [PATCH perf-tools-next v3 5/5] perf ftrace: Filter extraneous functions in function graph tracer

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf ftrace: Target filter state leaks for leaf functions at non-zero depth --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf ftrace: Target filter state leaks for leaf functions at non-zero depth --

commit 1481567e7a5dda207ce5b61e39c1b59803ca97c4
Author: Aaron Tomlin <[email protected]>

perf ftrace: Filter extraneous functions in function graph tracer

This commit enhances perf ftrace to filter out extraneous top-level functions and detached orphan calls from the function graph output when using a target filter.

Quoted 4 lines
tools/perf/util/ftrace.c +0 -0
index 5cae01a28529f..96a41077dccbf 100644

[ ... ]

Quoted 4 lines
tools/perf/util/ftrace.c +0 -0
@@ -509,6 +564,92 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 	has_open_brace = (strchr(func_start, '{') != NULL);
 	has_semicolon = (strchr(func_start, ';') != NULL);
 

[ ... ]

tools/perf/util/ftrace.c int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL · line 651

Quoted 8 lines
tools/perf/util/ftrace.c +8 -0
+		print_line(ftrace, line, out);
+		if (has_filter && cs->in_target && cs->depth == 0 &&
+		    has_semicolon && !has_open_brace) {
+			cs->in_target = false;
+			cs->orphan_depth = 0;
+		}
+		return 0;
+	}

[Severity: Medium]
When a targeted function is a leaf call (no braces, only a semicolon) and occurs at a call stack depth greater than 0, doesn't the hardcoded check for cs->depth == 0 prevent the filter state from resetting?

It appears that cs->in_target will remain true indefinitely if the target is not at the root depth, causing subsequent non-targeted sibling functions and their descendants within the same parent frame to bypass the filter and incorrectly appear in the trace output.

[ ... ]

tools/perf/util/ftrace.c int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL · line 812

Quoted 13 lines
tools/perf/util/ftrace.c +6 -0
@@ -645,6 +801,12 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 		}
 	}
 
+	if (has_filter && cs->in_target && cs->depth == 0 &&
+	    has_semicolon && !has_open_brace) {
+		cs->in_target = false;
+		cs->orphan_depth = 0;
+	}
+
 	for (i = 0; i < num_inlines; i++)
 		free((void *)inlined_names[i]);
 	return 0;

[Severity: Medium]
The identical logic exists here for the use_inline code path. Should these checks verify cs->depth == cs->target_root_depth instead of cs->depth == 0 to properly reset the filter for leaf targets at non-zero depths?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
#5 lore raw

Re: [PATCH perf-tools-next v3 2/5] perf ftrace: Support filtering return address comments

In reply to Aaron Tomlin
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Low] perf: build failure in cmd_ftrace() due to extra OPT_CALLBACK argument - [Low] perf…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Low] perf: build failure in cmd_ftrace() due to extra OPT_CALLBACK argument
- [Low] perf: ftrace_parse_retaddr() is dead code and never used for filtering
- [Low] perf: missing document title underline breaks asciidoc generation --

commit 5dab431981f96da9cbf4caa8e862a4c0e461a1ff
Author: Aaron Tomlin <[email protected]>

perf ftrace: Support filtering return address comments

This commit introduces support for filtering out return address comments from function graph trace output via a standalone --filter-retaddr flag. It adds parsing and filtering logic, updates the command-line interface, and includes a dedicated unit test suite.

tools/perf/Documentation/perf-ftrace.txt line 2

Quoted 7 lines
tools/perf/Documentation/perf-ftrace.txt +0 -1
index 3f3808e513fe8..3dbb7e8e783e9 100644
@@ -1,5 +1,4 @@
 perf-ftrace(1)
-==============

[Severity: Low]
Does removing the document title underline break the asciidoc generation?

Asciidoctor strictly requires the document title to be underlined with equal signs of at least the same length as the text. Removing it renders the document malformed.

[ ... ]

Quoted 4 lines
tools/perf/builtin-ftrace.c +0 -0
index d6604be593a34..f3221089a4793 100644

[ ... ]

tools/perf/builtin-ftrace.c int cmd_ftrace(int argc, const char **argv) · line 1837

Quoted 6 lines
tools/perf/builtin-ftrace.c +1 -0
@@ -1827,11 +1833,14 @@ int cmd_ftrace(int argc, const char **argv)
 		     "Set nograph filter on given functions", parse_filter_func),
 	OPT_CALLBACK(0, "graph-opts", &ftrace, "options",
 		     "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,verbose,thresh=<n>,depth=<n>",
+		     "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,noduration,verbose,thresh=<n>,depth=<n>,tail",
 		     parse_graph_tracer_opts),

[Severity: Low]
Does this extra string literal cause a build failure?

The OPT_CALLBACK macro for graph-opts was modified by adding a new help string literal without removing the old one. Because they are separated by a comma, the C preprocessor treats them as two distinct arguments, causing the macro to be invoked with 7 arguments instead of 6.

[ ... ]

Quoted 5 lines
tools/perf/util/ftrace.c new file +0 -0
new file mode 100644
index 0000000000000..344d36293f5ca

[ ... ]

tools/perf/util/ftrace.c

Quoted 2 lines
tools/perf/util/ftrace.c new file +2 -0
+bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset)
+{

[Severity: Low]
Is ftrace_parse_retaddr() ever used outside of the test suite?

The commit message claims this function is implemented to robustly match return address comments before stripping them. However, looking at the implementation of ftrace_filter_retaddr() below:

[ ... ]

tools/perf/util/ftrace.c line 76

Quoted 9 lines
tools/perf/util/ftrace.c new file +9 -0
+void ftrace_filter_retaddr(char *str)
+{
+	char *start = strstr(str, "/* <-");
+	char *end;
+
+	if (!start)
+		return;
+
+	end = strstr(start, "*/");

It appears ftrace_filter_retaddr() performs string replacement using strstr and memmove directly, bypassing ftrace_parse_retaddr() entirely. This leaves the parsing function unused in the actual execution path.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#11 lore raw

Re: [PATCH perf-tools-next v3 4/5] perf ftrace: Support omitting execution duration in function graph tracer

In reply to Aaron Tomlin
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Low] perf: formatting corruption in make_blank_prefix() with verbose enabled - [Low] perf: profile…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Low] perf: formatting corruption in make_blank_prefix() with verbose enabled
- [Low] perf: profile subcommand silently drops records with noduration
--

commit d0f3a0ada6164d2372b897fe5b75d0f425aca5e4
Author: Aaron Tomlin <[email protected]>

perf ftrace: Support omitting execution duration in function graph tracer

The Linux kernel's function graph tracer displays the execution duration of completed functions by default via the funcgraph-duration trace option. Whilst informative, in scenarios where the user is solely interested in the call graph topology, call flow sequence, or inlined hierarchy, the duration column can add visual clutter and widen each trace line.

[ ... ]

Quoted 4 lines
tools/perf/builtin-ftrace.c +0 -0
index 504007064e240..926726c5d9a88 100644

[ ... ]

tools/perf/builtin-ftrace.c static int parse_graph_tracer_opts(const struct option *opt, · line 1783

Quoted 5 lines
tools/perf/builtin-ftrace.c +1 -0
@@ -1754,6 +1780,7 @@ static int parse_graph_tracer_opts(const struct option *opt,
 		{ .name = "retaddr",		.value_ptr = &ftrace->graph_retaddr },
 		{ .name = "nosleep-time",	.value_ptr = &ftrace->graph_nosleep_time },
 		{ .name = "noirqs",		.value_ptr = &ftrace->graph_noirqs },
+		{ .name = "noduration",		.value_ptr = &ftrace->graph_noduration },

[Severity: Low]
Does adding noduration here allow it to be used with the profile subcommand?

The profile subcommand inherently requires function duration to compute statistics. When the duration column is omitted, parse_func_duration() sees no " us" suffix and silently drops the record:

tools/perf/builtin-ftrace.c:parse_func_duration() {
    ...
	if (strncmp(p, " us", 3)) {
		pr_debug("non-usec time found.. ignoring\n");
		return 0;
	}
    ...
}

This results in an empty output instead of rejecting the invalid option combination. Should the option be rejected when used with profile?

Quoted 3 lines
tools/perf/builtin-ftrace.c +0 -0
 		{ .name = "verbose",		.value_ptr = &ftrace->graph_verbose },
 		{ .name = "thresh",		.value_ptr = &ftrace->graph_thresh },
 		{ .name = "depth",		.value_ptr = &ftrace->graph_depth },

[ ... ]

Quoted 4 lines
tools/perf/util/ftrace.c +0 -0
index 8737bb52921e1..5cae01a28529f 100644

[ ... ]

Quoted 4 lines
tools/perf/util/ftrace.c +0 -0
@@ -346,9 +348,14 @@ static const char *find_func_column(const struct perf_ftrace *ftrace,
 			return NULL;
 	}
 

tools/perf/util/ftrace.c static const char *find_func_column(const struct perf_ftrace *ftrace, · line 358

Quoted 10 lines
tools/perf/util/ftrace.c +7 -2
-	p = strchr(p + 1, '|');
-	if (!p)
+	if (!ftrace->graph_noduration) {
+		p = strchr(p + 1, '|');
+		if (!p)
+			return NULL;
+	} else if (strstr(p, " => ")) {
+		/* Context switch marker: " CPU) comm-pid => comm-pid" */
 		return NULL;
+	}

[Severity: Low]
Will this corrupt the comm-pid column when verbose is enabled alongside noduration?

When both verbose and noduration are enabled, find_func_column() returns the CPU ')' or the separator after it. make_blank_prefix() uses the final pipe_char and overwrites everything after the ')' with spaces:

tools/perf/util/ftrace.c:make_blank_prefix() {
    ...
	/* Replace duration between ')' and '|' with spaces */
	if (paren && paren < line + len) {
		char *end = buf + (pipe_char - line);
    ...
		for (c = buf + (paren - line) + 1; c < end; c++)
			*c = ' ';
	}
    ...
}

Since there is no duration field, does this blindly overwrite the comm-pid column and its trailing separator?

Quoted 3 lines
tools/perf/util/ftrace.c +0 -0
 	*cpu = (int)val;
 	return p;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
#2 lore raw

Re: [PATCH perf-tools-next v3 1/5] perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming

In reply to Aaron Tomlin
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] perf: dropped stdout error check causes infinite loop on EPIPE - [Medium] perf: uninitialized…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] perf: dropped stdout error check causes infinite loop on EPIPE
- [Medium] perf: uninitialized heap memory read in __cmd_ftrace() strbuf
--

commit 0a06afad8e6a1e6bf21a4b8a3c19582ed3d99cd1
Author: Aaron Tomlin <[email protected]>

perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming

This commit refactors __cmd_ftrace() to use dynamic heap allocation for the read buffer and a strbuf to accumulate streamed characters into lines. This reduces the stack frame size by over 96%.

Quoted 4 lines
tools/perf/builtin-ftrace.c +0 -0
index 6017493ff1797..d6604be593a34 100644

[ ... ]

Quoted 5 lines
tools/perf/builtin-ftrace.c +0 -1
@@ -810,11 +821,19 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 			break;
 
 		if (pollfd.revents & POLLIN) {
-			int n = read(trace_fd, buf, sizeof(buf));

tools/perf/builtin-ftrace.c static int __cmd_ftrace(struct perf_ftrace *ftrace) · line 830

Quoted 10 lines
tools/perf/builtin-ftrace.c +6 -3
-			if (n < 0)
-				break;
-			if (fwrite(buf, n, 1, stdout) != 1)
+			int n = read(trace_fd, buf, TRACE_BUF_SIZE);
+
+			if (n <= 0)
 				break;
+			for (int i = 0; i < n; i++) {
+				if (buf[i] == '\n') {
+					fprintf(stdout, "%s\n", linebuf.buf);

[Severity: Medium]
Can this print uninitialized heap memory?

Earlier in this function, strbuf_init() allocates the buffer but does not explicitly null-terminate it:

tools/perf/util/strbuf.c:strbuf_grow() {
    ...
    buf = realloc(sb->alloc ? sb->buf : NULL, nr * sizeof(*buf));
    if (!buf)
        return -ENOMEM;

    sb->buf = buf;
    sb->alloc = nr;
    return 0;
}

If the very first character read from trace_pipe is '\n' (for example, from an empty trace line), linebuf.buf is passed to fprintf() as a string before any null byte has been written to it.

[Severity: Medium]
Does the removal of the stdout error check cause an infinite loop?

The original code explicitly checked if fwrite() failed to gracefully exit the polling loop if stdout closes (for example, when the pager exits or when piped to head).

Because trace_pipe provides endless continuous data while tracing is active, ignoring the fprintf() error forces the tool to endlessly read trace data and discard it, never terminating.

Quoted 9 lines
tools/perf/builtin-ftrace.c +6 -0
+					strbuf_setlen(&linebuf, 0);
+				} else {
+					if (strbuf_addch(&linebuf, buf[i]) < 0)
+						goto out_close_fd;
+				}
+			}
 			/* flush output since stdout is in full buffering mode due to pager */
 			fflush(stdout);
 		}

[ ... ]

Quoted 4 lines
tools/perf/builtin-ftrace.c +0 -0
@@ -832,15 +851,31 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 
 	/* read remaining buffer contents */
 	while (true) {

tools/perf/builtin-ftrace.c static int __cmd_ftrace(struct perf_ftrace *ftrace) · line 860

Quoted 10 lines
tools/perf/builtin-ftrace.c +5 -3
-		int n = read(trace_fd, buf, sizeof(buf));
+		int n = read(trace_fd, buf, TRACE_BUF_SIZE);
+
 		if (n <= 0)
 			break;
-		if (fwrite(buf, n, 1, stdout) != 1)
-			break;
+		for (int i = 0; i < n; i++) {
+			if (buf[i] == '\n') {
+				fprintf(stdout, "%s\n", linebuf.buf);

[Severity: Medium]
Does the lack of a return value check on fprintf() also cause an infinite loop here on a broken pipe?

The previous code broke out of the loop if fwrite() failed. Without checking the return value of fprintf(), an error writing to stdout is ignored.

Quoted 7 lines
tools/perf/builtin-ftrace.c +7 -0
+				strbuf_setlen(&linebuf, 0);
+			} else {
+				if (strbuf_addch(&linebuf, buf[i]) < 0)
+					goto out_close_fd;
+			}
+		}
+	}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#8 lore raw

Re: [PATCH perf-tools-next v3 3/5] perf ftrace: Support display of inlined functions in function graph tracer

In reply to Aaron Tomlin
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Medium] perf: machine__delete() used instead of machine__put() violating refcount API - [Low] perf…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Medium] perf: machine__delete() used instead of machine__put() violating refcount API
- [Low] perf: missing explicit POSIX libc headers in tools/perf/tests/ftrace.c
- [Low] perf ftrace: make_blank_prefix clobbers comm-pid column in verbose mode --

commit 9964df54e91214dc84e330cbac056548e3af8bc4
Author: Aaron Tomlin <[email protected]>

perf ftrace: Support display of inlined functions in function graph tracer

This commit introduces the --inline option to perf ftrace for the function graph tracer. It utilizes DWARF debug information to dynamically resolve and synthesize inlined function frames directly into the streamed call graph.

Quoted 13 lines
tools/perf/tests/ftrace.c +4 -0
index 596cecee844e3..e856bed00700a 100644
@@ -7,6 +7,10 @@
 #include <unistd.h>
 #include "tests.h"
 #include "debug.h"
+#include "dso.h"
+#include "machine.h"
+#include "map.h"
+#include "symbol.h"
 #include "util/ftrace.h"

[ ... ]

Quoted 3 lines
tools/perf/tests/ftrace.c +3 -0
+static int test_inline_processing(void)
+{
+	static const char line_entry[] = " 0)               |  wake_up_new_task() {";

[ ... ]

Quoted 12 lines
tools/perf/tests/ftrace.c +12 -0
+	struct perf_ftrace ftrace;
+	char *out_buf = NULL;
+	size_t out_len = 0;
+	FILE *out;
+	const char *unmatched_pos;
+	const char *inline_exit_pos;
+	int ret = TEST_OK;
+
+	memset(&ftrace, 0, sizeof(ftrace));
+	ftrace.use_inline = true;
+	ftrace.filter_retaddr = 1; /* Default when --graph-opts retaddr is not specified */
+	ftrace.tracer = "function_graph";

[ ... ]

tools/perf/tests/ftrace.c

Quoted 1 line
tools/perf/tests/ftrace.c +1 -0
+		out = open_memstream(&out_buf, &out_len);

[Severity: Low]
Is it possible that test_inline_processing() and other functions in this test file will fail to compile under musl libc?
The perf subsystem guidelines require files using POSIX standard libc functions, constants, or variables to directly include the corresponding headers. This code makes heavy use of string/memory functions like memset() and file streams like open_memstream(), but the file does not include <string.h> or <stdio.h>. Could these headers be explicitly included to avoid relying on implicit glibc namespace pollution?

[ ... ]

Quoted 5 lines
tools/perf/util/ftrace.c +0 -0
index 344d36293f5ca..8737bb52921e1 100644
@@ -1,13 +1,82 @@

[ ... ]

Quoted 12 lines
tools/perf/util/ftrace.c +12 -0
+static void make_blank_prefix(const char *line, const char *pipe_char, char *buf, size_t size)
+{
+	size_t len = pipe_char - line + 1;
+	const char *paren = strchr(line, ')');
+	char *c;
+
+	if (len >= size)
+		len = size - 1;
+
+	memcpy(buf, line, len);
+	buf[len] = '\0';
+

tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 303

Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+	/* Replace duration between ')' and '|' with spaces */
+	if (paren && paren < line + len) {
+		char *end = buf + (pipe_char - line);
+
+		if (end > buf + len)
+			end = buf + len;
+		for (c = buf + (paren - line) + 1; c < end; c++)
+			*c = ' ';
+	}
+}

[Severity: Low]
Does make_blank_prefix() inadvertently erase the comm-pid string when in verbose mode?
In function_graph verbose mode (--graph-opts verbose), the trace output includes a comm-pid column and a duration column between the CPU number and the function name. This loop overwrites everything between the closing parenthesis of the CPU number and the final pipe delimiter before the function name with spaces. While this correctly blanks out the duration column, it looks like it might unintentionally clobber the task comm-pid string and its pipe delimiter as well, breaking visual alignment.

[ ... ]

tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 702

Quoted 8 lines
tools/perf/util/ftrace.c +8 -0
+void perf_ftrace__cleanup_inlines(struct perf_ftrace *ftrace)
+{
+	if (ftrace->machine) {
+		machine__delete(ftrace->machine);
+		ftrace->machine = NULL;
+	}
+	ftrace_cpu_states__clear();
+}

[Severity: Medium]
Should machine__put() be used here instead of machine__delete()? The machine reference is allocated using machine__new_host() earlier in perf_ftrace__setup_inlines(). The perf subsystem guidelines mandate strict reference count checking, stating that every reference handle allocated via _new() must be strictly paired with a matching _put(). Bypassing the refcount API by calling machine__delete() directly can lead to memory leaks, use-after-free traps, and LSAN failures.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#3 lore raw

Re: [PATCH perf-tools-next v3 1/5] perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming

In reply to [email protected]
Quoted 17 lines · On Mon, Oct 12, 2026 at 12:41:36AM +0000, [email protected] wrote:

[Severity: Medium]
Can this print uninitialized heap memory?

Earlier in this function, strbuf_init() allocates the buffer but does not explicitly null-terminate it:

tools/perf/util/strbuf.c:strbuf_grow() {
    ...
    buf = realloc(sb->alloc ? sb->buf : NULL, nr * sizeof(*buf));
    if (!buf)
        return -ENOMEM;

    sb->buf = buf;
    sb->alloc = nr;
    return 0;
}
Quoted 3 lines

If the very first character read from trace_pipe is '\n' (for example, from an empty trace line), linebuf.buf is passed to fprintf() as a string before any null byte has been written to it.

So, when strbuf_init() allocates memory via strbuf_grow(), realloc() does not NUL-terminate sb->buf[0], leaving it uninitialised if an initial newline is processed immediately.

This will be addressed by explicitly initialising the buffer with strbuf_setlen(&linebuf, 0) immediately after strbuf_init(&linebuf, 512). For example, following the pattern in tools/perf/util/pfm.c:

    strbuf_init(&linebuf, 512);
    strbuf_setlen(&linebuf, 0);

This guarantees linebuf.buf is a valid empty string at offset 0 while preserving the pre-allocated capacity.

Quoted 10 lines

[Severity: Medium]
Does the removal of the stdout error check cause an infinite loop?

The original code explicitly checked if fwrite() failed to gracefully exit the polling loop if stdout closes (for example, when the pager exits or when piped to head).

Because trace_pipe provides endless continuous data while tracing is active, ignoring the fprintf() error forces the tool to endlessly read trace data and discard it, never terminating.

Indeed it does. When trace_pipe is active, read(trace_fd, ...) continuously yields data regardless of the stdout state. So, relying solely on (n <= 0) never triggers EOF or broken pipe detection.

This will be addressed in the next iteration.

Signature · 1 line
Aaron Tomlin
#6 lore raw

Re: [PATCH perf-tools-next v3 2/5] perf ftrace: Support filtering return address comments

In reply to [email protected]
Quoted 11 lines · On Mon, Oct 12, 2026 at 12:38:24AM +0000, [email protected] wrote:

tools/perf/Documentation/perf-ftrace.txt line 2

Quoted 7 lines
tools/perf/Documentation/perf-ftrace.txt +0 -1
index 3f3808e513fe8..3dbb7e8e783e9 100644
@@ -1,5 +1,4 @@
 perf-ftrace(1)
-==============

[Severity: Low]
Does removing the document title underline break the asciidoc generation?

Quoted 3 lines

Asciidoctor strictly requires the document title to be underlined with equal signs of at least the same length as the text. Removing it renders the document malformed.

This will be restored in the next iteration.

Quoted 15 lines
Quoted 4 lines
tools/perf/builtin-ftrace.c +0 -0
index d6604be593a34..f3221089a4793 100644

[ ... ]

tools/perf/builtin-ftrace.c int cmd_ftrace(int argc, const char **argv) · line 1837

Quoted 6 lines
tools/perf/builtin-ftrace.c +1 -0
@@ -1827,11 +1833,14 @@ int cmd_ftrace(int argc, const char **argv)
 		     "Set nograph filter on given functions", parse_filter_func),
 	OPT_CALLBACK(0, "graph-opts", &ftrace, "options",
 		     "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,verbose,thresh=<n>,depth=<n>",
+		     "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,noduration,verbose,thresh=<n>,depth=<n>,tail",
 		     parse_graph_tracer_opts),

[Severity: Low]
Does this extra string literal cause a build failure?

Quoted 4 lines

The OPT_CALLBACK macro for graph-opts was modified by adding a new help string literal without removing the old one. Because they are separated by a comma, the C preprocessor treats them as two distinct arguments, causing the macro to be invoked with 7 arguments instead of 6.

Yes. OPT_CALLBACK() accepts 6 arguments. So, the accidental duplicate string literal caused a preprocessor failure ("macro 'OPT_CALLBACK' passed 7 arguments, but takes just 6"), breaking bisectability.

I will revert the unintended edit to OPT_CALLBACK(0, "graph-opts", ...) in patch 2/5. Also, replace the help string in patch 4/5 where noduration is actually introduced.

Quoted 9 lines

[Severity: Low]
Is ftrace_parse_retaddr() ever used outside of the test suite?

The commit message claims this function is implemented to robustly match return address comments before stripping them. However, looking at the implementation of ftrace_filter_retaddr() below:

[ ... ]

Quoted 13 lines

tools/perf/builtin-ftrace.c int cmd_ftrace(int argc, const char **argv) · line 1846

Quoted 9 lines
tools/perf/builtin-ftrace.c +9 -0
+void ftrace_filter_retaddr(char *str)
+{
+	char *start = strstr(str, "/* <-");
+	char *end;
+
+	if (!start)
+		return;
+
+	end = strstr(start, "*/");

It appears ftrace_filter_retaddr() performs string replacement using strstr and memmove directly, bypassing ftrace_parse_retaddr() entirely. This leaves the parsing function unused in the actual execution path.

ftrace_parse_retaddr() is used in patch 3/5 to parse caller_sym and caller_offset for DWARF inlined function resolution, as well as in patch 5/5 to validate caller hierarchies when filtering with -G.

Also, ftrace_filter_retaddr() intentionally does not invoke ftrace_parse_retaddr() because comment stripping only requires locating the "/* <-" and "*/" delimiters to perform an in-place memmove(); decoding the symbol name and hex offset would add unnecessary overhead.

The commit message was poorly worded, conflating the stripping logic with the parsing helper. I will clarify the commit message in patch 2/5 to explicitly state that ftrace_parse_retaddr() is introduced as the parsing primitive for subsequent inline resolution.

Signature · 1 line
Aaron Tomlin
#9 lore raw

Re: [PATCH perf-tools-next v3 3/5] perf ftrace: Support display of inlined functions in function graph tracer

In reply to [email protected]
Quoted 9 lines · On Mon, Oct 12, 2026 at 12:43:38AM +0000, [email protected] wrote:

[Severity: Low]
Is it possible that test_inline_processing() and other functions in this test file will fail to compile under musl libc?
The perf subsystem guidelines require files using POSIX standard libc functions, constants, or variables to directly include the corresponding headers. This code makes heavy use of string/memory functions like memset() and file streams like open_memstream(), but the file does not include <string.h> or <stdio.h>. Could these headers be explicitly included to avoid relying on implicit glibc namespace pollution?

Both <stdio.h> and <string.h> are already directly included at the top of tools/perf/tests/ftrace.c (lines 4 and 6):

      #include <stdio.h>
      #include <stdlib.h>
      #include <string.h>
      #include <unistd.h>

They were introduced in patch 2/5 when the test file was created.

Quoted 13 lines

[Severity: Low]
Does make_blank_prefix() inadvertently erase the comm-pid string when in verbose mode?
In function_graph verbose mode (--graph-opts verbose), the trace output includes a comm-pid column and a duration column between the CPU number and the function name. This loop overwrites everything between the closing parenthesis of the CPU number and the final pipe delimiter before the function name with spaces. While this correctly blanks out the duration column, it looks like it might unintentionally clobber the task comm-pid string and its pipe delimiter as well, breaking visual alignment.

[ ... ]

Quoted 8 lines

tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 702

Quoted 8 lines
tools/perf/util/ftrace.c +8 -0
+void perf_ftrace__cleanup_inlines(struct perf_ftrace *ftrace)
+{
+	if (ftrace->machine) {
+		machine__delete(ftrace->machine);
+		ftrace->machine = NULL;
+	}
+	ftrace_cpu_states__clear();
+}

Yes, it does indeed clobber comm-pid. The loop assumed the duration column always started immediately after the CPU ")", which is only true in default layout mode. In verbose mode, that span includes "comm-pid | duration", wiping out both the process information and its column separator.

This will be addressed in the next iteration by:
1. Passing ftrace context to make_blank_prefix().

2. In verbose mode, searching backwards from pipe_char for the preceding "|" so that only the duration column between the two pipe delimiters is blanked out, preserving comm-pid.

3. Returning early if graph_noduration is set, as no duration column exists to blank out.

Quoted 8 lines

[Severity: Medium]
Should machine__put() be used here instead of machine__delete()? The machine reference is allocated using machine__new_host() earlier in perf_ftrace__setup_inlines(). The perf subsystem guidelines mandate strict reference count checking, stating that every reference handle allocated via _new() must be strictly paired with a matching _put(). Bypassing the refcount API by calling machine__delete() directly can lead to memory leaks, use-after-free traps, and LSAN failures.

machine__put() does not exist. Using machine__delete() is correct as seen throughout tools/perf e.g. __cmd_kallsyms() in tools/perf/builtin-kallsyms.c.

Signature · 1 line
Aaron Tomlin
#12 lore raw

Re: [PATCH perf-tools-next v3 4/5] perf ftrace: Support omitting execution duration in function graph tracer

In reply to [email protected]
Quoted 16 lines · On Mon, Oct 12, 2026 at 12:39:20AM +0000, [email protected] wrote:

[Severity: Low]
Does adding noduration here allow it to be used with the profile subcommand?

The profile subcommand inherently requires function duration to compute statistics. When the duration column is omitted, parse_func_duration() sees no " us" suffix and silently drops the record:

tools/perf/builtin-ftrace.c:parse_func_duration() {
    ...
	if (strncmp(p, " us", 3)) {
		pr_debug("non-usec time found.. ignoring\n");
		return 0;
	}
    ...
}
Quoted 2 lines

This results in an empty output instead of rejecting the invalid option combination. Should the option be rejected when used with profile?

Yes. Since profiling requires execution duration to compute min/max/average statistics and call counts, combining --graph-opts noduration with the profile subcommand is nonsensical and produces empty results.

In the next iteration, I have updated cmd_ftrace() in tools/perf/builtin-ftrace.c to explicitly reject this combination:

    case PERF_FTRACE_PROFILE:
        if (ftrace.graph_noduration) {
            pr_err("Error: 'noduration' option cannot be used with profile\n");
            ret = -EINVAL;
            goto out_delete_filters;
        }
        cmd_func = __cmd_profile;
        break;
Quoted 20 lines

[Severity: Low]
Will this corrupt the comm-pid column when verbose is enabled alongside noduration?

When both verbose and noduration are enabled, find_func_column() returns the CPU ')' or the separator after it. make_blank_prefix() uses the final pipe_char and overwrites everything after the ')' with spaces:

tools/perf/util/ftrace.c:make_blank_prefix() {
    ...
	/* Replace duration between ')' and '|' with spaces */
	if (paren && paren < line + len) {
		char *end = buf + (pipe_char - line);
    ...
		for (c = buf + (paren - line) + 1; c < end; c++)
			*c = ' ';
	}
    ...
}
Quoted 2 lines

Since there is no duration field, does this blindly overwrite the comm-pid column and its trailing separator?

In the initial implementation, make_blank_prefix() did indeed assume the duration column followed CPU ")" and would have clobbered comm-pid.

This has been resolved in the next iteration:
1. In make_blank_prefix(), we now return immediately when graph_noduration is set:

memcpy(buf, line, len);
buf[len] = '\0';

if (ftrace->graph_noduration)
return;

Because no duration column is present, no characters need to be blanked, leaving comm-pid untouched.

2. In verbose mode with duration enabled, make_blank_prefix() scans backwards from pipe_char for the preceding "|" delimiter, blanking only between the comm-pid delimiter and the final delimiter.

3. Unit test coverage has been added to tools/perf/tests/ftrace.c verifying that comm-pid is preserved in both verbose and verbose-noduration modes.

Signature · 1 line
Aaron Tomlin
#15 lore raw

Re: [PATCH perf-tools-next v3 5/5] perf ftrace: Filter extraneous functions in function graph tracer

In reply to [email protected]
Quoted 9 lines · On Mon, Oct 12, 2026 at 12:37:54AM +0000, [email protected] wrote:

[Severity: Medium]
When a targeted function is a leaf call (no braces, only a semicolon) and occurs at a call stack depth greater than 0, doesn't the hardcoded check for cs->depth == 0 prevent the filter state from resetting?

It appears that cs->in_target will remain true indefinitely if the target is not at the root depth, causing subsequent non-targeted sibling functions and their descendants within the same parent frame to bypass the filter and incorrectly appear in the trace output.

[ ... ]

Quoted 4 lines

[Severity: Medium]
The identical logic exists here for the use_inline code path. Should these checks verify cs->depth == cs->target_root_depth instead of cs->depth == 0 to properly reset the filter for leaf targets at non-zero depths?

Yes. When a targeted function is a leaf call occurring within an un-targeted parent frame (depth > 0), cs->target_root_depth records the depth of that leaf call, but the leaf call does not push a stack frame. The hardcoded cs->depth == 0 check therefore failed to trigger, leaving cs->in_target stuck at true and allowing subsequent sibling calls within the parent frame to leak into the output until the parent frame exited.

I have updated both code paths (with and without --inline) in tools/perf/util/ftrace.c to check cs->depth <= cs->target_root_depth and clear cs->target_root_depth:

    if (has_filter && cs->in_target && cs->depth <= cs->target_root_depth &&
        has_semicolon && !has_open_brace) {
        cs->in_target = false;
        cs->target_root_depth = 0;
        cs->orphan_depth = 0;
    }

This mirrors the exit delimiter ("}") logic and ensures that:
1. Leaf target functions executing at depth > 0 immediately reset the filter state, suppressing any following sibling calls.

2. Child leaf calls executing inside a non-leaf target function (where cs->depth > cs->target_root_depth) do not inadvertently clear the target filter.

I have also added test coverage in test_graph_target_filter() in tools/perf/tests/ftrace.c to verify that targeted leaf calls at depth > 0 suppress following sibling functions both with and without --inline.

Signature · 1 line
Aaron Tomlin