Skip to contentSkip to conversation

perf ftrace: Support inlined functions and display enhancements in function graph tracer

PATCH Applied Aaron Tomlin 3 participants 38 messages Oct 26 – Nov 4 perf events

v12 of 13 Applied Latest is v13 →

lore
Share & tools
lore
#0 lore raw

v12 cover perf ftrace: Support inlined functions and display enhancements in function graph tracer

The Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining (such as the scheduler, memory management, locking primitives, and RCU).

When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment (e.g. /* <-wake_up_new_task+0x1d1/0x3e0 */). By interrogating DWARF debug information from the kernel image (vmlinux) utilising libdw, perf ftrace resolves this return address to its inlined callchain and synthesises the intermediate inlined frames directly into the streamed call graph.

This patch series addresses these challenges across ten commits covering six key areas:

1. Signal handling and stack frame optimisation in __cmd_ftrace()

Separates robust signal handling (EINTR), non-blocking read retries, and pager exit handling into a dedicated patch, and replaces the 4096-byte stack array in __cmd_ftrace() with dynamic heap allocation alongside struct strbuf streaming, reducing the stack frame footprint by over 96% (down to ~160 bytes) while ensuring complete line-delimited records.

2. Return address comment filtering

Introduces ftrace_parse_retaddr() and ftrace_filter_retaddr() to automatically strip caller comments from trace output unless explicitly requested via --graph-opts retaddr, accompanied by a unit test suite in tools/perf/tests/ftrace.c.

3. Preparatory vmlinux lookup helper

Introduces vmlinux_path__find() in tools/perf/util/symbol.c and symbol.h to search standard vmlinux paths and return the first accessible kernel image, providing a generic helper across tools/perf and avoiding hardcoded access() checks in tests.

4. Inlined function call graph reconstruction

Introduces --inline and -k/--vmlinux, dynamically synthesising inlined frames marked with /* (inline) */. Correctly matches bare closing braces when funcgraph-tail is disabled, clamps stack depth to prevent indentation drift, and expands the automated test suite to verify inline resolution, comment filtering, and stack saturation recovery utilising vmlinux_path__find().

5. Execution duration suppression

Introduces --graph-opts noduration (accepting noduration=[0|1]) to toggle the kernel's funcgraph-duration tracing option, omitting the duration column when the user is focused purely on call graph topology and inlined hierarchy. Rejects noduration when invoked under 'perf ftrace profile'.

6. Extraneous function filtering in function graph tracer

Introduces userspace call graph filtering when targeted tracing (-G or --graph-funcs) is active. Suppresses transient unrelated entries arriving before filter activation as well as detached callee trees (such as ring-buffer wakeups) originating outside the target hierarchy. Dynamically resizes the synthetic call stack to eliminate saturation edge cases and accurately suppresses detached subtrees without orphan exit name collisions.

Synthesised inlined functions are rendered with an explicit /* (inline) */ annotation at both entry and exit:

# tracer: function_graph
#
# CPU FUNCTION CALLS
# | | | | |
0) enqueue_entity() {
0) update_cfs_group() { /* (inline) */
0) calc_concur_shares() {
0) tg_cpus() {
0) cpuset_num_cpus() {
0) class_rcu_constructor() { /* (inline) */
0) rcu_read_lock() { /* (inline) */
0) __rcu_read_lock();
0) } /* rcu_read_lock (inline) */
0) } /* class_rcu_constructor (inline) */
0) cgroup_e_css();
0) class_rcu_destructor() { /* (inline) */
0) rcu_read_unlock() { /* (inline) */
0) __rcu_read_unlock();
0) } /* rcu_read_unlock (inline) */
0) } /* class_rcu_destructor (inline) */
0) }
0) }
0) __calc_smp_shares();
0) reweight_entity();
0) } /* update_cfs_group (inline) */
0) }

Changes since v11:

- Replaced the single unsequenced min_raw_indent scalar with a monotonic suffix-minima array of (seq, raw_indent) watermarks so shallow exit watermarks are not lost on subsequent function entries or falsely applied to frames pushed after an earlier shallow exit

- Pruned each stack per-frame against subsequent watermarks on the other state and performed a 2-way sequence-ordered merge of surviving frames and watermarks in ftrace_task_state__rekey() to handle non-contiguous unknown-PID execution across multi-hop CPU migrations

- Tracked orphan subtree base indentation and event sequence numbers (orphan_base_indent and orphan_seq) in struct ftrace_task_state:
- Recovered cleanly when an orphan exit event is dropped inside a nested child of the target root by clearing orphan_depth when an exit matches or is shallower than orphan_base_indent, or when an entry at or shallower than orphan_base_indent descends directly from an active frame on the tracked stack
- Pruned stale orphan_depth across CPU migrations in ftrace_task_state__prune_stack() using subsequent watermarks on the peer state and preserved orphan tracking state across ftrace_task_state__rekey()
- Avoided recording indentation watermarks on detached orphan entries so suppressed orphan calls do not prematurely prune active frames

- Buffered interrupt entry markers ("==========>") in pending_irq_ent while inside a target callgraph and suppressed the corresponding "<==========" return marker (suppress_irq_ret) when the interrupt handler is filtered out as a detached orphan, preventing empty IRQ marker pairs from leaking into filtered output, and updated parse_graph_cpu() to bind the active task state from the comm-pid column in verbose mode

- Unwound stale non-target ancestor frames with raw_indent >= leading_spaces (and any trailing inlined frames) when an unmatched exit (match_idx < 0) occurs outside or resets the target filter state

- Expanded unit tests in tools/perf/tests/ftrace.c (test_graph_target_filter()) to ensure deterministic synthetic inline fallback and cover dropped orphan exits inside nested target children, orphan_depth pruning across task rekeying, IRQ marker suppression around orphan interrupt handlers, and stale ancestor unwinding on unmatched exits

- Link to v11: https://lore.kernel.org/lkml/[email protected]/

Changes since v10:

- Removed duplicated member fields members seq and min_raw_indent from struct ftrace_task_state

- Fixed call stack misordering and active frame pruning during CPU migration in ftrace_task_state__rekey():
- Tracked monotonic event sequence numbers (seq) on each synthetic stack frame
- Replaced whole-state older/newer classification with a split-and-splice
merge, partitioning existing->stack at the splice boundary where frames postdate ts, pruning only pre-ts exited ancestor frames from existing, and splicing ts->stack directly between ancestor and descendant frames via memmove()
- Prevented active intermediate frames entered on unbannered CPUs from being pruned and prevented exited ancestor frames from remaining on the stack when a task migrates and returns before a banner is emitted
- Added unit test scenario in tools/perf/tests/ftrace.c validating migration across unbannered CPUs and delayed context-switch rekeying

- Added tags "Tested-by:" and "Reviewed-by:" (Ian Rogers)

- Link to v10: https://lore.kernel.org/all/[email protected]/

Changes since v9:

- Reorganised the patch series from 6 to 10 commits following review feedback:
- Split error handling (signals on EINTR, non-blocking read retries, and pager/pipe exits) out of the strbuf streaming commit into a dedicated preparatory commit ("perf ftrace: Handle signals, non-blocking reads and pager exits in __cmd_ftrace()") (Namhyung Kim)
- Decomposed the monolithic inlined functions commit into four focused, bisectable commits covering basic task stack management, context-switch tracking, inlined function display, and automated unit tests (Namhyung Kim)

- Removed the standalone --filter-retaddr command-line option and --graph-opts filter-retaddr flag; return address comments are now automatically stripped from function graph output unless explicitly requested via --graph-opts retaddr (Namhyung Kim)

- Link to v9: https://lore.kernel.org/lkml/[email protected]/

Changes since v8:

- Preserved funcgraph-retval return value comments when filtering funcgraph-retaddr comments (/* <-caller ret=0x0 */ -> /* ret=0x0 */) in ftrace_filter_retaddr()

- Enhanced vmlinux_path__find() helper in tools/perf/util/symbol.c:
- Avoided redundant memory allocation in vmlinux_path__init() when vmlinux_path is already initialised
- Ensured symbol configuration is initialised via symbol__init() prior to path resolution
- Guaranteed vmlinux_path memory cleanup by invoking vmlinux_path__exit() unconditionally in symbol__exit()

- Tracked raw (pre-inline-expansion) indentation on synthetic stack frames, ensuring bare closing braces ("}") are accurately matched against corresponding non-inlined frames without indentation distortion

- Unwound stale frames on untracked caller entries (caller_idx < 0) whose raw indentation is greater than or equal to the new entry, and closed trailing open inlined frames resulting from dropped exit events

- Pre-bound single-PID tracing across all CPUs in perf_ftrace__setup_inlines(), ensuring consistent synthetic call stack tracking across CPU migration when kernel context-switch banners are suppressed (verif_pid prev_pid == -1)

- Strengthened error handling during DWARF/vmlinux initialisation in perf_ftrace__setup_inlines(), checking load return codes, emitting diagnostic warnings, and safely freeing ftrace->machine on failure

- Recognised interrupt/IRQ entry and exit markers ("=====>" and "<=====") alongside context-switch markers under --graph-opts noduration, preventing non-function lines from corrupting function column detection or popping active stack frames

- Handled leaf function lines with trailing comments
(e.g. funcgraph-retaddr or funcgraph-retval comments) in parse_func_duration() for 'perf ftrace profile', ensuring parentheses are stripped cleanly and leaf records match non-leaf entries in duration tracking

- Preserved tracked ancestor frames on cs->stack across dropped target root exits by verifying caller provenance
(caller_idx < cs->target_root_depth) before checking indentation bounds in ftrace_check_target_caller()

- Purged stale non-target frames below target root upon target root entry when caller_idx < 0, preventing unrelated ancestor frames from prematurely terminating subsequent target calls

- Prevented cs->orphan_depth lockouts when both an orphan exit and enclosing target root exit are dropped by resetting cs->orphan_depth on any line at or outside target root indentation (leading_spaces <= cs->target_root_indent), and allowing ancestor exits to fall through to match frames on cs->stack

- Preserved interrupt/IRQ markers inside active target callgraphs while suppressing those outside target filters under -G or --graph-funcs

- Fixed test_stack_saturation() to avoid premature frame unwinding, ensuring depth clamping and recovery are tested as intended

- Expanded automated test coverage in tools/perf/tests/ftrace.c:
- Added test_context_switch_handling() for CPU migration and task tracking
- Added test_column_layouts() covering standard, verbose, and noduration formats including IRQ and context-switch markers
- Added regression tests in test_graph_target_filter() covering double dropped exits, ancestor frame preservation, leaf duration parsing, and deep orphan recovery

- Link to v8: https://lore.kernel.org/lkml/[email protected]/

Changes since v7:

- Rebased onto latest perf-tools-next/perf-tools-next (commit 1c90f0a3c8a1 ("perf kvm: Cast tv_usec to long to fix build on MIPS 32-bits"))

- Link to v7: https://lore.kernel.org/all/[email protected]/

Changes since v6:

- Expanded the series from 5 to 6 patches by introducing a standalone preparatory commit ("perf symbol: Introduce vmlinux_path__find() helper") in tools/perf/util/symbol.c and symbol.h to locate accessible kernel images via configured vmlinux paths, replacing ad-hoc access("vmlinux", R_OK) checks in unit tests (Ian Rogers)

- Explicitly included stdio.h in tools/perf/util/ftrace.h to declare the FILE type for ftrace_process_fgraph_line(), ensuring clean compilation under musl libc and non-glibc environments

- Prevented name collisions between orphan callee exits and active target frames in ftrace_process_fgraph_line()

- Allowed dropped-orphan exit recovery on closing braces only when the exit explicitly names the target root function itself, ensuring that lost orphan exits do not swallow the target root's exit while protecting intermediate callee frames from collisions

- Replaced the fixed-size array in struct ftrace_task_state with a dynamically resized stack grown via realloc(), completely eliminating cs->overflow_depth and its associated failure modes:
- Fixed skipped filter teardown when a target function begins at the stack limit, ensuring standard teardown runs upon root exit and preventing subsequent sibling functions from leaking into filtered output

- Fixed orphan filter bypass during deep call stacks by preserving all caller frames on the stack, ensuring unknown callers (caller_idx < 0) are accurately identified as orphans and suppressed

- Link to v6: https://lore.kernel.org/all/[email protected]/

Changes since v5:

- Prevented call stack desynchronisation on dropped orphan exit events by clearing orphan_depth when a named exit matches a tracked frame or an entry descends from the target callgraph

- Fixed premature target filter termination on child exits by capturing cs->target_root_depth after unwinding stale frames above the caller

- Ensured synthesised caller inlines above cs->target_root_depth are cleanly closed upon target function exit before resetting filter state

- Prevented legitimate deep calls exceeding FTRACE_MAX_STACK from being falsely dropped as orphans by checking cs->overflow_depth

- Expanded unit tests to cover dropped orphan exits, target root depth synchronisation after stale siblings, and deep call stack overflows

- Provided support for glob pattern matching with -G or --graph-funcs

- Link to v5: https://lore.kernel.org/all/[email protected]/

Changes since v4:

- Handled non-fatal EAGAIN, EWOULDBLOCK, and EINTR in __cmd_ftrace() polling and draining loops to prevent premature session termination on non-blocking reads or signal interruptions

- Eliminated an unused local variable endptr in ftrace_parse_retaddr() by passing NULL to strtoull()

- Prevented stack buffer overflows in make_blank_prefix() by clamping the blanking loop bounds and end pointer to the destination buffer length when processing trace records with exceptionally long prefixes

- Tracked call stack saturation with overflow_depth to prevent untracked function exits with bare closing braces from prematurely popping tracked frames from the parser stack on deep call chains exceeding FTRACE_MAX_STACK

- Ensured stale non-inlined callee frames resulting from dropped function exits do not block inactive inlined frames from being closed upon subsequent entries by unwinding the stack down to caller_idx + 1 + common

- Fixed filter state not resetting for targeted leaf calls when --inline is active: synthesised inline frames inflated cs->depth beyond cs->target_root_depth, causing subsequent sibling functions to be falsely classified as detached orphans and bypassing inline unwinding; resolved by tracking target root entries (is_target_root), explicitly unwinding all synthesised inlines, and resetting filter state upon leaf completion

- Expanded unit test coverage to verify recovery across dropped exit events and bare closing braces, and to verify that targeted leaf calls with synthesised inlined frames properly close inlines and reset filter state

- Link to v4: https://lore.kernel.org/lkml/[email protected]/

Changes since v3:

- Explicitly rejected the noduration option when invoked under the profile subcommand, since profiling inherently requires function duration to compute timing statistics

- Fixed make_blank_prefix() clobbering the comm-pid column in verbose mode by searching backwards for the preceding "|" delimiter, and returning early when graph_noduration is active

- Fixed filter state not resetting for targeted leaf calls at non-zero call stack depths by checking cs->depth <= cs->target_root_depth and clearing cs->target_root_depth, preventing subsequent non-targeted sibling functions within the parent frame from leaking into trace output

- Expanded unit test coverage to verify comm-pid preservation under verbose and verbose-noduration inline modes, and to verify that targeted leaf calls at depth > 0 properly reset the target filter state.

- Link to v3: https://lore.kernel.org/lkml/[email protected]/

Changes since v2:

- Expanded the series from 4 to 5 patches by adding a dedicated commit ("perf ftrace: Filter extraneous functions in function graph tracer") to enforce strict target callgraph boundaries when -G or --graph-funcs is specified

- Suppressed unrelated top-level functions that leak through the transient tracefs filter initialisation window before RCU synchronisation completes

- Filtered detached/orphan calls from appearing as nested children within the target graph by validating caller return addresses against the active target stack

- Tracked nested orphan scope depth (cs->orphan_depth) to cleanly suppress child calls and closing delimiters of detached subtrees without desynchronising the target stack

- Added automated unit test coverage in tools/perf/tests/ftrace.c (test_graph_target_filter()) validating top-level suppression (both with and without --inline) and detached orphan wakeup suppression

- Link to v2: https://lore.kernel.org/lkml/[email protected]/

Changes since v1:

- Modularised the implementation into a 4-patch bisectable series rather than a monolithic commit

- Resolved potential infinite loop on poll() upon trace_pipe EOF

- Fixed stuck call stack frames when funcgraph-tail is disabled by supporting bare closing delimiters matching LIFO stack frames

- Prevented unbounded indentation growth on deep call trees by clamping inline expansion and synchronising cs->inlined_depth to frames successfully pushed onto cs->stack

- Added dedicated unit tests covering bare closing braces and stack saturation recovery

- Refined option documentation in perf-ftrace.txt

- Link to v1: https://lore.kernel.org/lkml/[email protected]/

Aaron Tomlin (10):
perf ftrace: Handle signals, non-blocking reads and pager exits in

    __cmd_ftrace()

perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming perf ftrace: Support filtering return address comments perf ftrace: Add basic task call stack management
perf ftrace: Support context switch tracking in function graph tracer perf ftrace: Support display of inlined functions in function graph
tracer
perf symbol: Introduce vmlinux_path__find() helper
perf test: Add test for ftrace inline and task stack tracking perf ftrace: Support omitting execution duration in function graph
tracer
perf ftrace: Filter extraneous functions in function graph tracer

 tools/perf/Documentation/perf-ftrace.txt |   14 +
 tools/perf/builtin-ftrace.c              |  202 +-
 tools/perf/tests/Build                   |    1 +
 tools/perf/tests/builtin-test.c          |    1 +
 tools/perf/tests/ftrace.c                | 4000 ++++++++++++++++++++++
 tools/perf/tests/tests.h                 |    1 +
 tools/perf/util/Build                    |    1 +
 tools/perf/util/ftrace.c                 | 2053 +++++++++++
 tools/perf/util/ftrace.h                 |   28 +
 tools/perf/util/symbol.c                 |   38 +-
 tools/perf/util/symbol.h                 |    2 +
 11 files changed, 6312 insertions(+), 29 deletions(-)
 create mode 100644 tools/perf/tests/ftrace.c
 create mode 100644 tools/perf/util/ftrace.c
Signature · 1 line
2.55.0
#1 lore raw

PATCH v12 1/10 perf ftrace: Handle signals, non-blocking reads and pager exits in __cmd_ftrace()

In __cmd_ftrace(), trace_fd is configured for non-blocking I/O via fcntl(trace_fd, F_SETFL, O_NONBLOCK) and polled using poll(). However, the polling loop previously treated any negative return code from poll() or read() as an unrecoverable failure.

If poll() or read() is interrupted by a non-fatal signal (i.e. SIGWINCH upon terminal resizing), errno is set to EINTR. Furthermore, read() on trace_pipe can return EAGAIN/EWOULDBLOCK even after poll() indicates POLLIN (for instance, under concurrent readers, spurious wakeups, or block-mode trace iterator options). In all such cases, breaking from the loop causes perf ftrace to terminate tracing prematurely.

Additionally, output errors on stdout (i.e. when a pager or consumer process exits, closing the downstream pipe) were ignored by fflush(), leaving __cmd_ftrace() polling indefinitely until the workload completes.

Resolve these concerns by retrying poll() upon receiving EINTR, retrying read() on EAGAIN, EWOULDBLOCK, and EINTR, and breaking on EOF (n == 0). Furthermore, detect stdout errors and broken pipes via fflush() and ferror() to exit immediately when the downstream output sink closes, and handle EINTR and ferror(stdout) cleanly in the subsequent buffer draining loop.

Fixes d01f4e8db22c ("perf ftrace: Introduce new 'ftrace' tool")
---
 tools/perf/builtin-ftrace.c | 27 ++++++++++++++++++++++-----
 1 file changed, 22 insertions(+), 5 deletions(-)
tools/perf/builtin-ftrace.c +22 -5
index 6017493ff179..7cdc9ae2fbbb 100644
@@ -14,6 +14,7 @@
 #include <poll.h>
 #include <semaphore.h>
 #include <signal.h>
+#include <stdio.h>
 #include <stdlib.h>
 #include <unistd.h>
 #include <linux/capability.h>
@@ -806,17 +807,27 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	}
 
 	while (!done) {
-		if (poll(&pollfd, 1, -1) < 0)
+		if (poll(&pollfd, 1, -1) < 0) {
+			if (errno == EINTR)
+				continue;
 			break;
+		}
 
 		if (pollfd.revents & POLLIN) {
 			int n = read(trace_fd, buf, sizeof(buf));
-			if (n < 0)
+
+			if (n < 0) {
+				if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
+					continue;
+				break;
+			}
+			if (n == 0)
 				break;
 			if (fwrite(buf, n, 1, stdout) != 1)
 				break;
 			/* flush output since stdout is in full buffering mode due to pager */
-			fflush(stdout);
+			if (fflush(stdout) < 0 || ferror(stdout))
+				break;
 		}
 	}
 
@@ -831,9 +842,15 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	}
 
 	/* read remaining buffer contents */
-	while (true) {
+	while (!ferror(stdout)) {
 		int n = read(trace_fd, buf, sizeof(buf));
-		if (n <= 0)
+
+		if (n < 0) {
+			if (errno == EINTR)
+				continue;
+			break;
+		}
+		if (n == 0)
 			break;
 		if (fwrite(buf, n, 1, stdout) != 1)
 			break;
Signature · 1 line
2.55.0
#2 lore raw

Re: [PATCH perf-tools-next v12 01/10] perf ftrace: Handle signals, non-blocking reads and pager exits in __cmd_ftrace()

Sashiko has reviewed this patch and found no issues.
Quoted 24 lines

In __cmd_ftrace(), trace_fd is configured for non-blocking I/O via fcntl(trace_fd, F_SETFL, O_NONBLOCK) and polled using poll(). However, the polling loop previously treated any negative return code from poll() or read() as an unrecoverable failure.

If poll() or read() is interrupted by a non-fatal signal (i.e. SIGWINCH upon terminal resizing), errno is set to EINTR. Furthermore, read() on trace_pipe can return EAGAIN/EWOULDBLOCK even after poll() indicates POLLIN (for instance, under concurrent readers, spurious wakeups, or block-mode trace iterator options). In all such cases, breaking from the loop causes perf ftrace to terminate tracing prematurely.

Additionally, output errors on stdout (i.e. when a pager or consumer process exits, closing the downstream pipe) were ignored by fflush(), leaving __cmd_ftrace() polling indefinitely until the workload completes.

Resolve these concerns by retrying poll() upon receiving EINTR, retrying read() on EAGAIN, EWOULDBLOCK, and EINTR, and breaking on EOF (n == 0). Furthermore, detect stdout errors and broken pipes via fflush() and ferror() to exit immediately when the downstream output sink closes, and handle EINTR and ferror(stdout) cleanly in the subsequent buffer draining loop.

Quoted 5 lines
Fixes d01f4e8db22c ("perf ftrace: Introduce new 'ftrace' tool")

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
maintainer Acked-by on v12 1/10
#3 lore raw

Re: [PATCH perf-tools-next v12 01/10] perf ftrace: Handle signals, non-blocking reads and pager exits in __cmd_ftrace()

In reply to Aaron Tomlin
Quoted 24 lines · Aaron Tomlin

In __cmd_ftrace(), trace_fd is configured for non-blocking I/O via fcntl(trace_fd, F_SETFL, O_NONBLOCK) and polled using poll(). However, the polling loop previously treated any negative return code from poll() or read() as an unrecoverable failure.

If poll() or read() is interrupted by a non-fatal signal (i.e. SIGWINCH upon terminal resizing), errno is set to EINTR. Furthermore, read() on trace_pipe can return EAGAIN/EWOULDBLOCK even after poll() indicates POLLIN (for instance, under concurrent readers, spurious wakeups, or block-mode trace iterator options). In all such cases, breaking from the loop causes perf ftrace to terminate tracing prematurely.

Additionally, output errors on stdout (i.e. when a pager or consumer process exits, closing the downstream pipe) were ignored by fflush(), leaving __cmd_ftrace() polling indefinitely until the workload completes.

Resolve these concerns by retrying poll() upon receiving EINTR, retrying read() on EAGAIN, EWOULDBLOCK, and EINTR, and breaking on EOF (n == 0). Furthermore, detect stdout errors and broken pipes via fflush() and ferror() to exit immediately when the downstream output sink closes, and handle EINTR and ferror(stdout) cleanly in the subsequent buffer draining loop.

Quoted 5 lines
Fixes d01f4e8db22c ("perf ftrace: Introduce new 'ftrace' tool")

Thanks,
Namhyung

Quoted 68 lines
---
 tools/perf/builtin-ftrace.c | 27 ++++++++++++++++++++++-----
 1 file changed, 22 insertions(+), 5 deletions(-)
tools/perf/builtin-ftrace.c +22 -5
index 6017493ff179..7cdc9ae2fbbb 100644
@@ -14,6 +14,7 @@
 #include <poll.h>
 #include <semaphore.h>
 #include <signal.h>
+#include <stdio.h>
 #include <stdlib.h>
 #include <unistd.h>
 #include <linux/capability.h>
@@ -806,17 +807,27 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	}
 
 	while (!done) {
-		if (poll(&pollfd, 1, -1) < 0)
+		if (poll(&pollfd, 1, -1) < 0) {
+			if (errno == EINTR)
+				continue;
 			break;
+		}
 
 		if (pollfd.revents & POLLIN) {
 			int n = read(trace_fd, buf, sizeof(buf));
-			if (n < 0)
+
+			if (n < 0) {
+				if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
+					continue;
+				break;
+			}
+			if (n == 0)
 				break;
 			if (fwrite(buf, n, 1, stdout) != 1)
 				break;
 			/* flush output since stdout is in full buffering mode due to pager */
-			fflush(stdout);
+			if (fflush(stdout) < 0 || ferror(stdout))
+				break;
 		}
 	}
 
@@ -831,9 +842,15 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	}
 
 	/* read remaining buffer contents */
-	while (true) {
+	while (!ferror(stdout)) {
 		int n = read(trace_fd, buf, sizeof(buf));
-		if (n <= 0)
+
+		if (n < 0) {
+			if (errno == EINTR)
+				continue;
+			break;
+		}
+		if (n == 0)
 			break;
 		if (fwrite(buf, n, 1, stdout) != 1)
 			break;
Signature · 1 line
2.55.0
#4 lore raw

Re: [PATCH perf-tools-next v12 01/10] perf ftrace: Handle signals, non-blocking reads and pager exits in __cmd_ftrace()

Quoted 31 lines · Namhyung Kim
Quoted 24 lines · Aaron Tomlin

In __cmd_ftrace(), trace_fd is configured for non-blocking I/O via fcntl(trace_fd, F_SETFL, O_NONBLOCK) and polled using poll(). However, the polling loop previously treated any negative return code from poll() or read() as an unrecoverable failure.

If poll() or read() is interrupted by a non-fatal signal (i.e. SIGWINCH upon terminal resizing), errno is set to EINTR. Furthermore, read() on trace_pipe can return EAGAIN/EWOULDBLOCK even after poll() indicates POLLIN (for instance, under concurrent readers, spurious wakeups, or block-mode trace iterator options). In all such cases, breaking from the loop causes perf ftrace to terminate tracing prematurely.

Additionally, output errors on stdout (i.e. when a pager or consumer process exits, closing the downstream pipe) were ignored by fflush(), leaving __cmd_ftrace() polling indefinitely until the workload completes.

Resolve these concerns by retrying poll() upon receiving EINTR, retrying read() on EAGAIN, EWOULDBLOCK, and EINTR, and breaking on EOF (n == 0). Furthermore, detect stdout errors and broken pipes via fflush() and ferror() to exit immediately when the downstream output sink closes, and handle EINTR and ferror(stdout) cleanly in the subsequent buffer draining loop.

Quoted 5 lines
Fixes d01f4e8db22c ("perf ftrace: Introduce new 'ftrace' tool")
Quoted 4 lines

Thanks,
Namhyung

Thank you,

Signature · 1 line
Aaron Tomlin
maintainer Applied
#5 lore raw

Re: [PATCH perf-tools-next v12 01/10] perf ftrace: Handle signals, non-blocking reads and pager exits in __cmd_ftrace()

Quoted 26 lines · Aaron Tomlin
Quoted 25 lines · Namhyung Kim
Quoted 24 lines · Aaron Tomlin

In __cmd_ftrace(), trace_fd is configured for non-blocking I/O via fcntl(trace_fd, F_SETFL, O_NONBLOCK) and polled using poll(). However, the polling loop previously treated any negative return code from poll() or read() as an unrecoverable failure.

If poll() or read() is interrupted by a non-fatal signal (i.e. SIGWINCH upon terminal resizing), errno is set to EINTR. Furthermore, read() on trace_pipe can return EAGAIN/EWOULDBLOCK even after poll() indicates POLLIN (for instance, under concurrent readers, spurious wakeups, or block-mode trace iterator options). In all such cases, breaking from the loop causes perf ftrace to terminate tracing prematurely.

Additionally, output errors on stdout (i.e. when a pager or consumer process exits, closing the downstream pipe) were ignored by fflush(), leaving __cmd_ftrace() polling indefinitely until the workload completes.

Resolve these concerns by retrying poll() upon receiving EINTR, retrying read() on EAGAIN, EWOULDBLOCK, and EINTR, and breaking on EOF (n == 0). Furthermore, detect stdout errors and broken pipes via fflush() and ferror() to exit immediately when the downstream output sink closes, and handle EINTR and ferror(stdout) cleanly in the subsequent buffer draining loop.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Fixes d01f4e8db22c ("perf ftrace: Introduce new 'ftrace' tool")

Applied to perf-tools-next, thanks!

Best regards
Namhyung

#6 lore raw

PATCH v12 2/10 perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming

In reply to Aaron Tomlin

In __cmd_ftrace(), incoming trace data from trace_pipe was historically buffered in a fixed 4096-byte array allocated directly on the stack frame. Furthermore, reads were written directly to stdout via fwrite() in arbitrary chunk sizes, rather than as discrete, well-delimited trace lines.

Refactor __cmd_ftrace() to employ dynamic heap allocation for the read buffer alongside a dynamic struct strbuf to accumulate streamed characters into whole lines. Pre-allocating the strbuf ensures that line processing incurs zero heap reallocations during steady-state polling, whilst the stack frame of __cmd_ftrace() is reduced by over 96% (down to ~160 bytes). Emitting complete lines provides discrete records necessary for subsequent call graph parsing and inlined function reconstruction.

---
 tools/perf/builtin-ftrace.c | 56 ++++++++++++++++++++++++++++++-------
 1 file changed, 46 insertions(+), 10 deletions(-)
tools/perf/builtin-ftrace.c +46 -10
index 7cdc9ae2fbbb..65983259b07e 100644
@@ -42,7 +42,10 @@
 #include "util/units.h"
 #include "util/parse-sublevel-options.h"
 
+#include "util/strbuf.h"
+
 #define DEFAULT_TRACER  "function_graph"
+#define TRACE_BUF_SIZE  4096
 
 static volatile sig_atomic_t workload_exec_errno;
 static volatile sig_atomic_t done;
@@ -739,7 +742,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 {
 	char *trace_file;
 	int trace_fd;
-	char buf[4096];
+	char *buf = NULL;
+	struct strbuf linebuf = STRBUF_INIT;
 	struct pollfd pollfd = {
 		.events = POLLIN,
 	};
@@ -768,10 +772,18 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 
 	setup_pager();
 
+	buf = malloc(TRACE_BUF_SIZE);
+	if (!buf) {
+		pr_err("failed to allocate trace buffer\n");
+		goto out_reset;
+	}
+	strbuf_init(&linebuf, 512);
+	strbuf_setlen(&linebuf, 0);
+
 	trace_file = get_tracing_instance_file("trace_pipe");
 	if (!trace_file) {
 		pr_err("failed to open trace_pipe\n");
-		goto out_reset;
+		goto out_free;
 	}
 
 	trace_fd = open(trace_file, O_RDONLY);
@@ -780,7 +792,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 
 	if (trace_fd < 0) {
 		pr_err("failed to open trace_pipe\n");
-		goto out_reset;
+		goto out_free;
 	}
 
 	fcntl(trace_fd, F_SETFL, O_NONBLOCK);
@@ -814,7 +826,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 		}
 
 		if (pollfd.revents & POLLIN) {
-			int n = read(trace_fd, buf, sizeof(buf));
+			int n = read(trace_fd, buf, TRACE_BUF_SIZE);
 
 			if (n < 0) {
 				if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
@@ -823,8 +835,16 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 			}
 			if (n == 0)
 				break;
-			if (fwrite(buf, n, 1, stdout) != 1)
-				break;
+			for (int i = 0; i < n; i++) {
+				if (buf[i] == '\n') {
+					if (fprintf(stdout, "%s\n", linebuf.buf) < 0)
+						break;
+					strbuf_setlen(&linebuf, 0);
+				} else {
+					if (strbuf_addch(&linebuf, buf[i]) < 0)
+						goto out_close_fd;
+				}
+			}
 			/* flush output since stdout is in full buffering mode due to pager */
 			if (fflush(stdout) < 0 || ferror(stdout))
 				break;
@@ -834,7 +854,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	write_tracing_file("tracing_on", "0");
 
 	if (workload_exec_errno) {
-		const char *emsg = str_error_r(workload_exec_errno, buf, sizeof(buf));
+		const char *emsg = str_error_r(workload_exec_errno, buf, TRACE_BUF_SIZE);
 		/* flush stdout first so below error msg appears at the end. */
 		fflush(stdout);
 		pr_err("workload failed: %s\n", emsg);
@@ -843,7 +863,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 
 	/* read remaining buffer contents */
 	while (!ferror(stdout)) {
-		int n = read(trace_fd, buf, sizeof(buf));
+		int n = read(trace_fd, buf, TRACE_BUF_SIZE);
 
 		if (n < 0) {
 			if (errno == EINTR)
@@ -852,12 +872,28 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 		}
 		if (n == 0)
 			break;
-		if (fwrite(buf, n, 1, stdout) != 1)
-			break;
+		for (int i = 0; i < n; i++) {
+			if (buf[i] == '\n') {
+				if (fprintf(stdout, "%s\n", linebuf.buf) < 0)
+					break;
+				strbuf_setlen(&linebuf, 0);
+			} else {
+				if (strbuf_addch(&linebuf, buf[i]) < 0)
+					goto out_close_fd;
+			}
+		}
+	}
+
+	if (!ferror(stdout) && linebuf.len > 0) {
+		fprintf(stdout, "%s\n", linebuf.buf);
+		strbuf_setlen(&linebuf, 0);
 	}
 
 out_close_fd:
 	close(trace_fd);
+out_free:
+	free(buf);
+	strbuf_release(&linebuf);
 out_reset:
 	exit_tracing_instance();
 out:
Signature · 1 line
2.55.0
#7 lore raw

Re: [PATCH perf-tools-next v12 02/10] perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming

Sashiko has reviewed this patch and found no issues.
Quoted 15 lines

In __cmd_ftrace(), incoming trace data from trace_pipe was historically buffered in a fixed 4096-byte array allocated directly on the stack frame. Furthermore, reads were written directly to stdout via fwrite() in arbitrary chunk sizes, rather than as discrete, well-delimited trace lines.

Refactor __cmd_ftrace() to employ dynamic heap allocation for the read buffer alongside a dynamic struct strbuf to accumulate streamed characters into whole lines. Pre-allocating the strbuf ensures that line processing incurs zero heap reallocations during steady-state polling, whilst the stack frame of __cmd_ftrace() is reduced by over 96% (down to ~160 bytes). Emitting complete lines provides discrete records necessary for subsequent call graph parsing and inlined function reconstruction.

Quoted 3 lines

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#8 lore raw

PATCH v12 3/10 perf ftrace: Support filtering return address comments

In reply to Aaron Tomlin

When tracing with the Linux kernel's function graph tracer, enabling funcgraph-retaddr records the return address of each function entry as a comment in the trace stream (e.g. /* <-load_elf_phdrs+0x6c/0xb0 */). Whilst valuable for tracing execution origins, these trailing comments can add considerable visual clutter to the call graph, particularly when examining deeply nested call trees or when synthesising higher-level abstractions.

Introduce support for filtering out return address comments from function graph trace output whenever return address display is not explicitly requested via --graph-opts retaddr.

Key aspects of this implementation:

1. Return address parsing and filtering

Implement ftrace_filter_retaddr() in tools/perf/util/ftrace.c to locate return address comments and strip them in-place with memmove(3).

Additionally, introduce ftrace_parse_retaddr() to parse caller symbol names and offsets across diverse output styles (standard format, return values, and bracketed addresses), establishing the parsing foundation for subsequent inlined callchain resolution.

2. Automatic comment filtering

In __cmd_ftrace(), filter return address comments from whole trace lines whenever !ftrace->graph_retaddr, ensuring clean output unless the user explicitly requests return address comments via --graph-opts retaddr.

3. Automated verification

Add a dedicated unit test suite ("Ftrace return address processing") in tools/perf/tests/ftrace.c to validate comment parsing and string filtering across all supported formats.

---
 tools/perf/builtin-ftrace.c     |  6 +++
 tools/perf/tests/Build          |  1 +
 tools/perf/tests/builtin-test.c |  1 +
 tools/perf/tests/ftrace.c       | 81 ++++++++++++++++++++++++++++
 tools/perf/tests/tests.h        |  1 +
 tools/perf/util/Build           |  1 +
 tools/perf/util/ftrace.c        | 93 +++++++++++++++++++++++++++++++++
 tools/perf/util/ftrace.h        |  3 ++
 8 files changed, 187 insertions(+)
 create mode 100644 tools/perf/tests/ftrace.c
 create mode 100644 tools/perf/util/ftrace.c
tools/perf/builtin-ftrace.c +6 -0
index 65983259b07e..e00621703dd7 100644
@@ -837,6 +837,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 				break;
 			for (int i = 0; i < n; i++) {
 				if (buf[i] == '\n') {
+					if (!ftrace->graph_retaddr)
+						ftrace_filter_retaddr(linebuf.buf);
 					if (fprintf(stdout, "%s\n", linebuf.buf) < 0)
 						break;
 					strbuf_setlen(&linebuf, 0);
@@ -874,6 +876,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 			break;
 		for (int i = 0; i < n; i++) {
 			if (buf[i] == '\n') {
+				if (!ftrace->graph_retaddr)
+					ftrace_filter_retaddr(linebuf.buf);
 				if (fprintf(stdout, "%s\n", linebuf.buf) < 0)
 					break;
 				strbuf_setlen(&linebuf, 0);
@@ -885,6 +889,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	}
 
 	if (!ferror(stdout) && linebuf.len > 0) {
+		if (!ftrace->graph_retaddr)
+			ftrace_filter_retaddr(linebuf.buf);
 		fprintf(stdout, "%s\n", linebuf.buf);
 		strbuf_setlen(&linebuf, 0);
 	}
tools/perf/tests/Build +1 -0
index 395f2866094a..136138932fcb 100644
@@ -72,6 +72,7 @@ perf-test-y += hwmon_pmu.o
 perf-test-y += tool_pmu.o
 perf-test-y += subcmd-help.o
 perf-test-y += kallsyms-split.o
+perf-test-y += ftrace.o
 
 ifeq ($(SRCARCH),$(filter $(SRCARCH),x86 arm arm64 powerpc riscv))
 perf-test-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
tools/perf/tests/builtin-test.c +1 -0
index 880eece0bc12..640d6d327215 100644
@@ -156,6 +156,7 @@ static struct test_suite *generic_tests[] = {
 	&suite__perf_hooks,
 	&suite__unit_number__scnprint,
 	&suite__mem2node,
+	&suite__ftrace,
 	&suite__time_utils,
 	&suite__jit_write_elf,
 	&suite__pfm,
tools/perf/tests/ftrace.c new file +81 -0

81 lines not on this page · raw message

tools/perf/tests/tests.h +1 -0
index 5982c4e62cd3..cd12ae85ccfa 100644
@@ -180,6 +180,7 @@ DECLARE_SUITE(perf_hooks);
 DECLARE_SUITE(unit_number__scnprint);
 DECLARE_SUITE(mem2node);
 DECLARE_SUITE(maps);
+DECLARE_SUITE(ftrace);
 DECLARE_SUITE(time_utils);
 DECLARE_SUITE(jit_write_elf);
 DECLARE_SUITE(api_io);
tools/perf/util/Build +1 -0
index 2c1f880c4c47..5b082d09006d 100644
@@ -169,6 +169,7 @@ perf-util-y += list_sort.o
 perf-util-y += mutex.o
 perf-util-y += sharded_mutex.o
 perf-util-y += intel-tpebs.o
+perf-util-y += ftrace.o
 
 perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_counter.o
 perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_counter_cgroup.o
tools/perf/util/ftrace.c new file +93 -0

93 lines not on this page · raw message

tools/perf/util/ftrace.h +3 -0
index 950f2efafad2..a3a679f74e9d 100644
@@ -93,4 +93,7 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)
 
 #endif  /* HAVE_BPF_SKEL */
 
+bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset);
+void ftrace_filter_retaddr(char *str);
+
 #endif  /* __PERF_FTRACE_H__ */
Signature · 1 line
2.55.0
#9 lore raw

Re: [PATCH perf-tools-next v12 03/10] perf ftrace: Support filtering return address comments

Sashiko has reviewed this patch and found no issues.
Quoted 14 lines

When tracing with the Linux kernel's function graph tracer, enabling funcgraph-retaddr records the return address of each function entry as a comment in the trace stream (e.g. /* <-load_elf_phdrs+0x6c/0xb0 */). Whilst valuable for tracing execution origins, these trailing comments can add considerable visual clutter to the call graph, particularly when examining deeply nested call trees or when synthesising higher-level abstractions.

Introduce support for filtering out return address comments from function graph trace output whenever return address display is not explicitly requested via --graph-opts retaddr.

Key aspects of this implementation:

Quoted 7 lines

1. Return address parsing and filtering
[ ... ]
parsing and string filtering across all supported formats.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#10 lore raw

PATCH v12 4/10 perf ftrace: Add basic task call stack management

In reply to Aaron Tomlin

The function_graph tracer records kernel function entries and exits with nested indentation. To perform call graph analysis and subsequent inlined function reconstruction, userspace must track the active call stack on a per-task and per-CPU basis.

Introduce basic call stack management for the function graph tracer:

1. Data structures

Define struct ftrace_stack_frame to track function names, indentation, and inlined status.

Define struct ftrace_task_state with a dynamically resisable call stack array, tracked in a hash table keyed by PID alongside per-CPU active task bindings (ftrace_cpu_tasks).

2. Trace stream processing

Implement find_func_column() to isolate context columns (CPU, timestamps, duration) from function graph trace lines across standard and verbose layouts.

Implement ftrace_process_fgraph_line() to parse function entries ("{") and exits ("}"), pushing and popping frames on the tracked stack and matching bare closing braces against preceding frames.

3. Integration in __cmd_ftrace()

Stream complete trace lines through ftrace_process_fgraph_line() in __cmd_ftrace() and clean up allocated CPU states on exit via ftrace_cpu_states__clear().

---
 tools/perf/builtin-ftrace.c |  40 ++--
 tools/perf/util/ftrace.c    | 394 ++++++++++++++++++++++++++++++++++++
 tools/perf/util/ftrace.h    |  30 +++
 3 files changed, 452 insertions(+), 12 deletions(-)
tools/perf/builtin-ftrace.c +28 -12
index e00621703dd7..e3909cff287b 100644
@@ -744,6 +744,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	int trace_fd;
 	char *buf = NULL;
 	struct strbuf linebuf = STRBUF_INIT;
+	int ret = 0;
 	struct pollfd pollfd = {
 		.events = POLLIN,
 	};
@@ -837,14 +838,20 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 				break;
 			for (int i = 0; i < n; i++) {
 				if (buf[i] == '\n') {
-					if (!ftrace->graph_retaddr)
-						ftrace_filter_retaddr(linebuf.buf);
-					if (fprintf(stdout, "%s\n", linebuf.buf) < 0)
+					if (ftrace_process_fgraph_line(ftrace,
+								       linebuf.buf,
+								       stdout) < 0) {
+						ret = -1;
+						goto out_close_fd;
+					}
+					if (ferror(stdout))
 						break;
 					strbuf_setlen(&linebuf, 0);
 				} else {
-					if (strbuf_addch(&linebuf, buf[i]) < 0)
+					if (strbuf_addch(&linebuf, buf[i]) < 0) {
+						ret = -1;
 						goto out_close_fd;
+					}
 				}
 			}
 			/* flush output since stdout is in full buffering mode due to pager */
@@ -876,22 +883,30 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 			break;
 		for (int i = 0; i < n; i++) {
 			if (buf[i] == '\n') {
-				if (!ftrace->graph_retaddr)
-					ftrace_filter_retaddr(linebuf.buf);
-				if (fprintf(stdout, "%s\n", linebuf.buf) < 0)
+				if (ftrace_process_fgraph_line(ftrace,
+							       linebuf.buf,
+							       stdout) < 0) {
+					ret = -1;
+					goto out_close_fd;
+				}
+				if (ferror(stdout))
 					break;
 				strbuf_setlen(&linebuf, 0);
 			} else {
-				if (strbuf_addch(&linebuf, buf[i]) < 0)
+				if (strbuf_addch(&linebuf, buf[i]) < 0) {
+					ret = -1;
 					goto out_close_fd;
+				}
 			}
 		}
 	}
 
 	if (!ferror(stdout) && linebuf.len > 0) {
-		if (!ftrace->graph_retaddr)
-			ftrace_filter_retaddr(linebuf.buf);
-		fprintf(stdout, "%s\n", linebuf.buf);
+		if (ftrace_process_fgraph_line(ftrace, linebuf.buf,
+					       stdout) < 0) {
+			ret = -1;
+			goto out_close_fd;
+		}
 		strbuf_setlen(&linebuf, 0);
 	}
 
@@ -901,9 +916,10 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	free(buf);
 	strbuf_release(&linebuf);
 out_reset:
+	ftrace_cpu_states__clear();
 	exit_tracing_instance();
 out:
-	return (done && !workload_exec_errno) ? 0 : -1;
+	return (ret == 0 && done && !workload_exec_errno) ? 0 : -1;
 }
 
 static void make_histogram(struct perf_ftrace *ftrace, int buckets[],
tools/perf/util/ftrace.c +394 -0

408 lines not on this page · raw message

tools/perf/util/ftrace.h +30 -0
index a3a679f74e9d..fa2974f46582 100644
@@ -1,6 +1,9 @@
 #ifndef __PERF_FTRACE_H__
 #define __PERF_FTRACE_H__
 
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/types.h>
 #include <linux/list.h>
 
 #include "target.h"
@@ -93,7 +96,34 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)
 
 #endif  /* HAVE_BPF_SKEL */
 
+#define FTRACE_MAX_CPUS 4096
+#define FTRACE_MAX_STACK 256
+#define FTRACE_PID_HASH_BITS 10
+#define FTRACE_PID_HASH_SIZE (1 << FTRACE_PID_HASH_BITS)
+
+#define FTRACE_IDLE_PID(cpu)       ((pid_t)(-1 - (cpu)))
+#define FTRACE_UNKNOWN_PID(cpu)    ((pid_t)(-1 - FTRACE_MAX_CPUS - (cpu)))
+
+struct ftrace_stack_frame {
+	char *name;
+	bool inlined;
+	int indent;
+	int raw_indent;
+};
+
+struct ftrace_task_state {
+	struct hlist_node	node;
+	pid_t			pid;
+	struct ftrace_stack_frame *stack;
+	int			max_stack;
+	int			depth;
+	int			inlined_depth;
+	int			overflow_depth;
+};
+
 bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset);
 void ftrace_filter_retaddr(char *str);
+void ftrace_cpu_states__clear(void);
+int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out);
 
 #endif  /* __PERF_FTRACE_H__ */
Signature · 1 line
2.55.0
#11 lore raw

Re: [PATCH perf-tools-next v12 04/10] perf ftrace: Add basic task call stack management

Sashiko has reviewed this patch and found no issues.
Quoted 16 lines

The function_graph tracer records kernel function entries and exits with nested indentation. To perform call graph analysis and subsequent inlined function reconstruction, userspace must track the active call stack on a per-task and per-CPU basis.

Introduce basic call stack management for the function graph tracer:

1. Data structures

Define struct ftrace_stack_frame to track function names, indentation, and inlined status.

Define struct ftrace_task_state with a dynamically resisable call stack array, tracked in a hash table keyed by PID alongside per-CPU active task bindings (ftrace_cpu_tasks).
[ ... ]

Quoted 5 lines
        ftrace_cpu_states__clear().

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
maintainer
#12 lore raw

Re: [PATCH perf-tools-next v12 04/10] perf ftrace: Add basic task call stack management

In reply to Aaron Tomlin
Quoted 36 lines · Aaron Tomlin

The function_graph tracer records kernel function entries and exits with nested indentation. To perform call graph analysis and subsequent inlined function reconstruction, userspace must track the active call stack on a per-task and per-CPU basis.

Introduce basic call stack management for the function graph tracer:

1. Data structures

Define struct ftrace_stack_frame to track function names, indentation, and inlined status.

Define struct ftrace_task_state with a dynamically resisable call stack array, tracked in a hash table keyed by PID alongside per-CPU active task bindings (ftrace_cpu_tasks).

2. Trace stream processing

Implement find_func_column() to isolate context columns (CPU, timestamps, duration) from function graph trace lines across standard and verbose layouts.

Implement ftrace_process_fgraph_line() to parse function entries ("{") and exits ("}"), pushing and popping frames on the tracked stack and matching bare closing braces against preceding frames.

3. Integration in __cmd_ftrace()

Stream complete trace lines through ftrace_process_fgraph_line() in __cmd_ftrace() and clean up allocated CPU states on exit via ftrace_cpu_states__clear().

---

[SNIP]

Quoted 43 lines
tools/perf/util/ftrace.c +43 -0
+/**
+ * ftrace_process_fgraph_line - Process and output a function graph trace line
+ * @ftrace: Perf ftrace context
+ * @line: The trace line to process
+ * @out: File stream to output to
+ *
+ * Parse function entries and exits, maintain the task's call stack, and
+ * output the trace line.
+ *
+ * Return: 0 on success, or a negative error code on failure.
+ */
+int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out)
+{
+	const char *pipe_char;
+	const char *after_pipe;
+	const char *func_start;
+	struct ftrace_task_state *cs;
+	int cpu = -1;
+	int leading_spaces;
+	int top_raw_indent;
+	int match_idx;
+	int i;
+	bool is_exit;
+	bool has_open_brace;
+	bool has_semicolon;
+
+	pipe_char = find_func_column(ftrace, line, &cpu);
+	if (!pipe_char)
+		return print_line(ftrace, line, out);
+
+	cs = get_cpu_state(ftrace, cpu);
+	if (!cs)
+		return print_line(ftrace, line, out);
+
+	after_pipe = pipe_char + 1;
+	func_start = skip_spaces(after_pipe);
+	leading_spaces = func_start - after_pipe;
+
+	if (*func_start == '\0')
+		return print_line(ftrace, line, out);
+
+	is_exit = (*func_start == '}');
+

tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 411

Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+	if (is_exit) {
+		char exit_name[128];
+
+		extract_exit_func_name(func_start, exit_name, sizeof(exit_name));
+
+		top_raw_indent = -1;
+		for (i = cs->depth - 1; i >= 0; i--) {
+			top_raw_indent = cs->stack[i].raw_indent;
+			break;
+		}

This loop doesn't look natural. Why not just use an if statement?

Quoted 39 lines
tools/perf/util/ftrace.c +39 -0
+
+		if (cs->overflow_depth > 0 &&
+		    (top_raw_indent < 0 || leading_spaces > top_raw_indent)) {
+			cs->overflow_depth--;
+			return print_line(ftrace, line, out);
+		}
+
+		match_idx = -1;
+		if (exit_name[0] != '\0') {
+			for (i = cs->depth - 1; i >= 0; i--) {
+				if (cs->stack[i].name &&
+				    cs->stack[i].raw_indent == leading_spaces &&
+				    strcmp(cs->stack[i].name, exit_name) == 0) {
+					match_idx = i;
+					break;
+				}
+			}
+		}
+
+		if (match_idx < 0 && exit_name[0] == '\0') {
+			for (i = cs->depth - 1; i >= 0; i--) {
+				if (cs->stack[i].raw_indent == leading_spaces) {
+					match_idx = i;
+					break;
+				}
+			}
+		}
+
+		if (match_idx >= 0) {
+			cs->overflow_depth = 0;
+			while (cs->depth > match_idx) {
+				free(cs->stack[cs->depth - 1].name);
+				cs->depth--;
+			}
+		} else if (cs->overflow_depth > 0) {
+			cs->overflow_depth--;
+		}
+
+		return print_line(ftrace, line, out);

tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 460

Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+	}
+
+	has_open_brace = (strchr(func_start, '{') != NULL);
+	has_semicolon = (strchr(func_start, ';') != NULL);
+
+	top_raw_indent = -1;
+	for (i = cs->depth - 1; i >= 0; i--) {
+		top_raw_indent = cs->stack[i].raw_indent;
+		break;
+	}

Ditto.

Quoted 48 lines
tools/perf/util/ftrace.c +27 -0
+
+	if (cs->overflow_depth > 0 && top_raw_indent >= 0 &&
+	    leading_spaces <= top_raw_indent)
+		cs->overflow_depth = 0;
+
+	if (has_open_brace && !has_semicolon) {
+		char entry_name[128];
+
+		extract_entry_func_name(func_start, entry_name, sizeof(entry_name));
+		if (cs->depth >= cs->max_stack &&
+		    ftrace_task_state__grow_stack(cs) < 0) {
+			cs->overflow_depth++;
+		} else {
+			cs->stack[cs->depth].name = strdup(entry_name);
+			if (!cs->stack[cs->depth].name) {
+				pr_err("Not enough memory\n");
+				return -ENOMEM;
+			}
+			cs->stack[cs->depth].inlined = false;
+			cs->stack[cs->depth].indent = leading_spaces;
+			cs->stack[cs->depth].raw_indent = leading_spaces;
+			cs->depth++;
+		}
+	}
+
+	return print_line(ftrace, line, out);
+}
tools/perf/util/ftrace.h +6 -0
index a3a679f74e9d..fa2974f46582 100644
@@ -1,6 +1,9 @@
 #ifndef __PERF_FTRACE_H__
 #define __PERF_FTRACE_H__
 
+#include <stdbool.h>
+#include <stdio.h>
+#include <sys/types.h>
 #include <linux/list.h>
 
 #include "target.h"
@@ -93,7 +96,34 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)
 
 #endif  /* HAVE_BPF_SKEL */
 
+#define FTRACE_MAX_CPUS 4096
+#define FTRACE_MAX_STACK 256
+#define FTRACE_PID_HASH_BITS 10

tools/perf/util/ftrace.h perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused) · line 111

Quoted 10 lines
tools/perf/util/ftrace.h +10 -0
+#define FTRACE_PID_HASH_SIZE (1 << FTRACE_PID_HASH_BITS)
+
+#define FTRACE_IDLE_PID(cpu)       ((pid_t)(-1 - (cpu)))
+#define FTRACE_UNKNOWN_PID(cpu)    ((pid_t)(-1 - FTRACE_MAX_CPUS - (cpu)))
+
+struct ftrace_stack_frame {
+	char *name;
+	bool inlined;
+	int indent;
+	int raw_indent;

Can you please add comments that explain why it has two indents? Also inlined field may be added later.

tools/perf/util/ftrace.h perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused) · line 121

Quoted 10 lines
tools/perf/util/ftrace.h +10 -0
+};
+
+struct ftrace_task_state {
+	struct hlist_node	node;
+	pid_t			pid;
+	struct ftrace_stack_frame *stack;
+	int			max_stack;
+	int			depth;
+	int			inlined_depth;
+	int			overflow_depth;

Ditto. What are these depths?

Thanks,
Namhyung

Quoted 11 lines
tools/perf/util/ftrace.h +4 -0
+};
+
 bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset);
 void ftrace_filter_retaddr(char *str);
+void ftrace_cpu_states__clear(void);
+int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out);
 
 #endif  /* __PERF_FTRACE_H__ */
Signature · 1 line
2.55.0
#13 lore raw

Re: [PATCH perf-tools-next v12 04/10] perf ftrace: Add basic task call stack management

Quoted 85 lines · Namhyung Kim
Quoted 36 lines · Aaron Tomlin

The function_graph tracer records kernel function entries and exits with nested indentation. To perform call graph analysis and subsequent inlined function reconstruction, userspace must track the active call stack on a per-task and per-CPU basis.

Introduce basic call stack management for the function graph tracer:

1. Data structures

Define struct ftrace_stack_frame to track function names, indentation, and inlined status.

Define struct ftrace_task_state with a dynamically resisable call stack array, tracked in a hash table keyed by PID alongside per-CPU active task bindings (ftrace_cpu_tasks).

2. Trace stream processing

Implement find_func_column() to isolate context columns (CPU, timestamps, duration) from function graph trace lines across standard and verbose layouts.

Implement ftrace_process_fgraph_line() to parse function entries ("{") and exits ("}"), pushing and popping frames on the tracked stack and matching bare closing braces against preceding frames.

3. Integration in __cmd_ftrace()

Stream complete trace lines through ftrace_process_fgraph_line() in __cmd_ftrace() and clean up allocated CPU states on exit via ftrace_cpu_states__clear().

---

[SNIP]

Quoted 47 lines
tools/perf/util/ftrace.c +47 -0
+/**
+ * ftrace_process_fgraph_line - Process and output a function graph trace line
+ * @ftrace: Perf ftrace context
+ * @line: The trace line to process
+ * @out: File stream to output to
+ *
+ * Parse function entries and exits, maintain the task's call stack, and
+ * output the trace line.
+ *
+ * Return: 0 on success, or a negative error code on failure.
+ */
+int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out)
+{
+	const char *pipe_char;
+	const char *after_pipe;
+	const char *func_start;
+	struct ftrace_task_state *cs;
+	int cpu = -1;
+	int leading_spaces;
+	int top_raw_indent;
+	int match_idx;
+	int i;
+	bool is_exit;
+	bool has_open_brace;
+	bool has_semicolon;
+
+	pipe_char = find_func_column(ftrace, line, &cpu);
+	if (!pipe_char)
+		return print_line(ftrace, line, out);
+
+	cs = get_cpu_state(ftrace, cpu);
+	if (!cs)
+		return print_line(ftrace, line, out);
+
+	after_pipe = pipe_char + 1;
+	func_start = skip_spaces(after_pipe);
+	leading_spaces = func_start - after_pipe;
+
+	if (*func_start == '\0')
+		return print_line(ftrace, line, out);
+
+	is_exit = (*func_start == '}');
+
+	if (is_exit) {
+		char exit_name[128];
+
+		extract_exit_func_name(func_start, exit_name, sizeof(exit_name));
Quoted 8 lines

tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 411

Quoted 6 lines
tools/perf/util/ftrace.c +6 -0
+
+		top_raw_indent = -1;
+		for (i = cs->depth - 1; i >= 0; i--) {
+			top_raw_indent = cs->stack[i].raw_indent;
+			break;
+		}

This loop doesn't look natural. Why not just use an if statement?

Hi Namhyung,

Thank you. I completely agree.

A loop that unconditionally breaks on the first iteration is unnecessary here. I will replace it with a simple conditional check in the next revision:

    top_raw_indent = -1;
    if (cs->depth > 0)
        top_raw_indent = cs->stack[cs->depth - 1].raw_indent;

All subsequent similar instances will be addressed accordingly as well.

Quoted 8 lines

tools/perf/util/ftrace.h perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused) · line 111

Quoted 5 lines
tools/perf/util/ftrace.h +5 -0
+struct ftrace_stack_frame {
+	char *name;
+	bool inlined;
+	int indent;
+	int raw_indent;

Can you please add comments that explain why it has two indents? Also inlined field may be added later.

Certainly. The two indent fields serve distinct purposes once inlined frames are injected into the output.

- raw_indent: Records the original leading whitespace count after "|" as emitted by the kernel's function_graph tracer. Because the kernel only traces real stack frames, raw_indent is used to match entry and exit frames against the raw trace stream.

- indent: Records the adjusted visual indentation (in spaces) used when rendering the frame to output, accounting for any extra indentation added by preceding synthetic inlined frames so that the closing "}" aligns with its opening "{".

Agreed, these fields are not used yet. I will move inlined and inlined_depth into the subsequent patch that introduces inlined function display, and add explanatory comments to both structures.

Quoted 4 lines
Quoted 4 lines
tools/perf/util/ftrace.h +4 -0
+};
+
+struct ftrace_task_state {
+	struct hlist_node	node;
Quoted 8 lines

tools/perf/util/ftrace.h perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused) · line 121

Quoted 6 lines
tools/perf/util/ftrace.h +6 -0
+	pid_t			pid;
+	struct ftrace_stack_frame *stack;
+	int			max_stack;
+	int			depth;
+	int			inlined_depth;
+	int			overflow_depth;

Ditto. What are these depths?

- max_stack: Current allocated capacity (in frames) of the dynamically resized 'stack' array.

- depth: Current number of active frames stored on stack.

- inlined_depth: Number of active synthetic inlined frames currently on stack, used to compute the extra visual indentation offset for child frames. This will also be moved to the later patch.

- overflow_depth: Number of nested call frames entered while stack is saturated (at the maximum depth cap or on allocation failure), used so their matching exit braces can be consumed without prematurely popping valid parent frames.

I will document all of these fields with comments in the next revision.

Please see an example. When synthetic inlined frames are injected, the visual display column (indent) shifts right relative to the kernel stream (raw_indent):

1. What kernel function_graph emits:

| foo() { <-- raw_indent = 2
| bar() { <-- raw_indent = 4
| } /* bar */ <-- raw_indent = 4
| } /* foo */ <-- raw_indent = 2

2. Rendered output and stack state:

When perf injects inlined function inline_helper() onto cs->stack, inlined_depth becomes 1. Every subsequent child frame shifts right by (inlined_depth * 2) spaces so bar() nests visually inside inline_helper(), while still retaining raw_indent = 4 to match the kernel's closing "}" event:

Rendered Output cs->stack[idx] and ftrace_task_state
--------------- ---------------------------
| foo() { [0] foo:
: raw_indent=2, indent=2, inlined=false
: (depth=1, inlined_depth=0)
:
| inline_helper() { [1] inline_helper:
: raw_indent=4, indent=4, inlined=true
: (depth=2, inlined_depth=1)
:
| bar() { [2] bar:
: raw_indent=4, indent=6, inlined=false
: (depth=3, inlined_depth=1)
: ^
: '-- indent = raw_indent (4) +
: inlined_depth * 2 (2) = 6
:
| } /* bar */ <-- kernel function_graph emits "}"
: at raw_indent=4. Matched against [2]
: via raw_indent=4, but printed at
: [2].indent (6 spaces)
:
| } /* inline_helper */ <-- Popped automatically; printed at
: [1].indent (4 spaces)
:
| } /* foo */ <-- kernel function_graph emits "}"
at raw_indent=2. Matched against [0] via raw_indent=2, printed at [0].indent (2 spaces)

Thanks,

Signature · 1 line
Aaron Tomlin
maintainer
#14 lore raw

Re: [PATCH perf-tools-next v12 04/10] perf ftrace: Add basic task call stack management

Hello,

Quoted 204 lines · Aaron Tomlin
Quoted 85 lines · Namhyung Kim
Quoted 36 lines · Aaron Tomlin

The function_graph tracer records kernel function entries and exits with nested indentation. To perform call graph analysis and subsequent inlined function reconstruction, userspace must track the active call stack on a per-task and per-CPU basis.

Introduce basic call stack management for the function graph tracer:

1. Data structures

Define struct ftrace_stack_frame to track function names, indentation, and inlined status.

Define struct ftrace_task_state with a dynamically resisable call stack array, tracked in a hash table keyed by PID alongside per-CPU active task bindings (ftrace_cpu_tasks).

2. Trace stream processing

Implement find_func_column() to isolate context columns (CPU, timestamps, duration) from function graph trace lines across standard and verbose layouts.

Implement ftrace_process_fgraph_line() to parse function entries ("{") and exits ("}"), pushing and popping frames on the tracked stack and matching bare closing braces against preceding frames.

3. Integration in __cmd_ftrace()

Stream complete trace lines through ftrace_process_fgraph_line() in __cmd_ftrace() and clean up allocated CPU states on exit via ftrace_cpu_states__clear().

---

[SNIP]

Quoted 47 lines
tools/perf/util/ftrace.c +47 -0
+/**
+ * ftrace_process_fgraph_line - Process and output a function graph trace line
+ * @ftrace: Perf ftrace context
+ * @line: The trace line to process
+ * @out: File stream to output to
+ *
+ * Parse function entries and exits, maintain the task's call stack, and
+ * output the trace line.
+ *
+ * Return: 0 on success, or a negative error code on failure.
+ */
+int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out)
+{
+	const char *pipe_char;
+	const char *after_pipe;
+	const char *func_start;
+	struct ftrace_task_state *cs;
+	int cpu = -1;
+	int leading_spaces;
+	int top_raw_indent;
+	int match_idx;
+	int i;
+	bool is_exit;
+	bool has_open_brace;
+	bool has_semicolon;
+
+	pipe_char = find_func_column(ftrace, line, &cpu);
+	if (!pipe_char)
+		return print_line(ftrace, line, out);
+
+	cs = get_cpu_state(ftrace, cpu);
+	if (!cs)
+		return print_line(ftrace, line, out);
+
+	after_pipe = pipe_char + 1;
+	func_start = skip_spaces(after_pipe);
+	leading_spaces = func_start - after_pipe;
+
+	if (*func_start == '\0')
+		return print_line(ftrace, line, out);
+
+	is_exit = (*func_start == '}');
+
+	if (is_exit) {
+		char exit_name[128];
+
+		extract_exit_func_name(func_start, exit_name, sizeof(exit_name));
Quoted 8 lines

tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 411

Quoted 6 lines
tools/perf/util/ftrace.c +6 -0
+
+		top_raw_indent = -1;
+		for (i = cs->depth - 1; i >= 0; i--) {
+			top_raw_indent = cs->stack[i].raw_indent;
+			break;
+		}

This loop doesn't look natural. Why not just use an if statement?

Hi Namhyung,

Thank you. I completely agree.

A loop that unconditionally breaks on the first iteration is unnecessary here. I will replace it with a simple conditional check in the next revision:

    top_raw_indent = -1;
    if (cs->depth > 0)
        top_raw_indent = cs->stack[cs->depth - 1].raw_indent;

All subsequent similar instances will be addressed accordingly as well.

Quoted 8 lines

tools/perf/util/ftrace.h perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused) · line 111

Quoted 5 lines
tools/perf/util/ftrace.h +5 -0
+struct ftrace_stack_frame {
+	char *name;
+	bool inlined;
+	int indent;
+	int raw_indent;

Can you please add comments that explain why it has two indents? Also inlined field may be added later.

Certainly. The two indent fields serve distinct purposes once inlined frames are injected into the output.

- raw_indent: Records the original leading whitespace count after "|" as emitted by the kernel's function_graph tracer. Because the kernel only traces real stack frames, raw_indent is used to match entry and exit frames against the raw trace stream.

- indent: Records the adjusted visual indentation (in spaces) used when rendering the frame to output, accounting for any extra indentation added by preceding synthetic inlined frames so that the closing "}" aligns with its opening "{".

Agreed, these fields are not used yet. I will move inlined and inlined_depth into the subsequent patch that introduces inlined function display, and add explanatory comments to both structures.

Quoted 4 lines
Quoted 4 lines
tools/perf/util/ftrace.h +4 -0
+};
+
+struct ftrace_task_state {
+	struct hlist_node	node;
Quoted 8 lines

tools/perf/util/ftrace.h perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused) · line 121

Quoted 6 lines
tools/perf/util/ftrace.h +6 -0
+	pid_t			pid;
+	struct ftrace_stack_frame *stack;
+	int			max_stack;
+	int			depth;
+	int			inlined_depth;
+	int			overflow_depth;

Ditto. What are these depths?

- max_stack: Current allocated capacity (in frames) of the dynamically resized 'stack' array.

- depth: Current number of active frames stored on stack.

- inlined_depth: Number of active synthetic inlined frames currently on stack, used to compute the extra visual indentation offset for child frames. This will also be moved to the later patch.

- overflow_depth: Number of nested call frames entered while stack is saturated (at the maximum depth cap or on allocation failure), used so their matching exit braces can be consumed without prematurely popping valid parent frames.

I will document all of these fields with comments in the next revision.

Please see an example. When synthetic inlined frames are injected, the visual display column (indent) shifts right relative to the kernel stream (raw_indent):

1. What kernel function_graph emits:

| foo() { <-- raw_indent = 2
| bar() { <-- raw_indent = 4
| } /* bar */ <-- raw_indent = 4
| } /* foo */ <-- raw_indent = 2

2. Rendered output and stack state:

When perf injects inlined function inline_helper() onto cs->stack, inlined_depth becomes 1. Every subsequent child frame shifts right by (inlined_depth * 2) spaces so bar() nests visually inside inline_helper(), while still retaining raw_indent = 4 to match the kernel's closing "}" event:

Rendered Output cs->stack[idx] and ftrace_task_state
--------------- ---------------------------
| foo() { [0] foo:
: raw_indent=2, indent=2, inlined=false
: (depth=1, inlined_depth=0)
:
| inline_helper() { [1] inline_helper:
: raw_indent=4, indent=4, inlined=true
: (depth=2, inlined_depth=1)
:
| bar() { [2] bar:
: raw_indent=4, indent=6, inlined=false
: (depth=3, inlined_depth=1)
: ^
: '-- indent = raw_indent (4) +
: inlined_depth * 2 (2) = 6
:
| } /* bar */ <-- kernel function_graph emits "}"
: at raw_indent=4. Matched against [2]
: via raw_indent=4, but printed at
: [2].indent (6 spaces)

Quoted 9 lines

:
| } /* inline_helper */ <-- Popped automatically; printed at
: [1].indent (4 spaces)
:
| } /* foo */ <-- kernel function_graph emits "}"
at raw_indent=2. Matched against [0] via raw_indent=2, printed at [0].indent (2 spaces)

Thanks for the example, the comment also look good.

Namhyung

#15 lore raw

Re: [PATCH perf-tools-next v12 04/10] perf ftrace: Add basic task call stack management

Quoted 217 lines · Namhyung Kim

Hello,

Quoted 204 lines · Aaron Tomlin
Quoted 85 lines · Namhyung Kim
Quoted 36 lines · Aaron Tomlin

The function_graph tracer records kernel function entries and exits with nested indentation. To perform call graph analysis and subsequent inlined function reconstruction, userspace must track the active call stack on a per-task and per-CPU basis.

Introduce basic call stack management for the function graph tracer:

1. Data structures

Define struct ftrace_stack_frame to track function names, indentation, and inlined status.

Define struct ftrace_task_state with a dynamically resisable call stack array, tracked in a hash table keyed by PID alongside per-CPU active task bindings (ftrace_cpu_tasks).

2. Trace stream processing

Implement find_func_column() to isolate context columns (CPU, timestamps, duration) from function graph trace lines across standard and verbose layouts.

Implement ftrace_process_fgraph_line() to parse function entries ("{") and exits ("}"), pushing and popping frames on the tracked stack and matching bare closing braces against preceding frames.

3. Integration in __cmd_ftrace()

Stream complete trace lines through ftrace_process_fgraph_line() in __cmd_ftrace() and clean up allocated CPU states on exit via ftrace_cpu_states__clear().

---

[SNIP]

Quoted 47 lines
tools/perf/util/ftrace.c +47 -0
+/**
+ * ftrace_process_fgraph_line - Process and output a function graph trace line
+ * @ftrace: Perf ftrace context
+ * @line: The trace line to process
+ * @out: File stream to output to
+ *
+ * Parse function entries and exits, maintain the task's call stack, and
+ * output the trace line.
+ *
+ * Return: 0 on success, or a negative error code on failure.
+ */
+int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out)
+{
+	const char *pipe_char;
+	const char *after_pipe;
+	const char *func_start;
+	struct ftrace_task_state *cs;
+	int cpu = -1;
+	int leading_spaces;
+	int top_raw_indent;
+	int match_idx;
+	int i;
+	bool is_exit;
+	bool has_open_brace;
+	bool has_semicolon;
+
+	pipe_char = find_func_column(ftrace, line, &cpu);
+	if (!pipe_char)
+		return print_line(ftrace, line, out);
+
+	cs = get_cpu_state(ftrace, cpu);
+	if (!cs)
+		return print_line(ftrace, line, out);
+
+	after_pipe = pipe_char + 1;
+	func_start = skip_spaces(after_pipe);
+	leading_spaces = func_start - after_pipe;
+
+	if (*func_start == '\0')
+		return print_line(ftrace, line, out);
+
+	is_exit = (*func_start == '}');
+
+	if (is_exit) {
+		char exit_name[128];
+
+		extract_exit_func_name(func_start, exit_name, sizeof(exit_name));
Quoted 8 lines

tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 411

Quoted 6 lines
tools/perf/util/ftrace.c +6 -0
+
+		top_raw_indent = -1;
+		for (i = cs->depth - 1; i >= 0; i--) {
+			top_raw_indent = cs->stack[i].raw_indent;
+			break;
+		}

This loop doesn't look natural. Why not just use an if statement?

Hi Namhyung,

Thank you. I completely agree.

A loop that unconditionally breaks on the first iteration is unnecessary here. I will replace it with a simple conditional check in the next revision:

    top_raw_indent = -1;
    if (cs->depth > 0)
        top_raw_indent = cs->stack[cs->depth - 1].raw_indent;

All subsequent similar instances will be addressed accordingly as well.

Quoted 8 lines

tools/perf/util/ftrace.h perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused) · line 111

Quoted 5 lines
tools/perf/util/ftrace.h +5 -0
+struct ftrace_stack_frame {
+	char *name;
+	bool inlined;
+	int indent;
+	int raw_indent;

Can you please add comments that explain why it has two indents? Also inlined field may be added later.

Certainly. The two indent fields serve distinct purposes once inlined frames are injected into the output.

- raw_indent: Records the original leading whitespace count after "|" as emitted by the kernel's function_graph tracer. Because the kernel only traces real stack frames, raw_indent is used to match entry and exit frames against the raw trace stream.

- indent: Records the adjusted visual indentation (in spaces) used when rendering the frame to output, accounting for any extra indentation added by preceding synthetic inlined frames so that the closing "}" aligns with its opening "{".

Agreed, these fields are not used yet. I will move inlined and inlined_depth into the subsequent patch that introduces inlined function display, and add explanatory comments to both structures.

Quoted 4 lines
Quoted 4 lines
tools/perf/util/ftrace.h +4 -0
+};
+
+struct ftrace_task_state {
+	struct hlist_node	node;
Quoted 8 lines

tools/perf/util/ftrace.h perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused) · line 121

Quoted 6 lines
tools/perf/util/ftrace.h +6 -0
+	pid_t			pid;
+	struct ftrace_stack_frame *stack;
+	int			max_stack;
+	int			depth;
+	int			inlined_depth;
+	int			overflow_depth;

Ditto. What are these depths?

- max_stack: Current allocated capacity (in frames) of the dynamically resized 'stack' array.

- depth: Current number of active frames stored on stack.

- inlined_depth: Number of active synthetic inlined frames currently on stack, used to compute the extra visual indentation offset for child frames. This will also be moved to the later patch.

- overflow_depth: Number of nested call frames entered while stack is saturated (at the maximum depth cap or on allocation failure), used so their matching exit braces can be consumed without prematurely popping valid parent frames.

I will document all of these fields with comments in the next revision.

Please see an example. When synthetic inlined frames are injected, the visual display column (indent) shifts right relative to the kernel stream (raw_indent):

1. What kernel function_graph emits:

| foo() { <-- raw_indent = 2
| bar() { <-- raw_indent = 4
| } /* bar */ <-- raw_indent = 4
| } /* foo */ <-- raw_indent = 2

2. Rendered output and stack state:

When perf injects inlined function inline_helper() onto cs->stack, inlined_depth becomes 1. Every subsequent child frame shifts right by (inlined_depth * 2) spaces so bar() nests visually inside inline_helper(), while still retaining raw_indent = 4 to match the kernel's closing "}" event:

Rendered Output cs->stack[idx] and ftrace_task_state
--------------- ---------------------------
| foo() { [0] foo:
: raw_indent=2, indent=2, inlined=false
: (depth=1, inlined_depth=0)
:
| inline_helper() { [1] inline_helper:
: raw_indent=4, indent=4, inlined=true
: (depth=2, inlined_depth=1)
:
| bar() { [2] bar:
: raw_indent=4, indent=6, inlined=false
: (depth=3, inlined_depth=1)
: ^
: '-- indent = raw_indent (4) +
: inlined_depth * 2 (2) = 6
:
| } /* bar */ <-- kernel function_graph emits "}"
: at raw_indent=4. Matched against [2]
: via raw_indent=4, but printed at
: [2].indent (6 spaces)

Quoted 9 lines

:
| } /* inline_helper */ <-- Popped automatically; printed at
: [1].indent (4 spaces)
:
| } /* foo */ <-- kernel function_graph emits "}"
at raw_indent=2. Matched against [0] via raw_indent=2, printed at [0].indent (2 spaces)

Quoted 1 line

Thanks for the example, the comment also look good.

Hi Namhyung,

Thank you and you are very welcome.

Kind regards,

Signature · 1 line
Aaron Tomlin
#16 lore raw

PATCH v12 5/10 perf ftrace: Support context switch tracking in function graph tracer

In reply to Aaron Tomlin

When tracing multiple tasks across multiple CPUs, kernel context switches cause different threads of execution to interleave on the same CPU. Without context switch awareness, interleaved function calls corrupt the tracked call stack.

Add context switch detection and per-task state tracking:

1. Context switch banner parsing

Implement parse_context_switch() to parse context switch banners (e.g. " 0) <idle>-0 => migrati-19") in the trace stream, extracting the CPU number and both outgoing and incoming task PIDs.

2. Task state rekeying and migration

Implement ftrace_task_state__rekey() to reassign initial unknown task states to real PIDs upon discovering outgoing task identity. Track monotonic trace event sequence numbers on each frame so merges can locate the exact splice boundary when tasks migrate across CPUs and back before delayed banners are emitted. Prune exited frames from pre-migration calls while preserving intermediate active frames and splicing them directly into the task call stack.

3. Verbose mode and single-PID binding

In find_func_column(), parse the comm-pid column in verbose mode to bind active task state on each line record.

In get_cpu_state(), pre-bind single-PID workloads across all CPUs when context-switch banners are absent.

---
 tools/perf/util/ftrace.c | 430 ++++++++++++++++++++++++++++++++++++++-
 tools/perf/util/ftrace.h |  10 +
 2 files changed, 436 insertions(+), 4 deletions(-)
tools/perf/util/ftrace.c +426 -4
index 95db57e70d11..fc385d680911 100644
@@ -13,11 +13,14 @@
 #include <errno.h>
 #include <sys/types.h>
 #include "debug.h"
+#include "evlist.h"
 #include "util/ftrace.h"
 #include "util/string2.h"
+#include <perf/threadmap.h>
 
 static struct hlist_head ftrace_task_hash[FTRACE_PID_HASH_SIZE];
 static struct ftrace_task_state *ftrace_cpu_tasks[FTRACE_MAX_CPUS];
+static u64 ftrace_event_seq;
 
 /**
  * ftrace_cpu_states__clear - Free allocated CPU and task stack states
@@ -36,12 +39,14 @@ void ftrace_cpu_states__clear(void)
 			for (j = 0; j < ts->depth; j++)
 				free(ts->stack[j].name);
 			free(ts->stack);
+			free(ts->watermarks);
 			hlist_del_init(&ts->node);
 			free(ts);
 		}
 	}
 
 	memset(ftrace_cpu_tasks, 0, sizeof(ftrace_cpu_tasks));
+	ftrace_event_seq = 0;
 }
 
 /**
@@ -98,6 +103,119 @@ static struct ftrace_task_state *get_task_state(pid_t pid)
 	return ts;
 }
 
+/**
+ * ftrace_task_state__record_indent - Record an indentation watermark for a task
+ * @cs: Task state to update
+ * @seq: Event sequence number of the observed trace record
+ * @raw_indent: Raw indentation (leading spaces) of the observed trace record
+ *
+ * Maintain a monotonic suffix-minima array of (seq, raw_indent) pairs. Any
+ * previously recorded watermarks with an indentation greater than or equal to
+ * @raw_indent are dominated by this newer, shallower (or equal) event for all
+ * earlier frames and are popped before appending the new watermark.
+ *
+ * Return: 0 on success, or -ENOMEM if memory allocation fails.
+ */
+static int ftrace_task_state__record_indent(struct ftrace_task_state *cs,
+					    u64 seq, int raw_indent)
+{
+	while (cs->nr_watermarks > 0 &&
+	       cs->watermarks[cs->nr_watermarks - 1].raw_indent >= raw_indent)
+		cs->nr_watermarks--;
+
+	if (cs->nr_watermarks >= cs->max_watermarks) {
+		int new_max = cs->max_watermarks ? cs->max_watermarks * 2 : 16;
+		struct ftrace_indent_watermark *new_wm;
+
+		new_wm = realloc(cs->watermarks, new_max * sizeof(*new_wm));
+		if (!new_wm)
+			return -ENOMEM;
+		cs->watermarks = new_wm;
+		cs->max_watermarks = new_max;
+	}
+
+	cs->watermarks[cs->nr_watermarks].seq = seq;
+	cs->watermarks[cs->nr_watermarks].raw_indent = raw_indent;
+	cs->nr_watermarks++;
+	return 0;
+}
+
+/**
+ * ftrace_task_state__min_indent_after - Query minimum indentation after a sequence number
+ * @cs: Task state whose watermarks are queried
+ * @frame_seq: Sequence number of the frame to check against
+ *
+ * Because @cs->watermarks stores suffix minima in strictly increasing order of
+ * both sequence number and indentation, the first watermark with seq > @frame_seq
+ * holds the minimum raw indentation observed on @cs after @frame_seq.
+ *
+ * Return: The minimum raw indentation observed on @cs after @frame_seq, or -1
+ *         if no events were recorded on @cs after @frame_seq.
+ */
+static int ftrace_task_state__min_indent_after(const struct ftrace_task_state *cs,
+					       u64 frame_seq)
+{
+	int i;
+
+	for (i = 0; i < cs->nr_watermarks; i++) {
+		if (cs->watermarks[i].seq > frame_seq)
+			return cs->watermarks[i].raw_indent;
+	}
+
+	return -1;
+}
+
+/**
+ * ftrace_task_state__prune_stack - Prune frames that exited on another task state
+ * @cs: Task state whose call stack should be pruned
+ * @other: Other task state representing the same task on another CPU
+ *
+ * Remove any frames on @cs->stack that are known to have exited based on
+ * subsequent trace events recorded in @other->watermarks (i.e. events with
+ * seq > frame->seq and raw_indent <= frame->raw_indent). Synthesised inline
+ * frames are retained if and only if their enclosing real frame survives.
+ */
+static void ftrace_task_state__prune_stack(struct ftrace_task_state *cs,
+					   const struct ftrace_task_state *other)
+{
+	bool keep_inlines = false;
+	bool seen_real = false;
+	int i, dst = 0;
+
+	for (i = cs->depth - 1; i >= 0; i--) {
+		struct ftrace_stack_frame *frame = &cs->stack[i];
+		int min_indent = ftrace_task_state__min_indent_after(other, frame->seq);
+		bool prune;
+
+		if (frame->inlined) {
+			if (seen_real)
+				prune = !keep_inlines;
+			else
+				prune = (min_indent >= 0);
+		} else {
+			prune = (min_indent >= 0 && frame->raw_indent >= min_indent);
+			seen_real = true;
+			keep_inlines = !prune;
+		}
+
+		if (prune) {
+			free(frame->name);
+			frame->name = NULL;
+			if (frame->inlined && cs->inlined_depth > 0)
+				cs->inlined_depth--;
+		}
+	}
+
+	for (i = 0; i < cs->depth; i++) {
+		if (cs->stack[i].name) {
+			if (dst != i)
+				cs->stack[dst] = cs->stack[i];
+			dst++;
+		}
+	}
+	cs->depth = dst;
+}
+
 /**
  * ftrace_task_state__grow_stack - Double the capacity of a task's call stack
  * @cs: Task state whose stack should be resized
@@ -124,6 +242,182 @@ static int ftrace_task_state__grow_stack(struct ftrace_task_state *cs)
 	return 0;
 }
 
+/**
+ * ftrace_task_state__rekey - Reassign a task state to a new PID
+ * @ts: The task state to re-key
+ * @new_pid: The new process ID to associate with the state
+ *
+ * Rehash @ts under @new_pid. If a task state for @new_pid already exists,
+ * merge or replace it and update active CPU references accordingly.
+ *
+ * Return: A pointer to the active task state associated with @new_pid.
+ */
+static struct ftrace_task_state *ftrace_task_state__rekey(struct ftrace_task_state *ts,
+							  pid_t new_pid)
+{
+	struct ftrace_task_state *existing;
+	u32 key;
+	int j, c;
+	int to_copy;
+
+	if (ts->pid == new_pid)
+		return ts;
+
+	hlist_del_init(&ts->node);
+
+	existing = find_task_state(new_pid);
+	if (existing) {
+		int orig_existing_depth;
+		int total_needed;
+
+		for (c = 0; c < FTRACE_MAX_CPUS; c++) {
+			if (ftrace_cpu_tasks[c] == ts)
+				ftrace_cpu_tasks[c] = existing;
+		}
+
+		/*
+		 * Prune frames on existing and ts that exited during subsequent
+		 * execution recorded on the other state.
+		 */
+		ftrace_task_state__prune_stack(existing, ts);
+		ftrace_task_state__prune_stack(ts, existing);
+
+		orig_existing_depth = existing->depth;
+		to_copy = ts->depth;
+		total_needed = orig_existing_depth + to_copy;
+
+		while (total_needed > existing->max_stack) {
+			if (ftrace_task_state__grow_stack(existing) < 0)
+				break;
+		}
+		if (total_needed > existing->max_stack) {
+			if (existing->max_stack > orig_existing_depth)
+				to_copy = existing->max_stack - orig_existing_depth;
+			else
+				to_copy = 0;
+		}
+
+		if (to_copy > 0) {
+			int e_idx = 0, t_idx = 0, out_idx = 0;
+
+			if (orig_existing_depth > 0) {
+				memmove(&existing->stack[to_copy],
+					&existing->stack[0],
+					orig_existing_depth * sizeof(existing->stack[0]));
+			}
+
+			while (e_idx < orig_existing_depth && t_idx < to_copy) {
+				if (existing->stack[to_copy + e_idx].seq <=
+				    ts->stack[t_idx].seq) {
+					existing->stack[out_idx++] =
+						existing->stack[to_copy + e_idx++];
+				} else {
+					if (ts->stack[t_idx].inlined)
+						existing->inlined_depth++;
+					existing->stack[out_idx++] =
+						ts->stack[t_idx++];
+				}
+			}
+			while (t_idx < to_copy) {
+				if (ts->stack[t_idx].inlined)
+					existing->inlined_depth++;
+				existing->stack[out_idx++] = ts->stack[t_idx++];
+			}
+			while (e_idx < orig_existing_depth) {
+				existing->stack[out_idx++] =
+					existing->stack[to_copy + e_idx++];
+			}
+		}
+
+		for (j = to_copy; j < ts->depth; j++)
+			free(ts->stack[j].name);
+
+		existing->depth = orig_existing_depth + to_copy;
+
+		if (ts->seq > existing->seq)
+			existing->seq = ts->seq;
+
+		if (ts->nr_watermarks > 0) {
+			struct ftrace_indent_watermark *old_wm = existing->watermarks;
+			int old_nr = existing->nr_watermarks;
+			int e_wm = 0, t_wm = 0;
+
+			existing->watermarks = NULL;
+			existing->max_watermarks = 0;
+			existing->nr_watermarks = 0;
+
+			while (e_wm < old_nr && t_wm < ts->nr_watermarks) {
+				if (old_wm[e_wm].seq <= ts->watermarks[t_wm].seq) {
+					ftrace_task_state__record_indent(
+						existing,
+						old_wm[e_wm].seq,
+						old_wm[e_wm].raw_indent);
+					e_wm++;
+				} else {
+					ftrace_task_state__record_indent(
+						existing,
+						ts->watermarks[t_wm].seq,
+						ts->watermarks[t_wm].raw_indent);
+					t_wm++;
+				}
+			}
+			while (e_wm < old_nr) {
+				ftrace_task_state__record_indent(
+					existing,
+					old_wm[e_wm].seq,
+					old_wm[e_wm].raw_indent);
+				e_wm++;
+			}
+			while (t_wm < ts->nr_watermarks) {
+				ftrace_task_state__record_indent(
+					existing,
+					ts->watermarks[t_wm].seq,
+					ts->watermarks[t_wm].raw_indent);
+				t_wm++;
+			}
+			free(old_wm);
+		}
+
+		free(ts->watermarks);
+		free(ts->stack);
+		free(ts);
+		return existing;
+	}
+
+	ts->pid = new_pid;
+	key = hash_32((u32)new_pid, FTRACE_PID_HASH_BITS);
+	hlist_add_head(&ts->node, &ftrace_task_hash[key]);
+	return ts;
+}
+
+/**
+ * ftrace__get_single_pid - Retrieve the single target process ID being traced
+ * @ftrace: Perf ftrace context
+ *
+ * Check whether @ftrace is configured to trace a single process without
+ * child inheritance. If the thread map contains exactly one valid process
+ * ID and child task inheritance is disabled, return that PID so that
+ * initial per-CPU task states can be bound directly to the target process.
+ *
+ * Return: The target process ID if tracing a single process without
+ *         inheritance, or -1 otherwise.
+ */
+static pid_t ftrace__get_single_pid(struct perf_ftrace *ftrace)
+{
+	struct perf_thread_map *threads;
+	pid_t pid;
+
+	if (!ftrace || !ftrace->evlist || ftrace->inherit)
+		return -1;
+
+	threads = evlist__core(ftrace->evlist)->threads;
+	if (!threads || perf_thread_map__nr(threads) != 1)
+		return -1;
+
+	pid = perf_thread_map__pid(threads, 0);
+	return (pid > 0) ? pid : -1;
+}
+
 /**
  * get_cpu_state - Get or allocate active task stack state for a given CPU
  * @ftrace: Perf ftrace context
@@ -132,17 +426,107 @@ static int ftrace_task_state__grow_stack(struct ftrace_task_state *cs)
  * Return: A pointer to the CPU's active task stack state, or NULL on error or
  * out-of-bounds CPU index.
  */
-static struct ftrace_task_state *get_cpu_state(struct perf_ftrace *ftrace __maybe_unused, int cpu)
+static struct ftrace_task_state *get_cpu_state(struct perf_ftrace *ftrace, int cpu)
 {
+	pid_t single_pid;
+
 	if (cpu < 0 || cpu >= FTRACE_MAX_CPUS)
 		return NULL;
 
-	if (!ftrace_cpu_tasks[cpu])
-		ftrace_cpu_tasks[cpu] = get_task_state(FTRACE_UNKNOWN_PID(cpu));
+	if (!ftrace_cpu_tasks[cpu]) {
+		single_pid = ftrace__get_single_pid(ftrace);
+		if (single_pid > 0)
+			ftrace_cpu_tasks[cpu] = get_task_state(single_pid);
+		else
+			ftrace_cpu_tasks[cpu] = get_task_state(FTRACE_UNKNOWN_PID(cpu));
+	}
 
 	return ftrace_cpu_tasks[cpu];
 }
 
+/**
+ * parse_context_switch - Parse context switch banner and update CPU's task
+ * @line: The trace line containing the banner
+ *
+ * Check if @line is a context switch banner (e.g. " 0) <idle>-0 => migrati-19").
+ * If so, extract the CPU number and both the outgoing and incoming task PIDs,
+ * re-key any initial unknown task state on this CPU to the outgoing PID, and
+ * update the active task mapping for that CPU to the incoming PID.
+ *
+ * Return: true if @line was a context switch banner, false otherwise.
+ */
+static bool parse_context_switch(const char *line)
+{
+	const char *arrow = strstr(line, " => ");
+	const char *p;
+	const char *next;
+	const char *hyphen;
+	const char *h;
+	char *endptr;
+	char *end_pid;
+	long cpu_val;
+	long prev_pid;
+	long next_pid;
+	pid_t real_prev;
+	pid_t real_next;
+
+	if (!arrow)
+		return false;
+
+	/* Find CPU number before ')' */
+	p = skip_spaces(line);
+	cpu_val = strtol(p, &endptr, 10);
+	if (endptr == p || *endptr != ')')
+		return false;
+
+	if (cpu_val < 0 || cpu_val >= FTRACE_MAX_CPUS)
+		return false;
+
+	/*
+	 * Parse outgoing task PID before " => " (" CPU) comm-pid => ...").
+	 * Scan backwards from arrow to find the hyphen before the numeric PID.
+	 */
+	h = arrow - 1;
+	while (h > endptr && *h == ' ')
+		h--;
+	while (h > endptr && *h != '-')
+		h--;
+	if (h <= endptr || *h != '-')
+		return false;
+
+	prev_pid = strtol(h + 1, &end_pid, 10);
+	if (end_pid == h + 1)
+		return false;
+
+	/* Parse target task PID after " => " (" => comm-pid") */
+	next = skip_spaces(arrow + 4);
+	hyphen = strrchr(next, '-');
+	if (!hyphen)
+		return false;
+
+	next_pid = strtol(hyphen + 1, &end_pid, 10);
+	if (end_pid == hyphen + 1)
+		return false;
+
+	real_prev = (prev_pid == 0) ? FTRACE_IDLE_PID(cpu_val) : (pid_t)prev_pid;
+	real_next = (next_pid == 0) ? FTRACE_IDLE_PID(cpu_val) : (pid_t)next_pid;
+
+	/*
+	 * If no context switch was seen previously on this CPU, its initial
+	 * pre-banner trace records were accumulated under an unknown synthetic
+	 * PID. Re-key that state to real_prev so the initial task's stack and
+	 * frames are preserved when it schedules back in.
+	 */
+	if (ftrace_cpu_tasks[cpu_val] &&
+	    ftrace_cpu_tasks[cpu_val]->pid == FTRACE_UNKNOWN_PID(cpu_val)) {
+		ftrace_cpu_tasks[cpu_val] =
+			ftrace_task_state__rekey(ftrace_cpu_tasks[cpu_val], real_prev);
+	}
+
+	ftrace_cpu_tasks[cpu_val] = get_task_state(real_next);
+	return true;
+}
+
 /**
  * ftrace_parse_retaddr - Parse return address comment from trace line
  * @str: The trace line containing the comment
@@ -307,10 +691,32 @@ static const char *find_func_column(const struct perf_ftrace *ftrace,
 	p = endptr;
 
 	if (ftrace->graph_verbose) {
+		const char *proc_start = p;
+
 		p = strchr(p, '|');
 		if (!p)
 			return NULL;
 
+		if (val >= 0 && val < FTRACE_MAX_CPUS) {
+			const char *h = p - 1;
+
+			while (h >= proc_start && *h != '-')
+				h--;
+			if (h >= proc_start && *h == '-') {
+				long pid_val = strtol(h + 1, NULL, 10);
+				pid_t task_pid = (pid_val == 0) ?
+						 FTRACE_IDLE_PID(val) :
+						 (pid_t)pid_val;
+
+				ftrace_cpu_tasks[(int)val] = get_task_state(task_pid);
+			}
+		}
+
+		/*
+		 * In verbose mode, latency-format is also enabled by
+		 * set_tracing_funcgraph_verbose(), emitting a "<lat-fmt> |"
+		 * column after the comm-pid column.
+		 */
 		p = strchr(p + 1, '|');
 		if (!p)
 			return NULL;
@@ -383,13 +789,17 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 	bool has_semicolon;
 
 	pipe_char = find_func_column(ftrace, line, &cpu);
-	if (!pipe_char)
+	if (!pipe_char) {
+		parse_context_switch(line);
 		return print_line(ftrace, line, out);
+	}
 
 	cs = get_cpu_state(ftrace, cpu);
 	if (!cs)
 		return print_line(ftrace, line, out);
 
+	cs->seq = ++ftrace_event_seq;
+
 	after_pipe = pipe_char + 1;
 	func_start = skip_spaces(after_pipe);
 	leading_spaces = func_start - after_pipe;
@@ -402,6 +812,8 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 	if (is_exit) {
 		char exit_name[128];
 
+		ftrace_task_state__record_indent(cs, cs->seq, leading_spaces);
+
 		extract_exit_func_name(func_start, exit_name, sizeof(exit_name));
 
 		top_raw_indent = -1;
@@ -453,6 +865,9 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 	has_open_brace = (strchr(func_start, '{') != NULL);
 	has_semicolon = (strchr(func_start, ';') != NULL);
 
+	if (has_open_brace || has_semicolon)
+		ftrace_task_state__record_indent(cs, cs->seq, leading_spaces);
+
 	top_raw_indent = -1;
 	for (i = cs->depth - 1; i >= 0; i--) {
 		top_raw_indent = cs->stack[i].raw_indent;
@@ -466,6 +881,12 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 	if (has_open_brace && !has_semicolon) {
 		char entry_name[128];
 
+		while (cs->depth > 0 &&
+		       cs->stack[cs->depth - 1].raw_indent >= leading_spaces) {
+			free(cs->stack[cs->depth - 1].name);
+			cs->depth--;
+		}
+
 		extract_entry_func_name(func_start, entry_name, sizeof(entry_name));
 		if (cs->depth >= cs->max_stack &&
 		    ftrace_task_state__grow_stack(cs) < 0) {
@@ -479,6 +900,7 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 			cs->stack[cs->depth].inlined = false;
 			cs->stack[cs->depth].indent = leading_spaces;
 			cs->stack[cs->depth].raw_indent = leading_spaces;
+			cs->stack[cs->depth].seq = cs->seq;
 			cs->depth++;
 		}
 	}
tools/perf/util/ftrace.h +10 -0
index fa2974f46582..45f5090d0115 100644
@@ -109,6 +109,12 @@ struct ftrace_stack_frame {
 	bool inlined;
 	int indent;
 	int raw_indent;
+	u64 seq;
+};
+
+struct ftrace_indent_watermark {
+	u64 seq;
+	int raw_indent;
 };
 
 struct ftrace_task_state {
@@ -119,6 +125,10 @@ struct ftrace_task_state {
 	int			depth;
 	int			inlined_depth;
 	int			overflow_depth;
+	u64			seq;
+	struct ftrace_indent_watermark *watermarks;
+	int			max_watermarks;
+	int			nr_watermarks;
 };
 
 bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset);
Signature · 1 line
2.55.0
#17 lore raw

Re: [PATCH perf-tools-next v12 05/10] perf ftrace: Support context switch tracking in function graph tracer

Sashiko has reviewed this patch and found no issues.
Quoted 13 lines

When tracing multiple tasks across multiple CPUs, kernel context switches cause different threads of execution to interleave on the same CPU. Without context switch awareness, interleaved function calls corrupt the tracked call stack.

Add context switch detection and per-task state tracking:

1. Context switch banner parsing

Implement parse_context_switch() to parse context switch banners (e.g. " 0) <idle>-0 => migrati-19") in the trace stream, extracting the CPU number and both outgoing and incoming task PIDs.

Quoted 8 lines

2. Task state rekeying and migration

[ ... ]
when context-switch banners are absent.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
maintainer
#18 lore raw

Re: [PATCH perf-tools-next v12 05/10] perf ftrace: Support context switch tracking in function graph tracer

In reply to Aaron Tomlin
Quoted 9 lines · Aaron Tomlin

When tracing multiple tasks across multiple CPUs, kernel context switches cause different threads of execution to interleave on the same CPU. Without context switch awareness, interleaved function calls corrupt the tracked call stack.

Add context switch detection and per-task state tracking:

1. Context switch banner parsing

Quoted 8 lines

Implement parse_context_switch() to parse context switch banners (e.g. " 0) <idle>-0 => migrati-19") in the trace stream, extracting the CPU number and both outgoing and incoming task PIDs.

2. Task state rekeying and migration

Implement ftrace_task_state__rekey() to reassign initial unknown task states to real PIDs upon discovering outgoing task identity.

Yeah, it's unfortunate that the default ftrace output doesn't show PIDs.

Quoted 58 lines

Track monotonic trace event sequence numbers on each frame so merges can locate the exact splice boundary when tasks migrate across CPUs and back before delayed banners are emitted. Prune exited frames from pre-migration calls while preserving intermediate active frames and splicing them directly into the task call stack.

3. Verbose mode and single-PID binding

In find_func_column(), parse the comm-pid column in verbose mode to bind active task state on each line record.

In get_cpu_state(), pre-bind single-PID workloads across all CPUs when context-switch banners are absent.

---
 tools/perf/util/ftrace.c | 430 ++++++++++++++++++++++++++++++++++++++-
 tools/perf/util/ftrace.h |  10 +
 2 files changed, 436 insertions(+), 4 deletions(-)
tools/perf/util/ftrace.c +5 -0
index 95db57e70d11..fc385d680911 100644
@@ -13,11 +13,14 @@
 #include <errno.h>
 #include <sys/types.h>
 #include "debug.h"
+#include "evlist.h"
 #include "util/ftrace.h"
 #include "util/string2.h"
+#include <perf/threadmap.h>
 
 static struct hlist_head ftrace_task_hash[FTRACE_PID_HASH_SIZE];
 static struct ftrace_task_state *ftrace_cpu_tasks[FTRACE_MAX_CPUS];
+static u64 ftrace_event_seq;
 
 /**
  * ftrace_cpu_states__clear - Free allocated CPU and task stack states
@@ -36,12 +39,14 @@ void ftrace_cpu_states__clear(void)
 			for (j = 0; j < ts->depth; j++)
 				free(ts->stack[j].name);
 			free(ts->stack);
+			free(ts->watermarks);
 			hlist_del_init(&ts->node);
 			free(ts);
 		}
 	}
 
 	memset(ftrace_cpu_tasks, 0, sizeof(ftrace_cpu_tasks));
+	ftrace_event_seq = 0;
 }
 
 /**

tools/perf/util/ftrace.c static struct ftrace_task_state *get_task_state(pid_t pid) · line 108

Quoted 6 lines
tools/perf/util/ftrace.c +3 -0
@@ -98,6 +103,119 @@ static struct ftrace_task_state *get_task_state(pid_t pid)
 	return ts;
 }
 
+/**
+ * ftrace_task_state__record_indent - Record an indentation watermark for a task
+ * @cs: Task state to update

Why is it called 'cs' instead of 'ts'? Same goes to the later functions too.

Quoted 85 lines
tools/perf/util/ftrace.c +85 -0
+ * @seq: Event sequence number of the observed trace record
+ * @raw_indent: Raw indentation (leading spaces) of the observed trace record
+ *
+ * Maintain a monotonic suffix-minima array of (seq, raw_indent) pairs. Any
+ * previously recorded watermarks with an indentation greater than or equal to
+ * @raw_indent are dominated by this newer, shallower (or equal) event for all
+ * earlier frames and are popped before appending the new watermark.
+ *
+ * Return: 0 on success, or -ENOMEM if memory allocation fails.
+ */
+static int ftrace_task_state__record_indent(struct ftrace_task_state *cs,
+					    u64 seq, int raw_indent)
+{
+	while (cs->nr_watermarks > 0 &&
+	       cs->watermarks[cs->nr_watermarks - 1].raw_indent >= raw_indent)
+		cs->nr_watermarks--;
+
+	if (cs->nr_watermarks >= cs->max_watermarks) {
+		int new_max = cs->max_watermarks ? cs->max_watermarks * 2 : 16;
+		struct ftrace_indent_watermark *new_wm;
+
+		new_wm = realloc(cs->watermarks, new_max * sizeof(*new_wm));
+		if (!new_wm)
+			return -ENOMEM;
+		cs->watermarks = new_wm;
+		cs->max_watermarks = new_max;
+	}
+
+	cs->watermarks[cs->nr_watermarks].seq = seq;
+	cs->watermarks[cs->nr_watermarks].raw_indent = raw_indent;
+	cs->nr_watermarks++;
+	return 0;
+}
+
+/**
+ * ftrace_task_state__min_indent_after - Query minimum indentation after a sequence number
+ * @cs: Task state whose watermarks are queried
+ * @frame_seq: Sequence number of the frame to check against
+ *
+ * Because @cs->watermarks stores suffix minima in strictly increasing order of
+ * both sequence number and indentation, the first watermark with seq > @frame_seq
+ * holds the minimum raw indentation observed on @cs after @frame_seq.
+ *
+ * Return: The minimum raw indentation observed on @cs after @frame_seq, or -1
+ *         if no events were recorded on @cs after @frame_seq.
+ */
+static int ftrace_task_state__min_indent_after(const struct ftrace_task_state *cs,
+					       u64 frame_seq)
+{
+	int i;
+
+	for (i = 0; i < cs->nr_watermarks; i++) {
+		if (cs->watermarks[i].seq > frame_seq)
+			return cs->watermarks[i].raw_indent;
+	}
+
+	return -1;
+}
+
+/**
+ * ftrace_task_state__prune_stack - Prune frames that exited on another task state
+ * @cs: Task state whose call stack should be pruned
+ * @other: Other task state representing the same task on another CPU
+ *
+ * Remove any frames on @cs->stack that are known to have exited based on
+ * subsequent trace events recorded in @other->watermarks (i.e. events with
+ * seq > frame->seq and raw_indent <= frame->raw_indent). Synthesised inline
+ * frames are retained if and only if their enclosing real frame survives.
+ */
+static void ftrace_task_state__prune_stack(struct ftrace_task_state *cs,
+					   const struct ftrace_task_state *other)
+{
+	bool keep_inlines = false;
+	bool seen_real = false;
+	int i, dst = 0;
+
+	for (i = cs->depth - 1; i >= 0; i--) {
+		struct ftrace_stack_frame *frame = &cs->stack[i];
+		int min_indent = ftrace_task_state__min_indent_after(other, frame->seq);
+		bool prune;
+
+		if (frame->inlined) {
+			if (seen_real)
+				prune = !keep_inlines;
+			else

tools/perf/util/ftrace.c static struct ftrace_task_state *get_task_state(pid_t pid) · line 203

Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+				prune = (min_indent >= 0);
+		} else {
+			prune = (min_indent >= 0 && frame->raw_indent >= min_indent);
+			seen_real = true;
+			keep_inlines = !prune;
+		}
+
+		if (prune) {
+			free(frame->name);
+			frame->name = NULL;

You can use zfree(&frame->name);

Quoted 37 lines
tools/perf/util/ftrace.c +30 -0
+			if (frame->inlined && cs->inlined_depth > 0)
+				cs->inlined_depth--;
+		}
+	}
+
+	for (i = 0; i < cs->depth; i++) {
+		if (cs->stack[i].name) {
+			if (dst != i)
+				cs->stack[dst] = cs->stack[i];
+			dst++;
+		}
+	}
+	cs->depth = dst;
+}
+
 /**
  * ftrace_task_state__grow_stack - Double the capacity of a task's call stack
  * @cs: Task state whose stack should be resized
@@ -124,6 +242,182 @@ static int ftrace_task_state__grow_stack(struct ftrace_task_state *cs)
 	return 0;
 }
 
+/**
+ * ftrace_task_state__rekey - Reassign a task state to a new PID
+ * @ts: The task state to re-key
+ * @new_pid: The new process ID to associate with the state
+ *
+ * Rehash @ts under @new_pid. If a task state for @new_pid already exists,
+ * merge or replace it and update active CPU references accordingly.
+ *
+ * Return: A pointer to the active task state associated with @new_pid.
+ */
+static struct ftrace_task_state *ftrace_task_state__rekey(struct ftrace_task_state *ts,
+							  pid_t new_pid)
+{
+	struct ftrace_task_state *existing;
+	u32 key;

tools/perf/util/ftrace.c static int ftrace_task_state__grow_stack(struct ftrace_task_state *cs) · line 269

Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+	int j, c;
+	int to_copy;
+
+	if (ts->pid == new_pid)
+		return ts;
+
+	hlist_del_init(&ts->node);
+
+	existing = find_task_state(new_pid);
+	if (existing) {

I think it's better to handle the else case first as it's much shorter. You can save an indentation level for the true case then.

tools/perf/util/ftrace.c static int ftrace_task_state__grow_stack(struct ftrace_task_state *cs) · line 276

Quoted 7 lines
tools/perf/util/ftrace.c +7 -0
+		int orig_existing_depth;
+		int total_needed;
+
+		for (c = 0; c < FTRACE_MAX_CPUS; c++) {
+			if (ftrace_cpu_tasks[c] == ts)
+				ftrace_cpu_tasks[c] = existing;
+		}

I don't this is necessary. IIUC the rekey function is supposed to be called for a specific CPU and caller will update the CPU state.

tools/perf/util/ftrace.c static int ftrace_task_state__grow_stack(struct ftrace_task_state *cs) · line 283

Quoted 7 lines
tools/perf/util/ftrace.c +7 -0
+
+		/*
+		 * Prune frames on existing and ts that exited during subsequent
+		 * execution recorded on the other state.
+		 */
+		ftrace_task_state__prune_stack(existing, ts);
+		ftrace_task_state__prune_stack(ts, existing);

I'm not sure about this. In this context, it knows the 'ts' is uptodate and the 'existing' contains the old state. Does it need to prune the ts using the existing?

IIUC this code assumes either ts or existing may have uptodate state like in general cases but it would make the code unnecessarily longer.

Quoted 86 lines
tools/perf/util/ftrace.c +86 -0
+
+		orig_existing_depth = existing->depth;
+		to_copy = ts->depth;
+		total_needed = orig_existing_depth + to_copy;
+
+		while (total_needed > existing->max_stack) {
+			if (ftrace_task_state__grow_stack(existing) < 0)
+				break;
+		}
+		if (total_needed > existing->max_stack) {
+			if (existing->max_stack > orig_existing_depth)
+				to_copy = existing->max_stack - orig_existing_depth;
+			else
+				to_copy = 0;
+		}
+
+		if (to_copy > 0) {
+			int e_idx = 0, t_idx = 0, out_idx = 0;
+
+			if (orig_existing_depth > 0) {
+				memmove(&existing->stack[to_copy],
+					&existing->stack[0],
+					orig_existing_depth * sizeof(existing->stack[0]));
+			}
+
+			while (e_idx < orig_existing_depth && t_idx < to_copy) {
+				if (existing->stack[to_copy + e_idx].seq <=
+				    ts->stack[t_idx].seq) {
+					existing->stack[out_idx++] =
+						existing->stack[to_copy + e_idx++];
+				} else {
+					if (ts->stack[t_idx].inlined)
+						existing->inlined_depth++;
+					existing->stack[out_idx++] =
+						ts->stack[t_idx++];
+				}
+			}
+			while (t_idx < to_copy) {
+				if (ts->stack[t_idx].inlined)
+					existing->inlined_depth++;
+				existing->stack[out_idx++] = ts->stack[t_idx++];
+			}
+			while (e_idx < orig_existing_depth) {
+				existing->stack[out_idx++] =
+					existing->stack[to_copy + e_idx++];
+			}
+		}
+
+		for (j = to_copy; j < ts->depth; j++)
+			free(ts->stack[j].name);
+
+		existing->depth = orig_existing_depth + to_copy;
+
+		if (ts->seq > existing->seq)
+			existing->seq = ts->seq;
+
+		if (ts->nr_watermarks > 0) {
+			struct ftrace_indent_watermark *old_wm = existing->watermarks;
+			int old_nr = existing->nr_watermarks;
+			int e_wm = 0, t_wm = 0;
+
+			existing->watermarks = NULL;
+			existing->max_watermarks = 0;
+			existing->nr_watermarks = 0;
+
+			while (e_wm < old_nr && t_wm < ts->nr_watermarks) {
+				if (old_wm[e_wm].seq <= ts->watermarks[t_wm].seq) {
+					ftrace_task_state__record_indent(
+						existing,
+						old_wm[e_wm].seq,
+						old_wm[e_wm].raw_indent);
+					e_wm++;
+				} else {
+					ftrace_task_state__record_indent(
+						existing,
+						ts->watermarks[t_wm].seq,
+						ts->watermarks[t_wm].raw_indent);
+					t_wm++;
+				}
+			}
+			while (e_wm < old_nr) {
+				ftrace_task_state__record_indent(
+					existing,
+					old_wm[e_wm].seq,
+					old_wm[e_wm].raw_indent);
+				e_wm++;

tools/perf/util/ftrace.c static int ftrace_task_state__grow_stack(struct ftrace_task_state *cs) · line 379

Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+			}
+			while (t_wm < ts->nr_watermarks) {
+				ftrace_task_state__record_indent(
+					existing,
+					ts->watermarks[t_wm].seq,
+					ts->watermarks[t_wm].raw_indent);
+				t_wm++;
+			}
+			free(old_wm);
+		}

I'm curious if it really needs to have watermarks. I guess it's enough to have a watermark for each task (actually for each CPUs that are running unidentified tasks) and only updates it on context switch.

Thanks,
Namhyung

Quoted 12 lines
tools/perf/util/ftrace.c +12 -0
+
+		free(ts->watermarks);
+		free(ts->stack);
+		free(ts);
+		return existing;
+	}
+
+	ts->pid = new_pid;
+	key = hash_32((u32)new_pid, FTRACE_PID_HASH_BITS);
+	hlist_add_head(&ts->node, &ftrace_task_hash[key]);
+	return ts;
+}
#19 lore raw

Re: [PATCH perf-tools-next v12 05/10] perf ftrace: Support context switch tracking in function graph tracer

Quoted 14 lines · Namhyung Kim
Quoted 13 lines · Aaron Tomlin

When tracing multiple tasks across multiple CPUs, kernel context switches cause different threads of execution to interleave on the same CPU. Without context switch awareness, interleaved function calls corrupt the tracked call stack.

Add context switch detection and per-task state tracking:

1. Context switch banner parsing

Implement parse_context_switch() to parse context switch banners (e.g. " 0) <idle>-0 => migrati-19") in the trace stream, extracting the CPU number and both outgoing and incoming task PIDs.

Quoted 6 lines
Quoted 4 lines

2. Task state rekeying and migration

Implement ftrace_task_state__rekey() to reassign initial unknown task states to real PIDs upon discovering outgoing task identity.

Yeah, it's unfortunate that the default ftrace output doesn't show PIDs.

Hi Namhyung,

Thank you for your feedback.

Yes, it is indeed.

Quoted 38 lines
Quoted 38 lines
tools/perf/util/ftrace.c +5 -0
index 95db57e70d11..fc385d680911 100644
@@ -13,11 +13,14 @@
 #include <errno.h>
 #include <sys/types.h>
 #include "debug.h"
+#include "evlist.h"
 #include "util/ftrace.h"
 #include "util/string2.h"
+#include <perf/threadmap.h>
 
 static struct hlist_head ftrace_task_hash[FTRACE_PID_HASH_SIZE];
 static struct ftrace_task_state *ftrace_cpu_tasks[FTRACE_MAX_CPUS];
+static u64 ftrace_event_seq;
 
 /**
  * ftrace_cpu_states__clear - Free allocated CPU and task stack states
@@ -36,12 +39,14 @@ void ftrace_cpu_states__clear(void)
 			for (j = 0; j < ts->depth; j++)
 				free(ts->stack[j].name);
 			free(ts->stack);
+			free(ts->watermarks);
 			hlist_del_init(&ts->node);
 			free(ts);
 		}
 	}
 
 	memset(ftrace_cpu_tasks, 0, sizeof(ftrace_cpu_tasks));
+	ftrace_event_seq = 0;
 }
 
 /**
@@ -98,6 +103,119 @@ static struct ftrace_task_state *get_task_state(pid_t pid)
 	return ts;
 }
 
Quoted 6 lines

tools/perf/util/ftrace.c static struct ftrace_task_state *get_task_state(pid_t pid) · line 108

Quoted 3 lines
tools/perf/util/ftrace.c +3 -0
+/**
+ * ftrace_task_state__record_indent - Record an indentation watermark for a task
+ * @cs: Task state to update

Why is it called 'cs' instead of 'ts'? Same goes to the later functions too.

Agreed. That is a relic from when the struct was named ftrace_cpu_state. I will rename "cs" to "ts" consistently across all functions in the series.

Quoted 5 lines

tools/perf/util/ftrace.c static struct ftrace_task_state *get_task_state(pid_t pid) · line 203

Quoted 3 lines
tools/perf/util/ftrace.c +3 -0
+		if (prune) {
+			free(frame->name);
+			frame->name = NULL;

You can use zfree(&frame->name);

Acknowledged. I will switch to zfree(&frame->name).

Quoted 10 lines
Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+static struct ftrace_task_state *ftrace_task_state__rekey(struct ftrace_task_state *ts,
+							  pid_t new_pid)
+{
+	struct ftrace_task_state *existing;
+	u32 key;
+	int j, c;
+	int to_copy;
+
+	if (ts->pid == new_pid)
+		return ts;
Quoted 8 lines

tools/perf/util/ftrace.c static int ftrace_task_state__grow_stack(struct ftrace_task_state *cs) · line 269

Quoted 5 lines
tools/perf/util/ftrace.c +5 -0
+
+	hlist_del_init(&ts->node);
+
+	existing = find_task_state(new_pid);
+	if (existing) {

I think it's better to handle the else case first as it's much shorter. You can save an indentation level for the true case then.

Agreed.

Quoted 11 lines

tools/perf/util/ftrace.c static int ftrace_task_state__grow_stack(struct ftrace_task_state *cs) · line 276

Quoted 8 lines
tools/perf/util/ftrace.c +7 -0
+		int orig_existing_depth;
+		int total_needed;
+
+		for (c = 0; c < FTRACE_MAX_CPUS; c++) {
+			if (ftrace_cpu_tasks[c] == ts)
+				ftrace_cpu_tasks[c] = existing;
+		}

I don't this is necessary. IIUC the rekey function is supposed to be called for a specific CPU and caller will update the CPU state.

Fair. Since ts is keyed to a single CPU's synthetic unknown PID (FTRACE_UNKNOWN_PID(cpu)) and parse_context_switch() immediately updates ftrace_cpu_tasks[cpu_val], no other CPU slot can reference ts. I will remove this loop.

Quoted 12 lines

tools/perf/util/ftrace.c static int ftrace_task_state__grow_stack(struct ftrace_task_state *cs) · line 283

Quoted 7 lines
tools/perf/util/ftrace.c +7 -0
+
+		/*
+		 * Prune frames on existing and ts that exited during subsequent
+		 * execution recorded on the other state.
+		 */
+		ftrace_task_state__prune_stack(existing, ts);
+		ftrace_task_state__prune_stack(ts, existing);

I'm not sure about this. In this context, it knows the 'ts' is uptodate and the 'existing' contains the old state. Does it need to prune the ts using the existing?

Quoted 2 lines

IIUC this code assumes either ts or existing may have uptodate state like in general cases but it would make the code unnecessarily longer.

The reason this handled both directions is due to how trace_pipe merges per-CPU buffers when a task starts on CPU 0 (unidentified, tracked in ts under FTRACE_UNKNOWN_PID(0)) and migrates to CPU 1:

1. Task X starts on CPU 0 before any context switch banner is seen on CPU 0, so its initial frames are recorded in ts.

2. Task X migrates to CPU 1. If CPU 1's incoming banner ("1) idle => X") and subsequent function exits are read from trace_pipe before CPU 0's outgoing banner ("0) X => idle") is flushed, existing is created on CPU 1 first and actually holds the newer state, while ts holds the older pre-migration frames from CPU 0.

3. Conversely, if Task X was already known on CPU 1 (existing) and migrated to an unbannered CPU 0 (ts), then ts holds the newer state and existing holds the older state.

That said, I agree that maintaining a dynamic per-task watermarks[] array for arbitrary multi-hop migrations before the first banner is superfluous. Replacing the dynamic watermarks array with a single scalar minimum-indent watermark on struct ftrace_task_state makes this much simpler and removes ftrace_task_state__record_indent(), ftrace_task_state__min_indent_after(), and the watermark merge loop in ftrace_task_state__rekey().

Thank you!

Kind regards,

Signature · 1 line
Aaron Tomlin
maintainer
#20 lore raw

Re: [PATCH perf-tools-next v12 05/10] perf ftrace: Support context switch tracking in function graph tracer

Quoted 2 lines · Aaron Tomlin
Quoted 1 line · Namhyung Kim

On Sun, Oct 25, 2026 at 10:57:30PM -0400, Aaron Tomlin wrote:

[SNIP]

Quoted 28 lines
Quoted 12 lines

tools/perf/util/ftrace.c static int ftrace_task_state__grow_stack(struct ftrace_task_state *cs) · line 283

Quoted 7 lines
tools/perf/util/ftrace.c +7 -0
+
+		/*
+		 * Prune frames on existing and ts that exited during subsequent
+		 * execution recorded on the other state.
+		 */
+		ftrace_task_state__prune_stack(existing, ts);
+		ftrace_task_state__prune_stack(ts, existing);

I'm not sure about this. In this context, it knows the 'ts' is uptodate and the 'existing' contains the old state. Does it need to prune the ts using the existing?

Quoted 2 lines

IIUC this code assumes either ts or existing may have uptodate state like in general cases but it would make the code unnecessarily longer.

The reason this handled both directions is due to how trace_pipe merges per-CPU buffers when a task starts on CPU 0 (unidentified, tracked in ts under FTRACE_UNKNOWN_PID(0)) and migrates to CPU 1:

1. Task X starts on CPU 0 before any context switch banner is seen on CPU 0, so its initial frames are recorded in ts.

2. Task X migrates to CPU 1. If CPU 1's incoming banner ("1) idle => X") and subsequent function exits are read from trace_pipe before CPU 0's outgoing banner ("0) X => idle") is flushed, existing is created on CPU 1 first and actually holds the newer state, while ts holds the older pre-migration frames from CPU 0.

Quoted 3 lines

3. Conversely, if Task X was already known on CPU 1 (existing) and migrated to an unbannered CPU 0 (ts), then ts holds the newer state and existing holds the older state.

IIRC we use the global trace pipe so I expect its output in time order. How can it see CPU 1's sched switch before the CPU 0's? Maybe some time drifts on each CPU? Anyway, if that happens in practice, yeah then we can check in both directions.

Quoted 7 lines

That said, I agree that maintaining a dynamic per-task watermarks[] array for arbitrary multi-hop migrations before the first banner is superfluous. Replacing the dynamic watermarks array with a single scalar minimum-indent watermark on struct ftrace_task_state makes this much simpler and removes ftrace_task_state__record_indent(), ftrace_task_state__min_indent_after(), and the watermark merge loop in ftrace_task_state__rekey().

Sounds good, thanks!
Namhyung

#21 lore raw

Re: [PATCH perf-tools-next v12 05/10] perf ftrace: Support context switch tracking in function graph tracer

Quoted 32 lines · Namhyung Kim
Quoted 2 lines · Aaron Tomlin
Quoted 1 line · Namhyung Kim

On Sun, Oct 25, 2026 at 10:57:30PM -0400, Aaron Tomlin wrote:

[SNIP]

Quoted 28 lines
Quoted 12 lines

tools/perf/util/ftrace.c static int ftrace_task_state__grow_stack(struct ftrace_task_state *cs) · line 283

Quoted 7 lines
tools/perf/util/ftrace.c +7 -0
+
+		/*
+		 * Prune frames on existing and ts that exited during subsequent
+		 * execution recorded on the other state.
+		 */
+		ftrace_task_state__prune_stack(existing, ts);
+		ftrace_task_state__prune_stack(ts, existing);

I'm not sure about this. In this context, it knows the 'ts' is uptodate and the 'existing' contains the old state. Does it need to prune the ts using the existing?

Quoted 2 lines

IIUC this code assumes either ts or existing may have uptodate state like in general cases but it would make the code unnecessarily longer.

The reason this handled both directions is due to how trace_pipe merges per-CPU buffers when a task starts on CPU 0 (unidentified, tracked in ts under FTRACE_UNKNOWN_PID(0)) and migrates to CPU 1:

1. Task X starts on CPU 0 before any context switch banner is seen on CPU 0, so its initial frames are recorded in ts.

2. Task X migrates to CPU 1. If CPU 1's incoming banner ("1) idle => X") and subsequent function exits are read from trace_pipe before CPU 0's outgoing banner ("0) X => idle") is flushed, existing is created on CPU 1 first and actually holds the newer state, while ts holds the older pre-migration frames from CPU 0.

Quoted 8 lines
Quoted 3 lines

3. Conversely, if Task X was already known on CPU 1 (existing) and migrated to an unbannered CPU 0 (ts), then ts holds the newer state and existing holds the older state.

IIRC we use the global trace pipe so I expect its output in time order. How can it see CPU 1's sched switch before the CPU 0's? Maybe some time drifts on each CPU? Anyway, if that happens in practice, yeah then we can check in both directions.

Hi Namhyung,

Yes, global trace_pipe is in timestamp order.

The reason is that function_graph does not emit context-switch banners at the time of sched_switch; instead, verif_pid() in kernel/trace/trace_functions_graph.c emits "CPU) prev => next" lazily only when the next traced function entry/return event is formatted on that CPU.

So if Task X runs traced functions on CPU 0 (i.e. where prev_pid was -1, no banner was printed yet) and then schedules out on CPU 0 to an untraced task (e.g. under -T|-G filters or idle) before migrating to CPU 1:

- CPU 0 emits no fgraph events while the untraced task runs, so CPU 0's "0) X => ..." banner is not emitted yet

- Task X immediately hits traced functions on CPU 1, which triggers verif_pid() on CPU 1 and emits "1) ... => X" (and subsequent exits) at an earlier timestamp

- Only later, when a task on CPU 0 next hits a traced function, does verif_pid() on CPU 0 emit "0) X => ..."

Quoted 8 lines
Quoted 6 lines

That said, I agree that maintaining a dynamic per-task watermarks[] array for arbitrary multi-hop migrations before the first banner is superfluous. Replacing the dynamic watermarks array with a single scalar minimum-indent watermark on struct ftrace_task_state makes this much simpler and removes ftrace_task_state__record_indent(), ftrace_task_state__min_indent_after(), and the watermark merge loop in ftrace_task_state__rekey().

Sounds good, thanks!

Acknowledged!

Please ignore my v13. Sashiko found additional valid concerns which I need to address first.

Thank you,

Signature · 1 line
Aaron Tomlin
#22 lore raw

PATCH v12 6/10 perf ftrace: Support display of inlined functions in function graph tracer

In reply to Aaron Tomlin

The Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining.

Address this by introducing the --inline option to perf ftrace for the function_graph tracer.

When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment (e.g. /* <-irq_work_queue_on+0x81/0xa0 */). By interrogating DWARF debug information from the kernel image (vmlinux) utilising libdw, perf ftrace resolves this return address to its inlined callchain and synthesises the intermediate inlined frames directly into the streamed call graph.

Key aspects of this implementation:

1. DWARF inline resolution

Implement ftrace_resolve_inlines() using libdw to locate inlined subroutines at the instruction address corresponding to the caller's return address.

Implement perf_ftrace__setup_inlines() and perf_ftrace__cleanup_inlines() to initialise machine symbol tracking, load vmlinux symbols, and verify kernel CONFIG_FUNCTION_GRAPH_RETADDR support.

2. Inline frame synthesis

In ftrace_process_fgraph_line(), when --inline is active:
- Interrogate return addresses using ftrace_parse_retaddr() and resolve inlines via ftrace_resolve_inlines().

- Identify common inlined ancestors on the tracked stack and unwind inactive frames.

- Open new inlined frames with /* (inline) */ and indent by inlined_depth * 2.

- On function exit, close matching inlined frames with } /* func (inline) */.

3. Command-line options and documentation

Add --inline and -k/--vmlinux options in tools/perf/builtin-ftrace.c and document their usage in tools/perf/Documentation/perf-ftrace.txt.

---
 tools/perf/Documentation/perf-ftrace.txt |  13 +
 tools/perf/builtin-ftrace.c              |  27 +-
 tools/perf/util/ftrace.c                 | 637 ++++++++++++++++++++---
 tools/perf/util/ftrace.h                 |  44 +-
 4 files changed, 605 insertions(+), 116 deletions(-)
tools/perf/Documentation/perf-ftrace.txt +13 -0
index 3f3808e513fe..0af6b0c1f134 100644
@@ -134,6 +134,19 @@ OPTIONS for 'perf ftrace trace'
 	  - depth=<n>    - Set max depth for function graph tracer to follow.
 	  - tail         - Print function name at the end.
 
+--inline::
+	Show inlined functions in function_graph tracer. This requires
+	kernel support for the funcgraph-retaddr option and a vmlinux with
+	debug symbols. Inlined functions are displayed in the call graph
+	hierarchy, and their inlined status is indicated with `/* (inline) */`
+	on entry and exit, e.g. `func() { /* (inline) */` and `} /* func (inline) */`.
+	By default, raw return address comments (`/* <-caller+offset */`)
+	are hidden in the output; specify `--graph-opts retaddr` to show them.
+
+-k::
+--vmlinux=::
+	Path to the vmlinux file containing debug symbols for resolving
+	inlined functions.
 
 OPTIONS for 'perf ftrace latency'
 ---------------------------------
tools/perf/builtin-ftrace.c +24 -3
index e3909cff287b..1f652d1e2754 100644
@@ -575,9 +575,14 @@ static int set_tracing_funcgraph_retval(struct perf_ftrace *ftrace)
 
 static int set_tracing_funcgraph_retaddr(struct perf_ftrace *ftrace)
 {
-	if (ftrace->graph_retaddr) {
-		if (write_tracing_option_file("funcgraph-retaddr", "1") < 0)
+	if (ftrace->graph_retaddr || ftrace->use_inline) {
+		if (write_tracing_option_file("funcgraph-retaddr", "1") < 0) {
+			if (ftrace->use_inline) {
+				pr_err("failed to set funcgraph-retaddr option in tracing\n");
+				pr_err("Hint: ensure kernel is compiled with CONFIG_FUNCTION_GRAPH_RETADDR=y\n");
+			}
 			return -1;
+		}
 	}
 
 	return 0;
@@ -771,6 +776,9 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 		goto out_reset;
 	}
 
+	if (perf_ftrace__setup_inlines(ftrace) < 0)
+		goto out_reset;
+
 	setup_pager();
 
 	buf = malloc(TRACE_BUF_SIZE);
@@ -916,7 +924,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 	free(buf);
 	strbuf_release(&linebuf);
 out_reset:
-	ftrace_cpu_states__clear();
+	perf_ftrace__cleanup_inlines(ftrace);
 	exit_tracing_instance();
 out:
 	return (ret == 0 && done && !workload_exec_errno) ? 0 : -1;
@@ -1872,6 +1880,10 @@ int cmd_ftrace(int argc, const char **argv)
 		     "Size of per cpu buffer, needs to use a B, K, M or G suffix.", parse_buffer_size),
 	OPT_BOOLEAN(0, "inherit", &ftrace.inherit,
 		    "Trace children processes"),
+	OPT_BOOLEAN(0, "inline", &ftrace.use_inline,
+		    "Show inlined functions in function_graph tracer"),
+	OPT_STRING('k', "vmlinux", &symbol_conf.vmlinux_name, "file",
+		   "vmlinux pathname"),
 	OPT_INTEGER('D', "delay", &ftrace.target.initial_delay,
 		    "Number of milliseconds to wait before starting tracing after program start"),
 	OPT_PARENT(common_options),
@@ -1987,6 +1999,15 @@ int cmd_ftrace(int argc, const char **argv)
 
 	switch (subcmd) {
 	case PERF_FTRACE_TRACE:
+		select_tracer(&ftrace);
+
+		if (ftrace.use_inline) {
+			if (strcmp(ftrace.tracer, "function_graph") != 0) {
+				pr_err("Error: --inline option is only supported with function_graph tracer\n");
+				ret = -EINVAL;
+				goto out_delete_filters;
+			}
+		}
 		cmd_func = __cmd_ftrace;
 		break;
 	case PERF_FTRACE_LATENCY:
tools/perf/util/ftrace.c +558 -79

786 lines not on this page · raw message

tools/perf/util/ftrace.h +10 -34

60 lines not on this page · raw message

Signature · 1 line
2.55.0
#23 lore raw

Re: [PATCH perf-tools-next v12 06/10] perf ftrace: Support display of inlined functions in function graph tracer

Sashiko has reviewed this patch and found no issues.
Quoted 18 lines

The Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining.

Address this by introducing the --inline option to perf ftrace for the function_graph tracer.

When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment [ ... ]
tools/perf/Documentation/perf-ftrace.txt.

Quoted 3 lines

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
maintainer
#24 lore raw

Re: [PATCH perf-tools-next v12 06/10] perf ftrace: Support display of inlined functions in function graph tracer

In reply to Aaron Tomlin
Quoted 58 lines · Aaron Tomlin

The Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining.

Address this by introducing the --inline option to perf ftrace for the function_graph tracer.

When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment (e.g. /* <-irq_work_queue_on+0x81/0xa0 */). By interrogating DWARF debug information from the kernel image (vmlinux) utilising libdw, perf ftrace resolves this return address to its inlined callchain and synthesises the intermediate inlined frames directly into the streamed call graph.

Key aspects of this implementation:

1. DWARF inline resolution

Implement ftrace_resolve_inlines() using libdw to locate inlined subroutines at the instruction address corresponding to the caller's return address.

Implement perf_ftrace__setup_inlines() and perf_ftrace__cleanup_inlines() to initialise machine symbol tracking, load vmlinux symbols, and verify kernel CONFIG_FUNCTION_GRAPH_RETADDR support.

2. Inline frame synthesis

In ftrace_process_fgraph_line(), when --inline is active:
- Interrogate return addresses using ftrace_parse_retaddr() and resolve inlines via ftrace_resolve_inlines().

- Identify common inlined ancestors on the tracked stack and unwind inactive frames.

- Open new inlined frames with /* (inline) */ and indent by inlined_depth * 2.

- On function exit, close matching inlined frames with } /* func (inline) */.

3. Command-line options and documentation

Add --inline and -k/--vmlinux options in tools/perf/builtin-ftrace.c and document their usage in tools/perf/Documentation/perf-ftrace.txt.

---

[SNIP]

Quoted 8 lines
tools/perf/util/ftrace.c +3 -1
@@ -664,11 +834,16 @@ static void extract_exit_func_name(const char *str, char *buf, size_t size)
  * @cpu: Pointer to store the CPU number
  *
  * Locate the context columns that precede the function call column of a
- * function_graph trace record.
+ * function_graph trace record. Depending on the funcgraph-* options set
+ * up by perf, a record looks like:
+ *

tools/perf/util/ftrace.c static void extract_exit_func_name(const char *str, char *buf, size_t size) · line 847

Quoted 10 lines
tools/perf/util/ftrace.c +5 -2
+ *   [ abstime |]  CPU)  [ comm-pid |]  [ duration |]  func
+ *
+ * The abstime and comm-pid columns are only present with --graph-opts
+ * verbose.
  *
  * Return: A pointer to the last character of the context columns (the final
- * '|' delimiter, or the CPU ')' when no delimiter follows it). Returns NULL
- * if @line is not a trace record.
+ * '|' delimiter). Returns NULL if @line is not a trace record.
  */

It'd be nice if you could move the comments to the original commit that introduced the function.

Quoted 2 lines
tools/perf/util/ftrace.c +0 -0
 static const char *find_func_column(const struct perf_ftrace *ftrace,
 				    const char *line, int *cpu)

[SNIP]

Quoted 42 lines
tools/perf/util/ftrace.h +3 -23
index 45f5090d0115..7b9e0919ae8d 100644
@@ -42,6 +42,8 @@ struct perf_ftrace {
 	int			graph_verbose;
 	int			graph_thresh;
 	int			graph_tail;
+	bool			use_inline;
+	struct machine		*machine;
 };
 
 struct filter_entry {
@@ -96,44 +98,18 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)
 
 #endif  /* HAVE_BPF_SKEL */
 
-#define FTRACE_MAX_CPUS 4096
-#define FTRACE_MAX_STACK 256
-#define FTRACE_PID_HASH_BITS 10
-#define FTRACE_PID_HASH_SIZE (1 << FTRACE_PID_HASH_BITS)
+struct machine;
 
-#define FTRACE_IDLE_PID(cpu)       ((pid_t)(-1 - (cpu)))
-#define FTRACE_UNKNOWN_PID(cpu)    ((pid_t)(-1 - FTRACE_MAX_CPUS - (cpu)))
-
-struct ftrace_stack_frame {
-	char *name;
-	bool inlined;
-	int indent;
-	int raw_indent;
-	u64 seq;
-};
-
-struct ftrace_indent_watermark {
-	u64 seq;
-	int raw_indent;
-};
-
-struct ftrace_task_state {
-	struct hlist_node	node;
-	pid_t			pid;

tools/perf/util/ftrace.h perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused) · line 132

Quoted 10 lines
tools/perf/util/ftrace.h +0 -10
-	struct ftrace_stack_frame *stack;
-	int			max_stack;
-	int			depth;
-	int			inlined_depth;
-	int			overflow_depth;
-	u64			seq;
-	struct ftrace_indent_watermark *watermarks;
-	int			max_watermarks;
-	int			nr_watermarks;
-};

Hmm.. probably better to define the structs in the .c file in the first place?

Thanks,
Namhyung

Quoted 17 lines
tools/perf/util/ftrace.h +7 -1
+#define FTRACE_MAX_INLINES 32
 
 bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset);
 void ftrace_filter_retaddr(char *str);
-void ftrace_cpu_states__clear(void);
+int ftrace_resolve_inlines(struct machine *machine,
+			   const char *sym_name, u64 offset,
+			   const char **inlined_names, int max_inlines);
 int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out);
+void ftrace_cpu_states__clear(void);
+int perf_ftrace__setup_inlines(struct perf_ftrace *ftrace);
+void perf_ftrace__cleanup_inlines(struct perf_ftrace *ftrace);
 
 #endif  /* __PERF_FTRACE_H__ */
Signature · 1 line
2.55.0
#25 lore raw

Re: [PATCH perf-tools-next v12 06/10] perf ftrace: Support display of inlined functions in function graph tracer

Quoted 13 lines · Namhyung Kim
Quoted 13 lines
tools/perf/util/ftrace.c +7 -1
@@ -664,11 +834,16 @@ static void extract_exit_func_name(const char *str, char *buf, size_t size)
  * @cpu: Pointer to store the CPU number
  *
  * Locate the context columns that precede the function call column of a
- * function_graph trace record.
+ * function_graph trace record. Depending on the funcgraph-* options set
+ * up by perf, a record looks like:
+ *
+ *   [ abstime |]  CPU)  [ comm-pid |]  [ duration |]  func
+ *
+ * The abstime and comm-pid columns are only present with --graph-opts
+ * verbose.
  *
Quoted 8 lines

tools/perf/util/ftrace.c static void extract_exit_func_name(const char *str, char *buf, size_t size) · line 847

Quoted 5 lines
tools/perf/util/ftrace.c +1 -2
  * Return: A pointer to the last character of the context columns (the final
- * '|' delimiter, or the CPU ')' when no delimiter follows it). Returns NULL
- * if @line is not a trace record.
+ * '|' delimiter). Returns NULL if @line is not a trace record.
  */

It'd be nice if you could move the comments to the original commit that introduced the function.

Agreed. This was an unintended artifact from an earlier rebase. I will move the comment to patch 04/10 where find_func_column() is introduced.

Quoted 56 lines
Quoted 42 lines
tools/perf/util/ftrace.h +3 -23
index 45f5090d0115..7b9e0919ae8d 100644
@@ -42,6 +42,8 @@ struct perf_ftrace {
 	int			graph_verbose;
 	int			graph_thresh;
 	int			graph_tail;
+	bool			use_inline;
+	struct machine		*machine;
 };
 
 struct filter_entry {
@@ -96,44 +98,18 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)
 
 #endif  /* HAVE_BPF_SKEL */
 
-#define FTRACE_MAX_CPUS 4096
-#define FTRACE_MAX_STACK 256
-#define FTRACE_PID_HASH_BITS 10
-#define FTRACE_PID_HASH_SIZE (1 << FTRACE_PID_HASH_BITS)
+struct machine;
 
-#define FTRACE_IDLE_PID(cpu)       ((pid_t)(-1 - (cpu)))
-#define FTRACE_UNKNOWN_PID(cpu)    ((pid_t)(-1 - FTRACE_MAX_CPUS - (cpu)))
-
-struct ftrace_stack_frame {
-	char *name;
-	bool inlined;
-	int indent;
-	int raw_indent;
-	u64 seq;
-};
-
-struct ftrace_indent_watermark {
-	u64 seq;
-	int raw_indent;
-};
-
-struct ftrace_task_state {
-	struct hlist_node	node;
-	pid_t			pid;

tools/perf/util/ftrace.h perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused) · line 132

Quoted 10 lines
tools/perf/util/ftrace.h +0 -10
-	struct ftrace_stack_frame *stack;
-	int			max_stack;
-	int			depth;
-	int			inlined_depth;
-	int			overflow_depth;
-	u64			seq;
-	struct ftrace_indent_watermark *watermarks;
-	int			max_watermarks;
-	int			nr_watermarks;
-};

Hmm.. probably better to define the structs in the .c file in the first place?

Quoted 2 lines

Thanks,
Namhyung

Agreed. Since these structures and macros are internal to tools/perf/util/ftrace.c, I will define them directly in ftrace.c in patch 04/10 from the start.

Kind regards,

Signature · 1 line
Aaron Tomlin
maintainer
#26 lore raw

Re: [PATCH perf-tools-next v12 06/10] perf ftrace: Support display of inlined functions in function graph tracer

In reply to Aaron Tomlin
Quoted 58 lines · Aaron Tomlin

The Linux kernel's function graph tracer operates at the machine instruction level via compiler instrumentation (-fpatchable-function-entry), recording entry and return events for physical function calls. Consequently, functions inlined by the compiler are invisible in the resulting call graph, as no discrete call or return instructions are emitted for them. While technically expected, this omission frequently obscures the logical execution flow when analysing kernel subsystems heavily reliant upon inlining.

Address this by introducing the --inline option to perf ftrace for the function_graph tracer.

When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the funcgraph-retaddr trace option records the caller's return address on each function entry, manifested in the trace stream as a comment (e.g. /* <-irq_work_queue_on+0x81/0xa0 */). By interrogating DWARF debug information from the kernel image (vmlinux) utilising libdw, perf ftrace resolves this return address to its inlined callchain and synthesises the intermediate inlined frames directly into the streamed call graph.

Key aspects of this implementation:

1. DWARF inline resolution

Implement ftrace_resolve_inlines() using libdw to locate inlined subroutines at the instruction address corresponding to the caller's return address.

Implement perf_ftrace__setup_inlines() and perf_ftrace__cleanup_inlines() to initialise machine symbol tracking, load vmlinux symbols, and verify kernel CONFIG_FUNCTION_GRAPH_RETADDR support.

2. Inline frame synthesis

In ftrace_process_fgraph_line(), when --inline is active:
- Interrogate return addresses using ftrace_parse_retaddr() and resolve inlines via ftrace_resolve_inlines().

- Identify common inlined ancestors on the tracked stack and unwind inactive frames.

- Open new inlined frames with /* (inline) */ and indent by inlined_depth * 2.

- On function exit, close matching inlined frames with } /* func (inline) */.

3. Command-line options and documentation

Add --inline and -k/--vmlinux options in tools/perf/builtin-ftrace.c and document their usage in tools/perf/Documentation/perf-ftrace.txt.

---

[SNIP]

Quoted 6 lines
tools/perf/util/ftrace.c +6 -0
+/**
+ * ftrace_resolve_inlines - Resolve inlined functions for a return address
+ * @machine: Perf machine context
+ * @sym_name: Name of the caller function
+ * @offset: Instruction offset within the caller function
+ * @inlined_names: Array to store the resolved inline function names

tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 673

Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+ * @max_inlines: Maximum number of inline frames to resolve
+ *
+ * Use DWARF debug information from the kernel image to resolve a caller's
+ * return address to its inlined callchain.
+ *
+ * Return: The number of inline functions successfully resolved.
+ */
+int ftrace_resolve_inlines(struct machine *machine,
+			   const char *sym_name, u64 offset,
+			   const char **inlined_names, int max_inlines)

Why not remove 'const' from the inlined_names?

Quoted 31 lines
tools/perf/util/ftrace.c +31 -0
+{
+	struct map *map = NULL;
+	struct symbol *sym;
+	struct dso *dso;
+	struct inline_node *node;
+	struct inline_list *ilist;
+	u64 ip, addr;
+	int count = 0;
+	int j;
+
+	if (!machine || !offset)
+		return 0;
+
+	sym = machine__find_kernel_symbol_by_name(machine, sym_name, &map);
+	if (!sym || !map)
+		return 0;
+
+	dso = map__dso(map);
+	if (!dso) {
+		map__put(map);
+		return 0;
+	}
+
+	ip = sym->start + offset - 1;
+	addr = map__rip_2objdump(map, ip);
+
+	node = inlines__tree_find(dso__inlined_nodes(dso), addr);
+	if (!node) {
+		node = dso__parse_addr_inlines(dso, addr, sym);
+		if (node)
+			inlines__tree_insert(dso__inlined_nodes(dso), node);

tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 714

Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+	}
+
+	if (!node) {
+		map__put(map);
+		return 0;
+	}
+
+	list_for_each_entry_reverse(ilist, &node->val, list) {
+		if (ilist->symbol && symbol__inlined(ilist->symbol)) {
+			if (count < max_inlines) {

You can reduce the indentation by inverting the condition.

tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 715

Quoted 1 line
tools/perf/util/ftrace.c +1 -0
+				char *name = strdup(ilist->symbol->name);

Why not just save the pointer? You may not use some of them, also it copies the string again when it's added to the stack. It'd make the error handling a bit easier.

Quoted 15 lines
tools/perf/util/ftrace.c +15 -0
+
+				if (!name) {
+					for (j = 0; j < count; j++)
+						free((void *)inlined_names[j]);
+					map__put(map);
+					return -ENOMEM;
+				}
+				inlined_names[count++] = name;
+			}
+		}
+	}
+
+	map__put(map);
+	return count;
+}

[SNIP]

Quoted 297 lines

* @line: The trace line to process
* @out: File stream to output to
*
- * Parse function entries and exits, maintain the task's call stack, and
- * output the trace line.
+ * Process a trace line from the function_graph tracer. If --inline is set,
+ * this function parses the return address comment, resolves it to its
+ * inlined callchain, and synthesises intermediate frames into the streamed
+ * call graph.
*
* Return: 0 on success, or a negative error code on failure.
*/
int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out) {

tools/perf/util/ftrace.c +193 -20
+	char caller_sym[128];
+	char blank_prefix[128];
+	const char *inlined_names[FTRACE_MAX_INLINES];
 	const char *pipe_char;
 	const char *after_pipe;
 	const char *func_start;
-	struct ftrace_task_state *cs;
+	struct ftrace_cpu_state *cs;
+	u64 caller_offset = 0;
+	int num_inlines = 0;
 	int cpu = -1;
 	int leading_spaces;
-	int top_raw_indent;
+	int extra_indent;
+	int caller_idx;
+	int active_inlines;
+	int common;
 	int match_idx;
-	int i;
+	int top_raw_indent;
+	int i, j;
 	bool is_exit;
 	bool has_open_brace;
 	bool has_semicolon;
+	bool has_retaddr = false;
+
+	if (!ftrace->use_inline)
+		return print_line(ftrace, line, out);
 
 	pipe_char = find_func_column(ftrace, line, &cpu);
 	if (!pipe_char) {
@@ -799,6 +989,7 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 		return print_line(ftrace, line, out);
 
 	cs->seq = ++ftrace_event_seq;
+	make_blank_prefix(ftrace, line, pipe_char, blank_prefix, sizeof(blank_prefix));
 
 	after_pipe = pipe_char + 1;
 	func_start = skip_spaces(after_pipe);
@@ -818,20 +1009,32 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 
 		top_raw_indent = -1;
 		for (i = cs->depth - 1; i >= 0; i--) {
-			top_raw_indent = cs->stack[i].raw_indent;
-			break;
+			if (!cs->stack[i].inlined) {
+				top_raw_indent = cs->stack[i].raw_indent;
+				break;
+			}
 		}
 
+		/*
+		 * If currently overflowed beyond FTRACE_MAX_STACK, any exit whose
+		 * indentation is deeper than the top tracked frame belongs to the
+		 * overflow region.
+		 */
 		if (cs->overflow_depth > 0 &&
 		    (top_raw_indent < 0 || leading_spaces > top_raw_indent)) {
 			cs->overflow_depth--;
-			return print_line(ftrace, line, out);
+			extra_indent = cs->inlined_depth * 2;
+			fprintf(out, "%.*s%*s%s\n",
+				(int)(pipe_char - line + 1), line,
+				leading_spaces + extra_indent, "", func_start);
+			return 0;
 		}
 
+		/* Find matching frame on stack */
 		match_idx = -1;
 		if (exit_name[0] != '\0') {
 			for (i = cs->depth - 1; i >= 0; i--) {
-				if (cs->stack[i].name &&
+				if (!cs->stack[i].inlined && cs->stack[i].name &&
 				    cs->stack[i].raw_indent == leading_spaces &&
 				    strcmp(cs->stack[i].name, exit_name) == 0) {
 					match_idx = i;
@@ -840,9 +1043,14 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 			}
 		}
 
+		/*
+		 * If bare brace ('}'), match the topmost non-inlined frame
+		 * with matching indentation.
+		 */
 		if (match_idx < 0 && exit_name[0] == '\0') {
 			for (i = cs->depth - 1; i >= 0; i--) {
-				if (cs->stack[i].raw_indent == leading_spaces) {
+				if (!cs->stack[i].inlined &&
+				    cs->stack[i].raw_indent == leading_spaces) {
 					match_idx = i;
 					break;
 				}
@@ -851,17 +1059,51 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 
 		if (match_idx >= 0) {
 			cs->overflow_depth = 0;
-			while (cs->depth > match_idx) {
+		} else if (cs->overflow_depth > 0) {
+			cs->overflow_depth--;
+			extra_indent = cs->inlined_depth * 2;
+			fprintf(out, "%.*s%*s%s\n",
+				(int)(pipe_char - line + 1), line,
+				leading_spaces + extra_indent, "", func_start);
+			return 0;
+		}
+
+		if (match_idx >= 0) {
+			/* Close all inlined and orphaned frames above match_idx */
+			while (cs->depth > 0 && cs->depth - 1 > match_idx) {
+				struct ftrace_stack_frame *top = &cs->stack[cs->depth - 1];
+
+				if (top->inlined) {
+					int exit_indent = top->indent;
+
+					if (cs->inlined_depth > 0)
+						cs->inlined_depth--;
+					fprintf(out, "%s%*s} /* %s (inline) */\n",
+						blank_prefix, exit_indent, "", top->name);
+				}
+				free(top->name);
+				cs->depth--;
+			}
+
+			/* Pop the matched real frame */
+			if (cs->depth > 0 && match_idx == cs->depth - 1 &&
+			    !cs->stack[cs->depth - 1].inlined) {
 				free(cs->stack[cs->depth - 1].name);
 				cs->depth--;
 			}
-		} else if (cs->overflow_depth > 0) {
-			cs->overflow_depth--;
+
+			extra_indent = cs->inlined_depth * 2;
+		} else {
+			extra_indent = 0;
 		}
 
-		return print_line(ftrace, line, out);
+		fprintf(out, "%.*s%*s%s\n",
+			(int)(pipe_char - line + 1), line,
+			leading_spaces + extra_indent, "", func_start);
+		return 0;
 	}
 
+	/* It's an entry or a leaf call */
 	has_open_brace = (strchr(func_start, '{') != NULL);
 	has_semicolon = (strchr(func_start, ';') != NULL);
 
@@ -870,40 +1112,277 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 
 	top_raw_indent = -1;
 	for (i = cs->depth - 1; i >= 0; i--) {
-		top_raw_indent = cs->stack[i].raw_indent;
-		break;
+		if (!cs->stack[i].inlined) {
+			top_raw_indent = cs->stack[i].raw_indent;
+			break;
+		}
 	}
 
 	if (cs->overflow_depth > 0 && top_raw_indent >= 0 &&
 	    leading_spaces <= top_raw_indent)
 		cs->overflow_depth = 0;
 
-	if (has_open_brace && !has_semicolon) {
-		char entry_name[128];
+	if (ftrace_parse_retaddr(func_start, caller_sym, sizeof(caller_sym), &caller_offset)) {
+		has_retaddr = true;
+		num_inlines = ftrace_resolve_inlines(ftrace->machine, caller_sym,
+						      caller_offset, inlined_names,
+						      FTRACE_MAX_INLINES);
+		if (num_inlines < 0)
+			return num_inlines;
+	}
+
+	/* Determine which inlined frames are currently open above the caller */
+	caller_idx = -1;
+	if (has_retaddr && cs->overflow_depth == 0) {
+		for (i = cs->depth - 1; i >= 0; i--) {
+			if (!cs->stack[i].inlined && cs->stack[i].name &&
+			    cs->stack[i].raw_indent < leading_spaces &&
+			    strcmp(cs->stack[i].name, caller_sym) == 0) {
+				caller_idx = i;
+				break;
+			}
+		}
+	}
+
+	/* If caller is not on the tracked stack, do not synthesise inlines */
+	if (has_retaddr && caller_idx < 0) {
+		for (i = 0; i < num_inlines; i++)
+			free((void *)inlined_names[i]);
+		num_inlines = 0;
+	}
+
+	active_inlines = 0;
+	if (caller_idx >= 0) {
+		for (i = caller_idx + 1; i < cs->depth; i++) {
+			if (cs->stack[i].inlined)
+				active_inlines++;
+			else
+				break;
+		}
+	}
+
+	/* Find common prefix of inlined functions */
+	common = 0;
+	if (caller_idx >= 0) {
+		while (common < active_inlines && common < num_inlines &&
+		       cs->stack[caller_idx + 1 + common].name &&
+		       !strcmp(cs->stack[caller_idx + 1 + common].name,
+			       inlined_names[common])) {
+			common++;
+		}
+	}
+
+	/* Close stale callee frames and inactive inlined frames above caller */
+	if (caller_idx >= 0)
+		cs->overflow_depth = 0;
+
+	while (caller_idx >= 0 && cs->depth > caller_idx + 1 + common) {
+		struct ftrace_stack_frame *top = &cs->stack[cs->depth - 1];
+
+		if (top->inlined) {
+			int exit_indent = top->indent;
+
+			if (cs->inlined_depth > 0)
+				cs->inlined_depth--;
+			fprintf(out, "%s%*s} /* %s (inline) */\n",
+				blank_prefix, exit_indent, "", top->name);
+		}
+		free(top->name);
+		cs->depth--;
+	}
+
+	if (caller_idx < 0 && cs->overflow_depth == 0) {
+		/*
+		 * If caller is not on the tracked stack, unwind any frames whose
+		 * raw indentation is greater than or equal to the new entry's
+		 * indentation (their exit events were dropped), and close any
+		 * trailing inlined frames that were left open from prior calls.
+		 */
+		while (cs->depth > 0) {
+			struct ftrace_stack_frame *top = &cs->stack[cs->depth - 1];
+
+			if (top->inlined) {
+				int exit_indent = top->indent;
+
+				if (cs->inlined_depth > 0)
+					cs->inlined_depth--;
+				fprintf(out, "%s%*s} /* %s (inline) */\n",
+					blank_prefix, exit_indent, "", top->name);
+				free(top->name);
+				cs->depth--;
+			} else if (top->raw_indent >= leading_spaces) {
+				free(top->name);
+				cs->depth--;
+			} else {
+				break;
+			}
+		}
+	}
 
-		while (cs->depth > 0 &&
-		       cs->stack[cs->depth - 1].raw_indent >= leading_spaces) {
-			free(cs->stack[cs->depth - 1].name);
-			cs->depth--;
+	/* Open new inlined frames */
+	for (i = common; i < num_inlines; i++) {
+		int entry_indent = leading_spaces + cs->inlined_depth * 2;
+
+		if (!inlined_names[i]) {
+			pr_err("Not enough memory\n");
+			for (j = 0; j < num_inlines; j++)
+				free((void *)inlined_names[j]);
+			return -ENOMEM;
+		}
+		if (cs->depth >= cs->max_stack &&
+		    ftrace_task_state__grow_stack(cs) < 0)
+			break;
+
+		fprintf(out, "%s%*s%s() { /* (inline) */\n",
+			blank_prefix, entry_indent, "", inlined_names[i]);

tools/perf/util/ftrace.c int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL · line 1247

Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+		cs->stack[cs->depth].name = strdup(inlined_names[i]);
+		if (!cs->stack[cs->depth].name) {
+			pr_err("Not enough memory\n");
+			for (j = 0; j < num_inlines; j++)
+				free((void *)inlined_names[j]);
+			return -ENOMEM;
+		}
+		cs->stack[cs->depth].inlined = true;
+		cs->stack[cs->depth].indent = entry_indent;
+		cs->stack[cs->depth].raw_indent = -1;

I'm curious if it's possible to get rid of the indent and raw_indent. I think you can get the indent from the depth easily. The raw indent would be from any non-inlined parent if needed. Also having inlined depth in the cs/ts seems unnecessary. What do you think?

Quoted 71 lines
tools/perf/util/ftrace.c +54 -12
+		cs->stack[cs->depth].seq = cs->seq;
+		cs->depth++;
+		cs->inlined_depth++;
+	}
+
+	/* Print current line with updated indentation */
+	extra_indent = cs->inlined_depth * 2;
+
+	if (!ftrace->graph_retaddr) {
+		char *filtered = strdup(func_start);
+
+		if (filtered) {
+			ftrace_filter_retaddr(filtered);
+			fprintf(out, "%.*s%*s%s\n",
+				(int)(pipe_char - line + 1), line,
+				leading_spaces + extra_indent, "", filtered);
+			free(filtered);
+		} else {
+			pr_err("Not enough memory\n");
+			for (i = 0; i < num_inlines; i++)
+				free((void *)inlined_names[i]);
+			return -ENOMEM;
 		}
+	} else {
+		fprintf(out, "%.*s%*s%s\n",
+			(int)(pipe_char - line + 1), line,
+			leading_spaces + extra_indent, "", func_start);
+	}
+
+	/* If non-leaf entry, push onto stack */
+	if (has_open_brace && !has_semicolon) {
+		char entry_name[128];
 
 		extract_entry_func_name(func_start, entry_name, sizeof(entry_name));
 		if (cs->depth >= cs->max_stack &&
 		    ftrace_task_state__grow_stack(cs) < 0) {
-			cs->overflow_depth++;
-		} else {
-			cs->stack[cs->depth].name = strdup(entry_name);
-			if (!cs->stack[cs->depth].name) {
-				pr_err("Not enough memory\n");
-				return -ENOMEM;
-			}
-			cs->stack[cs->depth].inlined = false;
-			cs->stack[cs->depth].indent = leading_spaces;
-			cs->stack[cs->depth].raw_indent = leading_spaces;
-			cs->stack[cs->depth].seq = cs->seq;
-			cs->depth++;
+			for (i = 0; i < num_inlines; i++)
+				free((void *)inlined_names[i]);
+			return -ENOMEM;
+		}
+
+		cs->stack[cs->depth].name = strdup(entry_name);
+		if (!cs->stack[cs->depth].name) {
+			pr_err("Not enough memory\n");
+			for (i = 0; i < num_inlines; i++)
+				free((void *)inlined_names[i]);
+			return -ENOMEM;
+		}
+		cs->stack[cs->depth].inlined = false;
+		cs->stack[cs->depth].indent = leading_spaces + extra_indent;
+		cs->stack[cs->depth].raw_indent = leading_spaces;
+		cs->stack[cs->depth].seq = cs->seq;
+		cs->depth++;
+	}
+
+	for (i = 0; i < num_inlines; i++)
+		free((void *)inlined_names[i]);
+	return 0;
+}

[SNIP]

Quoted 43 lines
tools/perf/util/ftrace.h +3 -24
index 45f5090d0115..7b9e0919ae8d 100644
@@ -42,6 +42,8 @@ struct perf_ftrace {
 	int			graph_verbose;
 	int			graph_thresh;
 	int			graph_tail;
+	bool			use_inline;
+	struct machine		*machine;
 };
 
 struct filter_entry {
@@ -96,44 +98,18 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)
 
 #endif  /* HAVE_BPF_SKEL */
 
-#define FTRACE_MAX_CPUS 4096
-#define FTRACE_MAX_STACK 256
-#define FTRACE_PID_HASH_BITS 10
-#define FTRACE_PID_HASH_SIZE (1 << FTRACE_PID_HASH_BITS)
+struct machine;
 
-#define FTRACE_IDLE_PID(cpu)       ((pid_t)(-1 - (cpu)))
-#define FTRACE_UNKNOWN_PID(cpu)    ((pid_t)(-1 - FTRACE_MAX_CPUS - (cpu)))
-
-struct ftrace_stack_frame {
-	char *name;
-	bool inlined;
-	int indent;
-	int raw_indent;
-	u64 seq;
-};
-
-struct ftrace_indent_watermark {
-	u64 seq;
-	int raw_indent;
-};
-
-struct ftrace_task_state {
-	struct hlist_node	node;
-	pid_t			pid;
-	struct ftrace_stack_frame *stack;

tools/perf/util/ftrace.h perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused) · line 103

Quoted 10 lines
tools/perf/util/ftrace.h +1 -9
-	int			max_stack;
-	int			depth;
-	int			inlined_depth;
-	int			overflow_depth;
-	u64			seq;
-	struct ftrace_indent_watermark *watermarks;
-	int			max_watermarks;
-	int			nr_watermarks;
-};
+#define FTRACE_MAX_INLINES 32

Isn't 32 too big? I guess 8 would be enough..

Thanks,
Namhyung

Quoted 16 lines
tools/perf/util/ftrace.h +6 -1
 bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset);
 void ftrace_filter_retaddr(char *str);
-void ftrace_cpu_states__clear(void);
+int ftrace_resolve_inlines(struct machine *machine,
+			   const char *sym_name, u64 offset,
+			   const char **inlined_names, int max_inlines);
 int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out);
+void ftrace_cpu_states__clear(void);
+int perf_ftrace__setup_inlines(struct perf_ftrace *ftrace);
+void perf_ftrace__cleanup_inlines(struct perf_ftrace *ftrace);
 
 #endif  /* __PERF_FTRACE_H__ */
Signature · 1 line
2.55.0
#27 lore raw

Re: [PATCH perf-tools-next v12 06/10] perf ftrace: Support display of inlined functions in function graph tracer

Quoted 64 lines
Quoted 6 lines
tools/perf/util/ftrace.c +6 -0
+/**
+ * ftrace_resolve_inlines - Resolve inlined functions for a return address
+ * @machine: Perf machine context
+ * @sym_name: Name of the caller function
+ * @offset: Instruction offset within the caller function
+ * @inlined_names: Array to store the resolved inline function names

tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 673

Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+ * @max_inlines: Maximum number of inline frames to resolve
+ *
+ * Use DWARF debug information from the kernel image to resolve a caller's
+ * return address to its inlined callchain.
+ *
+ * Return: The number of inline functions successfully resolved.
+ */
+int ftrace_resolve_inlines(struct machine *machine,
+			   const char *sym_name, u64 offset,
+			   const char **inlined_names, int max_inlines)

Why not remove 'const' from the inlined_names?

Quoted 31 lines
tools/perf/util/ftrace.c +31 -0
+{
+	struct map *map = NULL;
+	struct symbol *sym;
+	struct dso *dso;
+	struct inline_node *node;
+	struct inline_list *ilist;
+	u64 ip, addr;
+	int count = 0;
+	int j;
+
+	if (!machine || !offset)
+		return 0;
+
+	sym = machine__find_kernel_symbol_by_name(machine, sym_name, &map);
+	if (!sym || !map)
+		return 0;
+
+	dso = map__dso(map);
+	if (!dso) {
+		map__put(map);
+		return 0;
+	}
+
+	ip = sym->start + offset - 1;
+	addr = map__rip_2objdump(map, ip);
+
+	node = inlines__tree_find(dso__inlined_nodes(dso), addr);
+	if (!node) {
+		node = dso__parse_addr_inlines(dso, addr, sym);
+		if (node)
+			inlines__tree_insert(dso__inlined_nodes(dso), node);

tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 714

Quoted 10 lines
tools/perf/util/ftrace.c +10 -0
+	}
+
+	if (!node) {
+		map__put(map);
+		return 0;
+	}
+
+	list_for_each_entry_reverse(ilist, &node->val, list) {
+		if (ilist->symbol && symbol__inlined(ilist->symbol)) {
+			if (count < max_inlines) {

You can reduce the indentation by inverting the condition.

Quoted 5 lines

tools/perf/util/ftrace.c void ftrace_filter_retaddr(char *str) · line 715

Quoted 1 line
tools/perf/util/ftrace.c +1 -0
+				char *name = strdup(ilist->symbol->name);

Why not just save the pointer? You may not use some of them, also it copies the string again when it's added to the stack. It'd make the error handling a bit easier.

Since the inline_node is cached in the DSO's inlined_nodes tree for the lifetime of ftrace->machine, we can store ilist->symbol->name directly in inlined_names without strdup() and without freeing each entry in the caller. I will also invert the condition in the loop to flatten the indentation.

Quoted 8 lines

tools/perf/util/ftrace.c int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL · line 1247

Quoted 3 lines
tools/perf/util/ftrace.c +3 -0
+		cs->stack[cs->depth].inlined = true;
+		cs->stack[cs->depth].indent = entry_indent;
+		cs->stack[cs->depth].raw_indent = -1;

I'm curious if it's possible to get rid of the indent and raw_indent. I think you can get the indent from the depth easily. The raw indent would be from any non-inlined parent if needed. Also having inlined depth in the cs/ts seems unnecessary. What do you think?

We still need to record the trace line's indentation rather than deriving it purely from ts->depth * 2 because tracing can start mid-callchain (or after ring-buffer drops), where ts->depth == 0 has a non-zero kernel indentation (e.g. 6 or 8 spaces), and bare "}" exits must be matched against that kernel indentation.

However, we do not need both indent and raw_indent on struct ftrace_stack_frame, nor do we need inlined_depth in struct ftrace_task_state:

1. In struct ftrace_stack_frame, indent was only used for inlined frames and raw_indent was only used for non-inlined frames, so they can be consolidated into a single indent field.

2. In struct ftrace_task_state, inlined_depth can be dropped altogether and computed from the active inlined frames on ts->stack when needed.

I will simplify both structures accordingly.

Quoted 3 lines
Quoted 1 line

+#define FTRACE_MAX_INLINES 32

Isn't 32 too big? I guess 8 would be enough..

No, 8 is not enough in practice. I will prefer to keep 32. For instance, after a -O2 kernel build:

❯ addr2line -f -i -e vmlinux 0xffffffff816916e9

      __do_trace_rss_stat
      include/trace/events/kmem.h:427 (discriminator 20)
      trace_rss_stat
      include/trace/events/kmem.h:427 (discriminator 20)
      mm_trace_rss_stat
      mm/memory.c:169 (discriminator 20)
      add_mm_counter
      include/linux/mm.h:3469 (discriminator 20)
      add_mm_rss_vec
      mm/memory.c:492 (discriminator 20)
      zap_pte_range
      mm/memory.c:2087 (discriminator 20)
      zap_pmd_range
      mm/memory.c:2151 (discriminator 20)
      zap_pud_range
      mm/memory.c:2179 (discriminator 20)
      zap_p4d_range
      mm/memory.c:2200 (discriminator 20)
      __zap_vma_range
      mm/memory.c:2240 (discriminator 20)

Another example:

❯ addr2line -f -i -e vmlinux 0xffffffff821c34af

      constant_test_bit
      arch/x86/include/asm/bitops.h:202
      arch_test_bit
      arch/x86/include/asm/bitops.h:232
      _test_bit
      include/asm-generic/bitops/instrumented-non-atomic.h:142
      test_ti_thread_flag
      include/linux/thread_info.h:133
      test_tsk_thread_flag
      include/linux/sched.h:2102
      task_sigpending
      include/linux/sched/signal.h:384
      fatal_signal_pending
      include/linux/sched/signal.h:406
      ptrace_report_syscall
      include/linux/ptrace.h:427
      ptrace_report_syscall_exit
      include/linux/ptrace.h:477
      arch_ptrace_report_syscall_exit
      include/linux/entry-common.h:233
      syscall_exit_work
      include/linux/entry-common.h:266
      syscall_exit_to_user_mode_work
      include/linux/entry-common.h:297
      syscall_exit_to_user_mode
      include/linux/entry-common.h:334
      do_syscall_64
      arch/x86/entry/syscall_64.c:89

Kind regards,

Signature · 1 line
Aaron Tomlin
maintainer
#28 lore raw

Re: [PATCH perf-tools-next v12 06/10] perf ftrace: Support display of inlined functions in function graph tracer

On Mon, Nov 02, 2026 at 02:20:47PM -0500, Aaron Tomlin wrote:
[SNIP]

Quoted 9 lines
Quoted 8 lines

tools/perf/util/ftrace.c int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL · line 1247

Quoted 3 lines
tools/perf/util/ftrace.c +3 -0
+		cs->stack[cs->depth].inlined = true;
+		cs->stack[cs->depth].indent = entry_indent;
+		cs->stack[cs->depth].raw_indent = -1;

I'm curious if it's possible to get rid of the indent and raw_indent. I think you can get the indent from the depth easily. The raw indent would be from any non-inlined parent if needed. Also having inlined depth in the cs/ts seems unnecessary. What do you think?

Quoted 5 lines

We still need to record the trace line's indentation rather than deriving it purely from ts->depth * 2 because tracing can start mid-callchain (or after ring-buffer drops), where ts->depth == 0 has a non-zero kernel indentation (e.g. 6 or 8 spaces), and bare "}" exits must be matched against that kernel indentation.

Right, but it may start from non-zero depth then. Probably save the starting depth in the ts struct.

Quoted 9 lines

However, we do not need both indent and raw_indent on struct ftrace_stack_frame, nor do we need inlined_depth in struct ftrace_task_state:

1. In struct ftrace_stack_frame, indent was only used for inlined frames and raw_indent was only used for non-inlined frames, so they can be consolidated into a single indent field.

Quoted 5 lines

2. In struct ftrace_task_state, inlined_depth can be dropped altogether and computed from the active inlined frames on ts->stack when needed.

I will simplify both structures accordingly.

Thanks!

Quoted 53 lines
Quoted 4 lines
Quoted 2 lines

+#define FTRACE_MAX_INLINES 32

Isn't 32 too big? I guess 8 would be enough..

No, 8 is not enough in practice. I will prefer to keep 32. For instance, after a -O2 kernel build:

❯ addr2line -f -i -e vmlinux 0xffffffff816916e9

      __do_trace_rss_stat
      include/trace/events/kmem.h:427 (discriminator 20)
      trace_rss_stat
      include/trace/events/kmem.h:427 (discriminator 20)
      mm_trace_rss_stat
      mm/memory.c:169 (discriminator 20)
      add_mm_counter
      include/linux/mm.h:3469 (discriminator 20)
      add_mm_rss_vec
      mm/memory.c:492 (discriminator 20)
      zap_pte_range
      mm/memory.c:2087 (discriminator 20)
      zap_pmd_range
      mm/memory.c:2151 (discriminator 20)
      zap_pud_range
      mm/memory.c:2179 (discriminator 20)
      zap_p4d_range
      mm/memory.c:2200 (discriminator 20)
      __zap_vma_range
      mm/memory.c:2240 (discriminator 20)

Another example:

❯ addr2line -f -i -e vmlinux 0xffffffff821c34af

      constant_test_bit
      arch/x86/include/asm/bitops.h:202
      arch_test_bit
      arch/x86/include/asm/bitops.h:232
      _test_bit
      include/asm-generic/bitops/instrumented-non-atomic.h:142
      test_ti_thread_flag
      include/linux/thread_info.h:133
      test_tsk_thread_flag
      include/linux/sched.h:2102
      task_sigpending
      include/linux/sched/signal.h:384
      fatal_signal_pending
      include/linux/sched/signal.h:406
      ptrace_report_syscall
      include/linux/ptrace.h:427
      ptrace_report_syscall_exit
      include/linux/ptrace.h:477
      arch_ptrace_report_syscall_exit
      include/linux/entry-common.h:233
Quoted 8 lines
      syscall_exit_work
      include/linux/entry-common.h:266
      syscall_exit_to_user_mode_work
      include/linux/entry-common.h:297
      syscall_exit_to_user_mode
      include/linux/entry-common.h:334
      do_syscall_64
      arch/x86/entry/syscall_64.c:89

Oh, indeed. Thanks for providing this!

Namhyung

#29 lore raw

Re: [PATCH perf-tools-next v12 06/10] perf ftrace: Support display of inlined functions in function graph tracer

Quoted 61 lines · Namhyung Kim
Quoted 52 lines
Quoted 3 lines
Quoted 1 line

+#define FTRACE_MAX_INLINES 32

Isn't 32 too big? I guess 8 would be enough..

No, 8 is not enough in practice. I will prefer to keep 32. For instance, after a -O2 kernel build:

❯ addr2line -f -i -e vmlinux 0xffffffff816916e9

      __do_trace_rss_stat
      include/trace/events/kmem.h:427 (discriminator 20)
      trace_rss_stat
      include/trace/events/kmem.h:427 (discriminator 20)
      mm_trace_rss_stat
      mm/memory.c:169 (discriminator 20)
      add_mm_counter
      include/linux/mm.h:3469 (discriminator 20)
      add_mm_rss_vec
      mm/memory.c:492 (discriminator 20)
      zap_pte_range
      mm/memory.c:2087 (discriminator 20)
      zap_pmd_range
      mm/memory.c:2151 (discriminator 20)
      zap_pud_range
      mm/memory.c:2179 (discriminator 20)
      zap_p4d_range
      mm/memory.c:2200 (discriminator 20)
      __zap_vma_range
      mm/memory.c:2240 (discriminator 20)

Another example:

❯ addr2line -f -i -e vmlinux 0xffffffff821c34af

      constant_test_bit
      arch/x86/include/asm/bitops.h:202
      arch_test_bit
      arch/x86/include/asm/bitops.h:232
      _test_bit
      include/asm-generic/bitops/instrumented-non-atomic.h:142
      test_ti_thread_flag
      include/linux/thread_info.h:133
      test_tsk_thread_flag
      include/linux/sched.h:2102
      task_sigpending
      include/linux/sched/signal.h:384
      fatal_signal_pending
      include/linux/sched/signal.h:406
      ptrace_report_syscall
      include/linux/ptrace.h:427
      ptrace_report_syscall_exit
      include/linux/ptrace.h:477
      arch_ptrace_report_syscall_exit
      include/linux/entry-common.h:233
Quoted 8 lines
      syscall_exit_work
      include/linux/entry-common.h:266
      syscall_exit_to_user_mode_work
      include/linux/entry-common.h:297
      syscall_exit_to_user_mode
      include/linux/entry-common.h:334
      do_syscall_64
      arch/x86/entry/syscall_64.c:89
Quoted 3 lines

Oh, indeed. Thanks for providing this!

Namhyung

No problem!

Kind regards,

Signature · 1 line
Aaron Tomlin
#30 lore raw

PATCH v12 7/10 perf symbol: Introduce vmlinux_path__find() helper

In reply to Aaron Tomlin

The vmlinux_path array maintains standard paths used by perf to locate the kernel image (e.g. "vmlinux", "/boot/vmlinux", "/lib/modules/%s/build/vmlinux", "/usr/lib/debug/..."). While vmlinux_path__init() populates these candidates and dso__load_vmlinux_path() loads symbols into a DSO/map, callers (such as test suites or probing tools) often need to verify the existence of an accessible vmlinux on disk before attempting machine setup or test execution.

Introduce vmlinux_path__find() to search through the configured vmlinux_path entries (or symbol_conf.vmlinux_name if explicitly set) and return the path to the first readable vmlinux file, or NULL if none is accessible.

---
 tools/perf/util/symbol.c | 38 +++++++++++++++++++++++++++++++++++++-
 tools/perf/util/symbol.h |  2 ++
 2 files changed, 39 insertions(+), 1 deletion(-)
tools/perf/util/symbol.c +37 -1
index b61a574ce9a3..fd4a4441718e 100644
@@ -2337,6 +2337,9 @@ static int vmlinux_path__init(struct perf_env *env)
 	const char *kernel_version;
 	unsigned int i;
 
+	if (vmlinux_path)
+		return 0;
+
 	vmlinux_path = malloc(sizeof(char *) * (ARRAY_SIZE(vmlinux_paths) +
 			      ARRAY_SIZE(vmlinux_paths_upd)));
 	if (vmlinux_path == NULL)
@@ -2372,6 +2375,38 @@ static int vmlinux_path__init(struct perf_env *env)
 	return -1;
 }
 
+/**
+ * vmlinux_path__find - Locate the first accessible vmlinux file
+ *
+ * Check symbol_conf.vmlinux_name first if set, then search through the
+ * initialized paths in vmlinux_path.
+ *
+ * Return: A pointer to the path of the first readable vmlinux file, or NULL if
+ * none is accessible.
+ */
+const char *vmlinux_path__find(void)
+{
+	int i;
+
+	if (symbol_conf.vmlinux_name && access(symbol_conf.vmlinux_name, R_OK) == 0)
+		return symbol_conf.vmlinux_name;
+
+	if (symbol__init(NULL) < 0)
+		return NULL;
+
+	if (!vmlinux_path) {
+		if (vmlinux_path__init(NULL) < 0)
+			return NULL;
+	}
+
+	for (i = 0; i < vmlinux_path__nr_entries; ++i) {
+		if (access(vmlinux_path[i], R_OK) == 0)
+			return vmlinux_path[i];
+	}
+
+	return NULL;
+}
+
 int setup_list(struct strlist **list, const char *list_str,
 		      const char *list_name)
 {
@@ -2598,6 +2633,8 @@ int symbol__init(struct perf_env *env)
 
 void symbol__exit(void)
 {
+	vmlinux_path__exit();
+
 	if (!symbol_conf.initialized)
 		return;
 
@@ -2608,7 +2645,6 @@ void symbol__exit(void)
 	intlist__delete(symbol_conf.tid_list);
 	intlist__delete(symbol_conf.pid_list);
 	intlist__delete(symbol_conf.addr_list);
-	vmlinux_path__exit();
 	symbol_conf.sym_list = symbol_conf.dso_list = symbol_conf.comm_list = NULL;
 	symbol_conf.bt_stop_list = NULL;
 	symbol_conf.initialized = false;
tools/perf/util/symbol.h +2 -0
index d0bac824c79c..6d1eb08076ab 100644
@@ -184,6 +184,8 @@ static inline int __symbol__join_symfs(char *bf, size_t size, const char *path)
 extern int vmlinux_path__nr_entries;
 extern char **vmlinux_path;
 
+const char *vmlinux_path__find(void);
+
 static inline void *symbol__priv(struct symbol *sym)
 {
 	return ((void *)sym) - symbol_conf.priv_size;
Signature · 1 line
2.55.0
#31 lore raw

Re: [PATCH perf-tools-next v12 07/10] perf symbol: Introduce vmlinux_path__find() helper

Sashiko has reviewed this patch and found no issues.
Quoted 8 lines

The vmlinux_path array maintains standard paths used by perf to locate the kernel image (e.g. "vmlinux", "/boot/vmlinux", "/lib/modules/%s/build/vmlinux", "/usr/lib/debug/..."). While vmlinux_path__init() populates these candidates and dso__load_vmlinux_path() loads symbols into a DSO/map, callers (such as test suites or probing tools) often need to verify the existence of an accessible vmlinux on disk before attempting machine setup or test execution.

Quoted 7 lines

Introduce vmlinux_path__find() to search through the configured vmlinux_path entries (or symbol_conf.vmlinux_name if explicitly set) and return the path to the first readable vmlinux file, or NULL if none is accessible.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
#32 lore raw

PATCH v12 8/10 perf test: Add test for ftrace inline and task stack tracking

In reply to Aaron Tomlin

Add an automated unit test suite ("Ftrace inline processing") in tools/perf/tests/ftrace.c to validate the function graph call stack tracking and inlined function reconstruction implementation.

The test suite verifies:

1. DWARF inline symbol resolution

Uses vmlinux_path__find() to locate the kernel image, setting up symbol resolution and validating activate_task inlining within wake_up_new_task at offset 0x1d1.

2. Inline presentation and return address filtering

Verifies that synthesised inlined entries and exits are correctly annotated with /* (inline) */, that return address comments are filtered out by default, and that comments are preserved when graph_retaddr is requested.

3. Bare closing brace matching

Verifies that bare closing braces without trailing function names (the default when funcgraph-tail is disabled) correctly pop and match their corresponding LIFO stack frames.

4. Task context-switch tracking and CPU migration

Verifies call stack isolation across context switch banners and task migration across different CPUs.

5. Deep stack growth and recovery

Verifies that dynamic call stack growth and depth recovery operate cleanly under deep call stacks exceeding the initial FTRACE_MAX_STACK capacity.

---
 tools/perf/tests/ftrace.c | 1377 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 1368 insertions(+), 9 deletions(-)
tools/perf/tests/ftrace.c +1368 -9
index 42604811a2fb..734d1e97bf07 100644
@@ -7,7 +7,14 @@
 #include <unistd.h>
 #include "tests.h"
 #include "debug.h"
+#include "dso.h"
+#include "machine.h"
+#include "map.h"
+#include "symbol.h"
+#include "srcline.h"
 #include "util/ftrace.h"
+#include "evlist.h"
+#include "thread_map.h"
 
 static int test_parse_retaddr(void)
 {
@@ -63,19 +70,1371 @@ static int test_filter_retaddr(void)
 	return TEST_OK;
 }
 
-static int test__ftrace(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
+static void ensure_synthetic_inline(struct machine *machine,
+				    const char *caller, u64 offset,
+				    const char *inline_name)
 {
-	int ret;
+	struct map *kmap = NULL;
+	struct symbol *sym;
 
-	ret = test_parse_retaddr();
-	if (ret != TEST_OK)
-		return ret;
+	if (!machine || !offset)
+		return;
 
-	ret = test_filter_retaddr();
-	if (ret != TEST_OK)
-		return ret;
+	sym = machine__find_kernel_symbol_by_name(machine, caller, &kmap);
+	if (sym && kmap) {
+		struct dso *dso = map__dso(kmap);
+		u64 ip = sym->start + offset - 1;
+		u64 addr = map__rip_2objdump(kmap, ip);
+		struct inline_node *node;
 
+		node = inlines__tree_find(dso__inlined_nodes(dso), addr);
+		if (!node) {
+			struct symbol *isym = new_inline_sym(dso, sym, inline_name);
+
+			node = zalloc(sizeof(*node));
+			if (node && isym) {
+				INIT_LIST_HEAD(&node->val);
+				node->addr = addr;
+				if (inline_list__append(isym, NULL, node) != 0) {
+					symbol__delete(isym);
+					free(node);
+				} else {
+					inlines__tree_insert(dso__inlined_nodes(dso), node);
+				}
+			} else {
+				if (isym)
+					symbol__delete(isym);
+				free(node);
+			}
+		} else {
+			struct inline_list *pos, *outer = NULL;
+
+			list_for_each_entry_reverse(pos, &node->val, list) {
+				if (pos->symbol && symbol__inlined(pos->symbol)) {
+					outer = pos;
+					break;
+				}
+			}
+			if (!outer || strcmp(outer->symbol->name, inline_name)) {
+				struct symbol *isym = new_inline_sym(dso, sym, inline_name);
+
+				if (isym) {
+					inline_node__clear_frames(node);
+					if (inline_list__append(isym, NULL, node) != 0)
+						symbol__delete(isym);
+				}
+			}
+		}
+		map__put(kmap);
+	}
+}
+
+#define TEST_CHECK(text, cond)					\
+do {								\
+	if (!(cond)) {						\
+		pr_debug("FAILED %s:%d %s\n", __FILE__, __LINE__, text); \
+		ret = TEST_FAIL;				\
+		goto out_cleanup;				\
+	}							\
+} while (0)
+
+static int test_inline_processing(void)
+{
+	static const char line_entry[] = " 0)               |  wake_up_new_task() {";
+	static const char line_enqueue[] =
+		" 0)               |    enqueue_task() { /* <-wake_up_new_task+0x1d1/0x3e0 */";
+	static const char line_exit_enqueue[] = " 0) + 10.000 us   |    } /* enqueue_task */";
+	static const char line_exit_task[] = " 0) + 20.000 us   |  } /* wake_up_new_task */";
+	static const char line_bare_enqueue[] = " 0) + 10.000 us   |    }";
+	static const char line_bare_task[] = " 0) + 20.000 us   |  }";
+	static const char line_unmatched[] = " 0) +  5.000 us   |    } /* unrelated_func */";
+	static const char line_verbose_entry[] =
+		" 1234.567890 |   0)   bash-1234    |  ....   |               |  wake_up_new_task() {";
+	static const char line_verbose_enqueue[] =
+		" 1234.567890 |   0)   bash-1234    |  ....   |               |    enqueue_task() { /* <-wake_up_new_task+0x1d1/0x3e0 */";
+	static const char line_verbose_exit_enqueue[] =
+		" 1234.567890 |   0)   bash-1234    |  ....   |   1.234 us    |    } /* enqueue_task */";
+	static const char line_verbose_exit_task[] =
+		" 1234.567890 |   0)   bash-1234    |  ....   |   2.567 us    |  } /* wake_up_new_task */";
+	static const char line_verbose_inline_entry[] =
+		" 1234.567890 |   0)   bash-1234    |  ....   |               |    activate_task() { /* (inline) */";
+	static const char line_verbose_inline_exit[] =
+		" 1234.567890 |   0)   bash-1234    |  ....   |               |    } /* activate_task (inline) */";
+	struct perf_ftrace ftrace;
+	char *out_buf = NULL;
+	size_t out_len = 0;
+	FILE *out;
+	const char *unmatched_pos;
+	const char *inline_exit_pos;
+	const char *vmlinux;
+	const char *saved_vmlinux = symbol_conf.vmlinux_name;
+	bool saved_ignore_buildid = symbol_conf.ignore_vmlinux_buildid;
+	int ret = TEST_OK;
+
+	memset(&ftrace, 0, sizeof(ftrace));
+	ftrace.use_inline = true;
+	/* By default, graph_retaddr is 0 (return address comments are filtered) */
+	ftrace.tracer = "function_graph";
+
+	vmlinux = vmlinux_path__find();
+	if (!vmlinux)
+		return TEST_SKIP;
+
+	/* If vmlinux is present, test resolving symbols */
+	{
+		const char *inlined_names[FTRACE_MAX_INLINES];
+		int num_inlines;
+		int j;
+
+		symbol_conf.vmlinux_name = vmlinux;
+		symbol_conf.ignore_vmlinux_buildid = true;
+		ret = perf_ftrace__setup_inlines(&ftrace);
+		if (ret < 0 || !ftrace.machine) {
+			ret = TEST_SKIP;
+			goto out_cleanup;
+		}
+
+		num_inlines = ftrace_resolve_inlines(ftrace.machine,
+						     "wake_up_new_task", 0x1d1,
+						     inlined_names,
+						     FTRACE_MAX_INLINES);
+		if (num_inlines <= 0 || strcmp(inlined_names[0], "activate_task") != 0) {
+			pr_debug("vmlinux does not inline activate_task at wake_up_new_task+0x1d1; falling back to synthetic inline\n");
+			for (j = 0; j < num_inlines; j++)
+				free((void *)inlined_names[j]);
+			ensure_synthetic_inline(ftrace.machine, "wake_up_new_task", 0x1d1,
+						"activate_task");
+			num_inlines = ftrace_resolve_inlines(ftrace.machine,
+							     "wake_up_new_task", 0x1d1,
+							     inlined_names,
+							     FTRACE_MAX_INLINES);
+			if (num_inlines <= 0 || strcmp(inlined_names[0], "activate_task") != 0) {
+				for (j = 0; j < num_inlines; j++)
+					free((void *)inlined_names[j]);
+				ret = TEST_SKIP;
+				goto out_cleanup;
+			}
+		}
+		for (j = 0; j < num_inlines; j++)
+			free((void *)inlined_names[j]);
+
+		if (ftrace_resolve_inlines(ftrace.machine, "wake_up_new_task", 0,
+					   inlined_names, FTRACE_MAX_INLINES) != 0) {
+			pr_debug("ftrace_resolve_inlines with offset 0 should return 0\n");
+			ret = TEST_FAIL;
+			goto out_cleanup;
+		}
+
+		out = open_memstream(&out_buf, &out_len);
+		if (!out) {
+			ret = TEST_FAIL;
+			goto out_cleanup;
+		}
+
+		/* Feed simulated function graph trace lines */
+		ftrace_process_fgraph_line(&ftrace, line_entry, out);
+		ftrace_process_fgraph_line(&ftrace, line_enqueue, out);
+		ftrace_process_fgraph_line(&ftrace, line_exit_enqueue, out);
+		ftrace_process_fgraph_line(&ftrace, line_exit_task, out);
+
+		fclose(out);
+		ftrace_cpu_states__clear();
+
+		pr_debug("Processed output:\n%s\n", out_buf);
+
+		/* Verify activate_task entry with inline hint */
+		TEST_CHECK("contains activate_task entry with inline hint",
+			   strstr(out_buf, "activate_task() { /* (inline) */") != NULL);
+
+		/* Verify activate_task exit with inline hint */
+		TEST_CHECK("contains activate_task exit with inline hint",
+			   strstr(out_buf, "} /* activate_task (inline) */") != NULL);
+
+		/* Verify return address comment was filtered out by default */
+		TEST_CHECK("retaddr comment filtered out",
+			   strstr(out_buf, "/* <-wake_up_new_task") == NULL);
+
+		free(out_buf);
+		out_buf = NULL;
+
+		/* Now test with graph_retaddr = 1 (user specified --graph-opts retaddr) */
+		ftrace.graph_retaddr = 1;
+		out = open_memstream(&out_buf, &out_len);
+		if (!out) {
+			ret = TEST_FAIL;
+			goto out_cleanup;
+		}
+
+		ftrace_process_fgraph_line(&ftrace, line_entry, out);
+		ftrace_process_fgraph_line(&ftrace, line_enqueue, out);
+		ftrace_process_fgraph_line(&ftrace, line_exit_enqueue, out);
+		ftrace_process_fgraph_line(&ftrace, line_exit_task, out);
+
+		fclose(out);
+		ftrace_cpu_states__clear();
+
+		/* Verify return address comment IS preserved when graph_retaddr = 1 */
+		TEST_CHECK("retaddr comment preserved when requested",
+			   strstr(out_buf, "/* <-wake_up_new_task") != NULL);
+
+		free(out_buf);
+		out_buf = NULL;
+
+		/* Test with bare closing braces (funcgraph-tail disabled by default) */
+		ftrace.graph_retaddr = 0;
+		out = open_memstream(&out_buf, &out_len);
+		if (!out) {
+			ret = TEST_FAIL;
+			goto out_cleanup;
+		}
+
+		ftrace_process_fgraph_line(&ftrace, line_entry, out);
+		ftrace_process_fgraph_line(&ftrace, line_enqueue, out);
+		ftrace_process_fgraph_line(&ftrace, line_bare_enqueue, out);
+		ftrace_process_fgraph_line(&ftrace, line_bare_task, out);
+
+		fclose(out);
+
+		TEST_CHECK("bare braces inline entry",
+			   strstr(out_buf, "activate_task() { /* (inline) */") != NULL);
+		TEST_CHECK("bare braces inline exit",
+			   strstr(out_buf, "} /* activate_task (inline) */") != NULL);
+
+		free(out_buf);
+		out_buf = NULL;
+
+		/* Test bare closing brace matches outer frame when inner callee exit was dropped */
+		out = open_memstream(&out_buf, &out_len);
+		if (!out) {
+			ret = TEST_FAIL;
+			goto out_cleanup;
+		}
+
+		ftrace_process_fgraph_line(&ftrace, line_entry, out);
+		ftrace_process_fgraph_line(&ftrace, line_enqueue, out);
+		/* line_bare_enqueue is dropped! */
+		ftrace_process_fgraph_line(&ftrace, line_bare_task, out);
+
+		fclose(out);
+
+		TEST_CHECK("dropped callee exit: inlined frame closed",
+			   strstr(out_buf, "} /* activate_task (inline) */") != NULL);
+		TEST_CHECK("dropped callee exit: outer bare brace emitted",
+			   strstr(out_buf, "|  }\n") != NULL);
+
+		free(out_buf);
+		out_buf = NULL;
+
+		/* Test that unmatched exit events do not pop active inlined frames */
+		out = open_memstream(&out_buf, &out_len);
+		if (!out) {
+			ret = TEST_FAIL;
+			goto out_cleanup;
+		}
+
+		ftrace_process_fgraph_line(&ftrace, line_entry, out);
+		ftrace_process_fgraph_line(&ftrace, line_enqueue, out);
+		ftrace_process_fgraph_line(&ftrace, line_exit_enqueue, out);
+		/* Unmatched exit event while activate_task is still active */
+		ftrace_process_fgraph_line(&ftrace, line_unmatched, out);
+		ftrace_process_fgraph_line(&ftrace, line_exit_task, out);
+
+		fclose(out);
+
+		unmatched_pos = strstr(out_buf, "} /* unrelated_func */");
+		inline_exit_pos = strstr(out_buf, "} /* activate_task (inline) */");
+
+		TEST_CHECK("unmatched exit present", unmatched_pos != NULL);
+		TEST_CHECK("inline exit present", inline_exit_pos != NULL);
+		TEST_CHECK("inline exit not prematurely popped before unmatched exit",
+			   inline_exit_pos > unmatched_pos);
+
+		free(out_buf);
+		out_buf = NULL;
+
+		/* Test recovery when an intermediate real function exit event is dropped */
+		out = open_memstream(&out_buf, &out_len);
+		if (!out) {
+			ret = TEST_FAIL;
+			goto out_cleanup;
+		}
+
+		ftrace_process_fgraph_line(&ftrace, line_entry, out);
+		ftrace_process_fgraph_line(&ftrace, line_enqueue, out);
+		/* line_exit_enqueue is dropped; outer wake_up_new_task exits directly */
+		ftrace_process_fgraph_line(&ftrace, line_exit_task, out);
+
+		fclose(out);
+
+		TEST_CHECK("inline exit synthesized despite dropped callee exit",
+			   strstr(out_buf, "} /* activate_task (inline) */") != NULL);
+
+		free(out_buf);
+		out_buf = NULL;
+
+		/* Test recovery when intermediate callee exit is dropped before sibling call */
+		out = open_memstream(&out_buf, &out_len);
+		if (!out) {
+			ret = TEST_FAIL;
+			goto out_cleanup;
+		}
+
+		ftrace_process_fgraph_line(&ftrace, line_entry, out);
+		ftrace_process_fgraph_line(&ftrace, line_enqueue, out);
+		/* line_exit_enqueue dropped; caller makes sibling call without inlines */
+		ftrace_process_fgraph_line(&ftrace,
+			" 0)               |    check_preempt_curr(); /* <-wake_up_new_task+0x0/0x3e0 */",
+			out);
+		ftrace_process_fgraph_line(&ftrace, line_exit_task, out);
+
+		fclose(out);
+
+		TEST_CHECK("inline exit synthesized when callee exit dropped before sibling call",
+			   strstr(out_buf, "} /* activate_task (inline) */") != NULL);
+
+		free(out_buf);
+		out_buf = NULL;
+
+		/* Test that callers not on active stack do not synthesise inlines */
+		out = open_memstream(&out_buf, &out_len);
+		if (!out) {
+			ret = TEST_FAIL;
+			goto out_cleanup;
+		}
+
+		ftrace_process_fgraph_line(&ftrace, line_entry, out);
+		/* housekeeping_any_cpu called by untraced ring_buffer_unlock_commit */
+		ftrace_process_fgraph_line(&ftrace,
+			" 0)               |    housekeeping_any_cpu(); /* <-ring_buffer_unlock_commit+0x1a0/0x1f0 */",
+			out);
+		ftrace_process_fgraph_line(&ftrace, line_exit_task, out);
+
+		fclose(out);
+
+		TEST_CHECK("untraced caller inlines not synthesised",
+			   strstr(out_buf, "rb_wakeups") == NULL &&
+			   strstr(out_buf, "rb_irq_work_queue") == NULL);
+
+		free(out_buf);
+		out_buf = NULL;
+
+		/* Test trailing inlined frame closed on untraced entry (caller_idx < 0) */
+		out = open_memstream(&out_buf, &out_len);
+		if (!out) {
+			ret = TEST_FAIL;
+			goto out_cleanup;
+		}
+
+		ftrace_process_fgraph_line(&ftrace, line_entry, out);
+		ftrace_process_fgraph_line(&ftrace, line_enqueue, out);
+		/* line_enqueue has inlined activate_task() open; next entry has caller_idx < 0 */
+		ftrace_process_fgraph_line(&ftrace,
+					   " 0)               |    irq_handler() {",
+					   out);
+		ftrace_process_fgraph_line(&ftrace,
+					   " 0)               |    } /* irq_handler */",
+					   out);
+		ftrace_process_fgraph_line(&ftrace, line_exit_task, out);
+
+		fclose(out);
+
+		TEST_CHECK("trailing inline frame closed before untraced call",
+			   strstr(out_buf, "} /* activate_task (inline) */") != NULL);
+		TEST_CHECK("untraced call not indented inside trailing inline frame",
+			   strstr(out_buf, "|    irq_handler() {") != NULL);
+
+		free(out_buf);
+		out_buf = NULL;
+
+		/* Test dropped exits unwound when new function starts at shallower indent */
+		out = open_memstream(&out_buf, &out_len);
+		if (!out) {
+			ret = TEST_FAIL;
+			goto out_cleanup;
+		}
+
+		ftrace_process_fgraph_line(&ftrace, line_entry, out);
+		ftrace_process_fgraph_line(&ftrace, line_enqueue, out);
+		/* Exits dropped! New root entry arrives at indent 2 */
+		ftrace_process_fgraph_line(&ftrace,
+					   " 0)               |  schedule() {",
+					   out);
+		ftrace_process_fgraph_line(&ftrace,
+					   " 0)               |  } /* schedule */",
+					   out);
+
+		fclose(out);
+
+		TEST_CHECK("stale inlines closed when new root arrives",
+			   strstr(out_buf, "} /* activate_task (inline) */") != NULL);
+		TEST_CHECK("new root entry printed without stale indentation",
+			   strstr(out_buf, "|  schedule() {") != NULL);
+
+		free(out_buf);
+		out_buf = NULL;
+
+		/* Test verbose mode prefix formatting for synthesised inlines */
+		ftrace.graph_verbose = true;
+		out = open_memstream(&out_buf, &out_len);
+		if (!out) {
+			ret = TEST_FAIL;
+			goto out_cleanup;
+		}
+
+		ftrace_process_fgraph_line(&ftrace, line_verbose_entry, out);
+		ftrace_process_fgraph_line(&ftrace, line_verbose_enqueue, out);
+		ftrace_process_fgraph_line(&ftrace, line_verbose_exit_enqueue, out);
+		ftrace_process_fgraph_line(&ftrace, line_verbose_exit_task, out);
+
+		fclose(out);
+		ftrace_cpu_states__clear();
+
+		TEST_CHECK("verbose inline entry preserves comm-pid",
+			   strstr(out_buf, line_verbose_inline_entry) != NULL);
+		TEST_CHECK("verbose inline exit preserves comm-pid",
+			   strstr(out_buf, line_verbose_inline_exit) != NULL);
+
+		free(out_buf);
+		out_buf = NULL;
+		ftrace.graph_verbose = false;
+	}
+
+out_cleanup:
+	free(out_buf);
+	perf_ftrace__cleanup_inlines(&ftrace);
+	symbol_conf.vmlinux_name = saved_vmlinux;
+	symbol_conf.ignore_vmlinux_buildid = saved_ignore_buildid;
+	return ret;
+}
+#undef TEST_CHECK
+
+static void feed_fgraph_entry(struct perf_ftrace *ftrace, int cpu, int depth,
+			      const char *name, const char *retaddr, FILE *out)
+{
+	char line[1024];
+	int indent = 2 + 2 * depth;
+
+	if (retaddr) {
+		snprintf(line, sizeof(line), " %d)               |%*s%s() { /* <-%s */",
+			 cpu, indent, "", name, retaddr);
+	} else {
+		snprintf(line, sizeof(line), " %d)               |%*s%s() {",
+			 cpu, indent, "", name);
+	}
+	ftrace_process_fgraph_line(ftrace, line, out);
+}
+
+static void feed_fgraph_exit(struct perf_ftrace *ftrace, int cpu, int depth,
+			     const char *name, FILE *out)
+{
+	char line[1024];
+	int indent = 2 + 2 * depth;
+
+	if (name) {
+		snprintf(line, sizeof(line), " %d)   1.000 us    |%*s} /* %s */",
+			 cpu, indent, "", name);
+	} else {
+		snprintf(line, sizeof(line), " %d)   1.000 us    |%*s}",
+			 cpu, indent, "");
+	}
+	ftrace_process_fgraph_line(ftrace, line, out);
+}
+
+static int test_stack_saturation(void)
+{
+	struct perf_ftrace ftrace;
+	char *out_buf = NULL;
+	size_t out_len = 0;
+	FILE *out;
+	const char *vmlinux;
+	int i;
+
+	memset(&ftrace, 0, sizeof(ftrace));
+	ftrace.use_inline = true;
+	ftrace_cpu_states__clear();
+
+	out = open_memstream(&out_buf, &out_len);
+	if (!out)
+		return TEST_FAIL;
+
+	/* Push root frame at depth 0 */
+	feed_fgraph_entry(&ftrace, 0, 0, "root_func", NULL, out);
+
+	/* Push 255 more frames to fill initial FTRACE_MAX_STACK (256) */
+	for (i = 1; i < 256; i++)
+		feed_fgraph_entry(&ftrace, 0, i, "sub_func", NULL, out);
+
+	/* Push 44 additional frames triggering dynamic stack expansion beyond 256 */
+	for (i = 0; i < 44; i++)
+		feed_fgraph_entry(&ftrace, 0, 256 + i, "overflow_func", NULL, out);
+
+	/* Pop the 44 expanded frames with bare closing braces */
+	for (i = 43; i >= 0; i--)
+		feed_fgraph_exit(&ftrace, 0, 256 + i, NULL, out);
+
+	/* Pop 255 tracked sub_func frames */
+	for (i = 255; i >= 1; i--)
+		feed_fgraph_exit(&ftrace, 0, i, NULL, out);
+
+	/* Pop root frame - must cleanly match root_func on stack, not popped prematurely */
+	feed_fgraph_exit(&ftrace, 0, 0, "root_func", out);
+
+	/* Feed one more function entry - it must have base indentation (no indentation drift) */
+	feed_fgraph_entry(&ftrace, 0, 0, "base_func", NULL, out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	/* Verify base_func starts at standard 2 spaces indentation without drift */
+	if (!strstr(out_buf, "|  base_func() {")) {
+		pr_debug("FAILED %s:%d no indentation drift after stack saturation\n",
+			 __FILE__, __LINE__);
+		free(out_buf);
+		return TEST_FAIL;
+	}
+
+	free(out_buf);
+	out_buf = NULL;
+
+	/* Test call depth recovery when events are dropped on deep stack */
+	out = open_memstream(&out_buf, &out_len);
+	if (!out)
+		return TEST_FAIL;
+
+	feed_fgraph_entry(&ftrace, 0, 0, "root_func", NULL, out);
+	for (i = 1; i < 256; i++)
+		feed_fgraph_entry(&ftrace, 0, i, "sub_func", NULL, out);
+	for (i = 0; i < 44; i++)
+		feed_fgraph_entry(&ftrace, 0, 256 + i, "overflow_func", NULL, out);
+
+	/* Dropped events: root_func directly calls recovery_func */
+	feed_fgraph_entry(&ftrace, 0, 1, "recovery_func", "root_func+0x20", out);
+	feed_fgraph_exit(&ftrace, 0, 1, NULL, out);
+	feed_fgraph_exit(&ftrace, 0, 0, "root_func", out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	if (!strstr(out_buf, "recovery_func() {")) {
+		pr_debug("FAILED %s:%d recovery_func missing after overflow recovery\n",
+			 __FILE__, __LINE__);
+		free(out_buf);
+		return TEST_FAIL;
+	}
+	if (!strstr(out_buf, "} /* root_func */")) {
+		pr_debug("FAILED %s:%d root_func exit missing after overflow recovery\n",
+			 __FILE__, __LINE__);
+		free(out_buf);
+		return TEST_FAIL;
+	}
+
+	free(out_buf);
+	out_buf = NULL;
+
+	/* Test that bare braces on deep stack do not prematurely pop tracked frames */
+	out = open_memstream(&out_buf, &out_len);
+	if (!out)
+		return TEST_FAIL;
+
+	feed_fgraph_entry(&ftrace, 0, 0, "root_func", NULL, out);
+	for (i = 1; i < 256; i++)
+		feed_fgraph_entry(&ftrace, 0, i, "sub_func", NULL, out);
+	for (i = 0; i < 10; i++)
+		feed_fgraph_entry(&ftrace, 0, 256 + i, "overflow_func", NULL, out);
+
+	/* Pop 10 overflow frames with bare braces */
+	for (i = 9; i >= 0; i--)
+		feed_fgraph_exit(&ftrace, 0, 256 + i, NULL, out);
+
+	/* Pop 255 sub_func frames with bare braces */
+	for (i = 255; i >= 1; i--)
+		feed_fgraph_exit(&ftrace, 0, i, NULL, out);
+
+	/* Root frame must still be on the stack to match caller */
+	feed_fgraph_entry(&ftrace, 0, 1, "child_func", "root_func+0x10", out);
+	feed_fgraph_exit(&ftrace, 0, 1, NULL, out);
+	feed_fgraph_exit(&ftrace, 0, 0, "root_func", out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	if (!strstr(out_buf, "child_func() {")) {
+		pr_debug("FAILED %s:%d child_func missing after bare braces in overflow\n",
+			 __FILE__, __LINE__);
+		free(out_buf);
+		return TEST_FAIL;
+	}
+	if (!strstr(out_buf, "} /* root_func */")) {
+		pr_debug("FAILED %s:%d root_func exit missing after bare braces in overflow\n",
+			 __FILE__, __LINE__);
+		free(out_buf);
+		return TEST_FAIL;
+	}
+
+	free(out_buf);
+	out_buf = NULL;
+
+	/* Test recursive caller lookup skips stale inner invocation */
+	out = open_memstream(&out_buf, &out_len);
+	if (!out)
+		return TEST_FAIL;
+
+	/* Outer recursive invocation at indent 2 */
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  rec_func() {", out);
+	/* Inner recursive invocation at indent 4 */
+	ftrace_process_fgraph_line(&ftrace,
+				   " 0)               |    rec_func() { /* <-rec_func+0x10 */",
+				   out);
+	/* Inner invocation exit is dropped by ring buffer */
+	/* Outer invocation calls sibling_func at indent 4 */
+	ftrace_process_fgraph_line(&ftrace,
+				   " 0)   1.000 us    |    sibling_func(); /* <-rec_func+0x20 */",
+				   out);
+	ftrace_process_fgraph_line(&ftrace, " 0)   2.000 us    |  } /* rec_func */", out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	if (!strstr(out_buf, "sibling_func();")) {
+		pr_debug("FAILED %s:%d sibling_func missing under recursive caller\n",
+			 __FILE__, __LINE__);
+		free(out_buf);
+		return TEST_FAIL;
+	}
+	if (!strstr(out_buf, "} /* rec_func */")) {
+		pr_debug("FAILED %s:%d rec_func exit missing after dropped inner exit\n",
+			 __FILE__, __LINE__);
+		free(out_buf);
+		return TEST_FAIL;
+	}
+	free(out_buf);
+	out_buf = NULL;
+
+	/* Test dropped inner recursive entry does not prematurely unwind outer caller */
+	out = open_memstream(&out_buf, &out_len);
+	if (!out)
+		return TEST_FAIL;
+
+	/* Outer recursive invocation at indent 2 */
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  rec_func() {", out);
+	/* Inner recursive entry at indent 4 is DROPPED by ring buffer */
+	/* Inner recursive invocation exits at indent 4 */
+	ftrace_process_fgraph_line(&ftrace, " 0)   1.000 us    |    } /* rec_func */", out);
+	/* Outer invocation is still active; calls child_func at indent 4 */
+	ftrace_process_fgraph_line(&ftrace,
+				   " 0)   1.000 us    |    child_func(); /* <-rec_func+0x20 */",
+				   out);
+	/* Outer invocation exits at indent 2 */
+	ftrace_process_fgraph_line(&ftrace, " 0)   2.000 us    |  } /* rec_func */", out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	if (!strstr(out_buf, "child_func();")) {
+		pr_debug("FAILED %s:%d child_func missing after dropped inner recursive entry\n",
+			 __FILE__, __LINE__);
+		free(out_buf);
+		return TEST_FAIL;
+	}
+	free(out_buf);
+	out_buf = NULL;
+
+	/* Test top inlined frame when cs->overflow_depth > 0 does not misclassify exit */
+	vmlinux = vmlinux_path__find();
+	if (vmlinux) {
+		const char *saved_vmlinux = symbol_conf.vmlinux_name;
+		bool saved_ignore_buildid = symbol_conf.ignore_vmlinux_buildid;
+		const char *inlined_names[FTRACE_MAX_INLINES];
+		int num_inlines;
+		int j;
+
+		symbol_conf.vmlinux_name = vmlinux;
+		symbol_conf.ignore_vmlinux_buildid = true;
+		if (perf_ftrace__setup_inlines(&ftrace) == 0 && ftrace.machine) {
+			ensure_synthetic_inline(ftrace.machine, "wake_up_new_task", 0x1d1,
+						"activate_task");
+			num_inlines = ftrace_resolve_inlines(ftrace.machine,
+							     "wake_up_new_task", 0x1d1,
+							     inlined_names,
+							     FTRACE_MAX_INLINES);
+			if (num_inlines > 0 && !strcmp(inlined_names[0], "activate_task")) {
+				out = open_memstream(&out_buf, &out_len);
+				if (!out) {
+					for (j = 0; j < num_inlines; j++)
+						free((void *)inlined_names[j]);
+					perf_ftrace__cleanup_inlines(&ftrace);
+					symbol_conf.vmlinux_name = saved_vmlinux;
+					symbol_conf.ignore_vmlinux_buildid = saved_ignore_buildid;
+					return TEST_FAIL;
+				}
+
+				/* Fill stack up to 254 with tracked frames */
+				feed_fgraph_entry(&ftrace, 0, 0, "root_func", NULL, out);
+				for (i = 1; i < 254; i++)
+					feed_fgraph_entry(&ftrace, 0, i, "sub_func", NULL, out);
+
+				/* Frame 254 is wake_up_new_task */
+				feed_fgraph_entry(&ftrace, 0, 254, "wake_up_new_task", NULL, out);
+
+				/*
+				 * Frame 255: enqueue_task resolves inline activate_task.
+				 * Inlined frame activate_task is pushed at stack[255]
+				 * (cs->depth = 256), and real frame enqueue_task dynamically
+				 * grows the stack capacity beyond 256.
+				 */
+				feed_fgraph_entry(&ftrace, 0, 255, "enqueue_task",
+						  "wake_up_new_task+0x1d1/0x3e0", out);
+
+				/*
+				 * Nested child call on deep stack: does NOT pop the
+				 * inlined frame!
+				 */
+				feed_fgraph_entry(&ftrace, 0, 256, "child_overflow",
+						  "enqueue_task+0x10", out);
+
+				/* Exit child_overflow */
+				feed_fgraph_exit(&ftrace, 0, 256, NULL, out);
+
+				/* Exit enqueue_task */
+				feed_fgraph_exit(&ftrace, 0, 255, NULL, out);
+
+				/* Exit wake_up_new_task: synthesises inline exit */
+				feed_fgraph_exit(&ftrace, 0, 254, NULL, out);
+
+				/* Pop remaining sub_func frames and root_func */
+				for (i = 253; i >= 1; i--)
+					feed_fgraph_exit(&ftrace, 0, i, NULL, out);
+				feed_fgraph_exit(&ftrace, 0, 0, "root_func", out);
+
+				/* Feed base frame to check for indentation drift */
+				feed_fgraph_entry(&ftrace, 0, 0, "drift_check", NULL, out);
+
+				fclose(out);
+				ftrace_cpu_states__clear();
+
+				if (!strstr(out_buf, "activate_task() { /* (inline) */") ||
+				    !strstr(out_buf, "} /* activate_task (inline) */") ||
+				    !strstr(out_buf, "|  drift_check() {")) {
+					pr_debug("FAILED %s:%d top inlined frame saturation failed\n",
+						 __FILE__, __LINE__);
+					free(out_buf);
+					for (j = 0; j < num_inlines; j++)
+						free((void *)inlined_names[j]);
+					perf_ftrace__cleanup_inlines(&ftrace);
+					symbol_conf.vmlinux_name = saved_vmlinux;
+					symbol_conf.ignore_vmlinux_buildid = saved_ignore_buildid;
+					return TEST_FAIL;
+				}
+
+				free(out_buf);
+				out_buf = NULL;
+			}
+			for (j = 0; j < num_inlines; j++)
+				free((void *)inlined_names[j]);
+			perf_ftrace__cleanup_inlines(&ftrace);
+		}
+		symbol_conf.vmlinux_name = saved_vmlinux;
+		symbol_conf.ignore_vmlinux_buildid = saved_ignore_buildid;
+	}
+
+	free(out_buf);
 	return TEST_OK;
 }
 
-DEFINE_SUITE("Ftrace return address processing", ftrace);
+static int test_column_layouts(void)
+{
+	static const struct {
+		const char *desc;
+				bool verbose;
+		const char *in;
+		const char *expected;
+	} cases[] = {
+		{
+			"default layout", false,
+			" 7)               |  wake_up_new_task() { /* <-kernel_clone+0x160/0x510 */",
+			" 7)               |  wake_up_new_task() {\n",
+		},
+		{
+			"verbose layout", true,
+			" 1234.567890 |   7)   bash-1234    |  ....   |               |  wake_up_new_task() { /* <-kernel_clone+0x160/0x510 */",
+			" 1234.567890 |   7)   bash-1234    |  ....   |               |  wake_up_new_task() {\n",
+		},
+	};
+	struct perf_ftrace ftrace;
+	char *out_buf = NULL;
+	size_t out_len = 0;
+	FILE *out;
+	size_t i;
+
+	for (i = 0; i < ARRAY_SIZE(cases); i++) {
+		memset(&ftrace, 0, sizeof(ftrace));
+		ftrace.use_inline = true;
+		ftrace.graph_verbose = cases[i].verbose;
+
+		out = open_memstream(&out_buf, &out_len);
+		if (!out)
+			return TEST_FAIL;
+
+		ftrace_process_fgraph_line(&ftrace, cases[i].in, out);
+		fclose(out);
+		ftrace_cpu_states__clear();
+
+		if (strcmp(out_buf, cases[i].expected)) {
+			pr_debug("%s: expected '%s', got '%s'\n",
+				 cases[i].desc, cases[i].expected, out_buf);
+			free(out_buf);
+			return TEST_FAIL;
+		}
+		free(out_buf);
+		out_buf = NULL;
+	}
+
+	return TEST_OK;
+}
+
+#define TEST_CS_ASSERT(text, cond)					\
+do {									\
+	if (!(cond)) {							\
+		pr_debug("FAILED %s:%d %s\n", __FILE__, __LINE__, text); \
+		ret = TEST_FAIL;					\
+		goto out_cleanup;					\
+	}								\
+} while (0)
+
+static int test_context_switch_handling(void)
+{
+	struct perf_ftrace ftrace;
+	struct perf_thread_map *threads;
+	const char *saved_vmlinux = symbol_conf.vmlinux_name;
+	bool saved_ignore_buildid = symbol_conf.ignore_vmlinux_buildid;
+	char *out_buf = NULL;
+	size_t out_len = 0;
+	const char *vmlinux;
+	struct map *kmap = NULL;
+	struct symbol *sym;
+	FILE *out;
+	int i;
+	int ret = TEST_OK;
+
+	vmlinux = vmlinux_path__find();
+	if (!vmlinux)
+		return TEST_SKIP;
+
+	memset(&ftrace, 0, sizeof(ftrace));
+	ftrace.use_inline = true;
+	symbol_conf.vmlinux_name = vmlinux;
+	symbol_conf.ignore_vmlinux_buildid = true;
+	if (perf_ftrace__setup_inlines(&ftrace) < 0 || !ftrace.machine) {
+		ret = TEST_SKIP;
+		goto out_cleanup_conf;
+	}
+
+	sym = machine__find_kernel_symbol_by_name(ftrace.machine, "wake_up_new_task", &kmap);
+	if (!sym || !kmap) {
+		ret = TEST_SKIP;
+		goto out_cleanup;
+	}
+	map__put(kmap);
+
+	ensure_synthetic_inline(ftrace.machine, "wake_up_new_task", 0x1d1,
+				"activate_task");
+
+	ftrace_cpu_states__clear();
+
+	out = open_memstream(&out_buf, &out_len);
+	if (!out) {
+		ret = TEST_FAIL;
+		goto out_cleanup;
+	}
+
+	/*
+	 * Initial Task 1 (PID 100) starts running on CPU 0 before any banner
+	 * has been emitted on that CPU (default kernel ftrace behaviour).
+	 * Frame wake_up_new_task at offset 0x1d1 resolves inline activate_task.
+	 */
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  wake_up_new_task() {", out);
+	ftrace_process_fgraph_line(&ftrace,
+				   " 0)               |    enqueue_task() { /* <-wake_up_new_task+0x1d1/0x3e0 */",
+				   out);
+
+	/* First context switch on CPU 0: Task 1 switches to <idle>-0 */
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)    task1-100   =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+
+	/* Idle task executes on CPU 0 - must NOT inherit Task 1's inlined call stack */
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  default_idle() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  } /* default_idle */", out);
+
+	/* Context switch to Task 2 (PID 200) on CPU 0 */
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)    <idle>-0    =>    task2-200 ", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+
+	/* Task 2 executes func2 - must NOT inherit Task 1's inlined call stack */
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  func2() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  } /* func2 */", out);
+
+	/* Task 1 migrates and resumes on CPU 1 */
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)    <idle>-0    =>    task1-100 ", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+
+	/* Task 1 exits enqueue_task, then wake_up_new_task on CPU 1 */
+	ftrace_process_fgraph_line(&ftrace, " 1)   1.000 us    |    } /* enqueue_task */", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)   2.000 us    |  } /* wake_up_new_task */", out);
+
+	/*
+	 * Test delayed context switch banner migration where a task state
+	 * with an inlined frame on CPU 2 is rekeyed from CPU 2 to CPU 3.
+	 */
+	ftrace_process_fgraph_line(&ftrace, " 2)               |  wake_up_new_task() {", out);
+	ftrace_process_fgraph_line(&ftrace,
+				   " 2)               |    enqueue_task() { /* <-wake_up_new_task+0x1d1/0x3e0 */",
+				   out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)    <idle>-0    =>    task3-300 ", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 2)    task3-300   =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)               |      func4() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)               |      } /* func4 */", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)               |    } /* enqueue_task */", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)               |  } /* wake_up_new_task */", out);
+
+	/*
+	 * Test delayed context switch banner migration where CPU 3 enters func6()
+	 * BEFORE CPU 2 emits its delayed context switch banner. ts from CPU 2
+	 * must be merged underneath existing on CPU 3.
+	 */
+	ftrace_process_fgraph_line(&ftrace, " 2)               |  func5() {", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)    <idle>-0    =>    task4-400 ", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)               |    func6() {", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 2)    task4-400   =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)               |    } /* func6 */", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)               |  } /* func5 */", out);
+
+	/*
+	 * Test single-task profiling across CPU migration where context switch
+	 * banners are suppressed by the kernel (verif_pid prev_pid == -1).
+	 */
+	ftrace.evlist = evlist__new();
+	if (ftrace.evlist) {
+		threads = thread_map__new_by_tid(500);
+		perf_evlist__set_maps(evlist__core(ftrace.evlist), NULL, threads);
+		perf_thread_map__put(threads);
+	}
+
+	ftrace_cpu_states__clear();
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  func_a() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)               |    func_b() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)               |    } /* func_b */", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  } /* func_a */", out);
+
+	evlist__put(ftrace.evlist);
+	ftrace.evlist = NULL;
+
+	/*
+	 * Test migration where task with active outer frame on CPU 0 migrates
+	 * to CPU 1 without an initial context switch banner, accumulating inner
+	 * frames in ts under FTRACE_UNKNOWN_PID(1). Rekeying must append inner
+	 * frames to existing stack rather than inverting the stack.
+	 */
+	ftrace_cpu_states__clear();
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  caller_outer() {", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)    task7-700   =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	/* Migrates to CPU 1 without banner; inner frame called */
+	ftrace_process_fgraph_line(&ftrace, " 1)               |    callee_inner() {", out);
+	/* Banner on CPU 1 rekeys CPU 1 to task7 */
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)    task7-700   =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	/* Resumes on CPU 0: callee_inner exits, followed by caller_outer */
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)    <idle>-0    =>    task7-700 ", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)               |    } /* callee_inner */", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  } /* caller_outer */", out);
+
+	/*
+	 * Test task migration where older stack has scheduler frames that already
+	 * exited before the task entered a callee on the new CPU.
+	 * CPU 0 has caller_outer (indent 2) -> schedule (indent 4) -> __schedule (indent 6).
+	 * Resumes on CPU 1 where schedule frames returned and callee_worker (indent 4) runs.
+	 * Rekeying must prune schedule and __schedule from older stack rather than
+	 * leaving stranded frames between caller_outer and callee_worker.
+	 */
+	ftrace_cpu_states__clear();
+	ftrace_process_fgraph_line(&ftrace, " 0) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)    <idle>-0    =>    task8-800 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 0) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  caller_outer() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)               |    schedule() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)               |      __schedule() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)               |    callee_worker() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 0) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)    task8-800   =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 0) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)    task8-800   =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 1) ------------------------------------------", out);
+	/* Resumes on CPU 0: bare braces pop callee_worker then caller_outer */
+	ftrace_process_fgraph_line(&ftrace, " 0) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)    <idle>-0    =>    task8-800 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 0) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)   1.000 us    |    }", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)   2.000 us    |  }", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  base_verify() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)   1.000 us    |  }", out);
+
+	/*
+	 * Test migration where task unwinds to depth 0 on CPU 1 before CPU 0
+	 * emits its delayed context switch banner. Rekeying must prune the
+	 * already-exited frames from the older stack rather than restoring them.
+	 */
+	ftrace_cpu_states__clear();
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  caller_deep() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)               |    callee_deep() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 1) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)    <idle>-0    =>    task9-900 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 1) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)   1.000 us    |    } /* callee_deep */", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)   2.000 us    |  } /* caller_deep */", out);
+	ftrace_process_fgraph_line(&ftrace, " 0) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)    task9-900   =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 0) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)               |  fresh_root() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)   1.000 us    |  } /* fresh_root */", out);
+
+	/*
+	 * Test migration where older stack has an inlined frame at the top
+	 * when rekeying. Pruning must not stall on raw_indent == -1.
+	 */
+	ftrace_cpu_states__clear();
+	ftrace_process_fgraph_line(&ftrace, " 2)               |  wake_up_new_task() {", out);
+	ftrace_process_fgraph_line(&ftrace,
+				   " 2)               |    enqueue_task() { /* <-wake_up_new_task+0x1d1/0x3e0 */",
+				   out);
+	ftrace_process_fgraph_line(&ftrace, " 3) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)    <idle>-0    =>    task10-1000 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 3) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)   1.000 us    |    } /* enqueue_task */", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)               |    other_work() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 2) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 2)    task10-1000 =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 2) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)   1.000 us    |    } /* other_work */", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)   2.000 us    |  } /* wake_up_new_task */", out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	/* Verify synthetic inlined entry was emitted for Task 1 */
+	TEST_CS_ASSERT("task1 emitted inlined activate_task entry",
+			strstr(out_buf, "activate_task() { /* (inline) */") != NULL);
+
+	/* Verify default_idle in Idle task started at base indentation without extra indent */
+	TEST_CS_ASSERT("idle task starts at base indentation",
+			strstr(out_buf, "|  default_idle() {") != NULL);
+	TEST_CS_ASSERT("idle task does not inherit extra inlined indent",
+			strstr(out_buf, "|    default_idle() {") == NULL);
+
+	/* Verify func2 in Task 2 started at base indentation "|  func2() {" without extra indent */
+	TEST_CS_ASSERT("task2 starts at base indentation",
+			strstr(out_buf, "|  func2() {") != NULL);
+	TEST_CS_ASSERT("task2 does not inherit extra inlined indent",
+			strstr(out_buf, "|    func2() {") == NULL);
+
+	/* Verify task1 synthesized inlined exit on CPU 1 */
+	TEST_CS_ASSERT("task1 emitted synthesized inlined exit on CPU 1",
+			strstr(out_buf,
+			       "1)               |    } /* activate_task (inline) */") != NULL);
+
+	/* Verify task3 preserved inlined stack across delayed context switch banner */
+	TEST_CS_ASSERT("task3 preserved inlined frame across delayed banner",
+			strstr(out_buf,
+			       "3)               |    } /* activate_task (inline) */") != NULL);
+
+	/* Verify task4 merged outer stack when entered before delayed banner */
+	TEST_CS_ASSERT("task4 merged outer stack when entered before delayed banner",
+			strstr(out_buf, "3)               |  } /* func5 */") != NULL);
+
+	/* Verify single-task stack tracked across CPUs without banners */
+	TEST_CS_ASSERT("single-task stack tracked across CPUs without banners",
+			strstr(out_buf, "0)               |  } /* func_a */") != NULL);
+
+	/* Verify migrated task with unbannered entry appended rather than inverted stack */
+	TEST_CS_ASSERT("migrated task unwound inner callee cleanly",
+			strstr(out_buf, "0)               |    } /* callee_inner */") != NULL);
+	TEST_CS_ASSERT("migrated task unwound outer caller cleanly",
+			strstr(out_buf, "0)               |  } /* caller_outer */") != NULL);
+
+	/* Verify pruned scheduler frames allowed clean bare brace unwinding */
+	TEST_CS_ASSERT("pruned scheduler frames allowed clean bare brace unwinding",
+			strstr(out_buf, "|  base_verify() {") != NULL);
+
+	/* Verify unwound migrated task starts at base indent */
+	TEST_CS_ASSERT("unwound migrated task starts at base indent",
+		       strstr(out_buf, " 1)               |  fresh_root() {") != NULL);
+
+	/* Verify inlined frame pruned without stalling on migration */
+	TEST_CS_ASSERT("inlined frame pruned without stalling on migration",
+		       strstr(out_buf, " 3)   1.000 us    |    } /* other_work */") != NULL &&
+		       strstr(out_buf, " 3)   2.000 us    |  } /* wake_up_new_task */") != NULL);
+
+	/*
+	 * Test migration where merged stack depth exceeds initial FTRACE_MAX_STACK (256)
+	 * triggers dynamic stack growth and unwinds without indentation drift or corruption.
+	 * CPU 2 starts without a context switch banner, accumulating under
+	 * FTRACE_UNKNOWN_PID(2).
+	 */
+	free(out_buf);
+	out_buf = NULL;
+	out = open_memstream(&out_buf, &out_len);
+	if (!out) {
+		ret = TEST_FAIL;
+		goto out_cleanup;
+	}
+
+	for (i = 0; i < 200; i++)
+		feed_fgraph_entry(&ftrace, 2, i, "deep_func", NULL, out);
+
+	/* Migrate to CPU 3, enter more frames */
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 3)    <idle>-0    =>    task5-500 ", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	for (i = 0; i < 100; i++)
+		feed_fgraph_entry(&ftrace, 3, 200 + i, "more_func", NULL, out);
+
+	/* Delayed context switch banner from CPU 2 rekeys and merges stacks (> 256) */
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 2)    task5-500   =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " ------------------------------------------", out);
+
+	/* Unwind all 300 frames on CPU 3 on dynamically resized stack */
+	for (i = 299; i >= 0; i--)
+		feed_fgraph_exit(&ftrace, 3, i, NULL, out);
+	feed_fgraph_entry(&ftrace, 3, 0, "after_merge", NULL, out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	TEST_CS_ASSERT("no indentation drift after saturated stack rekey",
+			strstr(out_buf, "|  after_merge() {") != NULL);
+
+	/*
+	 * Test migration where task exits to depth 0 on CPU 1 (setting min_raw_indent = 2),
+	 * migrates to CPU 0 entering outer frames wake_up_new_task (indent 2) -> enqueue_task (indent 4)
+	 * with activate_task inlined, migrates back to CPU 1 entering inner callee_c (indent 6).
+	 * Delayed rekeying of CPU 0 must not over-prune caller frames using
+	 * CPU 1's previous min_raw_indent.
+	 */
+	free(out_buf);
+	out_buf = NULL;
+	out = open_memstream(&out_buf, &out_len);
+	if (!out) {
+		ret = TEST_FAIL;
+		goto out_cleanup;
+	}
+
+	ftrace_cpu_states__clear();
+	ftrace_process_fgraph_line(&ftrace, " 1) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)    <idle>-0    =>    task11-1100 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 1) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)               |  root_prior() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)   1.000 us    |  } /* root_prior */", out);
+	/* Task 11 migrates to CPU 0 without banner, entering active caller frames with inline */
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  wake_up_new_task() {", out);
+	ftrace_process_fgraph_line(&ftrace,
+				   " 0)               |    enqueue_task() { /* <-wake_up_new_task+0x1d1/0x3e0 */",
+				   out);
+	/* Task 11 migrates back to CPU 1 and enters inner frame */
+	ftrace_process_fgraph_line(&ftrace, " 1) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)    <idle>-0    =>    task11-1100 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 1) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)               |      callee_c() {", out);
+	/* Delayed banner on CPU 0 triggers rekey */
+	ftrace_process_fgraph_line(&ftrace, " 0) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)    task11-1100 =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 0) ------------------------------------------", out);
+	/* Resumes on CPU 1: callee_c, enqueue_task (synthesizing activate_task exit), wake_up_new_task */
+	ftrace_process_fgraph_line(&ftrace, " 1)   1.000 us    |      } /* callee_c */", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)   2.000 us    |    } /* enqueue_task */", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)   3.000 us    |  } /* wake_up_new_task */", out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	/* Verify active caller frames preserved despite prior exit min_raw_indent */
+	TEST_CS_ASSERT("task11 emitted synthesized inlined exit on CPU 1",
+		       strstr(out_buf,
+			      " 1)               |    } /* activate_task (inline) */") != NULL);
+
+	/*
+	 * Test multi-stage rekeying where CPU 1 unwinds frames to depth 0 first
+	 * (recording min_raw_indent = 2) and merges into an existing state,
+	 * followed by CPU 0 rekeying its older UNKNOWN_PID state.
+	 * The merge into existing must preserve min_raw_indent so CPU 0's older
+	 * exited frames are properly pruned rather than resurrected.
+	 */
+	free(out_buf);
+	out_buf = NULL;
+	out = open_memstream(&out_buf, &out_len);
+	if (!out) {
+		ret = TEST_FAIL;
+		goto out_cleanup;
+	}
+
+	ftrace_cpu_states__clear();
+	/* CPU 0: Task 12 enters frames under UNKNOWN_PID(0) with inline */
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  wake_up_new_task() {", out);
+	ftrace_process_fgraph_line(&ftrace,
+				   " 0)               |    enqueue_task() { /* <-wake_up_new_task+0x1d1/0x3e0 */",
+				   out);
+	/* CPU 1: Task 12 unwinds both frames to depth 0 under UNKNOWN_PID(1) */
+	ftrace_process_fgraph_line(&ftrace, " 1)   1.000 us    |    } /* enqueue_task */", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)   2.000 us    |  } /* wake_up_new_task */", out);
+	/* CPU 2: pre-bind an empty state for task 12 */
+	ftrace_process_fgraph_line(&ftrace, " 2) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 2)    <idle>-0    =>    task12-1200 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 2)    task12-1200 =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 2) ------------------------------------------", out);
+	/* CPU 1 emits banner first: merges ts1 (depth 0, min_raw_indent 2) into existing */
+	ftrace_process_fgraph_line(&ftrace, " 1) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)    task12-1200 =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 1) ------------------------------------------", out);
+	/* CPU 0 emits banner second: merges ts0 into existing, must prune exited frames */
+	ftrace_process_fgraph_line(&ftrace, " 0) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)    task12-1200 =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 0) ------------------------------------------", out);
+	/* Task 12 resumes on CPU 1 and runs fresh function: bare braces must not pop stale frames */
+	ftrace_process_fgraph_line(&ftrace, " 1) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)    <idle>-0    =>    task12-1200 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 1) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)               |  fresh_task12() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)   1.000 us    |  }", out);
+
+	fclose(out);
+	ftrace_cpu_states__clear();
+
+	/* Verify multi-stage rekey preserved min_raw_indent to prune stale exited frames */
+	TEST_CS_ASSERT("fresh_task12 starts at base indent without resurrected frames",
+		       strstr(out_buf, " 1)               |  fresh_task12() {") != NULL);
+	TEST_CS_ASSERT("stale inlined frame not resurrected before fresh_task12",
+		       strstr(out_buf, "} /* activate_task (inline) */\n 1)               |  fresh_task12() {") == NULL);
+
+	/*
+	 * Test migration where task exits bar() (indent 4) and enters qux() (indent 4)
+	 * on unbannered CPU 1, then migrates back to CPU 0 entering child baz() (indent 6)
+	 * before CPU 1 emits its context switch banner.
+	 * Rekeying of CPU 1 must prune exited bar(), insert qux() between foo()
+	 * and baz(), and cleanly unwind baz, qux, and foo on CPU 0.
+	 */
+	free(out_buf);
+	out_buf = NULL;
+	out = open_memstream(&out_buf, &out_len);
+	if (!out) {
+		ret = TEST_FAIL;
+		goto out_cleanup;
+	}
+
+	ftrace_cpu_states__clear();
+	/* CPU 0: Task 13 enters root (indent 2) -> foo (indent 4) */
+	ftrace_process_fgraph_line(&ftrace, " 0) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)    <idle>-0    =>    task13-1300 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 0) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)               |  root() {", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)               |    foo() {", out);
+	/*
+	 * Task 13 migrates to CPU 1 without banner: exits foo,
+	 * enters wake_up_new_task -> enqueue_task.
+	 */
+	ftrace_process_fgraph_line(&ftrace, " 1)   1.000 us    |    } /* foo */", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)               |    wake_up_new_task() {", out);
+	ftrace_process_fgraph_line(&ftrace,
+				   " 1)               |      enqueue_task() { /* <-wake_up_new_task+0x1d1/0x3e0 */",
+				   out);
+	/* Task 13 migrates back to CPU 0 and enters child baz */
+	ftrace_process_fgraph_line(&ftrace, " 0)               |        baz() {", out);
+	/* Delayed banner on CPU 1 triggers rekey */
+	ftrace_process_fgraph_line(&ftrace, " 1) ------------------------------------------", out);
+	ftrace_process_fgraph_line(&ftrace, " 1)    task13-1300 =>    <idle>-0 ", out);
+	ftrace_process_fgraph_line(&ftrace, " 1) ------------------------------------------", out);
+	/*
+	 * Task 13 unwinds baz, enqueue_task (synthesizing activate_task),
+	 * wake_up_new_task, root on CPU 0.
+	 */
+	ftrace_process_fgraph_line(&ftrace, " 0)   1.000 us    |        } /* baz */", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)   2.000 us    |      } /* enqueue_task */", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)   3.000 us    |    } /* wake_up_new_task */", out);
+	ftrace_process_fgraph_line(&ftrace, " 0)   4.000 us    |  } /* root */", out);
+
+	fclose(out);
+	pr_debug("OUT_BUF:\n%s\n", out_buf);
+	ftrace_cpu_states__clear();
+
+	TEST_CS_ASSERT("task13 synthesized inlined exit for enqueue_task",
+		       strstr(out_buf, "} /* activate_task (inline) */") != NULL);
+
+out_cleanup:
+	if (out_buf)
+		free(out_buf);
+	perf_ftrace__cleanup_inlines(&ftrace);
+out_cleanup_conf:
+	symbol_conf.vmlinux_name = saved_vmlinux;
+	symbol_conf.ignore_vmlinux_buildid = saved_ignore_buildid;
+	return ret;
+}
+#undef TEST_CS_ASSERT
+
+static int test__parse_retaddr(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
+{
+	return test_parse_retaddr();
+}
+
+static int test__filter_retaddr(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
+{
+	return test_filter_retaddr();
+}
+
+static int test__column_layouts(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
+{
+	return test_column_layouts();
+}
+
+static int test__stack_saturation(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
+{
+	return test_stack_saturation();
+}
+
+static int test__context_switch_handling(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
+{
+	return test_context_switch_handling();
+}
+
+static int test__inline_processing(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
+{
+	return test_inline_processing();
+}
+
+static struct test_case ftrace_test_cases[] = {
+	TEST_CASE("Parse return address comments", parse_retaddr),
+	TEST_CASE("Filter return address comments", filter_retaddr),
+	TEST_CASE("Function graph column layouts", column_layouts),
+	TEST_CASE("Stack saturation and expansion", stack_saturation),
+	TEST_CASE("Context switch handling", context_switch_handling),
+	TEST_CASE("Inline symbol and trace processing", inline_processing),
+	{ .name = NULL, }
+};
+
+struct test_suite suite__ftrace = {
+	.desc = "Ftrace inline processing",
+	.test_cases = ftrace_test_cases,
+};
Signature · 1 line
2.55.0
#33 lore raw

Re: [PATCH perf-tools-next v12 08/10] perf test: Add test for ftrace inline and task stack tracking

Sashiko has reviewed this patch and found no issues.
Quoted 14 lines

Add an automated unit test suite ("Ftrace inline processing") in tools/perf/tests/ftrace.c to validate the function graph call stack tracking and inlined function reconstruction implementation.

The test suite verifies:

1. DWARF inline symbol resolution

Uses vmlinux_path__find() to locate the kernel image, setting up symbol resolution and validating activate_task inlining within wake_up_new_task at offset 0x1d1.

2. Inline presentation and return address filtering

Quoted 7 lines

Verifies that synthesised inlined entries and exits are
[ ... ]
FTRACE_MAX_STACK capacity.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
#34 lore raw

PATCH v12 9/10 perf ftrace: Support omitting execution duration in function graph tracer

In reply to Aaron Tomlin

The Linux kernel's function graph tracer displays the execution duration of completed functions by default via the funcgraph-duration trace option. Whilst informative, in scenarios where the user is solely interested in the call graph topology, call flow sequence, or inlined hierarchy, the duration column can add visual clutter and widen each trace line.

Introduce support for omitting function duration by exposing the noduration sub-option under --graph-opts. Consistent with the existing nosleep-time and noirqs sub-options, it accepts an optional value, so noduration=0 explicitly retains the duration column; when specified more than once, the last occurrence takes effect. When enabled, perf ftrace configures the kernel's tracing instance option funcgraph-duration to 0, and restores it to 1 upon resetting tracing options.

Because the profile subcommand fundamentally requires function execution duration to compute timing statistics and call counts, combining noduration with perf ftrace profile is rejected with an explicit error.

---
 tools/perf/Documentation/perf-ftrace.txt |   1 +
 tools/perf/builtin-ftrace.c              |  34 +++++++-
 tools/perf/tests/ftrace.c                | 104 ++++++++++++++++++++++-
 tools/perf/util/ftrace.c                 |  25 ++++--
 tools/perf/util/ftrace.h                 |   1 +
 5 files changed, 154 insertions(+), 11 deletions(-)
tools/perf/Documentation/perf-ftrace.txt +1 -0
index 0af6b0c1f134..ed2e45b8507f 100644
@@ -129,6 +129,7 @@ OPTIONS for 'perf ftrace trace'
 	  - retaddr      - Show function return address.
 	  - nosleep-time - Measure on-CPU time only for function_graph tracer.
 	  - noirqs       - Ignore functions that happen inside interrupt.
+	  - noduration   - Do not print function duration.
 	  - verbose      - Show process names, PIDs, timestamps, etc.
 	  - thresh=<n>   - Setup trace duration threshold in microseconds.
 	  - depth=<n>    - Set max depth for function graph tracer to follow.
tools/perf/builtin-ftrace.c +33 -1
index 1f652d1e2754..db9616ad68af 100644
@@ -300,6 +300,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("func_stack_trace", "0");
 	write_tracing_option_file("sleep-time", "1");
 	write_tracing_option_file("funcgraph-irqs", "1");
+	write_tracing_option_file("funcgraph-duration", "1");
 	write_tracing_option_file("funcgraph-proc", "0");
 	write_tracing_option_file("funcgraph-abstime", "0");
 	write_tracing_option_file("funcgraph-tail", "0");
@@ -599,6 +600,26 @@ static int set_tracing_funcgraph_irqs(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+/**
+ * set_tracing_funcgraph_duration - Set funcgraph-duration tracing option
+ * @ftrace: Perf ftrace context
+ *
+ * Configure the kernel's funcgraph-duration tracing option based on the
+ * ftrace->graph_noduration flag.
+ *
+ * Return: 0 on success, or a negative error code on failure.
+ */
+static int set_tracing_funcgraph_duration(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->graph_noduration)
+		return 0;
+
+	if (write_tracing_option_file("funcgraph-duration", "0") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int set_tracing_funcgraph_verbose(struct perf_ftrace *ftrace)
 {
 	if (!ftrace->graph_verbose)
@@ -708,6 +729,11 @@ static int set_tracing_options(struct perf_ftrace *ftrace)
 		return -1;
 	}
 
+	if (set_tracing_funcgraph_duration(ftrace) < 0) {
+		pr_err("failed to set tracing option funcgraph-duration\n");
+		return -1;
+	}
+
 	if (set_tracing_funcgraph_verbose(ftrace) < 0) {
 		pr_err("failed to set tracing option funcgraph-proc/funcgraph-abstime\n");
 		return -1;
@@ -1785,6 +1811,7 @@ static int parse_graph_tracer_opts(const struct option *opt,
 		{ .name = "retaddr",		.value_ptr = &ftrace->graph_retaddr },
 		{ .name = "nosleep-time",	.value_ptr = &ftrace->graph_nosleep_time },
 		{ .name = "noirqs",		.value_ptr = &ftrace->graph_noirqs },
+		{ .name = "noduration",		.value_ptr = &ftrace->graph_noduration },
 		{ .name = "verbose",		.value_ptr = &ftrace->graph_verbose },
 		{ .name = "thresh",		.value_ptr = &ftrace->graph_thresh },
 		{ .name = "depth",		.value_ptr = &ftrace->graph_depth },
@@ -1874,7 +1901,7 @@ int cmd_ftrace(int argc, const char **argv)
 	OPT_CALLBACK('g', "nograph-funcs", &ftrace.nograph_funcs, "func",
 		     "Set nograph filter on given functions", parse_filter_func),
 	OPT_CALLBACK(0, "graph-opts", &ftrace, "options",
-		     "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,verbose,thresh=<n>,depth=<n>",
+		     "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,noduration,verbose,thresh=<n>,depth=<n>,tail",
 		     parse_graph_tracer_opts),
 	OPT_CALLBACK('m', "buffer-size", &ftrace.percpu_buffer_size, "size",
 		     "Size of per cpu buffer, needs to use a B, K, M or G suffix.", parse_buffer_size),
@@ -2073,6 +2100,11 @@ int cmd_ftrace(int argc, const char **argv)
 		cmd_func = __cmd_latency;
 		break;
 	case PERF_FTRACE_PROFILE:
+		if (ftrace.graph_noduration) {
+			pr_err("Error: 'noduration' option cannot be used with profile\n");
+			ret = -EINVAL;
+			goto out_delete_filters;
+		}
 		cmd_func = __cmd_profile;
 		break;
 	case PERF_FTRACE_NONE:
tools/perf/tests/ftrace.c +101 -3

145 lines not on this page · raw message

tools/perf/util/ftrace.c +18 -7
index 2807846d7449..ac1fdf922c7d 100644
@@ -802,6 +802,9 @@ static void make_blank_prefix(const struct perf_ftrace *ftrace, const char *line
 	memcpy(buf, line, len);
 	buf[len] = '\0';
 
+	if (ftrace->graph_noduration)
+		return;
+
 	if (*pipe_char == '|') {
 		if (ftrace->graph_verbose) {
 			/* In verbose mode, duration starts after the comm-pid '|' */
@@ -840,10 +843,12 @@ static void make_blank_prefix(const struct perf_ftrace *ftrace, const char *line
  *   [ abstime |]  CPU)  [ comm-pid |]  [ duration |]  func
  *
  * The abstime and comm-pid columns are only present with --graph-opts
- * verbose.
+ * verbose, whereas the duration column, and thus its '|' delimiter, is
+ * omitted with --graph-opts noduration.
  *
  * Return: A pointer to the last character of the context columns (the final
- * '|' delimiter). Returns NULL if @line is not a trace record.
+ * '|' delimiter, or the CPU ')' when no delimiter follows it). Returns NULL
+ * if @line is not a trace record (e.g. a header or a context switch marker).
  */
 static const char *find_func_column(const struct perf_ftrace *ftrace,
 				    const char *line, int *cpu)
@@ -897,9 +902,14 @@ static const char *find_func_column(const struct perf_ftrace *ftrace,
 			return NULL;
 	}
 
-	p = strchr(p + 1, '|');
-	if (!p)
+	if (!ftrace->graph_noduration) {
+		p = strchr(p + 1, '|');
+		if (!p)
+			return NULL;
+	} else if (strstr(p, " => ") || strstr(p, "=====>") || strstr(p, "<=====")) {
+		/* Context switch or interrupt marker */
 		return NULL;
+	}
 
 	*cpu = (int)val;
 	return p;
@@ -995,11 +1005,12 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
 	func_start = skip_spaces(after_pipe);
 	leading_spaces = func_start - after_pipe;
 
-	if (*func_start == '\0')
-		return print_line(ftrace, line, out);
-
 	is_exit = (*func_start == '}');
 
+	if (*func_start == '\0' ||
+	    (!isalpha(*func_start) && *func_start != '_' && !is_exit))
+		return print_line(ftrace, line, out);
+
 	if (is_exit) {
 		char exit_name[128];
 
tools/perf/util/ftrace.h +1 -0
index 7b9e0919ae8d..0afaa9695691 100644
@@ -39,6 +39,7 @@ struct perf_ftrace {
 	int			graph_retaddr;
 	int			graph_nosleep_time;
 	int			graph_noirqs;
+	int			graph_noduration;
 	int			graph_verbose;
 	int			graph_thresh;
 	int			graph_tail;
Signature · 1 line
2.55.0
#35 lore raw

Re: [PATCH perf-tools-next v12 09/10] perf ftrace: Support omitting execution duration in function graph tracer

Sashiko has reviewed this patch and found no issues.
Quoted 16 lines

The Linux kernel's function graph tracer displays the execution duration of completed functions by default via the funcgraph-duration trace option. Whilst informative, in scenarios where the user is solely interested in the call graph topology, call flow sequence, or inlined hierarchy, the duration column can add visual clutter and widen each trace line.

Introduce support for omitting function duration by exposing the noduration sub-option under --graph-opts. Consistent with the existing nosleep-time and noirqs sub-options, it accepts an optional value, so noduration=0 explicitly retains the duration column; when specified more than once, the last occurrence takes effect. When enabled, perf ftrace configures the kernel's tracing instance option funcgraph-duration to 0, and restores it to 1 upon resetting tracing options.

Quoted 7 lines

Because the profile subcommand fundamentally requires function execution duration to compute timing statistics and call counts, combining noduration with perf ftrace profile is rejected with an explicit error.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
#36 lore raw

PATCH v12 10/10 perf ftrace: Filter extraneous functions in function graph tracer

In reply to Aaron Tomlin

When tracing with the function graph tracer using a targeted filter (-G or --graph-funcs), users occasionally observe unrelated functions in the trace output:

1. Top-level functions (e.g. most notably finish_task_switch.isra.0()) appearing before or alongside the requested target function. This occurs because clearing and rewriting set_graph_function creates a transient window where remote CPUs still observe an empty filter (tracing all functions) before RCU synchronisation completes, coupled with perf ftrace sleeping in poll() on trace_pipe right as tracing starts.

2. Detached/orphan calls inside the target callgraph (e.g. housekeeping_any_cpu() and arch_irq_work_raise()) appearing as child frames under the target function. These routines are invoked by rb_wakeups() to notify perf's poll() on trace_pipe whenever an event is committed, and execute while the target function is on the CPU.

Resolve these issues deterministically in userspace:

1. Target callgraph state tracking

Track active target callgraph state per-CPU in tools/perf/util/ftrace.c. Any entry not matching the requested -G targets is suppressed along with its nested children until its scope exits. When a target function is matched (at root or nested depths), record its root depth; for leaf target calls, immediately reset the filter state upon completion to ensure subsequent sibling functions remain suppressed. Support glob wildcard matching (-G 'prefix_*' and compiler clone suffixes).

2. Caller provenance and orphan suppression

Within an active target callgraph, validate that child entries descend from an active frame on the target stack using their return addresses (CONFIG_FUNCTION_GRAPH_RETADDR). Any detached calls originating outside the target hierarchy (i.e. ring-buffer wakeups or asynchronous interrupts) are suppressed along with their subtrees (cs->orphan_depth).

Exits within an orphan subtree are suppressed without querying the target stack, preventing common helpers
(i.e. _raw_spin_unlock) from colliding with or prematurely unspooling active target frames. Dropped-orphan recovery safely resets orphan depth whenever an entry or exit occurs at or outside the target root indentation.

3. Dynamic call stack allocation

Maintain the synthetic call stack as a dynamically reallocated array, growing on demand via realloc(). This supports arbitrarily deep call trees without fixed-size truncation, ensures unknown callers are unambiguously identified as orphans, and guarantees that standard filter teardown executes cleanly regardless of stack depth upon target function entry.

4. Automated verification

Add comprehensive regression tests in tools/perf/tests/ftrace.c (test_graph_target_filter()) validating top-level suppression, detached orphan suppression, glob matching, deep stack traversal, dropped-orphan recovery, and orphan exit collision prevention.

---
 tools/perf/builtin-ftrace.c |   42 +-
 tools/perf/tests/ftrace.c   | 2462 +++++++++++++++++++++++++++++++++++
 tools/perf/util/ftrace.c    |  838 ++++++++++--
 tools/perf/util/ftrace.h    |    8 +
 4 files changed, 3246 insertions(+), 104 deletions(-)
tools/perf/builtin-ftrace.c +30 -12
index db9616ad68af..1302c9cbfcd6 100644
@@ -576,13 +576,19 @@ static int set_tracing_funcgraph_retval(struct perf_ftrace *ftrace)
 
 static int set_tracing_funcgraph_retaddr(struct perf_ftrace *ftrace)
 {
-	if (ftrace->graph_retaddr || ftrace->use_inline) {
+	bool need_retaddr = ftrace->graph_retaddr || ftrace->use_inline ||
+			    (ftrace->subcmd == PERF_FTRACE_TRACE &&
+			     !list_empty(&ftrace->graph_funcs));
+
+	if (need_retaddr) {
 		if (write_tracing_option_file("funcgraph-retaddr", "1") < 0) {
 			if (ftrace->use_inline) {
 				pr_err("failed to set funcgraph-retaddr option in tracing\n");
 				pr_err("Hint: ensure kernel is compiled with CONFIG_FUNCTION_GRAPH_RETADDR=y\n");
+				return -1;
 			}
-			return -1;
+			if (ftrace->graph_retaddr)
+				return -1;
 		}
 	}
 
@@ -1381,6 +1387,8 @@ static int parse_func_duration(struct perf_ftrace *ftrace, char *line, size_t le
 {
 	char *p;
 	char *func;
+	char *semi;
+	char *ret_str;
 	double duration;
 
 	/* skip CPU */
@@ -1423,12 +1431,21 @@ static int parse_func_duration(struct perf_ftrace *ftrace, char *line, size_t le
 	func = skip_spaces(p);
 
 	/* skip the closing bracket and the start of comment */
-	if (*func == '}')
+	if (*func == '}') {
 		func += 5;
+		ret_str = strstr(func, " ret=");
+		if (ret_str)
+			*ret_str = '\0';
+		p = func + strlen(func) - 1;
+	} else {
+		semi = strchr(func, ';');
+		if (semi)
+			*semi = '\0';
+		p = func + strlen(func) - 1;
+	}
 
-	/* remove semi-colon or end of comment at the end */
-	p = line + len - 1;
-	while (!isalnum(*p) && *p != ']') {
+	/* remove parens, semi-colon or end of comment at the end */
+	while (p > func && !isalnum(*p) && *p != ']') {
 		*p = '\0';
 		--p;
 	}
@@ -1853,12 +1870,6 @@ static int parse_sort_key(const struct option *opt, const char *str, int unset)
 	return 0;
 }
 
-enum perf_ftrace_subcommand {
-	PERF_FTRACE_NONE,
-	PERF_FTRACE_TRACE,
-	PERF_FTRACE_LATENCY,
-	PERF_FTRACE_PROFILE,
-};
 
 int cmd_ftrace(int argc, const char **argv)
 {
@@ -2013,6 +2024,8 @@ int cmd_ftrace(int argc, const char **argv)
 	if (subcmd == PERF_FTRACE_NONE)
 		subcmd = PERF_FTRACE_TRACE;
 
+	ftrace.subcmd = subcmd;
+
 	argc = parse_options(argc, argv, options, ftrace_usage,
 			    PARSE_OPT_STOP_AT_NON_OPTION);
 	if (argc < 0) {
@@ -2034,6 +2047,11 @@ int cmd_ftrace(int argc, const char **argv)
 				ret = -EINVAL;
 				goto out_delete_filters;
 			}
+			if (ftrace.graph_thresh) {
+				pr_err("Error: --inline cannot be combined with --graph-opts thresh\n");
+				ret = -EINVAL;
+				goto out_delete_filters;
+			}
 		}
 		cmd_func = __cmd_ftrace;
 		break;
tools/perf/tests/ftrace.c +2462 -0

2,480 lines not on this page · raw message

tools/perf/util/ftrace.c +746 -92

435 lines not on this page · raw message

tools/perf/util/ftrace.h +8 -0
index 0afaa9695691..1dc808b8a7df 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.55.0

Part of the diff is not shown here. View the raw message.

#37 lore raw

Re: [PATCH perf-tools-next v12 10/10] perf ftrace: Filter extraneous functions in function graph tracer

Sashiko has reviewed this patch and found no issues.
Quoted 13 lines

When tracing with the function graph tracer using a targeted filter (-G or --graph-funcs), users occasionally observe unrelated functions in the trace output:

1. Top-level functions (e.g. most notably finish_task_switch.isra.0()) appearing before or alongside the requested target function. This occurs because clearing and rewriting set_graph_function creates a transient window where remote CPUs still observe an empty filter (tracing all functions) before RCU synchronisation completes, coupled with perf ftrace sleeping in poll() on trace_pipe right as tracing starts.

Quoted 8 lines

2. Detached/orphan calls inside the target callgraph (e.g. housekeeping_any_cpu() and arch_irq_work_raise())
[ ... ]
prevention.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10