Skip to contentSkip to conversation

perf trace: Symbolise kernel virtual addresses and function pointers

PATCH In review Aaron Tomlin 1 participant 5 messages Aug 3 perf events

lore
Share & tools
lore
#0 lore raw

cover perf trace: Symbolise kernel virtual addresses and function pointers

When inspecting kernel execution flows using 'perf trace' (e.g., when monitoring workqueues delayed work items, timer callbacks, etc.), tracepoint payload arguments containing raw kernel virtual addresses are currently rendered as hexadecimal values (e.g., 0xffffffff81234567).

This requires manual symbol lookups against /proc/kallsyms or vmlinux to identify the underlying kernel function being executed.

This patch series enhances 'perf trace' by introducing kernel virtual address and function pointer symbolisation using perf's native symbol engine (i.e., machine__find_kernel_symbol()).

Before:

    workqueue:workqueue_execute_end(work: 0xffffffffab2f1420, function: 0xffffffffa8046b50)

After:

    workqueue:workqueue_execute_end(work: 0xffff8ac2c420f270, function: wb_update_bandwidth_workfn)

Patch 1 adds the 'syscall_arg__scnprintf_ksym' ('SCA_KSYM') beautifier which resolves virtual addresses via machine__find_kernel_symbol(), formatting them as 'symbol_name+offset' (or "NULL" with a hex fallback).

Patch 2 updates event format initialisation in syscall_arg_fmt__init_array() to automatically assign SCA_KSYM to tracepoint fields named 'function', 'fn', 'work', 'action', or 'callsite', as well as fields typed as function pointers.

Patch 3 extends BTF pretty-printing in trace__btf_scnprintf() with btf_is_func_ptr() to automatically detect BTF function prototypes and symbolise kernel function pointers without requiring manual field table configuration.

Aaron Tomlin (3):
perf trace: Introduce kernel symbol beautifier for virtual addresses perf trace: Auto-assign kernel symbol beautifier to function pointer
fields
perf trace: Enhance BTF type formatting to symbolise kernel function
pointers

 tools/perf/builtin-trace.c       | 76 ++++++++++++++++++++++++++++----
 tools/perf/trace/beauty/beauty.h |  3 ++
 2 files changed, 71 insertions(+), 8 deletions(-)
Signature · 1 line
2.55.0
#1 lore raw

PATCH 1/3 perf trace: Introduce kernel symbol beautifier for virtual addresses

Currently, when 'perf trace' formats tracepoint payloads or system call arguments containing raw kernel virtual addresses (e.g., a work item function pointer 'work_func_t' in 'workqueue:workqueue_execute_start'), it prints them as raw hexadecimal values (e.g., 0xffffffff81234567). This impairs readability when tracing kernel execution flows.

Introduce a dedicated kernel symbol beautifier, 'syscall_arg__scnprintf_ksym' (i.e., SCA_KSYM), to resolve kernel virtual addresses to human-readable symbol names and offsets (e.g., 'flush_to_ldisc').

The beautifier looks up the virtual address in the machine kernel maps via machine__find_kernel_symbol(). If a valid kernel symbol is found, the symbol name and offset are printed without requiring --libtraceevent; if the address is zero, "NULL" is rendered; otherwise, it gracefully falls back to hexadecimal formatting.

---
 tools/perf/builtin-trace.c       | 24 ++++++++++++++++++++++++
 tools/perf/trace/beauty/beauty.h |  3 +++
 2 files changed, 27 insertions(+)
tools/perf/builtin-trace.c +24 -0
index 5a3e043b5b72..cbc5bdc0cc12 100644
@@ -749,6 +749,30 @@ size_t syscall_arg__scnprintf_ptr(char *bf, size_t size, struct syscall_arg *arg
 	return syscall_arg__scnprintf_hex(bf, size, arg);
 }
 
+size_t syscall_arg__scnprintf_ksym(char *bf, size_t size, struct syscall_arg *arg)
+{
+	if (arg->val == 0)
+		return scnprintf(bf, size, "NULL");
+
+	if (arg->trace && arg->trace->host) {
+		struct map *map;
+		struct symbol *sym = machine__find_kernel_symbol(arg->trace->host,
+								 arg->val, &map);
+
+		if (sym) {
+			u64 start = map__unmap_ip(map, sym->start);
+			u64 offset = arg->val - start;
+
+			if (offset == 0)
+				return scnprintf(bf, size, "%s", sym->name);
+			return scnprintf(bf, size, "%s+0x%" PRIx64,
+					 sym->name, offset);
+		}
+	}
+
+	return syscall_arg__scnprintf_hex(bf, size, arg);
+}
+
 size_t syscall_arg__scnprintf_int(char *bf, size_t size, struct syscall_arg *arg)
 {
 	return scnprintf(bf, size, "%d", arg->val);
tools/perf/trace/beauty/beauty.h +3 -0
index 58a3206481ae..0f4801c61a5b 100644
@@ -160,6 +160,9 @@ size_t syscall_arg__scnprintf_hex(char *bf, size_t size, struct syscall_arg *arg
 size_t syscall_arg__scnprintf_ptr(char *bf, size_t size, struct syscall_arg *arg);
 #define SCA_PTR syscall_arg__scnprintf_ptr
 
+size_t syscall_arg__scnprintf_ksym(char *bf, size_t size, struct syscall_arg *arg);
+#define SCA_KSYM syscall_arg__scnprintf_ksym
+
 size_t syscall_arg__scnprintf_int(char *bf, size_t size, struct syscall_arg *arg);
 #define SCA_INT syscall_arg__scnprintf_int
 
Signature · 1 line
2.55.0
#2 lore raw

PATCH 2/3 perf trace: Auto-assign kernel symbol beautifier to function pointer fields

In reply to Aaron Tomlin

Tracepoint fields that convey kernel function pointers, such as 'function', 'fn', 'work', 'action', and 'callsite' are currently formatted as generic hexadecimal pointers by default.

Enhance syscall_arg_fmt__init_array() to automatically detect these fields by name and type signature (e.g., typedefs ending with '_func_t' or '_fn', or C function pointer types containing '(*)') and assign 'SCA_KSYM' as their default beautifier.

Additionally, register common function pointer field names within the sorted 'syscall_arg_fmts__by_name' lookup table. This ensures tracepoint arguments such as 'workqueue:workqueue_execute_start.function' are symbolised automatically without requiring explicit per-event configuration. For example:

    ❯ sudo tools/perf/perf trace --event workqueue:workqueue_execute_end --max-events 2 --show-cpu
         0.000 [000] kworker/u32:15/236682 workqueue:workqueue_execute_end(work: 0xffffffffab2f1420, function: toggle_allocation_gate)
         0.132 [000] kworker/u32:15/236682 workqueue:workqueue_execute_end(work: 0xffff8ac2c1adc010, function: flush_to_ldisc)
---
 tools/perf/builtin-trace.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
tools/perf/builtin-trace.c +13 -0
index cbc5bdc0cc12..d3401b64340a 100644
@@ -2090,8 +2090,13 @@ static int syscall__alloc_arg_fmts(struct syscall *sc, int nr_args)
 }
 
 static const struct syscall_arg_fmt syscall_arg_fmts__by_name[] = {
+	{ .name = "action",	.scnprintf = SCA_KSYM, },
+	{ .name = "callsite",	.scnprintf = SCA_KSYM, },
+	{ .name = "fn",		.scnprintf = SCA_KSYM, },
+	{ .name = "function",	.scnprintf = SCA_KSYM, },
 	{ .name = "msr",	.scnprintf = SCA_X86_MSR,	  .strtoul = STUL_X86_MSR,	   },
 	{ .name = "vector",	.scnprintf = SCA_X86_IRQ_VECTORS, .strtoul = STUL_X86_IRQ_VECTORS, },
+	{ .name = "work",	.scnprintf = SCA_KSYM, },
 };
 
 static int syscall_arg_fmt__cmp(const void *name, const void *fmtp)
@@ -2192,6 +2197,14 @@ syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_field
 		    ((len >= 4 && strcmp(field->name + len - 4, "name") == 0) ||
 		     strstr(field->name, "path") != NULL)) {
 			arg->scnprintf = SCA_FILENAME;
+		} else if (strcmp(field->name, "fn") == 0 ||
+			   strcmp(field->name, "function") == 0 ||
+			   strcmp(field->name, "callsite") == 0 ||
+			   strcmp(field->name, "action") == 0 ||
+			   (field->type && (strstr(field->type, "(*)") != NULL ||
+					    strstr(field->type, "_func_t") != NULL ||
+					    strstr(field->type, "_fn") != NULL))) {
+			arg->scnprintf = SCA_KSYM;
 		} else if ((field->flags & TEP_FIELD_IS_POINTER) || strstr(field->name, "addr") ||
 			   field_has_hex_fmt(field, len))
 			arg->scnprintf = SCA_PTR;
Signature · 1 line
2.55.0
#3 lore raw

PATCH 3/3 perf trace: Enhance BTF type formatting to symbolise kernel function pointers

In reply to Aaron Tomlin

When BTF (BPF Type Format) metadata is loaded from vmlinux, 'perf trace' can inspect the precise C types of tracepoint and system call parameters. However, function pointer arguments are currently not recognised during BTF pretty-printing and default to hexadecimal output.

Introduce btf_is_func_ptr() to inspect BTF type hierarchies (i.e., traversing pointers, typedefs, and type modifiers) to determine whether a parameter resolves to a function prototype ('BTF_KIND_FUNC_PROTO').

Generalise BTF type caching via syscall_arg_fmt__cache_btf_type() to handle structs, unions, enums, and function pointers alike. When a field is identified as a kernel function pointer, trace__btf_scnprintf() routes its value to syscall_arg__scnprintf_ksym(), enabling automatic zero-config symbolisation of kernel function pointers whenever BTF is available.

---
 tools/perf/builtin-trace.c | 39 ++++++++++++++++++++++++++++++--------
 1 file changed, 31 insertions(+), 8 deletions(-)
tools/perf/builtin-trace.c +31 -8
index d3401b64340a..60dddcf93ed0 100644
@@ -973,21 +973,43 @@ static size_t syscall_arg__scnprintf_getrandom_flags(char *bf, size_t size,
 #define SCA_GETRANDOM_FLAGS syscall_arg__scnprintf_getrandom_flags
 
 #ifdef HAVE_LIBBPF_SUPPORT
-static void syscall_arg_fmt__cache_btf_enum(struct syscall_arg_fmt *arg_fmt, struct btf *btf, char *type)
+static bool btf_is_func_ptr(const struct btf *btf, const struct btf_type *type)
+{
+	while (type) {
+		if (btf_is_ptr(type)) {
+			type = btf__type_by_id(btf, type->type);
+			return type && btf_is_func_proto(type);
+		}
+		if (btf_is_typedef(type) || btf_is_mod(type)) {
+			type = btf__type_by_id(btf, type->type);
+		} else {
+			break;
+		}
+	}
+	return false;
+}
+
+static void syscall_arg_fmt__cache_btf_type(struct syscall_arg_fmt *arg_fmt,
+					    struct btf *btf, char *type)
 {
 	int id;
 
-	type = strstr(type, "enum ");
 	if (type == NULL)
 		return;
 
-	type += 5; // skip "enum " to get the enumeration name
+	if (strstarts(type, "enum "))
+		type += 5;
+	else if (strstarts(type, "struct "))
+		type += 7;
+	else if (strstarts(type, "union "))
+		type += 6;
 
 	id = btf__find_by_name(btf, type);
 	if (id < 0)
 		return;
 
 	arg_fmt->type = btf__type_by_id(btf, id);
+	arg_fmt->type_id = id;
 }
 
 static bool syscall_arg__strtoul_btf_enum(char *bf, size_t size, struct syscall_arg *arg, u64 *val)
@@ -1022,8 +1044,7 @@ static bool syscall_arg__strtoul_btf_type(char *bf, size_t size, struct syscall_
 		return false;
 
 	if (arg->fmt->type == NULL) {
-		// See if this is an enum
-		syscall_arg_fmt__cache_btf_enum(arg->fmt, btf, type);
+		syscall_arg_fmt__cache_btf_type(arg->fmt, btf, type);
 	}
 
 	// Now let's see if we have a BTF type resolved
@@ -1110,8 +1131,7 @@ static size_t trace__btf_scnprintf(struct trace *trace, struct syscall_arg *arg,
 		return 0;
 
 	if (arg_fmt->type == NULL) {
-		// Check if this is an enum and if we have the BTF type for it.
-		syscall_arg_fmt__cache_btf_enum(arg_fmt, trace->btf, type);
+		syscall_arg_fmt__cache_btf_type(arg_fmt, trace->btf, type);
 	}
 
 	// Did we manage to find a BTF type for the syscall/tracepoint argument?
@@ -1122,6 +1142,8 @@ static size_t trace__btf_scnprintf(struct trace *trace, struct syscall_arg *arg,
 		return btf_enum_scnprintf(arg_fmt->type, trace->btf, bf, size, val);
 	else if (btf_is_struct(arg_fmt->type) || btf_is_union(arg_fmt->type))
 		return btf_struct_scnprintf(arg_fmt->type, trace->btf, bf, size, arg);
+	else if (btf_is_func_ptr(trace->btf, arg_fmt->type))
+		return syscall_arg__scnprintf_ksym(bf, size, arg);
 
 	return 0;
 }
@@ -2565,7 +2587,8 @@ static size_t syscall__scnprintf_args(struct syscall *sc, char *bf, size_t size,
 
 			default_scnprintf = sc->arg_fmt[arg.idx].scnprintf;
 
-			if (trace->force_btf || default_scnprintf == NULL || default_scnprintf == SCA_PTR) {
+			if (trace->force_btf || default_scnprintf == NULL ||
+			    default_scnprintf == SCA_PTR || default_scnprintf == SCA_KSYM) {
 				btf_printed = trace__btf_scnprintf(trace, &arg, bf + printed,
 								   size - printed, val, field->type);
 				if (btf_printed) {
Signature · 1 line
2.55.0
#4 lore raw
In reply to Aaron Tomlin
Quoted 39 lines · Aaron Tomlin

When inspecting kernel execution flows using 'perf trace' (e.g., when monitoring workqueues delayed work items, timer callbacks, etc.), tracepoint payload arguments containing raw kernel virtual addresses are currently rendered as hexadecimal values (e.g., 0xffffffff81234567).

This requires manual symbol lookups against /proc/kallsyms or vmlinux to identify the underlying kernel function being executed.

This patch series enhances 'perf trace' by introducing kernel virtual address and function pointer symbolisation using perf's native symbol engine (i.e., machine__find_kernel_symbol()).

Before:

    workqueue:workqueue_execute_end(work: 0xffffffffab2f1420, function: 0xffffffffa8046b50)

After:

    workqueue:workqueue_execute_end(work: 0xffff8ac2c420f270, function: wb_update_bandwidth_workfn)

Patch 1 adds the 'syscall_arg__scnprintf_ksym' ('SCA_KSYM') beautifier which resolves virtual addresses via machine__find_kernel_symbol(), formatting them as 'symbol_name+offset' (or "NULL" with a hex fallback).

Patch 2 updates event format initialisation in syscall_arg_fmt__init_array() to automatically assign SCA_KSYM to tracepoint fields named 'function', 'fn', 'work', 'action', or 'callsite', as well as fields typed as function pointers.

Patch 3 extends BTF pretty-printing in trace__btf_scnprintf() with btf_is_func_ptr() to automatically detect BTF function prototypes and symbolise kernel function pointers without requiring manual field table configuration.

Aaron Tomlin (3):
perf trace: Introduce kernel symbol beautifier for virtual addresses perf trace: Auto-assign kernel symbol beautifier to function pointer
fields
perf trace: Enhance BTF type formatting to symbolise kernel function
pointers

Quoted 6 lines
 tools/perf/builtin-trace.c       | 76 ++++++++++++++++++++++++++++----
 tools/perf/trace/beauty/beauty.h |  3 ++
 2 files changed, 71 insertions(+), 8 deletions(-)
Signature · 1 line
2.55.0

Hi Ian, Arnaldo, Namhyung,

Kindly ignore this series for now.

I would prefer to wait for series [1] and patch [2] to be merged. Thanks.

[1]: https://lore.kernel.org/lkml/[email protected]/
[2]: https://lore.kernel.org/lkml/[email protected]/

Kind regards,

Signature · 1 line
Aaron Tomlin