perf trace: Symbolise kernel virtual addresses and function pointers
v3 cover perf trace: Symbolise kernel virtual addresses and function pointers
When inspecting kernel execution flows using perf trace (e.g., when monitoring workqueues, delayed work items, timer callbacks, etc.), tracepoint payload arguments containing raw kernel virtual addresses are currently rendered as hexadecimal values (e.g., 0xffffffff81234567).
This requires manual symbol lookups against /proc/kallsyms or vmlinux to identify the underlying kernel function being executed.
This patch series enhances perf trace by introducing kernel virtual address and function pointer symbolisation using perf's native symbol engine (i.e., machine__find_kernel_symbol()).
Before:
workqueue:workqueue_execute_end(work: 0xffffffffab2f1420, function: 0xffffffffa8046b50)
After:
workqueue:workqueue_execute_end(work: 0xffff8ac2c420f270, function: wb_update_bandwidth_workfn)Patch 1 introduces the syscall_arg__scnprintf_ksym() (SCA_KSYM) beautifier, which resolves virtual addresses via machine__find_kernel_symbol(), formatting them as symbol_name+offset (or "NULL", with a graceful hexadecimal fallback upon lookup failure).
Patch 2 updates event format initialisation in syscall_arg_fmt__init_array() to automatically assign SCA_KSYM to tracepoint fields typed as function pointers (such as typedefs ending in "_func_t" or "_fn", or function prototypes matching "(*)"). It also registers common function pointer, callback, and callsite field names (e.g., "func", "function", "callback", "action", "handler", "caller", "location", "call_site") in syscall_arg_fmts__by_name[], and removes the legacy trace__field_is_ip() hex override in trace__fprintf_tp_fields().
Patch 3 extends BTF pretty-printing in trace__btf_scnprintf() with btf_is_func_ptr() to automatically traverse BTF type hierarchies (including nested typedefs and qualifiers) and route kernel function pointer arguments to SCA_KSYM when BTF metadata is available, with negative caching to prevent redundant BTF scans.
Patch 4 adds an automated regression test script, trace_ksym_beautifier.sh, under tools/perf/tests/shell/ to verify symbolisation across both default kallsyms and BTF routing paths.
Changes since v2:
- Populated syscall_arg_fmts__by_name[] with common function pointer, callback, handler, and callsite field names ("action", "call_site", "callback", "caller", "callsite", "cb", "fn", "func", "function", "handler", "location")
- Checked syscall_arg_fmt__find_by_name() prior to generic pointer fallbacks in syscall_arg_fmt__init_array(), and dropped the inline 64-bit size check to ensure 32-bit and cross-platform compatibility
- Removed legacy trace__field_is_ip() in trace__fprintf_tp_fields() to allow "call_site" and "caller_ip" to be beautified with SCA_KSYM rather than being forced to raw hexadecimal
- Simplified btf_is_func_ptr() to remove the internal pointer requirement, correctly identifying bare prototype typedefs as function pointers
- Prevented pointer enums (e.g., "enum foo *") from being misclassified and formatted as scalar enum values in syscall_arg_fmt__cache_btf_type(), trace__btf_scnprintf(), and syscall_arg__strtoul_btf_type()
- Added negative caching (btf_type_cached) to struct syscall_arg_fmt to avoid repeated BTF searches on every event for unresolvable or primitive types
- Widened btf_enum_scnprintf() to accept unsigned long val, eliminating narrowing truncation of 64-bit values to 32-bit signed integers
- Fixed trace_ksym_beautifier.sh:
- Quoted "$0" to support directory paths containing spaces
- Captured output in memory to eliminate temporary file leaks on early skip
- Hardened regex validation to require valid C symbol identifiers and reject raw hexadecimal addresses
- Used '%s' format specifiers in printf to prevent format string injection
- Link to v2: https://lore.kernel.org/lkml/[email protected]/
Changes since v1:
- Fixed reference leak of struct map in syscall_arg__scnprintf_ksym() by calling map__put() prior to returning Removed unreachable and erroneous entries ("action", "callsite", "call_site", "fn", "function", "work") from syscall_arg_fmts__by_name[]
- Restricted name-based SCA_KSYM auto-assignment in syscall_arg_fmt__init_array() to pointer or 64-bit address fields, preventing misclassification of non-pointer integer fields
- Updated btf_is_func_ptr() to fully unwrap typedefs and type modifiers below pointer targets
- Fixed BTF type name matching in syscall_arg_fmt__cache_btf_type() to handle leading modifiers and strip trailing pointer asterisks before lookup
- Synchronised arg->val with val in trace__btf_scnprintf() and widened val to unsigned long, fixing erroneous "NULL" output
- Added shell test script, tools/perf/tests/shell/trace_ksym_beautifier.sh, to verify kernel symbol beautification for both default kallsyms and BTF routing
- Link to v1: https://lore.kernel.org/lkml/[email protected]/
Aaron Tomlin (4):
perf trace: Introduce kernel symbol beautifier for virtual addresses perf trace: Auto-assign kernel symbol beautifier to function pointer
fields
perf trace: Enhance BTF type formatting to symbolise kernel function
pointers
perf tests: Add shell test for kernel symbol beautifier
tools/perf/builtin-trace.c | 187 ++++++++++++------ tools/perf/tests/shell/trace_btf_general.sh | 2 +- .../perf/tests/shell/trace_ksym_beautifier.sh | 43 ++++ tools/perf/trace/beauty/beauty.h | 3 + 4 files changed, 178 insertions(+), 57 deletions(-) create mode 100755 tools/perf/tests/shell/trace_ksym_beautifier.sh
Signature · 1 line
2.55.0
PATCH v3 1/4 perf trace: Introduce kernel symbol beautifier for virtual addresses
Currently, when 'perf trace' formats tracepoint payloads or system call arguments containing raw kernel virtual addresses (e.g., a work item function pointer work_func_t in workqueue:workqueue_execute_start), it prints them as raw hexadecimal values (e.g., 0xffffffff81234567). This impairs readability when tracing kernel execution flows.
Introduce a dedicated kernel symbol beautifier, syscall_arg__scnprintf_ksym (i.e., SCA_KSYM), to resolve kernel virtual addresses to human-readable symbol names and offsets (e.g., "flush_to_ldisc").
The beautifier looks up the virtual address in the machine kernel maps via machine__find_kernel_symbol(). If a valid kernel symbol is found, the symbol name and offset are printed without requiring --libtraceevent; if the address is zero, "NULL" is rendered; otherwise, it gracefully falls back to hexadecimal formatting.
--- tools/perf/builtin-trace.c | 29 +++++++++++++++++++++++++++++ tools/perf/trace/beauty/beauty.h | 3 +++ 2 files changed, 32 insertions(+)
tools/perf/builtin-trace.c +29 -0
@@ -750,6 +750,35 @@ size_t syscall_arg__scnprintf_ptr(char *bf, size_t size, struct syscall_arg *argreturn syscall_arg__scnprintf_hex (bf, size, arg); } +size_t syscall_arg__scnprintf_ksym (char *bf,size_t size,struct syscall_arg *arg) +{ +if (arg->val ==0 ) +return scnprintf (bf, size,"NULL" ); + +if (arg->trace && arg->trace->host) { +struct map *map =NULL ; +struct symbol *sym =machine__find_kernel_symbol (arg->trace->host, + arg->val, &map); + +if (sym) { +u64 start =map__unmap_ip (map, sym->start); +u64 offset = arg->val - start; +size_t printed; + +if (offset ==0 ) + printed =scnprintf (bf, size,"%s" , sym->name); +else + printed =scnprintf (bf, size,"%s+0x%" PRIx64, + sym->name, offset); +map__put (map); +return printed; + } +map__put (map); + } + +return syscall_arg__scnprintf_hex (bf, size, arg); +} +size_t syscall_arg__scnprintf_int (char *bf,size_t size,struct syscall_arg *arg) {return scnprintf (bf, size,"%d" , arg->val);
tools/perf/trace/beauty/beauty.h +3 -0
@@ -160,6 +160,9 @@ size_t syscall_arg__scnprintf_hex(char *bf, size_t size, struct syscall_arg *argsize_t syscall_arg__scnprintf_ptr (char *bf,size_t size,struct syscall_arg *arg);#define SCA_PTR syscall_arg__scnprintf_ptr +size_t syscall_arg__scnprintf_ksym (char *bf,size_t size,struct syscall_arg *arg); +#define SCA_KSYM syscall_arg__scnprintf_ksym +size_t syscall_arg__scnprintf_int (char *bf,size_t size,struct syscall_arg *arg);#define SCA_INT syscall_arg__scnprintf_int
Signature · 1 line
2.55.0
PATCH v3 2/4 perf trace: Auto-assign kernel symbol beautifier to function pointer fields
In reply to Aaron TomlinTracepoint fields that convey kernel function pointers, callbacks, and call sites, such as "function", "func", "fn", "callback", "action", "handler", "caller", "location", "callsite", and "call_site" are currently formatted as generic hexadecimal pointers by default.
Enhance syscall_arg_fmt__init_array() to automatically detect function pointer fields by type signature (e.g., typedefs ending with "_func_t" or "_fn", or C function pointer types containing "(*)") and assign SCA_KSYM as their default beautifier.
Additionally, register common function pointer and callback field names within the sorted syscall_arg_fmts__by_name lookup table. This ensures tracepoint arguments such as workqueue:workqueue_execute_start.function and csd:csd_function.func are symbolised automatically without requiring explicit per-event configuration. For example:
❯ sudo tools/perf/perf trace --event workqueue:workqueue_execute_end --max-events 2 --show-cpu
0.000 [000] kworker/u32:15/236682 workqueue:workqueue_execute_end(work: 0xffffffffab2f1420, function: toggle_allocation_gate)
0.132 [000] kworker/u32:15/236682 workqueue:workqueue_execute_end(work: 0xffff8ac2c1adc010, function: flush_to_ldisc)--- tools/perf/builtin-trace.c | 71 +++++++++++++++++++------------------- 1 file changed, 35 insertions(+), 36 deletions(-)
tools/perf/builtin-trace.c +35 -36
@@ -2096,6 +2096,17 @@ static int syscall__alloc_arg_fmts(struct syscall *sc, int nr_args) }static const struct syscall_arg_fmt syscall_arg_fmts__by_name[] = { + { .name ="action" , .scnprintf =SCA_KSYM , }, + { .name ="call_site" , .scnprintf =SCA_KSYM , }, + { .name ="callback" , .scnprintf =SCA_KSYM , }, + { .name ="caller" , .scnprintf =SCA_KSYM , }, + { .name ="callsite" , .scnprintf =SCA_KSYM , }, + { .name ="cb" , .scnprintf =SCA_KSYM , }, + { .name ="fn" , .scnprintf =SCA_KSYM , }, + { .name ="func" , .scnprintf =SCA_KSYM , }, + { .name ="function" , .scnprintf =SCA_KSYM , }, + { .name ="handler" , .scnprintf =SCA_KSYM , }, + { .name ="location" , .scnprintf =SCA_KSYM , }, { .name ="msr" , .scnprintf =SCA_X86_MSR , .strtoul =STUL_X86_MSR , }, { .name ="vector" , .scnprintf =SCA_X86_IRQ_VECTORS , .strtoul =STUL_X86_IRQ_VECTORS , }, };
@@ -2198,31 +2209,10 @@ syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_field ((len >= 4 && strcmp(field->name + len - 4, "name") == 0) || strstr(field->name, "path") != NULL)) { arg->scnprintf =SCA_FILENAME ; - } else if ((field->flags & TEP_FIELD_IS_POINTER) || strstr(field->name, "addr") || -field_has_hex_fmt (field, len)) - arg->scnprintf = SCA_PTR; - else if (strcmp(field->type, "pid_t") == 0) - arg->scnprintf = SCA_PID; - else if (strcmp(field->type, "umode_t") == 0) - arg->scnprintf = SCA_MODE_T; - else if ((field->flags & TEP_FIELD_IS_ARRAY) && strstr(field->type, "char")) { - arg->scnprintf =SCA_CHAR_ARRAY ; - arg->nr_entries = field->arraylen; - } else if ((strcmp(field->type, "int") == 0 || - strcmp(field->type, "unsigned int") == 0 || - strcmp(field->type, "long") == 0) && - len >= 2 && strcmp(field->name + len - 2, "fd") == 0) { -/* -* /sys/kernel/tracing/events/syscalls/sys_enter* -* grep -E 'field:.*fd;' .../format|sed -r 's/.*field:([a-z ]+) [a-z_]*fd.+/\1/g'|sort|uniq -c -* 65 int -* 23 unsigned int -* 7 unsigned long -*/ - arg->scnprintf =SCA_FD ; - } else if (strstr(field->type, "enum") && use_btf != NULL) { - *use_btf = true; - arg->strtoul =STUL_BTF_TYPE ; + } else if (field->type && (strstr(field->type, "(*)") != NULL || + strstr(field->type, "_func_t") != NULL || + strstr(field->type, "_fn") != NULL)) { + arg->scnprintf =SCA_KSYM ; } else {const struct syscall_arg_fmt *fmt =syscall_arg_fmt__find_by_name (field->name);
@@ -2230,6 +2220,24 @@ syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_fieldif (fmt) { arg->scnprintf = fmt->scnprintf; arg->strtoul = fmt->strtoul; + }else if ((field->flags &TEP_FIELD_IS_POINTER ) ||strstr (field->name,"addr" ) || +field_has_hex_fmt (field, len)) { + arg->scnprintf =SCA_PTR ; + }else if (strcmp (field->type,"pid_t" ) ==0 ) { + arg->scnprintf =SCA_PID ; + }else if (strcmp (field->type,"umode_t" ) ==0 ) { + arg->scnprintf =SCA_MODE_T ; + }else if ((field->flags &TEP_FIELD_IS_ARRAY ) &&strstr (field->type,"char" )) { + arg->scnprintf =SCA_CHAR_ARRAY ; + arg->nr_entries = field->arraylen; + }else if ((strcmp (field->type,"int" ) ==0 || +strcmp (field->type,"unsigned int" ) ==0 || +strcmp (field->type,"long" ) ==0 ) && + len >=2 &&strcmp (field->name + len -2 ,"fd" ) ==0 ) { + arg->scnprintf =SCA_FD ; + }else if (strstr (field->type,"enum" ) && use_btf !=NULL ) { + *use_btf = true; + arg->strtoul =STUL_BTF_TYPE ; } } }
@@ -3297,12 +3305,6 @@ static unsigned char bitmap_byte(const unsigned long *mask, int byte_idx)return b_val; } -static bool trace__field_is_ip (const char *name) -{ -return !strcmp (name,"__probe_ip" ) || - !strcmp (name,"caller_ip" ) || - !strcmp (name,"call_site" ); -}static size_t trace__fprintf_tp_fields (struct trace *trace,struct perf_sample *sample,struct thread *thread,void *augmented_args,int augmented_args_size)
@@ -3404,14 +3406,11 @@ static size_t trace__fprintf_tp_fields(struct trace *trace, struct perf_sample ** Suppress it by default to avoid cluttering the output. * If verbose mode is enabled, ensure it is formatted as a * hexadecimal memory address rather than a signed integer. -* -* caller_ip and call_site are also expected to be instruction -* pointers and should always be represented in hexadecimal. */ is_probe_ip =evsel__is_probe (evsel) && !strcmp (field->name,"__probe_ip" ); -if (is_probe_ip ||trace__field_is_ip (field->name)) { -if (is_probe_ip && !verbose) +if (is_probe_ip) { +if (!verbose)continue ; printed += scnprintf(bf + printed, size - printed,
Signature · 1 line
2.55.0
PATCH v3 3/4 perf trace: Enhance BTF type formatting to symbolise kernel function pointers
In reply to Aaron TomlinWhen BTF (BPF Type Format) metadata is loaded from vmlinux, 'perf trace' can inspect the precise C types of tracepoint and system call parameters. However, function pointer arguments are currently not recognised during BTF pretty-printing and default to hexadecimal output.
Introduce btf_is_func_ptr() to inspect BTF type hierarchies (i.e., traversing pointers, typedefs, and type modifiers) to determine whether a parameter resolves to a function prototype (BTF_KIND_FUNC_PROTO).
Generalise BTF type caching via syscall_arg_fmt__cache_btf_type() to handle structs, unions, enums, and function pointers alike. When a field is identified as a kernel function pointer, trace__btf_scnprintf() routes its value to syscall_arg__scnprintf_ksym(), enabling automatic zero-config symbolisation of kernel function pointers whenever BTF is available.
--- tools/perf/builtin-trace.c | 87 ++++++++++++++++----- tools/perf/tests/shell/trace_btf_general.sh | 2 +- 2 files changed, 68 insertions(+), 21 deletions(-)
tools/perf/builtin-trace.c +67 -20
@@ -122,6 +122,7 @@ struct syscall_arg_fmt {#ifdef HAVE_LIBBPF_SUPPORT const struct btf_type *type;int type_id;/* used in btf_dump */ +bool btf_type_cached;#endif };
@@ -979,21 +980,61 @@ static size_t syscall_arg__scnprintf_getrandom_flags(char *bf, size_t size,#define SCA_GETRANDOM_FLAGS syscall_arg__scnprintf_getrandom_flags#ifdef HAVE_LIBBPF_SUPPORT -static void syscall_arg_fmt__cache_btf_enum (struct syscall_arg_fmt *arg_fmt,struct btf *btf,char *type) +static bool btf_is_func_ptr (const struct btf *btf,const struct btf_type *type) { +while (type) { +if (btf_is_ptr (type) ||btf_is_typedef (type) ||btf_is_mod (type)) + type =btf__type_by_id (btf, type->type); +else +break ; + } +return type &&btf_is_func_proto (type); +} + +static void syscall_arg_fmt__cache_btf_type (struct syscall_arg_fmt *arg_fmt, +struct btf *btf,const char *type) +{ +char name[128 ]; +const char *pos; +size_t len =0 ;int id; - type =strstr (type,"enum " ); + arg_fmt->btf_type_cached = true; +if (type ==NULL )return ; - type +=5 ;// skip "enum " to get the enumeration name +/* Pointers to enums are memory addresses, not scalar enums */ +if (strstr (type,"enum " ) &&strchr (type,'*' )) +return ; + +if ((pos =strstr (type,"enum " )) !=NULL ) + pos +=5 ; +else if ((pos =strstr (type,"struct " )) !=NULL ) + pos +=7 ; +else if ((pos =strstr (type,"union " )) !=NULL ) + pos +=6 ; +else + pos = type; + +while (isspace (*pos)) + pos++; - id =btf__find_by_name (btf, type); +while ((isalnum (pos[len]) || pos[len] =='_' ) && len <sizeof (name) -1 ) { + name[len] = pos[len]; + len++; + } + name[len] ='\0' ; + +if (len ==0 ) +return ; + + id =btf__find_by_name (btf, name);if (id <0 )return ; arg_fmt->type =btf__type_by_id (btf, id); + arg_fmt->type_id = id; }static bool syscall_arg__strtoul_btf_enum (char *bf,size_t size,struct syscall_arg *arg,u64 *val)
@@ -1027,10 +1068,8 @@ static bool syscall_arg__strtoul_btf_type(char *bf, size_t size, struct syscall_if (btf ==NULL )return false; -if (arg->fmt->type ==NULL ) { -// See if this is an enum -syscall_arg_fmt__cache_btf_enum (arg->fmt, btf, type); - } +if (!arg->fmt->btf_type_cached) +syscall_arg_fmt__cache_btf_type (arg->fmt, btf, type);// Now let's see if we have a BTF type resolved bt = arg->fmt->type;
@@ -1038,19 +1077,22 @@ static bool syscall_arg__strtoul_btf_type(char *bf, size_t size, struct syscall_return false;// If it is an enum: -if (btf_is_enum (arg->fmt->type)) +if (btf_is_enum (arg->fmt->type)) { +if (type &&strchr (type,'*' )) +return false;return syscall_arg__strtoul_btf_enum (bf, size, arg, val); + }return false; } -static size_t btf_enum_scnprintf (const struct btf_type *type,struct btf *btf,char *bf,size_t size,int val) +static size_t btf_enum_scnprintf (const struct btf_type *type,struct btf *btf,char *bf,size_t size,unsigned long val) {struct btf_enum *be =btf_enum (type);const unsigned int nr_entries =btf_vlen (type);for (unsigned int i =0 ; i < nr_entries; ++i, ++be) { -if (be->val == val) { +if ((unsigned long )(__u32 )be->val == val || (unsigned long )be->val == val) {return scnprintf (bf, size,"%s" ,btf__name_by_offset (btf, be->name_off)); }
@@ -1108,33 +1150,37 @@ static size_t btf_struct_scnprintf(const struct btf_type *type, struct btf *btf, }static size_t trace__btf_scnprintf (struct trace *trace,struct syscall_arg *arg,char *bf, -size_t size,int val,char *type) +size_t size,unsigned long val,char *type) {struct syscall_arg_fmt *arg_fmt = arg->fmt;if (trace->btf ==NULL )return 0 ; -if (arg_fmt->type ==NULL ) { -// Check if this is an enum and if we have the BTF type for it. -syscall_arg_fmt__cache_btf_enum (arg_fmt, trace->btf, type); - } +if (!arg_fmt->btf_type_cached) +syscall_arg_fmt__cache_btf_type (arg_fmt, trace->btf, type);// Did we manage to find a BTF type for the syscall/tracepoint argument? if (arg_fmt->type ==NULL )return 0 ; -if (btf_is_enum (arg_fmt->type)) +if (btf_is_enum (arg_fmt->type)) { +if (type &&strchr (type,'*' )) +return 0 ;return btf_enum_scnprintf (arg_fmt->type, trace->btf, bf, size, val); -else if (btf_is_struct (arg_fmt->type) ||btf_is_union (arg_fmt->type)) + }else if (btf_is_struct (arg_fmt->type) ||btf_is_union (arg_fmt->type))return btf_struct_scnprintf (arg_fmt->type, trace->btf, bf, size, arg); +else if (btf_is_func_ptr (trace->btf, arg_fmt->type)) { + arg->val = val; +return syscall_arg__scnprintf_ksym (bf, size, arg); + }return 0 ; }#else // HAVE_LIBBPF_SUPPORT static size_t trace__btf_scnprintf (struct trace *trace__maybe_unused ,struct syscall_arg *arg__maybe_unused , -char *bf__maybe_unused ,size_t size__maybe_unused ,int val__maybe_unused , +char *bf__maybe_unused ,size_t size__maybe_unused ,unsigned long val__maybe_unused ,char *type__maybe_unused ) {return 0 ;
@@ -2566,7 +2612,8 @@ static size_t syscall__scnprintf_args(struct syscall *sc, char *bf, size_t size, default_scnprintf = sc->arg_fmt[arg.idx].scnprintf; -if (trace->force_btf || default_scnprintf ==NULL || default_scnprintf ==SCA_PTR ) { +if (trace->force_btf || default_scnprintf ==NULL || + default_scnprintf ==SCA_PTR || default_scnprintf ==SCA_KSYM ) { btf_printed =trace__btf_scnprintf (trace, &arg, bf + printed, size - printed, val, field->type);if (btf_printed) {
tools/perf/tests/shell/trace_btf_general.sh +1 -1
@@ -49,7 +49,7 @@ trace_test_buffer() { trace_test_struct_btf() { echo "Testing perf trace's struct augmentation" output="$(perf trace --sort-events -e clock_nanosleep --force-btf --max-events=1 -- sleep 1 2>&1)" - if ! echo "$output" | grep -q -E "^sleep/[0-9]+ clock_nanosleep\(0, 0, \{1,.*\}, 0x[0-9a-f]+\) += +[0-9]+$" + if ! echo "$output" | grep -q -E "sleep/[0-9]+ clock_nanosleep\(.*(struct __kernel_timespec|\{1,).*\) = [0-9]+" then printf "BTF struct augmentation test failed, output:\n$output\n" err=1
Signature · 1 line
2.55.0
PATCH v3 4/4 perf tests: Add shell test for kernel symbol beautifier
In reply to Aaron TomlinAdd a dedicated shell test script, trace_ksym_beautifier.sh, to verify that 'perf trace' properly symbolises kernel virtual addresses and function pointers using both the default kallsyms beautifier (SCA_KSYM) and BTF type routing i.e., --force-btf.
--- .../perf/tests/shell/trace_ksym_beautifier.sh | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 tools/perf/tests/shell/trace_ksym_beautifier.sh
tools/perf/tests/shell/trace_ksym_beautifier.sh new file +43 -0
@@ -0,0 +1,43 @@ +#!/bin/bash +# perf trace kernel symbol beautifier tests +# SPDX-License-Identifier: GPL-2.0 + +err=0 + +# shellcheck source=lib/probe.sh +. "$(dirname "$0")"/lib/probe.sh +skip_if_no_perf_trace || exit 2 +[ "$(id -u)" = 0 ] || exit 2 + +test_ksym_kallsyms() { + echo "Testing perf trace kernel symbol beautifier (default kallsyms)" + output="$(perf trace -e kmem:kmalloc --max-events=1 2>&1)" + if ! echo "$output" | grep -q -E "call_site: [a-zA-Z_][a-zA-Z0-9_]*" || echo "$output" | grep -q -E "call_site: 0x[0-9a-fA-F]+" + then + printf "Default kallsyms function symbolization failed, output:\n%s\n" "$output" + err=1 + fi +} + +test_ksym_btf() { + echo "Testing perf trace kernel symbol beautifier (BTF)" + if [ ! -f /sys/kernel/btf/vmlinux ]; then + echo "Skipping BTF test due to missing vmlinux BTF" + return + fi + + output="$(perf trace -e kmem:kmalloc --force-btf --max-events=1 2>&1)" + if ! echo "$output" | grep -q -E "call_site: [a-zA-Z_][a-zA-Z0-9_]*" || echo "$output" | grep -q -E "call_site: 0x[0-9a-fA-F]+" + then + printf "BTF function symbolization failed, output:\n%s\n" "$output" + err=1 + fi +} + +test_ksym_kallsyms + +if [ $err = 0 ]; then + test_ksym_btf +fi + +exit $err
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH perf-tools-next v3 1/4] perf trace: Introduce kernel symbol beautifier for virtual addresses
In reply to Aaron TomlinSashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Currently, when 'perf trace' formats tracepoint payloads or system call arguments containing raw kernel virtual addresses (e.g., a work item function pointer
work_func_tin workqueue:workqueue_execute_start), it prints them as raw hexadecimal values (e.g.,0xffffffff81234567). This impairs readability when tracing kernel execution flows.Introduce a dedicated kernel symbol beautifier,
syscall_arg__scnprintf_ksym(i.e.,SCA_KSYM), to resolve kernel virtual addresses to human-readable symbol names and offsets (e.g., "flush_to_ldisc").
Quoted 7 lines
The beautifier looks up the virtual address in the machine kernel maps via
machine__find_kernel_symbol(). If a valid kernel symbol is found, the symbol name and offset are printed without requiring --libtraceevent; if the address is zero, "NULL" is rendered; otherwise, it gracefully falls back to hexadecimal formatting.Signed-off-by Aaron Tomlin <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewRe: [PATCH perf-tools-next v3 3/4] perf trace: Enhance BTF type formatting to symbolise kernel function pointers
In reply to Aaron TomlinSashiko has reviewed this patch and found no issues. It looks great!
Quoted 10 lines
When BTF (BPF Type Format) metadata is loaded from vmlinux, 'perf trace' can inspect the precise C types of tracepoint and system call parameters. However, function pointer arguments are currently not recognised during BTF pretty-printing and default to hexadecimal output.
Introduce
btf_is_func_ptr()to inspect BTF type hierarchies (i.e., traversing pointers, typedefs, and type modifiers) to determine whether a parameter resolves to a function prototype (BTF_KIND_FUNC_PROTO).
Quoted 8 lines
Generalise BTF type caching via
syscall_arg_fmt__cache_btf_type()to handle structs, unions, enums, and function pointers alike. When a field is identified as a kernel function pointer,trace__btf_scnprintf()routes its value tosyscall_arg__scnprintf_ksym(), enabling automatic zero-config symbolisation of kernel function pointers whenever BTF is available.Signed-off-by Aaron Tomlin <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
[email protected]
AI reviewRe: [PATCH perf-tools-next v3 2/4] perf trace: Auto-assign kernel symbol beautifier to function pointer fields
In reply to Aaron TomlinThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Mapping highly generic field names like 'action', 'cb', and 'caller' globally to `SCA_KSYM`…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Mapping highly generic field names like 'action', 'cb', and 'caller' globally to SCA_KSYM short-circuits type-based logic, leaking perf userspace memory addresses instead of strings and breaking integers/enums.
- [Medium] Removal of trace__field_is_ip() and omission of caller_ip from the new lookup table causes kernel instruction pointers to be formatted as unreadable decimal integers.
--
--- Patch [2]: [PATCH perf-tools-next v3 2/4] perf trace: Auto-assign kernel symbol beautifier to function pointer fields --- commit d3017d506b83550526e2692582e2544d0b006d0f
Author: Aaron Tomlin <[email protected]>
perf trace: Auto-assign kernel symbol beautifier to function pointer fields
This patch enhances perf trace to automatically format kernel function pointer fields using the SCA_KSYM beautifier. It detects these fields by type signature or by matching common function pointer and callback field names in a lookup table.
Quoted 14 lines
tools/perf/builtin-trace.c +6
-0@@ -2096,6 +2096,17 @@ static int syscall__alloc_arg_fmts(struct syscall *sc, int nr_args) }static const struct syscall_arg_fmt syscall_arg_fmts__by_name[] = { + { .name ="action" , .scnprintf =SCA_KSYM , }, + { .name ="call_site" , .scnprintf =SCA_KSYM , }, + { .name ="callback" , .scnprintf =SCA_KSYM , }, + { .name ="caller" , .scnprintf =SCA_KSYM , }, + { .name ="callsite" , .scnprintf =SCA_KSYM , }, + { .name ="cb" , .scnprintf =SCA_KSYM , },
[ ... ]
Quoted 4 lines
tools/perf/builtin-trace.c +0
-0@@ -2230,6 +2220,24 @@ syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_fieldif (fmt) { arg->scnprintf = fmt->scnprintf; arg->strtoul = fmt->strtoul;
tools/perf/builtin-trace.c syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_field · line 2232
Quoted 10 lines
tools/perf/builtin-trace.c +10
-0+ }else if ((field->flags &TEP_FIELD_IS_POINTER ) ||strstr (field->name,"addr" ) || +field_has_hex_fmt (field, len)) { + arg->scnprintf =SCA_PTR ; + }else if (strcmp (field->type,"pid_t" ) ==0 ) { + arg->scnprintf =SCA_PID ; + }else if (strcmp (field->type,"umode_t" ) ==0 ) { + arg->scnprintf =SCA_MODE_T ; + }else if ((field->flags &TEP_FIELD_IS_ARRAY ) &&strstr (field->type,"char" )) { + arg->scnprintf =SCA_CHAR_ARRAY ; + arg->nr_entries = field->arraylen;
[Severity: High]
Does checking syscall_arg_fmt__find_by_name() earlier in this branch, before the type-based checks like string arrays or BTF enums, short-circuit those checks for generic field names?
If a field is named "caller", "action", or "cb" but is actually a dynamic string or integer, it seems this early name match will blindly assign SCA_KSYM.
Would this cause strings to be printed as raw kernel addresses by passing a userspace perf pointer to SCA_KSYM, permanently obfuscating the trace data?
[ ... ]
tools/perf/builtin-trace.c static unsigned char bitmap_byte(const unsigned long *mask, int byte_idx) · line 3305
Quoted 10 lines
tools/perf/builtin-trace.c +0
-6@@ -3297,12 +3305,6 @@ static unsigned char bitmap_byte(const unsigned long *mask, int byte_idx)return b_val; } -static bool trace__field_is_ip (const char *name) -{ -return !strcmp (name,"__probe_ip" ) || - !strcmp (name,"caller_ip" ) || - !strcmp (name,"call_site" ); -}
[Severity: Medium]
Since trace__field_is_ip() is removed here, should "caller_ip" have been added to the new syscall_arg_fmts__by_name lookup table?
Because "caller_ip" is typed as unsigned long (not a pointer), lacks "addr" in its name, and its TP_printk format strings do not match the strict =0x/=%p checks in field_has_hex_fmt(), it evades all heuristics.
Will this cause kernel instruction pointers like "caller_ip" (used heavily in XFS and OCFS2 tracepoints) to be formatted as unreadable base-10 decimal integers?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[email protected]
AI reviewRe: [PATCH perf-tools-next v3 4/4] perf tests: Add shell test for kernel symbol beautifier
In reply to Aaron TomlinThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The BTF type routing test does not actually test BTF symbolization, as it silently falls back to…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The BTF type routing test does not actually test BTF symbolization, as it silently falls back to the default kallsyms beautifier. --
--- Patch [4]: [PATCH perf-tools-next v3 4/4] perf tests: Add shell test for kernel symbol beautifier --- commit c8c43ea37c4e0f26131dd9e871af4241cd478b77
Author: Aaron Tomlin <[email protected]>
perf tests: Add shell test for kernel symbol beautifier
This commit adds a shell test script, trace_ksym_beautifier.sh, to verify that perf trace symbolises kernel virtual addresses and function pointers using both the default kallsyms beautifier and BTF type routing.
Quoted 5 lines
tools/perf/tests/shell/trace_ksym_beautifier.sh new file +0
-0
[ ... ]
tools/perf/tests/shell/trace_ksym_beautifier.sh line 29
Quoted 8 lines
tools/perf/tests/shell/trace_ksym_beautifier.sh new file +8
-0+test_ksym_btf() { + echo "Testing perf trace kernel symbol beautifier (BTF)" + if [ ! -f /sys/kernel/btf/vmlinux ]; then + echo "Skipping BTF test due to missing vmlinux BTF" + return + fi + + output="$(perf trace -e kmem:kmalloc --force-btf --max-events=1 2>&1)"
[Severity: Medium]
Does this actually test the BTF type routing as intended?
Looking at the kmem:kmalloc tracepoint, the call_site field is defined as an unsigned long, rather than a function pointer. When trace__btf_scnprintf() evaluates this, it determines it is an integer rather than a function pointer, and returns 0.
This causes perf to silently fall back to the default SCA_KSYM beautifier. Consequently, the test passes via the fallback path rather than truly exercising the BTF routing.
Would it be better to use a tracepoint with a field explicitly typed as a function pointer for the --force-btf test?
Quoted 6 lines
tools/perf/tests/shell/trace_ksym_beautifier.sh new file +6
-0+ if ! echo "$output" | grep -q -E "call_site: [a-zA-Z_][a-zA-Z0-9_]*" || echo "$output" | grep -q -E "call_site: 0x[0-9a-fA-F]+" + then + printf "BTF function symbolization failed, output:\n%s\n" "$output" + err=1 + fi +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Re: [PATCH perf-tools-next v3 2/4] perf trace: Auto-assign kernel symbol beautifier to function pointer fields
In reply to [email protected]Quoted 47 lines · On Thu, Aug 20, 2026 at 09:26:12PM +0000, [email protected] wrote:
commit
d3017d506b83550526e2692582e2544d0b006d0f
Author: Aaron Tomlin <[email protected]>perf trace: Auto-assign kernel symbol beautifier to function pointer fields
This patch enhances perf trace to automatically format kernel function pointer fields using the
SCA_KSYMbeautifier. It detects these fields by type signature or by matching common function pointer and callback field names in a lookup table.Quoted 14 lines
tools/perf/builtin-trace.c +6
-0@@ -2096,6 +2096,17 @@ static int syscall__alloc_arg_fmts(struct syscall *sc, int nr_args) }static const struct syscall_arg_fmt syscall_arg_fmts__by_name[] = { + { .name ="action" , .scnprintf =SCA_KSYM , }, + { .name ="call_site" , .scnprintf =SCA_KSYM , }, + { .name ="callback" , .scnprintf =SCA_KSYM , }, + { .name ="caller" , .scnprintf =SCA_KSYM , }, + { .name ="callsite" , .scnprintf =SCA_KSYM , }, + { .name ="cb" , .scnprintf =SCA_KSYM , },[ ... ]
Quoted 4 lines
tools/perf/builtin-trace.c +0
-0@@ -2230,6 +2220,24 @@ syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_fieldif (fmt) { arg->scnprintf = fmt->scnprintf; arg->strtoul = fmt->strtoul;tools/perf/builtin-trace.c syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_field · line 2232
Quoted 10 lines
tools/perf/builtin-trace.c +10
-0+ }else if ((field->flags &TEP_FIELD_IS_POINTER ) ||strstr (field->name,"addr" ) || +field_has_hex_fmt (field, len)) { + arg->scnprintf =SCA_PTR ; + }else if (strcmp (field->type,"pid_t" ) ==0 ) { + arg->scnprintf =SCA_PID ; + }else if (strcmp (field->type,"umode_t" ) ==0 ) { + arg->scnprintf =SCA_MODE_T ; + }else if ((field->flags &TEP_FIELD_IS_ARRAY ) &&strstr (field->type,"char" )) { + arg->scnprintf =SCA_CHAR_ARRAY ; + arg->nr_entries = field->arraylen;[Severity: High]
Does checkingsyscall_arg_fmt__find_by_name()earlier in this branch, before the type-based checks like string arrays or BTF enums, short-circuit those checks for generic field names?
Quoted 6 lines
If a field is named "caller", "action", or "cb" but is actually a dynamic string or integer, it seems this early name match will blindly assign
SCA_KSYM.Would this cause strings to be printed as raw kernel addresses by passing a userspace perf pointer to
SCA_KSYM, permanently obfuscating the trace data?
I agree. This is a valid concern.
To resolve, I will guard the SCA_KSYM assignment in syscall_arg_fmt__init_array(). Moving forward, a name-based lookup now assigns SCA_KSYM only if the field is an actual pointer (TEP_FIELD_IS_POINTER) or a pointer-sized non-array scalar (field->size == sizeof(void *) && !(field->flags & TEP_FIELD_IS_ARRAY)). If a field fails this guard, it falls through to subsequent type checks (i.e., SCA_CHAR_ARRAY, BTF enum inspection, SCA_FD, etc.). Non-SCA_KSYM entries in the lookup table (e.g., "msr" and "vector") continue to pass through unhindered. This should work.
Quoted 17 lines
[ ... ]
tools/perf/builtin-trace.c static unsigned char bitmap_byte(const unsigned long *mask, int byte_idx) · line 3305
Quoted 10 lines
tools/perf/builtin-trace.c +0
-6@@ -3297,12 +3305,6 @@ static unsigned char bitmap_byte(const unsigned long *mask, int byte_idx)return b_val; } -static bool trace__field_is_ip (const char *name) -{ -return !strcmp (name,"__probe_ip" ) || - !strcmp (name,"caller_ip" ) || - !strcmp (name,"call_site" ); -}[Severity: Medium]
Sincetrace__field_is_ip()is removed here, should "caller_ip" have been added to the newsyscall_arg_fmts__by_namelookup table?
Quoted 7 lines
Because "
caller_ip" is typed as unsigned long (not a pointer), lacks "addr" in its name, and itsTP_printkformat strings do not match the strict =0x/=%p checks infield_has_hex_fmt(), it evades all heuristics.Will this cause kernel instruction pointers like "
caller_ip" (used heavily in XFS and OCFS2 tracepoints) to be formatted as unreadable base-10 decimal integers?
Apologies, "caller_ip" was accidently omitted!
In the next iteration, I have added "caller_ip" to syscall_arg_fmts__by_name[] in strict alphabetical order to ensure bsearch() integrity. Furthermore, I have relaxed field_has_hex_fmt() to tolerate delimiters preceding %p/%# format specifiers, ensuring tracepoint formats utilising %pS/%ps are properly recognised too.
Kind regards,
Signature · 1 line
Aaron Tomlin