perf trace: Symbolise kernel virtual addresses and function pointers
v2 cover perf trace: Symbolise kernel virtual addresses and function pointers
When inspecting kernel execution flows using perf trace (e.g., when monitoring workqueues delayed work items, timer callbacks, etc.), tracepoint payload arguments containing raw kernel virtual addresses are currently rendered as hexadecimal values (e.g., 0xffffffff81234567).
This requires manual symbol lookups against /proc/kallsyms or vmlinux to identify the underlying kernel function being executed.
This patch series enhances perf trace by introducing kernel virtual address and function pointer symbolisation using perfs native symbol engine (i.e., machine__find_kernel_symbol()).
Before:
workqueue:workqueue_execute_end(work: 0xffffffffab2f1420, function: 0xffffffffa8046b50)
After:
workqueue:workqueue_execute_end(work: 0xffff8ac2c420f270, function: wb_update_bandwidth_workfn)Patch 1 introduces the syscall_arg__scnprintf_ksym() (SCA_KSYM) beautifier, which resolves virtual addresses via machine__find_kernel_symbol(), formatting them as symbol_name+offset (or "NULL", with a graceful hexadecimal fallback upon lookup failure).
Patch 2 updates event format initialisation in syscall_arg_fmt__init_array() to automatically assign SCA_KSYM to tracepoint fields typed as function pointers (such as typedefs ending in "_func_t" or "_fn", or function prototypes matching "(*)"), as well as pointer or 64-bit address fields named "fn", "function", "callsite", or "call_site".
Patch 3 extends BTF pretty-printing in trace__btf_scnprintf() with btf_is_func_ptr() to automatically traverse BTF type hierarchies (including nested typedefs and qualifiers) and route kernel function pointer arguments to SCA_KSYM when BTF metadata is available.
Patch 4 adds an automated regression test script, trace_ksym_beautifier.sh, under tools/perf/tests/shell/ to verify symbolisation across both default kallsyms and BTF routing paths.
Changes since v1:
- Fixed reference leak of struct map in syscall_arg__scnprintf_ksym() by calling map__put() prior to returning Removed unreachable and erroneous entries ("action", "callsite", "call_site", "fn", "function", "work") from syscall_arg_fmts__by_name[]
- Restricted name-based SCA_KSYM auto-assignment in syscall_arg_fmt__init_array() to pointer or 64-bit address fields, preventing misclassification of non-pointer integer fields
- Updated btf_is_func_ptr() to fully unwrap typedefs and type modifiers below pointer targets
- Fixed BTF type name matching in syscall_arg_fmt__cache_btf_type() to handle leading modifiers and strip trailing pointer asterisks before lookup
- Synchronised arg->val with val in trace__btf_scnprintf() and widened val to unsigned long, fixing erroneous "NULL" output
- Added shell test script, tools/perf/tests/shell/trace_ksym_beautifier.sh, to verify kernel symbol beautification for both default kallsyms and BTF routing
- Link to v1: https://lore.kernel.org/lkml/[email protected]/
Aaron Tomlin (4):
perf trace: Introduce kernel symbol beautifier for virtual addresses perf trace: Auto-assign kernel symbol beautifier to function pointer
fields
perf trace: Enhance BTF type formatting to symbolise kernel function
pointers
perf tests: Add shell test for kernel symbol beautifier
tools/perf/builtin-trace.c | 103 ++++++++++++++++-- tools/perf/tests/shell/trace_btf_general.sh | 2 +- .../perf/tests/shell/trace_ksym_beautifier.sh | 52 +++++++++ tools/perf/trace/beauty/beauty.h | 3 + 4 files changed, 148 insertions(+), 12 deletions(-) create mode 100755 tools/perf/tests/shell/trace_ksym_beautifier.sh
Signature · 1 line
2.55.0
PATCH v2 1/4 perf trace: Introduce kernel symbol beautifier for virtual addresses
Currently, when 'perf trace' formats tracepoint payloads or system call arguments containing raw kernel virtual addresses (e.g., a work item function pointer work_func_t in workqueue:workqueue_execute_start), it prints them as raw hexadecimal values (e.g., 0xffffffff81234567). This impairs readability when tracing kernel execution flows.
Introduce a dedicated kernel symbol beautifier, syscall_arg__scnprintf_ksym (i.e., SCA_KSYM), to resolve kernel virtual addresses to human-readable symbol names and offsets (e.g., "flush_to_ldisc").
The beautifier looks up the virtual address in the machine kernel maps via machine__find_kernel_symbol(). If a valid kernel symbol is found, the symbol name and offset are printed without requiring --libtraceevent; if the address is zero, "NULL" is rendered; otherwise, it gracefully falls back to hexadecimal formatting.
--- tools/perf/builtin-trace.c | 29 +++++++++++++++++++++++++++++ tools/perf/trace/beauty/beauty.h | 3 +++ 2 files changed, 32 insertions(+)
tools/perf/builtin-trace.c +29 -0
@@ -750,6 +750,35 @@ size_t syscall_arg__scnprintf_ptr(char *bf, size_t size, struct syscall_arg *argreturn syscall_arg__scnprintf_hex (bf, size, arg); } +size_t syscall_arg__scnprintf_ksym (char *bf,size_t size,struct syscall_arg *arg) +{ +if (arg->val ==0 ) +return scnprintf (bf, size,"NULL" ); + +if (arg->trace && arg->trace->host) { +struct map *map =NULL ; +struct symbol *sym =machine__find_kernel_symbol (arg->trace->host, + arg->val, &map); + +if (sym) { +u64 start =map__unmap_ip (map, sym->start); +u64 offset = arg->val - start; +size_t printed; + +if (offset ==0 ) + printed =scnprintf (bf, size,"%s" , sym->name); +else + printed =scnprintf (bf, size,"%s+0x%" PRIx64, + sym->name, offset); +map__put (map); +return printed; + } +map__put (map); + } + +return syscall_arg__scnprintf_hex (bf, size, arg); +} +size_t syscall_arg__scnprintf_int (char *bf,size_t size,struct syscall_arg *arg) {return scnprintf (bf, size,"%d" , arg->val);
tools/perf/trace/beauty/beauty.h +3 -0
@@ -160,6 +160,9 @@ size_t syscall_arg__scnprintf_hex(char *bf, size_t size, struct syscall_arg *argsize_t syscall_arg__scnprintf_ptr (char *bf,size_t size,struct syscall_arg *arg);#define SCA_PTR syscall_arg__scnprintf_ptr +size_t syscall_arg__scnprintf_ksym (char *bf,size_t size,struct syscall_arg *arg); +#define SCA_KSYM syscall_arg__scnprintf_ksym +size_t syscall_arg__scnprintf_int (char *bf,size_t size,struct syscall_arg *arg);#define SCA_INT syscall_arg__scnprintf_int
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH perf-tools-next v2 1/4] perf trace: Introduce kernel symbol beautifier for virtual addresses
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Currently, when 'perf trace' formats tracepoint payloads or system call arguments containing raw kernel virtual addresses (e.g., a work item function pointer
work_func_tin workqueue:workqueue_execute_start), it prints them as raw hexadecimal values (e.g.,0xffffffff81234567). This impairs readability when tracing kernel execution flows.Introduce a dedicated kernel symbol beautifier,
syscall_arg__scnprintf_ksym(i.e.,SCA_KSYM), to resolve kernel virtual addresses to human-readable symbol names and offsets (e.g., "flush_to_ldisc").
Quoted 7 lines
The beautifier looks up the virtual address in the machine kernel maps via
machine__find_kernel_symbol(). If a valid kernel symbol is found, the symbol name and offset are printed without requiring --libtraceevent; if the address is zero, "NULL" is rendered; otherwise, it gracefully falls back to hexadecimal formatting.Signed-off-by Aaron Tomlin <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
PATCH v2 2/4 perf trace: Auto-assign kernel symbol beautifier to function pointer fields
In reply to Aaron TomlinTracepoint fields that convey kernel function pointers, such as "function", "fn", "work", "action", and "callsite" are currently formatted as generic hexadecimal pointers by default.
Enhance syscall_arg_fmt__init_array() to automatically detect these fields by name and type signature (e.g., typedefs ending with "_func_t" or "_fn", or C function pointer types containing "(*)") and assign SCA_KSYM as their default beautifier.
Additionally, register common function pointer field names within the sorted syscall_arg_fmts__by_name lookup table. This ensures tracepoint arguments such as workqueue:workqueue_execute_start.function are symbolised automatically without requiring explicit per-event configuration. For example:
❯ sudo tools/perf/perf trace --event workqueue:workqueue_execute_end --max-events 2 --show-cpu
0.000 [000] kworker/u32:15/236682 workqueue:workqueue_execute_end(work: 0xffffffffab2f1420, function: toggle_allocation_gate)
0.132 [000] kworker/u32:15/236682 workqueue:workqueue_execute_end(work: 0xffff8ac2c1adc010, function: flush_to_ldisc)--- tools/perf/builtin-trace.c | 9 +++++++++ 1 file changed, 9 insertions(+)
tools/perf/builtin-trace.c +9 -0
@@ -2198,6 +2198,15 @@ syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_field ((len >= 4 && strcmp(field->name + len - 4, "name") == 0) || strstr(field->name, "path") != NULL)) { arg->scnprintf =SCA_FILENAME ; + } else if ((field->type && (strstr(field->type, "(*)") != NULL || + strstr(field->type, "_func_t") != NULL || + strstr(field->type, "_fn") != NULL)) || + (((field->flags & TEP_FIELD_IS_POINTER) || field->size == sizeof(u64)) && + (strcmp(field->name, "fn") == 0 || + strcmp(field->name, "function") == 0 || + strcmp(field->name, "callsite") == 0 || + strcmp(field->name, "call_site") == 0))) { + arg->scnprintf =SCA_KSYM ; } else if ((field->flags & TEP_FIELD_IS_POINTER) || strstr(field->name, "addr") ||field_has_hex_fmt (field, len)) arg->scnprintf = SCA_PTR;
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH perf-tools-next v2 2/4] perf trace: Auto-assign kernel symbol beautifier to function pointer fields
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Medium] The patch's commit message explicitly claims it will "register common function pointer field…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Medium] The patch's commit message explicitly claims it will "register common function pointer field names within the sorted syscall_arg_fmts__by_name lookup table." However, the code completely omits any modifications to this table and instead hardcodes the string comparisons inline into syscall_arg_fmt__init_array().
- [Medium] The hardcoded field->size == sizeof(u64) check breaks 32-bit compatibility and cross-platform analysis for function pointers.
- [Medium] The hardcoded list of function pointer field names misses standard tracepoint fields like 'func' and 'callback'.
--
--- Patch [2]: [PATCH perf-tools-next v2 2/4] perf trace: Auto-assign kernel symbol beautifier to function pointer fields --- commit b4ca96eef1273e64d4ffedaf8b32cf198586f462
Author: Aaron Tomlin <[email protected]>
perf trace: Auto-assign kernel symbol beautifier to function pointer fields
This commit automatically detects function pointer fields in tracepoints by name and type. It assigns the SCA_KSYM beautifier to them so they are symbolised automatically instead of printing as generic hex values.
Quoted 5 lines
Additionally, register common function pointer field names within the sorted
syscall_arg_fmts__by_namelookup table. This ensures tracepoint arguments such as workqueue:workqueue_execute_start.function are symbolised automatically without requiring explicit per-event configuration.
[Severity: Medium]
Does the patch implement this update to syscall_arg_fmts__by_name?
The commit message explicitly claims to register names within this lookup table, but the patch omits any changes to it and instead hardcodes the string comparisons inline in syscall_arg_fmt__init_array().
tools/perf/builtin-trace.c syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_field · line 2204
Quoted 12 lines
tools/perf/builtin-trace.c +4
-0@@ -2198,6 +2198,15 @@ syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_field ((len >= 4 && strcmp(field->name + len - 4, "name") == 0) || strstr(field->name, "path") != NULL)) { arg->scnprintf =SCA_FILENAME ; + } else if ((field->type && (strstr(field->type, "(*)") != NULL || + strstr(field->type, "_func_t") != NULL || + strstr(field->type, "_fn") != NULL)) || + (((field->flags & TEP_FIELD_IS_POINTER) || field->size == sizeof(u64)) &&
[Severity: Medium]
Does this field->size check cause a regression in 32-bit compatibility and cross-platform analysis for function pointers?
On 32-bit systems, function pointers are 4 bytes, so field->size is 4. Since sizeof(u64) is always 8 on the host compiling perf, this check fails on native 32-bit systems or when a 64-bit host parses a 32-bit perf.data file.
tools/perf/builtin-trace.c syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_field · line 2208
Quoted 4 lines
tools/perf/builtin-trace.c +4
-0+ (strcmp (field->name,"fn" ) ==0 || +strcmp (field->name,"function" ) ==0 || +strcmp (field->name,"callsite" ) ==0 || + strcmp(field->name, "call_site") == 0))) {
[Severity: Medium]
Does this hardcoded list miss standard tracepoint fields like 'func' and 'callback'?
Tracepoints using common standard names for function pointers, such as 'func' (e.g., in csd:csd_function, rcu, and wireless subsystems) and 'callback' (e.g., in ipi:ipi_raise), will fail this name check and will not be automatically symbolized by the beautifier.
Quoted 4 lines
tools/perf/builtin-trace.c +1
-0+ arg->scnprintf =SCA_KSYM ; }else if ((field->flags &TEP_FIELD_IS_POINTER ) ||strstr (field->name,"addr" ) ||field_has_hex_fmt (field, len)) arg->scnprintf =SCA_PTR ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH v2 3/4 perf trace: Enhance BTF type formatting to symbolise kernel function pointers
In reply to Aaron TomlinWhen BTF (BPF Type Format) metadata is loaded from vmlinux, 'perf trace' can inspect the precise C types of tracepoint and system call parameters. However, function pointer arguments are currently not recognised during BTF pretty-printing and default to hexadecimal output.
Introduce btf_is_func_ptr() to inspect BTF type hierarchies (i.e., traversing pointers, typedefs, and type modifiers) to determine whether a parameter resolves to a function prototype (BTF_KIND_FUNC_PROTO).
Generalise BTF type caching via syscall_arg_fmt__cache_btf_type() to handle structs, unions, enums, and function pointers alike. When a field is identified as a kernel function pointer, trace__btf_scnprintf() routes its value to syscall_arg__scnprintf_ksym(), enabling automatic zero-config symbolisation of kernel function pointers whenever BTF is available.
--- tools/perf/builtin-trace.c | 65 +++++++++++++++++---- tools/perf/tests/shell/trace_btf_general.sh | 2 +- 2 files changed, 55 insertions(+), 12 deletions(-)
tools/perf/builtin-trace.c +54 -11
@@ -979,21 +979,61 @@ static size_t syscall_arg__scnprintf_getrandom_flags(char *bf, size_t size,#define SCA_GETRANDOM_FLAGS syscall_arg__scnprintf_getrandom_flags#ifdef HAVE_LIBBPF_SUPPORT -static void syscall_arg_fmt__cache_btf_enum (struct syscall_arg_fmt *arg_fmt,struct btf *btf,char *type) +static bool btf_is_func_ptr (const struct btf *btf,const struct btf_type *type) { +bool has_ptr = false; + +while (type) { +if (btf_is_ptr (type)) { + has_ptr = true; + type =btf__type_by_id (btf, type->type); + }else if (btf_is_typedef (type) ||btf_is_mod (type)) { + type =btf__type_by_id (btf, type->type); + }else { +break ; + } + } +return has_ptr && type &&btf_is_func_proto (type); +} + +static void syscall_arg_fmt__cache_btf_type (struct syscall_arg_fmt *arg_fmt, +struct btf *btf,const char *type) +{ +char name[128 ]; +const char *pos; +size_t len =0 ;int id; - type =strstr (type,"enum " );if (type ==NULL )return ; - type +=5 ;// skip "enum " to get the enumeration name +if ((pos =strstr (type,"enum " )) !=NULL ) + pos +=5 ; +else if ((pos =strstr (type,"struct " )) !=NULL ) + pos +=7 ; +else if ((pos =strstr (type,"union " )) !=NULL ) + pos +=6 ; +else + pos = type; + +while (isspace (*pos)) + pos++; + +while ((isalnum (pos[len]) || pos[len] =='_' ) && len <sizeof (name) -1 ) { + name[len] = pos[len]; + len++; + } + name[len] ='\0' ; - id =btf__find_by_name (btf, type); +if (len ==0 ) +return ; + + id =btf__find_by_name (btf, name);if (id <0 )return ; arg_fmt->type =btf__type_by_id (btf, id); + arg_fmt->type_id = id; }static bool syscall_arg__strtoul_btf_enum (char *bf,size_t size,struct syscall_arg *arg,u64 *val)
@@ -1028,8 +1068,7 @@ static bool syscall_arg__strtoul_btf_type(char *bf, size_t size, struct syscall_return false;if (arg->fmt->type ==NULL ) { -// See if this is an enum -syscall_arg_fmt__cache_btf_enum (arg->fmt, btf, type); +syscall_arg_fmt__cache_btf_type (arg->fmt, btf, type); }// Now let's see if we have a BTF type resolved
@@ -1108,7 +1147,7 @@ static size_t btf_struct_scnprintf(const struct btf_type *type, struct btf *btf, }static size_t trace__btf_scnprintf (struct trace *trace,struct syscall_arg *arg,char *bf, -size_t size,int val,char *type) +size_t size,unsigned long val,char *type) {struct syscall_arg_fmt *arg_fmt = arg->fmt;
@@ -1116,8 +1155,7 @@ static size_t trace__btf_scnprintf(struct trace *trace, struct syscall_arg *arg,return 0 ;if (arg_fmt->type ==NULL ) { -// Check if this is an enum and if we have the BTF type for it. -syscall_arg_fmt__cache_btf_enum (arg_fmt, trace->btf, type); +syscall_arg_fmt__cache_btf_type (arg_fmt, trace->btf, type); }// Did we manage to find a BTF type for the syscall/tracepoint argument?
@@ -1128,13 +1166,17 @@ static size_t trace__btf_scnprintf(struct trace *trace, struct syscall_arg *arg,return btf_enum_scnprintf (arg_fmt->type, trace->btf, bf, size, val); else if (btf_is_struct(arg_fmt->type) || btf_is_union(arg_fmt->type)) return btf_struct_scnprintf(arg_fmt->type, trace->btf, bf, size, arg); + else if (btf_is_func_ptr(trace->btf, arg_fmt->type)) { + arg->val = val; +return syscall_arg__scnprintf_ksym (bf, size, arg); + } return 0; }#else // HAVE_LIBBPF_SUPPORT static size_t trace__btf_scnprintf (struct trace *trace__maybe_unused ,struct syscall_arg *arg__maybe_unused , -char *bf__maybe_unused ,size_t size__maybe_unused ,int val__maybe_unused , +char *bf__maybe_unused ,size_t size__maybe_unused ,unsigned long val__maybe_unused ,char *type__maybe_unused ) {return 0 ;
@@ -2567,7 +2609,8 @@ static size_t syscall__scnprintf_args(struct syscall *sc, char *bf, size_t size, default_scnprintf = sc->arg_fmt[arg.idx].scnprintf; -if (trace->force_btf || default_scnprintf ==NULL || default_scnprintf ==SCA_PTR ) { +if (trace->force_btf || default_scnprintf ==NULL || + default_scnprintf ==SCA_PTR || default_scnprintf ==SCA_KSYM ) { btf_printed =trace__btf_scnprintf (trace, &arg, bf + printed, size - printed, val, field->type);if (btf_printed) {
tools/perf/tests/shell/trace_btf_general.sh +1 -1
@@ -49,7 +49,7 @@ trace_test_buffer() { trace_test_struct_btf() { echo "Testing perf trace's struct augmentation" output="$(perf trace --sort-events -e clock_nanosleep --force-btf --max-events=1 -- sleep 1 2>&1)" - if ! echo "$output" | grep -q -E "^sleep/[0-9]+ clock_nanosleep\(0, 0, \{1,.*\}, 0x[0-9a-f]+\) += +[0-9]+$" + if ! echo "$output" | grep -q -E "sleep/[0-9]+ clock_nanosleep\(.*(struct __kernel_timespec|\{1,).*\) = [0-9]+" then printf "BTF struct augmentation test failed, output:\n$output\n" err=1
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH perf-tools-next v2 3/4] perf trace: Enhance BTF type formatting to symbolise kernel function pointers
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] `syscall_arg_fmt__cache_btf_type()` causes severe performance degradation due to un-cached O(N)…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] syscall_arg_fmt__cache_btf_type() causes severe performance degradation due to un-cached O(N) BTF lookups for standard C types.
- [Medium] Pointers to enums are incorrectly evaluated as enum values, leading to silent truncation and invalid enum string formatting.
- [Medium] Function pointer symbolization silently fails for inline function pointers and pointers to function prototype typedefs. --
--- Patch [3]: [PATCH perf-tools-next v2 3/4] perf trace: Enhance BTF type formatting to symbolise kernel function pointers --- commit 78dc038d874ecff66ce4494a6bee179567d6b15e
Author: Aaron Tomlin <[email protected]>
perf trace: Enhance BTF type formatting to symbolise kernel function pointers
This commit introduces btf_is_func_ptr() to inspect BTF type hierarchies to determine if a parameter is a function prototype. It generalizes BTF type caching to handle various types via syscall_arg_fmt__cache_btf_type(), allowing for automatic symbolization of kernel function pointers in perf trace.
Quoted 15 lines
tools/perf/builtin-trace.c +5
-1@@ -979,21 +979,61 @@ static size_t syscall_arg__scnprintf_getrandom_flags(char *bf, size_t size,#define SCA_GETRANDOM_FLAGS syscall_arg__scnprintf_getrandom_flags#ifdef HAVE_LIBBPF_SUPPORT -static void syscall_arg_fmt__cache_btf_enum (struct syscall_arg_fmt *arg_fmt,struct btf *btf,char *type) +static bool btf_is_func_ptr (const struct btf *btf,const struct btf_type *type) { +bool has_ptr = false; + +while (type) { +if (btf_is_ptr (type)) {
tools/perf/builtin-trace.c static size_t syscall_arg__scnprintf_getrandom_flags(char *bf, size_t size, · line 997
Quoted 10 lines
tools/perf/builtin-trace.c +10
-0+ has_ptr = true; + type =btf__type_by_id (btf, type->type); + }else if (btf_is_typedef (type) ||btf_is_mod (type)) { + type =btf__type_by_id (btf, type->type); + }else { +break ; + } + } +return has_ptr && type &&btf_is_func_proto (type); +}
[Severity: Medium]
Will this reliably detect function pointers if the pointer modifier was stripped during parsing?
If a parameter is an inline function pointer (e.g., void (*cb)(int)) or a pointer to a function prototype typedef (e.g., foo_t *cb), the string extraction in syscall_arg_fmt__cache_btf_type() strips the pointer and punctuation.
For foo_t, it resolves to a typedef of a prototype, so btf_is_func_ptr() evaluates has_ptr = false since the pointer aspect was stripped. Symbolization only seems to work for typedefs that intrinsically encapsulate the pointer.
Quoted 20 lines
tools/perf/builtin-trace.c +13
-2+ +static void syscall_arg_fmt__cache_btf_type (struct syscall_arg_fmt *arg_fmt, +struct btf *btf,const char *type) +{ +char name[128 ]; +const char *pos; +size_t len =0 ;int id; - type =strstr (type,"enum " );if (type ==NULL )return ; - type +=5 ;// skip "enum " to get the enumeration name +if ((pos =strstr (type,"enum " )) !=NULL ) + pos +=5 ; +else if ((pos =strstr (type,"struct " )) !=NULL ) + pos +=7 ; +else if ((pos =strstr (type,"union " )) !=NULL ) + pos +=6 ;
tools/perf/builtin-trace.c static size_t syscall_arg__scnprintf_getrandom_flags(char *bf, size_t size, · line 1025
Quoted 10 lines
tools/perf/builtin-trace.c +10
-0+else + pos = type; + +while (isspace (*pos)) + pos++; + +while ((isalnum (pos[len]) || pos[len] =='_' ) && len <sizeof (name) -1 ) { + name[len] = pos[len]; + len++; + }
[Severity: Medium]
Does extracting only alphanumeric characters incorrectly strip pointer modifiers and function pointer structures?
If a traced syscall takes an argument of type enum foo *, this logic extracts only the alphanumeric words, stripping the pointer modifier. This causes the parameter to resolve to the base BTF_KIND_ENUM type rather than a pointer.
tools/perf/builtin-trace.c static size_t syscall_arg__scnprintf_getrandom_flags(char *bf, size_t size, · line 1037
Quoted 13 lines
tools/perf/builtin-trace.c +6
-1+ name[len] ='\0' ; - id =btf__find_by_name (btf, type); +if (len ==0 ) +return ; + + id =btf__find_by_name (btf, name);if (id <0 )return ; arg_fmt->type =btf__type_by_id (btf, id); + arg_fmt->type_id = id; }
[Severity: High]
Does this logic lead to repeated, expensive searches if a type is not found?
If btf__find_by_name() fails to find a type (which it does for standard C types like unsigned long or void *), arg_fmt->type remains NULL.
trace__btf_scnprintf() checks if arg_fmt->type is NULL and repeatedly calls syscall_arg_fmt__cache_btf_type():
trace__btf_scnprintf() {
...
if (arg_fmt->type == NULL) {
syscall_arg_fmt__cache_btf_type(arg_fmt, trace->btf, type);
}
...
}This results in an un-cached linear search over all kernel BTF types for every single trace event when a type is not found, potentially causing a severe performance regression.
[ ... ]
Quoted 20 lines
tools/perf/builtin-trace.c +2
-3@@ -1108,7 +1147,7 @@ static size_t btf_struct_scnprintf(const struct btf_type *type, struct btf *btf, }static size_t trace__btf_scnprintf (struct trace *trace,struct syscall_arg *arg,char *bf, -size_t size,int val,char *type) +size_t size,unsigned long val,char *type) {struct syscall_arg_fmt *arg_fmt = arg->fmt;@@ -1116,8 +1155,7 @@ static size_t trace__btf_scnprintf(struct trace *trace, struct syscall_arg *arg,return 0 ;if (arg_fmt->type ==NULL ) { -// Check if this is an enum and if we have the BTF type for it. -syscall_arg_fmt__cache_btf_enum (arg_fmt, trace->btf, type); +syscall_arg_fmt__cache_btf_type (arg_fmt, trace->btf, type); }// Did we manage to find a BTF type for the syscall/tracepoint argument?
tools/perf/builtin-trace.c static size_t trace__btf_scnprintf(struct trace *trace, struct syscall_arg *arg, · line 1166
Quoted 1 line
tools/perf/builtin-trace.c +0
-0@@ -1128,13 +1166,17 @@ static size_t trace__btf_scnprintf(struct trace *trace, struct syscall_arg *arg,return btf_enum_scnprintf (arg_fmt->type, trace->btf, bf, size, val);
[Severity: Medium]
Can this truncate a pointer address to 32 bits?
Since val is an unsigned long (64 bits), passing it to btf_enum_scnprintf() which takes an int truncates the value.
If an argument of type enum foo * had its pointer modifier stripped by syscall_arg_fmt__cache_btf_type(), this will silently truncate the pointer address to 32 bits. If the truncated value happens to match an enum member, it incorrectly displays the enum string instead of the pointer address.
Quoted 9 lines
tools/perf/builtin-trace.c +4
-0else if (btf_is_struct (arg_fmt->type) ||btf_is_union (arg_fmt->type))return btf_struct_scnprintf (arg_fmt->type, trace->btf, bf, size, arg); +else if (btf_is_func_ptr (trace->btf, arg_fmt->type)) { + arg->val = val; +return syscall_arg__scnprintf_ksym (bf, size, arg); + }return 0 ; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
PATCH v2 4/4 perf tests: Add shell test for kernel symbol beautifier
In reply to Aaron TomlinAdd a dedicated shell test script, trace_ksym_beautifier.sh, to verify that 'perf trace' properly symbolises kernel virtual addresses and function pointers using both the default kallsyms beautifier (SCA_KSYM) and BTF type routing i.e., --force-btf.
--- .../perf/tests/shell/trace_ksym_beautifier.sh | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100755 tools/perf/tests/shell/trace_ksym_beautifier.sh
tools/perf/tests/shell/trace_ksym_beautifier.sh new file +52 -0
@@ -0,0 +1,52 @@ +#!/bin/bash +# perf trace kernel symbol beautifier tests +# SPDX-License-Identifier: GPL-2.0 + +err=0 +OUTPUT=$(mktemp /tmp/perf_trace_test.XXXXX) + +# shellcheck source=lib/probe.sh +. "$(dirname $0)"/lib/probe.sh +skip_if_no_perf_trace || exit 2 +[ "$(id -u)" = 0 ] || exit 2 + +cleanup() { + rm -f ${OUTPUT} +} + +trap cleanup EXIT TERM INT HUP + +test_ksym_kallsyms() { + echo "Testing perf trace kernel symbol beautifier (default kallsyms)" + perf trace -e kmem:kmalloc --max-events=1 > ${OUTPUT} 2>&1 + if ! grep -q -E "call_site: [a-zA-Z0-9_]+" ${OUTPUT} + then + printf "Default kallsyms function symbolization failed, output:\n$(cat ${OUTPUT})\n" + err=1 + fi +} + +test_ksym_btf() { + echo "Testing perf trace kernel symbol beautifier (BTF)" + if [ ! -f /sys/kernel/btf/vmlinux ]; then + echo "Skipping BTF test due to missing vmlinux BTF" + return + fi + + perf trace -e kmem:kmalloc --force-btf --max-events=1 > ${OUTPUT} 2>&1 + if ! grep -q -E "call_site: [a-zA-Z0-9_]+" ${OUTPUT} + then + printf "BTF function symbolization failed, output:\n$(cat ${OUTPUT})\n" + err=1 + fi +} + +test_ksym_kallsyms + +if [ $err = 0 ]; then + test_ksym_btf +fi + +cleanup + +exit $err
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH perf-tools-next v2 4/4] perf tests: Add shell test for kernel symbol beautifier
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider: - [Medium] The regular expression used to verify kernel symbol beautification incorrectly matches raw…
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Medium] The regular expression used to verify kernel symbol beautification incorrectly matches raw hexadecimal addresses, causing the test to falsely pass even if symbolization fails.
- [Medium] Temporary file created by mktemp is leaked on early exits.
- [Low] Unquoted $0 in dirname command substitution can cause the script to fail if the execution path contains spaces.
- [Low] Command output is directly interpolated into a printf format string, which will mangle output containing % or \ characters.
--
--- Patch [4]: [PATCH perf-tools-next v2 4/4] perf tests: Add shell test for kernel symbol beautifier --- commit 3b67c2553760890642357a39f8d2c206545b9020
Author: Aaron Tomlin <[email protected]>
perf tests: Add shell test for kernel symbol beautifier
Add a dedicated shell test script, trace_ksym_beautifier.sh, to verify that 'perf trace' properly symbolises kernel virtual addresses and function pointers using both the default kallsyms beautifier (SCA_KSYM) and BTF type routing i.e., --force-btf.
Quoted 6 lines
tools/perf/tests/shell/trace_ksym_beautifier.sh new file +0
-0
tools/perf/tests/shell/trace_ksym_beautifier.sh line 9
Quoted 9 lines
tools/perf/tests/shell/trace_ksym_beautifier.sh new file +9
-0@@ -0,0 +1,52 @@ +#!/bin/bash +# perf trace kernel symbol beautifier tests +# SPDX-License-Identifier: GPL-2.0 + +err=0 +OUTPUT=$(mktemp /tmp/perf_trace_test.XXXXX) + +# shellcheck source=lib/probe.sh +. "$(dirname $0)"/lib/probe.sh
[Severity: Low]
Does the unquoted $0 in this dirname substitution cause the script to fail if the execution path contains spaces?
tools/perf/tests/shell/trace_ksym_beautifier.sh line 11
Quoted 2 lines
tools/perf/tests/shell/trace_ksym_beautifier.sh new file +2
-0+skip_if_no_perf_trace || exit 2 +[ "$(id -u)" = 0 ] || exit 2
[Severity: Medium]
Since OUTPUT is created via mktemp before these early exit checks, and the cleanup trap isn't registered until later, do these exits leak the temporary file in /tmp?
tools/perf/tests/shell/trace_ksym_beautifier.sh line 22
Quoted 11 lines
tools/perf/tests/shell/trace_ksym_beautifier.sh new file +11
-0+ +cleanup() { + rm -f ${OUTPUT} +} + +trap cleanup EXIT TERM INT HUP + +test_ksym_kallsyms() { + echo "Testing perf trace kernel symbol beautifier (default kallsyms)" + perf trace -e kmem:kmalloc --max-events=1 > ${OUTPUT} 2>&1 + if ! grep -q -E "call_site: [a-zA-Z0-9_]+" ${OUTPUT}
[Severity: Medium]
Does this regular expression inadvertently match raw hexadecimal addresses? Because the character x and hex digits fall within the [a-zA-Z0-9_]+ character class, it appears this could falsely pass if symbolization fails and perf trace outputs un-symbolized addresses like 0xffffffff12345678.
tools/perf/tests/shell/trace_ksym_beautifier.sh line 24
Quoted 2 lines
tools/perf/tests/shell/trace_ksym_beautifier.sh new file +2
-0+ then + printf "Default kallsyms function symbolization failed, output:\n$(cat ${OUTPUT})\n"
[Severity: Low]
Could directly interpolating the command output into the printf format string cause issues? If the output contains % or \ characters, printf might interpret them as format specifiers or escape sequences.
tools/perf/tests/shell/trace_ksym_beautifier.sh line 37
Quoted 13 lines
tools/perf/tests/shell/trace_ksym_beautifier.sh new file +13
-0+ err=1 + fi +} + +test_ksym_btf() { + echo "Testing perf trace kernel symbol beautifier (BTF)" + if [ ! -f /sys/kernel/btf/vmlinux ]; then + echo "Skipping BTF test due to missing vmlinux BTF" + return + fi + + perf trace -e kmem:kmalloc --force-btf --max-events=1 > ${OUTPUT} 2>&1 + if ! grep -q -E "call_site: [a-zA-Z0-9_]+" ${OUTPUT}
[Severity: Medium]
Similar to the kallsyms test above, does this regex incorrectly match raw hex addresses when BTF symbolization fails?
tools/perf/tests/shell/trace_ksym_beautifier.sh line 39
Quoted 2 lines
tools/perf/tests/shell/trace_ksym_beautifier.sh new file +2
-0+ then + printf "BTF function symbolization failed, output:\n$(cat ${OUTPUT})\n"
[Severity: Low]
As with the previous printf, does embedding the output directly into the format string risk mangling the output if it contains format specifiers?
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Ian Rogers
reviewerQuoted 13 lines · Aaron Tomlin
When inspecting kernel execution flows using perf trace (e.g., when monitoring workqueues delayed work items, timer callbacks, etc.), tracepoint payload arguments containing raw kernel virtual addresses are currently rendered as hexadecimal values (e.g.,
0xffffffff81234567).This requires manual symbol lookups against
/proc/kallsymsor vmlinux to identify the underlying kernel function being executed.This patch series enhances perf trace by introducing kernel virtual address and function pointer symbolisation using perfs native symbol engine (i.e.,
machine__find_kernel_symbol()).
Quoted 5 lines
Before:
workqueue:workqueue_execute_end(work: 0xffffffffab2f1420, function: 0xffffffffa8046b50)
After:
workqueue:workqueue_execute_end(work: 0xffff8ac2c420f270, function: wb_update_bandwidth_workfn)This looks great and a test! I noticed Sashiko feedback on patches 2, 3 and 4. It looked appropriate and actionable. I'm looking forward to the next version and testing!
Thanks,
Ian
Quoted 66 lines
Patch 1 introduces the
syscall_arg__scnprintf_ksym()(SCA_KSYM) beautifier, which resolves virtual addresses viamachine__find_kernel_symbol(), formatting them as symbol_name+offset (or "NULL", with a graceful hexadecimal fallback upon lookup failure).Patch 2 updates event format initialisation in
syscall_arg_fmt__init_array()to automatically assignSCA_KSYMto tracepoint fields typed as function pointers (such as typedefs ending in "_func_t" or "_fn", or function prototypes matching "(*)"), as well as pointer or 64-bit address fields named "fn", "function", "callsite", or "call_site".Patch 3 extends BTF pretty-printing in
trace__btf_scnprintf()withbtf_is_func_ptr()to automatically traverse BTF type hierarchies (including nested typedefs and qualifiers) and route kernel function pointer arguments toSCA_KSYMwhen BTF metadata is available.Patch 4 adds an automated regression test script,
trace_ksym_beautifier.sh, undertools/perf/tests/shell/to verify symbolisation across both default kallsyms and BTF routing paths.Changes since v1:
- Fixed reference leak of struct map in
syscall_arg__scnprintf_ksym()by callingmap__put()prior to returning Removed unreachable and erroneous entries ("action", "callsite", "call_site", "fn", "function", "work") from syscall_arg_fmts__by_name[]- Restricted name-based
SCA_KSYMauto-assignment insyscall_arg_fmt__init_array()to pointer or 64-bit address fields, preventing misclassification of non-pointer integer fields- Updated
btf_is_func_ptr()to fully unwrap typedefs and type modifiers below pointer targets- Fixed BTF type name matching in
syscall_arg_fmt__cache_btf_type()to handle leading modifiers and strip trailing pointer asterisks before lookup- Synchronised
arg->valwith val intrace__btf_scnprintf()and widened val to unsigned long, fixing erroneous "NULL" output- Added shell test script,
tools/perf/tests/shell/trace_ksym_beautifier.sh, to verify kernel symbol beautification for both default kallsyms and BTF routing- Link to v1: https://lore.kernel.org/lkml/[email protected]/
Aaron Tomlin (4):
perf trace: Introduce kernel symbol beautifier for virtual addresses perf trace: Auto-assign kernel symbol beautifier to function pointer
fields
perf trace: Enhance BTF type formatting to symbolise kernel function
pointers
perf tests: Add shell test for kernel symbol beautifiertools/perf/builtin-trace.c | 103 ++++++++++++++++-- tools/perf/tests/shell/trace_btf_general.sh | 2 +- .../perf/tests/shell/trace_ksym_beautifier.sh | 52 +++++++++ tools/perf/trace/beauty/beauty.h | 3 + 4 files changed, 148 insertions(+), 12 deletions(-) create mode 100755 tools/perf/tests/shell/trace_ksym_beautifier.sh--
2.55.0
Quoted 20 lines · Ian Rogers
Quoted 13 lines · Aaron Tomlin
When inspecting kernel execution flows using perf trace (e.g., when monitoring workqueues delayed work items, timer callbacks, etc.), tracepoint payload arguments containing raw kernel virtual addresses are currently rendered as hexadecimal values (e.g.,
0xffffffff81234567).This requires manual symbol lookups against
/proc/kallsymsor vmlinux to identify the underlying kernel function being executed.This patch series enhances perf trace by introducing kernel virtual address and function pointer symbolisation using perfs native symbol engine (i.e.,
machine__find_kernel_symbol()).Quoted 5 lines
Before: workqueue:workqueue_execute_end(work: 0xffffffffab2f1420, function: 0xffffffffa8046b50) After: workqueue:workqueue_execute_end(work: 0xffff8ac2c420f270, function: wb_update_bandwidth_workfn)
Quoted 6 lines
This looks great and a test! I noticed Sashiko feedback on patches 2, 3 and 4. It looked appropriate and actionable. I'm looking forward to the next version and testing!
Thanks,
Ian
Hi Ian,
Thank you!
Kind regards,
Signature · 1 line
Aaron Tomlin