perf: Support the deferred unwinding infrastructure
Subsystems (8)
v16 cover perf: Support the deferred unwinding infrastructure
This is based on top of tip/perf/core commit: 6d48436560e91be85
Then I added the patches from Peter Zijlstra:
https://lore.kernel.org/all/[email protected]/
This series implements the perf interface to use deferred user space stack tracing.
The patches for the user space side should still work with this series:
https://lore.kernel.org/linux-trace-kernel/[email protected]
Patch 1 updates the deferred unwinding infrastructure. It adds a new function called: unwind_deferred_task_init(). This is used when a tracer (perf) only needs to follow a single task. The descriptor returned can be used the same way as the descriptor returned by unwind_deferred_init(), but the tracer must only use it on one task at a time.
Patch 2 adds the per task deferred stack traces to perf. It adds a new event type called PERF_RECORD_CALLCHAIN_DEFERRED that is recorded when a task is about to go back to user space and happens in a location that pages may be faulted in. It also adds a new callchain context called PERF_CONTEXT_USER_DEFERRED that is used as a place holder in a kernel callchain to append the deferred user space stack trace to.
Patch 3 adds the user stack trace context cookie in the kernel callchain right after the PERF_CONTEXT_USER_DEFERRED context so that the user space side can map the request to the deferred user space stack trace.
Patch 4 adds support for the per CPU perf events that will allow the kernel to associate each of the per CPU perf event buffers to a single application. This is needed so that when a request for a deferred stack trace happens on a task that then migrates to another CPU, it will know which CPU buffer to use to record the stack trace on. It is possible to have more than one perf user tool running and a request made by one perf tool should have the deferred trace go to the same perf tool's perf CPU event buffer. A global list of all the descriptors representing each perf tool that is using deferred stack tracing is created to manage this.
Changes since v15: https://lore.kernel.org/linux-trace-kernel/[email protected]/
- The main update was that I moved the code to do single task deferred stack tracing into the unwind code. That allowed to reuse the code for tracing all tasks, and simplified the perf code in doing so.
The first patch updates the unwind deferred code to have this infrastructure. It only added a new function:
unwind_deferred_task_init()
This is the same as unwind_deferred_init() but it is used when the tracer will only trace a single task. The descriptor returned will have its own task_work callback it will use and it allows for any number of callers, not a limited set like the "all task" deferred unwinding has.
- The new code also removed the need to expose the generation of the cookie.
Josh Poimboeuf (1):
perf: Support deferred user callchains
Steven Rostedt (3):
unwind: Add interface to allow tracing a single task
perf: Have the deferred request record the user context cookie
perf: Support deferred user callchains for per CPU events
----
include/linux/perf_event.h | 9 +- include/linux/unwind_deferred.h | 15 ++ include/uapi/linux/perf_event.h | 25 ++- kernel/bpf/stackmap.c | 4 +- kernel/events/callchain.c | 14 +- kernel/events/core.c | 362 +++++++++++++++++++++++++++++++++- kernel/unwind/deferred.c | 283 ++++++++++++++++++++++---- tools/include/uapi/linux/perf_event.h | 25 ++- 8 files changed, 686 insertions(+), 51 deletions(-)
PATCH v16 1/4 unwind: Add interface to allow tracing a single task
From: Steven Rostedt <[email protected]>
If a tracer (namely perf) is only tracing a single task, it doesn't need the full functionality of the deferred stacktrace infrastructure. That infrastructure has a limited number of users as it needs to handle multiple tracers that can trace multiple tasks at the same time, creating a multi to multi relationship.
But for a tracer that is tracing a single task, that creates a single to multi relationship (a tracer tracing a single task and a task that can have several tracers tracing it). This allows for data to be allocated at the time of initialization to let the tracer use its own task_work data structures to attach to the task.
Add a new interface called unwind_deferred_task_init() that works similar to the unwind_deferred_init(), but this interface is used when the tracer will only ever trace a single task at the same time.
The unwind_work descriptor that is initialized during this init function now has a struct callback_head field that is used to attach itself to a task_work. The work->bit for this task is set to the UNWIND_PENDING_BIT (but defined as UNWIND_TASK) to differentiate it from unwind_works that are tracing any task, as their work->bit will be one of the allocated bits in the unwind_mask.
The rest of the calls are the same. That is, the unwind_deferred_request() and unwind_deferred_cancel() are called on this, and these functions will know that this unwind_work descriptor traces a single task. Even the callback function works the same.
If a tracer were to try to use this unwind_work on multiple tasks at the same time, it will simply fail to attach to the second task if one is already pending a deferred stacktrace, and a WARN_ON is produced.
--- include/linux/unwind_deferred.h | 15 ++ kernel/unwind/deferred.c | 283 +++++++++++++++++++++++++++----- 2 files changed, 255 insertions(+), 43 deletions(-)
include/linux/unwind_deferred.h +15 -0
@@ -2,6 +2,7 @@#ifndef _LINUX_UNWIND_USER_DEFERRED_H#define _LINUX_UNWIND_USER_DEFERRED_H +#include <linux/rcuwait.h> #include <linux/task_work.h> #include <linux/unwind_user.h> #include <linux/unwind_deferred_types.h>
@@ -15,6 +16,9 @@ typedef void (*unwind_callback_t)(struct unwind_work *work,struct unwind_work {struct list_head list;unwind_callback_t func; +struct callback_head work; +struct task_struct *task; +struct rcuwait wait;int bit; };
@@ -32,11 +36,22 @@ enum { UNWIND_USED = BIT(UNWIND_USED_BIT) }; +/* +* UNWIND_PENDING is set in the task's info->unwind_mask when +* a deferred unwind is requested on that task. If the unwind +* descriptor is used only to trace a specific task, it's bit +* is the UNWIND_PENDING_BIT. This gets set as the work->bit +* and is to distinguish unwind_work descriptors that trace +* a single task from those that trace all tasks. +*/ +#define UNWIND_TASK UNWIND_PENDING_BIT +void unwind_task_init (struct task_struct *task);void unwind_task_free (struct task_struct *task);int unwind_user_faultable (struct unwind_stacktrace *trace); +int unwind_deferred_task_init (struct unwind_work *work,unwind_callback_t func);int unwind_deferred_init (struct unwind_work *work,unwind_callback_t func);int unwind_deferred_request (struct unwind_work *work,u64 *cookie);void unwind_deferred_cancel (struct unwind_work *work);
kernel/unwind/deferred.c +240 -43
378 lines not on this page · raw message
Signature · 1 line
2.50.1
PATCH v16 2/4 perf: Support deferred user callchains
In reply to Steven RostedtFrom: Josh Poimboeuf <[email protected]>
If the user fault unwind is available (the one that will be used for sframes), have perf be able to utilize it. Currently all user stack traces are done at the request site. This mostly happens in interrupt or NMI context where user space is only accessible if it is currently present in memory. It is possible that the user stack was swapped out and is not present, but mostly the use of sframes will require faulting in user pages which will not be possible from interrupt context. Instead, add a frame work that will delay the reading of the user space stack until the task goes back to user space where faulting in pages is possible. This is also advantageous as the user space stack doesn't change while in the kernel, and this will also remove duplicate entries of user space stacks for a long running system call being profiled.
A new perf context is created called PERF_CONTEXT_USER_DEFERRED. It is added to the kernel callchain, usually when an interrupt or NMI is triggered (but can be added to any callchain). When a deferred unwind is required, it uses the new deferred unwind infrastructure.
When tracing a single task and a user stack trace is required, perf will call unwind_deferred_request(). This will trigger a task_work that on task kernel space exit will call the perf function perf_event_deferred_task() with the user stacktrace and a cookie (an identifier for that stack trace).
This user stack trace will go into a new perf type called PERF_RECORD_CALLCHAIN_DEFERRED. The perf user space will need to attach this stack trace to each of the previous kernel callchains for that task with the PERF_CONTEXT_USER_DEFERRED context in them.
---
Changes since v15: https://lore.kernel.org/[email protected]
- Peter Zijlstra pointed out that the code mostly duplicated the code of the unwind infrastructure, and had the same bugs as it had. The unwind infrastructure was updated to allow a tracer to use it for a single task. The perf code now uses that which greatly simplified this version over the previous one.
include/linux/perf_event.h | 5 +- include/uapi/linux/perf_event.h | 20 ++++- kernel/bpf/stackmap.c | 4 +- kernel/events/callchain.c | 11 ++- kernel/events/core.c | 110 +++++++++++++++++++++++++- tools/include/uapi/linux/perf_event.h | 20 ++++- 6 files changed, 162 insertions(+), 8 deletions(-)
include/linux/perf_event.h +4 -1
@@ -53,6 +53,7 @@#include <linux/security.h> #include <linux/static_call.h> #include <linux/lockdep.h> +#include <linux/unwind_deferred.h> #include <asm/local.h>
@@ -880,6 +881,8 @@ struct perf_event {struct callback_head pending_task;unsigned int pending_work; +struct unwind_work unwind_work; +atomic_t event_limit;/* address range filters */
@@ -1720,7 +1723,7 @@ extern void perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pextern void perf_callchain_kernel (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs);extern struct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user, -u32 max_stack,bool crosstask,bool add_mark); +u32 max_stack,bool crosstask,bool add_mark,bool defer_user);extern int get_callchain_buffers (int max_stack);extern void put_callchain_buffers (void );extern struct perf_callchain_entry *get_callchain_entry (int *rctx);
include/uapi/linux/perf_event.h +19 -1
@@ -463,7 +463,8 @@ struct perf_event_attr { inherit_thread : 1, /* children only inherit if cloned with CLONE_THREAD */ remove_on_exec : 1, /* event is removed from task on exec */ sigtrap : 1, /* send synchronous SIGTRAP on event */ - __reserved_1 : 26; + defer_callchain: 1, /* generate PERF_RECORD_CALLCHAIN_DEFERRED records */ + __reserved_1 : 25;union {__u32 wakeup_events;/* wake up every n events */
@@ -1239,6 +1240,22 @@ enum perf_event_type {*/ PERF_RECORD_AUX_OUTPUT_HW_ID =21 , +/* +* This user callchain capture was deferred until shortly before +* returning to user space. Previous samples would have kernel +* callchains only and they need to be stitched with this to make full +* callchains. +* +* struct { +* struct perf_event_header header; +* u64 cookie; +* u64 nr; +* u64 ips[nr]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_CALLCHAIN_DEFERRED =22 , + PERF_RECORD_MAX, /* non-ABI */ };
@@ -1269,6 +1286,7 @@ enum perf_callchain_context {PERF_CONTEXT_HV = (__u64)-32 ,PERF_CONTEXT_KERNEL = (__u64)-128 ,PERF_CONTEXT_USER = (__u64)-512 , +PERF_CONTEXT_USER_DEFERRED = (__u64)-640 ,PERF_CONTEXT_GUEST = (__u64)-2048 , PERF_CONTEXT_GUEST_KERNEL = (__u64)-2176,
kernel/bpf/stackmap.c +2 -2
@@ -315,7 +315,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, max_depth = sysctl_perf_event_max_stack; trace =get_perf_callchain (regs, kernel, user, max_depth, - false, false); + false, false, false);if (unlikely (!trace))/* couldn't fetch the stack trace */
@@ -452,7 +452,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, trace =get_callchain_entry_for_task (task, max_depth); else trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask, false); + crosstask, false, false);if (unlikely (!trace) || trace->nr < skip) {if (may_fault)
kernel/events/callchain.c +10 -1
@@ -218,7 +218,7 @@ static void fixup_uretprobe_trampoline_entries(struct perf_callchain_entry *entrstruct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user, -u32 max_stack,bool crosstask,bool add_mark) +u32 max_stack,bool crosstask,bool add_mark,bool defer_user) {struct perf_callchain_entry *entry;struct perf_callchain_entry_ctx ctx;
@@ -251,6 +251,15 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, regs =task_pt_regs (current); } +if (defer_user) { +/* +* Foretell the coming of PERF_RECORD_CALLCHAIN_DEFERRED +* which can be stitched to this one. +*/ +perf_callchain_store_context (&ctx,PERF_CONTEXT_USER_DEFERRED ); + goto exit_put; + } +if (add_mark)perf_callchain_store_context (&ctx,PERF_CONTEXT_USER );
kernel/events/core.c +108 -2
146 lines not on this page · raw message
tools/include/uapi/linux/perf_event.h +19 -1
@@ -463,7 +463,8 @@ struct perf_event_attr { inherit_thread : 1, /* children only inherit if cloned with CLONE_THREAD */ remove_on_exec : 1, /* event is removed from task on exec */ sigtrap : 1, /* send synchronous SIGTRAP on event */ - __reserved_1 : 26; + defer_callchain: 1, /* generate PERF_RECORD_CALLCHAIN_DEFERRED records */ + __reserved_1 : 25;union {__u32 wakeup_events;/* wake up every n events */
@@ -1239,6 +1240,22 @@ enum perf_event_type {*/ PERF_RECORD_AUX_OUTPUT_HW_ID =21 , +/* +* This user callchain capture was deferred until shortly before +* returning to user space. Previous samples would have kernel +* callchains only and they need to be stitched with this to make full +* callchains. +* +* struct { +* struct perf_event_header header; +* u64 cookie; +* u64 nr; +* u64 ips[nr]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_CALLCHAIN_DEFERRED =22 , + PERF_RECORD_MAX, /* non-ABI */ };
@@ -1269,6 +1286,7 @@ enum perf_callchain_context {PERF_CONTEXT_HV = (__u64)-32 ,PERF_CONTEXT_KERNEL = (__u64)-128 ,PERF_CONTEXT_USER = (__u64)-512 , +PERF_CONTEXT_USER_DEFERRED = (__u64)-640 ,PERF_CONTEXT_GUEST = (__u64)-2048 , PERF_CONTEXT_GUEST_KERNEL = (__u64)-2176,
Signature · 1 line
2.50.1
PATCH v16 3/4 perf: Have the deferred request record the user context cookie
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
When a request to have a deferred unwind is made, have the cookie associated to the user context recorded in the event that represents that request. It is added after the PERF_CONTEXT_USER_DEFERRED in the callchain. That perf context is a marker of where to add the associated user space stack trace in the callchain. Adding the cookie after that marker will not affect the appending of the callchain as it will be overwritten by the user space stack in the perf tool.
The cookie will be used to match the cookie that is saved when the deferred callchain is recorded. The perf tool will be able to use the cooking saved at the request to know if the callchain that was recorded when the task goes back to user space is for that event. If there were dropped events after the request was made where it dropped the calltrace that happened when the task went back to user space and then came back into the kernel and a new request was dropped, but then the record started again and it recorded a new callchain going back to user space, this callchain would not be for the initial request. The cookie matching will prevent this scenario from happening.
The cookie prevents:
record kernel stack trace with PERF_CONTEXT_USER_DEFERRED
[ dropped events starts here ]
record user stack trace - DROPPED
[enters user space ]
[exits user space back to the kernel ]
record kernel stack trace with PERF_CONTEXT_USER_DEFERRED - DROPPED!
[ events stop being dropped here ]
record user stack trace
Without a differentiating "cookie" identifier, the user space tool will incorrectly attach the last recorded user stack trace to the first kernel stack trace with the PERF_CONTEXT_USER_DEFERRED, as using the TID is not enough to identify this situation.
--- include/linux/perf_event.h | 2 +- include/uapi/linux/perf_event.h | 5 +++++ kernel/bpf/stackmap.c | 4 ++-- kernel/events/callchain.c | 9 ++++++--- kernel/events/core.c | 12 ++++++------ tools/include/uapi/linux/perf_event.h | 5 +++++ 6 files changed, 25 insertions(+), 12 deletions(-)
include/linux/perf_event.h +1 -1
@@ -1723,7 +1723,7 @@ extern void perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pextern void perf_callchain_kernel (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs);extern struct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user, -u32 max_stack,bool crosstask,bool add_mark,bool defer_user); +u32 max_stack,bool crosstask,bool add_mark,u64 defer_cookie);extern int get_callchain_buffers (int max_stack);extern void put_callchain_buffers (void );extern struct perf_callchain_entry *get_callchain_entry (int *rctx);
include/uapi/linux/perf_event.h +5 -0
@@ -1282,6 +1282,11 @@ enum perf_bpf_event_type {#define PERF_MAX_STACK_DEPTH 127#define PERF_MAX_CONTEXTS_PER_STACK 8 +/* +* The PERF_CONTEXT_USER_DEFERRED has two items (context and cookie) +*/ +#define PERF_DEFERRED_ITEMS 2 +enum perf_callchain_context {PERF_CONTEXT_HV = (__u64)-32 ,PERF_CONTEXT_KERNEL = (__u64)-128 ,
kernel/bpf/stackmap.c +2 -2
@@ -315,7 +315,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, max_depth = sysctl_perf_event_max_stack; trace =get_perf_callchain (regs, kernel, user, max_depth, - false, false, false); + false, false,0 );if (unlikely (!trace))/* couldn't fetch the stack trace */
@@ -452,7 +452,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, trace =get_callchain_entry_for_task (task, max_depth); else trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask, false, false); + crosstask, false,0 );if (unlikely (!trace) || trace->nr < skip) {if (may_fault)
kernel/events/callchain.c +6 -3
@@ -218,7 +218,7 @@ static void fixup_uretprobe_trampoline_entries(struct perf_callchain_entry *entrstruct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user, -u32 max_stack,bool crosstask,bool add_mark,bool defer_user) +u32 max_stack,bool crosstask,bool add_mark,u64 defer_cookie) {struct perf_callchain_entry *entry;struct perf_callchain_entry_ctx ctx;
@@ -251,12 +251,15 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, regs =task_pt_regs (current); } -if (defer_user) { +if (defer_cookie) {/* * Foretell the coming of PERF_RECORD_CALLCHAIN_DEFERRED -* which can be stitched to this one. +* which can be stitched to this one, and add +* the cookie after it (it will be cut off when the +* user stack is copied to the callchain). */ perf_callchain_store_context (&ctx,PERF_CONTEXT_USER_DEFERRED ); +perf_callchain_store_context (&ctx, defer_cookie); goto exit_put; }
kernel/events/core.c +6 -6
@@ -8262,10 +8262,9 @@ static struct perf_callchain_entry __empty_callchain = { .nr = 0, };* 0 : if it performed the queuing * < 0 : if it did not get queued. */ -static int deferred_request (struct perf_event *event) +static int deferred_request (struct perf_event *event,u64 *defer_cookie) {struct unwind_work *work = &event->unwind_work; -u64 cookie;/* Only defer for task events */ if (!event->ctx->task)
@@ -8275,7 +8274,7 @@ static int deferred_request(struct perf_event *event) !user_mode(task_pt_regs(current)))return -EINVAL ; -return unwind_deferred_request (work, &cookie); +return unwind_deferred_request (work, defer_cookie); } struct perf_callchain_entry *
@@ -8288,6 +8287,7 @@ perf_callchain(struct perf_event *event, struct pt_regs *regs)bool crosstask = event->ctx->task && event->ctx->task != current;const u32 max_stack = event->attr.sample_max_stack;struct perf_callchain_entry *callchain; +u64 defer_cookie =0 ;bool defer_user =IS_ENABLED (CONFIG_UNWIND_USER ) && user && event->attr.defer_callchain;
@@ -8302,15 +8302,15 @@ perf_callchain(struct perf_event *event, struct pt_regs *regs)return &__empty_callchain;if (defer_user) { -int ret =deferred_request (event); +int ret =deferred_request (event, &defer_cookie);if (!ret)local_inc (&event->ctx->nr_no_switch_fast);else if (ret <0 ) - defer_user = false; + defer_cookie =0 ; } callchain =get_perf_callchain (regs, kernel, user, max_stack, - crosstask, true, defer_user); + crosstask, true, defer_cookie);return callchain ?: &__empty_callchain; }
tools/include/uapi/linux/perf_event.h +5 -0
@@ -1282,6 +1282,11 @@ enum perf_bpf_event_type {#define PERF_MAX_STACK_DEPTH 127#define PERF_MAX_CONTEXTS_PER_STACK 8 +/* +* The PERF_CONTEXT_USER_DEFERRED has two items (context and cookie) +*/ +#define PERF_DEFERRED_ITEMS 2 +enum perf_callchain_context {PERF_CONTEXT_HV = (__u64)-32 ,PERF_CONTEXT_KERNEL = (__u64)-128 ,
Signature · 1 line
2.50.1
PATCH v16 4/4 perf: Support deferred user callchains for per CPU events
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The deferred unwinder works fine for task events (events that trace only a specific task), as it can use a task_work from an interrupt or NMI and when the task goes back to user space it will call the event's callback to do the deferred unwinding.
But for per CPU events things are not so simple. When a per CPU event wants a deferred unwinding to occur, it cannot simply use a task_work as there's a many to many relationship. If the task migrates and another task is scheduled in where the per CPU event wants a deferred unwinding to occur on that task as well, and the task that migrated to another CPU has that CPU's event want to unwind it too, each CPU may need unwinding from more than one task, and each task may have requests from many CPUs.
The main issue is that from the kernel point of view, there's currently nothing that associates a per CPU event for one CPU to the per CPU events that cover the other CPUs for a given process. To the kernel, they are all just individual event buffers. This is problematic if a delayed request is made on one CPU and the task migrates to another CPU where the delayed user stack trace will be performed. The kernel needs to know which CPU buffer to add it to that belongs to the same process that initiated the deferred request.
To solve this, when a per CPU event is created that has defer_callchain attribute set, it will do a lookup from a global list (unwind_deferred_list), for a perf_unwind_deferred descriptor that has the id that matches the PID of the current task's group_leader. (The process ID for all the threads of a process)
If it is not found, then it will create one and add it to the global list. This descriptor contains an array of all possible CPUs, where each element is a perf_unwind_cpu descriptor.
The perf_unwind_cpu descriptor has a list of all the per CPU events that is tracing the matching CPU that corresponds to its index in the array, where the events belong to a task that has the same group_leader. It also has a processing bit and rcuwait to handle removal.
For each occupied perf_unwind_cpu descriptor in the array, the perf_deferred_unwind descriptor increments its nr_cpu_events. When a perf_unwind_cpu descriptor is empty, the nr_cpu_events is decremented. This is used to know when to free the perf_deferred_unwind descriptor, as when it becomes empty, it is no longer referenced.
Finally, the perf_deferred_unwind descriptor has an id that holds the PID of the group_leader for the tasks that the events were created by.
When a second (or more) per CPU event is created where the perf_deferred_unwind descriptor already exists, it just adds itself to the perf_unwind_cpu array of that descriptor. Updating the necessary counter. This is used to map different per CPU events to each other based on their group leader PID.
Each of these perf_deferred_unwind descriptors have a unwind_work that registers with the deferred unwind infrastructure via unwind_deferred_init(), where it also registers a callback to perf_event_deferred_cpu().
Now when a per CPU event requests a deferred unwinding, it calls unwind_deferred_request() with the associated perf_deferred_unwind descriptor. It is expected that the program that uses this has events on all CPUs, as the deferred trace may not be called on the CPU event that requested it. That is, the task may migrate and its user stack trace will be recorded on the CPU event of the CPU that it exits back to user space on.
--- include/linux/perf_event.h | 4 + kernel/events/core.c | 260 ++++++++++++++++++++++++++++++++++++- 2 files changed, 260 insertions(+), 4 deletions(-)
include/linux/perf_event.h +4 -0
@@ -733,6 +733,7 @@ struct swevent_hlist {struct bpf_prog ;struct perf_cgroup ;struct perf_buffer ; +struct perf_unwind_deferred ;struct pmu_event_list {raw_spinlock_t lock;
@@ -883,6 +884,9 @@ struct perf_event {struct unwind_work unwind_work; +struct perf_unwind_deferred *unwind_deferred; +struct list_head unwind_list; +atomic_t event_limit;/* address range filters */
kernel/events/core.c +256 -4
299 lines not on this page · raw message
Signature · 1 line
2.50.1
Peter Zijlstra
maintainerQuoted 9 lines · Steven Rostedt
include/linux/perf_event.h | 9 +- include/linux/unwind_deferred.h | 15 ++ include/uapi/linux/perf_event.h | 25 ++- kernel/bpf/stackmap.c | 4 +- kernel/events/callchain.c | 14 +- kernel/events/core.c | 362 +++++++++++++++++++++++++++++++++- kernel/unwind/deferred.c | 283 ++++++++++++++++++++++---- tools/include/uapi/linux/perf_event.h | 25 ++- 8 files changed, 686 insertions(+), 51 deletions(-)
After staring at this some, I mostly threw it all out and wrote the below.
I also have some hackery on the userspace patches to go along with this, and it all sits in my unwind/cleanup branch.
Trouble is, pretty much every unwind is 510 entries long -- this cannot be right. I'm sure there's a silly mistake in unwind/user.c but I'm too tired to find it just now. I'll try again tomorrow.
--- include/linux/perf_event.h | 2 include/linux/unwind_deferred.h | 12 ----- include/linux/unwind_deferred_types.h | 13 +++++ include/uapi/linux/perf_event.h | 21 ++++++++- kernel/bpf/stackmap.c | 4 - kernel/events/callchain.c | 14 +++++- kernel/events/core.c | 79 +++++++++++++++++++++++++++++++++- tools/include/uapi/linux/perf_event.h | 21 ++++++++- 8 files changed, 146 insertions(+), 20 deletions(-)
include/linux/perf_event.h +1 -1
@@ -1720,7 +1720,7 @@ extern void perf_callchain_user(struct pextern void perf_callchain_kernel (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs);extern struct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user, -u32 max_stack,bool crosstask,bool add_mark); +u32 max_stack,bool crosstask,bool add_mark,u64 defer_cookie);extern int get_callchain_buffers (int max_stack);extern void put_callchain_buffers (void );extern struct perf_callchain_entry *get_callchain_entry (int *rctx);
include/linux/unwind_deferred.h +0 -12
@@ -6,18 +6,6 @@#include <linux/unwind_user.h> #include <linux/unwind_deferred_types.h> -struct unwind_work ; - -typedef void (*unwind_callback_t )(struct unwind_work *work, -struct unwind_stacktrace *trace, -u64 cookie); - -struct unwind_work { -struct list_head list; -unwind_callback_t func; -int bit; -}; -#ifdef CONFIG_UNWIND_USER enum {
include/linux/unwind_deferred_types.h +13 -0
@@ -39,4 +39,17 @@ struct unwind_task_info { union unwind_task_id id; }; +struct unwind_work ; +struct unwind_stacktrace ; + +typedef void (*unwind_callback_t )(struct unwind_work *work, +struct unwind_stacktrace *trace, +u64 cookie); + +struct unwind_work { +struct list_head list; +unwind_callback_t func; +int bit; +}; +#endif /* _LINUX_UNWIND_USER_DEFERRED_TYPES_H */
include/uapi/linux/perf_event.h +20 -1
@@ -463,7 +463,9 @@ struct perf_event_attr { inherit_thread : 1, /* children only inherit if cloned with CLONE_THREAD */ remove_on_exec : 1, /* event is removed from task on exec */ sigtrap : 1, /* send synchronous SIGTRAP on event */ - __reserved_1 : 26; + defer_callchain: 1, /* request PERF_RECORD_CALLCHAIN_DEFERRED records */ + defer_output : 1, /* output PERF_RECORD_CALLCHAIN_DEFERRED records */ + __reserved_1 : 24;union {__u32 wakeup_events;/* wake up every n events */
@@ -1239,6 +1241,22 @@ enum perf_event_type {*/ PERF_RECORD_AUX_OUTPUT_HW_ID =21 , +/* +* This user callchain capture was deferred until shortly before +* returning to user space. Previous samples would have kernel +* callchains only and they need to be stitched with this to make full +* callchains. +* +* struct { +* struct perf_event_header header; +* u64 cookie; +* u64 nr; +* u64 ips[nr]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_CALLCHAIN_DEFERRED =22 , + PERF_RECORD_MAX, /* non-ABI */ };
@@ -1269,6 +1287,7 @@ enum perf_callchain_context {PERF_CONTEXT_HV = (__u64)-32 ,PERF_CONTEXT_KERNEL = (__u64)-128 ,PERF_CONTEXT_USER = (__u64)-512 , +PERF_CONTEXT_USER_DEFERRED = (__u64)-640 ,PERF_CONTEXT_GUEST = (__u64)-2048 , PERF_CONTEXT_GUEST_KERNEL = (__u64)-2176,
kernel/bpf/stackmap.c +2 -2
@@ -315,7 +315,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_re max_depth = sysctl_perf_event_max_stack; trace =get_perf_callchain (regs, kernel, user, max_depth, - false, false); + false, false,0 );if (unlikely (!trace))/* couldn't fetch the stack trace */
@@ -452,7 +452,7 @@ static long __bpf_get_stack(struct pt_re trace =get_callchain_entry_for_task (task, max_depth); else trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask, false); + crosstask, false,0 );if (unlikely (!trace) || trace->nr < skip) {if (may_fault)
kernel/events/callchain.c +13 -1
@@ -218,7 +218,7 @@ static void fixup_uretprobe_trampoline_estruct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user, -u32 max_stack,bool crosstask,bool add_mark) +u32 max_stack,bool crosstask,bool add_mark,u64 defer_cookie) {struct perf_callchain_entry *entry;struct perf_callchain_entry_ctx ctx;
@@ -251,6 +251,18 @@ get_perf_callchain(struct pt_regs *regs, regs =task_pt_regs (current); } +if (defer_cookie) { +/* +* Foretell the coming of PERF_RECORD_CALLCHAIN_DEFERRED +* which can be stitched to this one, and add +* the cookie after it (it will be cut off when the +* user stack is copied to the callchain). +*/ +perf_callchain_store_context (&ctx,PERF_CONTEXT_USER_DEFERRED ); +perf_callchain_store_context (&ctx, defer_cookie); + goto exit_put; + } +if (add_mark)perf_callchain_store_context (&ctx,PERF_CONTEXT_USER );
kernel/events/core.c +77 -2
117 lines not on this page · raw message
tools/include/uapi/linux/perf_event.h +20 -1
@@ -463,7 +463,9 @@ struct perf_event_attr { inherit_thread : 1, /* children only inherit if cloned with CLONE_THREAD */ remove_on_exec : 1, /* event is removed from task on exec */ sigtrap : 1, /* send synchronous SIGTRAP on event */ - __reserved_1 : 26; + defer_callchain: 1, /* request PERF_RECORD_CALLCHAIN_DEFERRED records */ + defer_output : 1, /* output PERF_RECORD_CALLCHAIN_DEFERRED records */ + __reserved_1 : 24;union {__u32 wakeup_events;/* wake up every n events */
@@ -1239,6 +1241,22 @@ enum perf_event_type {*/ PERF_RECORD_AUX_OUTPUT_HW_ID =21 , +/* +* This user callchain capture was deferred until shortly before +* returning to user space. Previous samples would have kernel +* callchains only and they need to be stitched with this to make full +* callchains. +* +* struct { +* struct perf_event_header header; +* u64 cookie; +* u64 nr; +* u64 ips[nr]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_CALLCHAIN_DEFERRED =22 , + PERF_RECORD_MAX, /* non-ABI */ };
@@ -1269,6 +1287,7 @@ enum perf_callchain_context {PERF_CONTEXT_HV = (__u64)-32 ,PERF_CONTEXT_KERNEL = (__u64)-128 ,PERF_CONTEXT_USER = (__u64)-512 , +PERF_CONTEXT_USER_DEFERRED = (__u64)-640 ,PERF_CONTEXT_GUEST = (__u64)-2048 , PERF_CONTEXT_GUEST_KERNEL = (__u64)-2176,
Steven Rostedt
maintainerQuoted 11 lines · On Thu, 23 Oct 2025 17:00:02 +0200 Peter Zijlstra <[email protected]> wrote:
kernel/events/core.c +11
-0+/* Deferred unwinding callback for task specific events */ +static void perf_unwind_deferred_callback (struct unwind_work *work, +struct unwind_stacktrace *trace,u64 cookie) +{ +struct perf_callchain_deferred_event deferred_event = { + .trace = trace, + .event = { + .header = { + .type =PERF_RECORD_CALLCHAIN_DEFERRED , + .misc =PERF_RECORD_MISC_USER , + .size = sizeof(deferred_event.event) +
kernel/events/core.c void perf_event_bpf_event(struct bpf_pro · line 10077
Quoted 10 lines
kernel/events/core.c +10
-0+ (trace->nr * sizeof(u64)), + }, + .cookie = cookie, + .nr = trace->nr, + }, + }; + +perf_iterate_sb (perf_callchain_deferred_output, &deferred_event,NULL ); +} +
So "perf_iterate_sb()" was the key point I was missing. I'm guessing it's basically a demultiplexer that distributes events to all the requestors?
If I had know this, I would have done it completely different.
-- Steve
Peter Zijlstra
maintainerQuoted 19 lines · Steven Rostedt
Quoted 16 lines · On Thu, 23 Oct 2025 17:00:02 +0200 Peter Zijlstra <[email protected]> wrote:
kernel/events/core.c +16
-0+/* Deferred unwinding callback for task specific events */ +static void perf_unwind_deferred_callback (struct unwind_work *work, +struct unwind_stacktrace *trace,u64 cookie) +{ +struct perf_callchain_deferred_event deferred_event = { + .trace = trace, + .event = { + .header = { + .type =PERF_RECORD_CALLCHAIN_DEFERRED , + .misc =PERF_RECORD_MISC_USER , + .size =sizeof (deferred_event.event) + + (trace->nr *sizeof (u64)), + }, + .cookie = cookie, + .nr = trace->nr, + },
Quoted 8 lines
kernel/events/core.c void perf_event_bpf_event(struct bpf_pro · line 10077
Quoted 5 lines
kernel/events/core.c +5
-0+ }; + +perf_iterate_sb (perf_callchain_deferred_output, &deferred_event,NULL ); +} +So "
perf_iterate_sb()" was the key point I was missing. I'm guessing it's basically a demultiplexer that distributes events to all the requestors?
A superset. Basically every event in the relevant context that 'wants' it.
It is what we use for all traditional side-band events (hence the _sb naming) like mmap, task creation/exit, etc.
I was under the impression the perf tool would create one software dummy event to listen specifically for these events per buffer, but alas, when I looked at the tool this does not appear to be the case.
As a result it is possible to receive these events multiple times. And since that is a problem that needs to be solved anyway, I didn't think it 'relevant' in this case.
Quoted 1 line
If I had know this, I would have done it completely different.
I did do mention it here:
https://lkml.kernel.org/r/[email protected]
Anyway, no worries. Onwards to figuring out WTF the unwinder doesn't seem to terminate properly.
Peter Zijlstra
maintainerArnaldo, Namhyung,
Quoted 9 lines · Peter Zijlstra
Quoted 2 lines
So "
perf_iterate_sb()" was the key point I was missing. I'm guessing it's basically a demultiplexer that distributes events to all the requestors?A superset. Basically every event in the relevant context that 'wants' it.
It is what we use for all traditional side-band events (hence the
_sbnaming) like mmap, task creation/exit, etc.
Quoted 7 lines
I was under the impression the perf tool would create one software dummy event to listen specifically for these events per buffer, but alas, when I looked at the tool this does not appear to be the case.
As a result it is possible to receive these events multiple times. And since that is a problem that needs to be solved anyway, I didn't think it 'relevant' in this case.
When I use:
perf record -ag -e cycles -e instructions
I get:
# event : name = cycles, , id = { }, type = 0 (PERF_TYPE_HARDWARE), size = 136, config = 0 (PERF_COUNT_HW_CPU_CYCLES), { sample_period, sample_freq } = 2000, sample_type = IP|TID|TIME|CALLCHAIN|CPU|PERIOD|IDENTIFIER, read_format = ID|LOST, disabled = 1, freq = 1, sample_id_all = 1, defer_callchain = 1
# event : name = instructions, , id = { }, type = 0 (PERF_TYPE_HARDWARE), size = 136, config = 0x1 (PERF_COUNT_HW_INSTRUCTIONS), { sample_period, sample_freq } = 2000, sample_type = IP|TID|TIME|CALLCHAIN|CPU|PERIOD|IDENTIFIER, read_format = ID|LOST, disabled = 1, freq = 1, sample_id_all = 1, defer_callchain = 1
# event : name = dummy:u, , id = { }, type = 1 (PERF_TYPE_SOFTWARE), size = 136, config = 0x9 (PERF_COUNT_SW_DUMMY), { sample_period, sample_freq } = 1, sample_type = IP|TID|TIME|CPU|IDENTIFIER, read_format = ID|LOST, exclude_kernel = 1, exclude_hv = 1, mmap = 1, comm = 1, task = 1, sample_id_all = 1, exclude_guest = 1, mmap2 = 1, comm_exec = 1, ksymbol = 1, bpf_event = 1, build_id = 1, defer_output = 1And we have this dummy event I spoke of above; and it has defer_output set, none of the others do. This is what I expected.
*However*, when I use:
perf record -g -e cycles -e instruction
I get:
# event : name = cycles, , id = { }, type = 0 (PERF_TYPE_HARDWARE), size = 136, config = 0 (PERF_COUNT_HW_CPU_CYCLES), { sample_period, sample_freq } = 2000, sample_type = IP|TID|TIME|CALLCHAIN|ID|PERIOD, read_format = ID|LOST, disabled = 1, inherit = 1, mmap = 1, comm = 1, freq = 1, enable_on_exec = 1, task = 1, sample_id_all = 1, mmap2 = 1, comm_exec = 1, ksymbol = 1, bpf_event = 1, build_id = 1, defer_callchain = 1, defer_output = 1
# event : name = instructions, , id = { }, type = 0 (PERF_TYPE_HARDWARE), size = 136, config = 0x1 (PERF_COUNT_HW_INSTRUCTIONS), { sample_period, sample_freq } = 2000, sample_type = IP|TID|TIME|CALLCHAIN|ID|PERIOD, read_format = ID|LOST, disabled = 1, inherit = 1, freq = 1, enable_on_exec = 1, sample_id_all = 1, defer_callchain = 1Which doesn't have a dummy event. Notably the first real event has defer_output set (and all the other sideband stuff like mmap, comm, etc.).
Is there a reason the !cpu mode doesn't have the dummy event? Anyway, it should all work, just unexpected inconsistency that confused me.
Namhyung Kim
maintainerHi Peter,
Quoted 44 lines · Peter Zijlstra
Arnaldo, Namhyung,
Quoted 9 lines · Peter Zijlstra
Quoted 2 lines
So "
perf_iterate_sb()" was the key point I was missing. I'm guessing it's basically a demultiplexer that distributes events to all the requestors?A superset. Basically every event in the relevant context that 'wants' it.
It is what we use for all traditional side-band events (hence the
_sbnaming) like mmap, task creation/exit, etc.Quoted 7 lines
I was under the impression the perf tool would create one software dummy event to listen specifically for these events per buffer, but alas, when I looked at the tool this does not appear to be the case.
As a result it is possible to receive these events multiple times. And since that is a problem that needs to be solved anyway, I didn't think it 'relevant' in this case.
When I use:
perf record -ag -e cycles -e instructions
I get:
# event : name = cycles, , id = { }, type = 0 (PERF_TYPE_HARDWARE), size = 136, config = 0 (PERF_COUNT_HW_CPU_CYCLES), { sample_period, sample_freq } = 2000, sample_type = IP|TID|TIME|CALLCHAIN|CPU|PERIOD|IDENTIFIER, read_format = ID|LOST, disabled = 1, freq = 1, sample_id_all = 1, defer_callchain = 1 # event : name = instructions, , id = { }, type = 0 (PERF_TYPE_HARDWARE), size = 136, config = 0x1 (PERF_COUNT_HW_INSTRUCTIONS), { sample_period, sample_freq } = 2000, sample_type = IP|TID|TIME|CALLCHAIN|CPU|PERIOD|IDENTIFIER, read_format = ID|LOST, disabled = 1, freq = 1, sample_id_all = 1, defer_callchain = 1 # event : name = dummy:u, , id = { }, type = 1 (PERF_TYPE_SOFTWARE), size = 136, config = 0x9 (PERF_COUNT_SW_DUMMY), { sample_period, sample_freq } = 1, sample_type = IP|TID|TIME|CPU|IDENTIFIER, read_format = ID|LOST, exclude_kernel = 1, exclude_hv = 1, mmap = 1, comm = 1, task = 1, sample_id_all = 1, exclude_guest = 1, mmap2 = 1, comm_exec = 1, ksymbol = 1, bpf_event = 1, build_id = 1, defer_output = 1And we have this dummy event I spoke of above; and it has
defer_outputset, none of the others do. This is what I expected.*However*, when I use:
perf record -g -e cycles -e instruction
I get:
# event : name = cycles, , id = { }, type = 0 (PERF_TYPE_HARDWARE), size = 136, config = 0 (PERF_COUNT_HW_CPU_CYCLES), { sample_period, sample_freq } = 2000, sample_type = IP|TID|TIME|CALLCHAIN|ID|PERIOD, read_format = ID|LOST, disabled = 1, inherit = 1, mmap = 1, comm = 1, freq = 1, enable_on_exec = 1, task = 1, sample_id_all = 1, mmap2 = 1, comm_exec = 1, ksymbol = 1, bpf_event = 1, build_id = 1, defer_callchain = 1, defer_output = 1 # event : name = instructions, , id = { }, type = 0 (PERF_TYPE_HARDWARE), size = 136, config = 0x1 (PERF_COUNT_HW_INSTRUCTIONS), { sample_period, sample_freq } = 2000, sample_type = IP|TID|TIME|CALLCHAIN|ID|PERIOD, read_format = ID|LOST, disabled = 1, inherit = 1, freq = 1, enable_on_exec = 1, sample_id_all = 1, defer_callchain = 1
Quoted 6 lines
Which doesn't have a dummy event. Notably the first real event has
defer_outputset (and all the other sideband stuff like mmap, comm, etc.).Is there a reason the !cpu mode doesn't have the dummy event? Anyway, it should all work, just unexpected inconsistency that confused me.
Right, I don't remember why. I think there's no reason doing it for system wide mode only.
Adrian, do you have any idea? I have a vague memory that you worked on this in the past.
Thanks,
Namhyung
Peter Zijlstra
maintainerQuoted 3 lines · Peter Zijlstra
Trouble is, pretty much every unwind is 510 entries long -- this cannot be right. I'm sure there's a silly mistake in
unwind/user.cbut I'm too tired to find it just now. I'll try again tomorrow.
PEBKAC
Peter Zijlstra
maintainerQuoted 7 lines · Peter Zijlstra
Quoted 3 lines · Peter Zijlstra
Trouble is, pretty much every unwind is 510 entries long -- this cannot be right. I'm sure there's a silly mistake in
unwind/user.cbut I'm too tired to find it just now. I'll try again tomorrow.PEBKAC
Anyway, while staring at this, I noted that the perf userspace unwind code has a few bits that are missing from the new shiny thing.
How about something like so? This add an optional arch specific unwinder at the very highest priority (bit 0) and uses that to do a few extra bits before disabling itself and falling back to whatever lower prio unwinder to do the actual unwinding.
--- arch/x86/events/core.c | 40 --------------------------- arch/x86/include/asm/unwind_user.h | 4 ++ arch/x86/include/asm/uprobes.h | 9 ++++++ arch/x86/kernel/unwind_user.c | 53 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/uprobes.c | 32 ++++++++++++++++++++++ include/linux/unwind_user_types.h | 5 ++- kernel/unwind/user.c | 7 ++++ 7 files changed, 109 insertions(+), 41 deletions(-)
arch/x86/events/core.c +0 -40
@@ -2845,46 +2845,6 @@ static unsigned long get_segment_base(unreturn get_desc_base (desc); } -#ifdef CONFIG_UPROBES -/* -* Heuristic-based check if uprobe is installed at the function entry. -* -* Under assumption of user code being compiled with frame pointers, -* `push %rbp/%ebp` is a good indicator that we indeed are. -* -* Similarly, `endbr64` (assuming 64-bit mode) is also a common pattern. -* If we get this wrong, captured stack trace might have one extra bogus -* entry, but the rest of stack trace will still be meaningful. -*/ -static bool is_uprobe_at_func_entry (struct pt_regs *regs) -{ -struct arch_uprobe *auprobe; - -if (!current->utask) -return false; - - auprobe = current->utask->auprobe; -if (!auprobe) -return false; - -/* push %rbp/%ebp */ -if (auprobe->insn[0 ] ==0x55 ) -return true; - -/* endbr64 (64-bit only) */ -if (user_64bit_mode (regs) &&is_endbr ((u32 *)auprobe->insn)) -return true; - -return false; -} - -#else -static bool is_uprobe_at_func_entry (struct pt_regs *regs) -{ -return false; -} -#endif /* CONFIG_UPROBES */ -#ifdef CONFIG_IA32_EMULATION #include <linux/compat.h>
arch/x86/include/asm/unwind_user.h +4 -0
@@ -8,4 +8,8 @@ .fp_off = -2*(ws), \ .use_fp = true, +#define HAVE_UNWIND_USER_ARCH 1 + +extern int unwind_user_next_arch (struct unwind_user_state *state); +#endif /* _ASM_X86_UNWIND_USER_H */
arch/x86/include/asm/uprobes.h +9 -0
@@ -62,4 +62,13 @@ struct arch_uprobe_task { unsigned int saved_tf; }; +#ifdef CONFIG_UPROBES +extern bool is_uprobe_at_func_entry (struct pt_regs *regs); +#else +static bool is_uprobe_at_func_entry (struct pt_regs *regs) +{ +return false; +} +#endif /* CONFIG_UPROBES */ +#endif /* _ASM_UPROBES_H */
arch/x86/kernel/unwind_user.c new file +53 -0
53 lines not on this page · raw message
arch/x86/kernel/uprobes.c +32 -0
@@ -1791,3 +1791,35 @@ bool arch_uretprobe_is_alive(struct retuelse return regs->sp <= ret->stack; } + +/* +* Heuristic-based check if uprobe is installed at the function entry. +* +* Under assumption of user code being compiled with frame pointers, +* `push %rbp/%ebp` is a good indicator that we indeed are. +* +* Similarly, `endbr64` (assuming 64-bit mode) is also a common pattern. +* If we get this wrong, captured stack trace might have one extra bogus +* entry, but the rest of stack trace will still be meaningful. +*/ +bool is_uprobe_at_func_entry (struct pt_regs *regs) +{ +struct arch_uprobe *auprobe; + +if (!current->utask) +return false; + + auprobe = current->utask->auprobe; +if (!auprobe) +return false; + +/* push %rbp/%ebp */ +if (auprobe->insn[0 ] ==0x55 ) +return true; + +/* endbr64 (64-bit only) */ +if (user_64bit_mode (regs) &&is_endbr ((u32 *)auprobe->insn)) +return true; + +return false; +}
include/linux/unwind_user_types.h +4 -1
@@ -3,13 +3,15 @@#define _LINUX_UNWIND_USER_TYPES_H#include <linux/types.h> +#include <linux/bits.h> /* * Unwind types, listed in priority order: lower numbers are attempted first if * available. */ enum unwind_user_type_bits { -UNWIND_USER_TYPE_FP_BIT =0 , +UNWIND_USER_TYPE_ARCH_BIT =0 , +UNWIND_USER_TYPE_FP_BIT ,NR_UNWIND_USER_TYPE_BITS , };
@@ -17,6 +19,7 @@ enum unwind_user_type_bits {enum unwind_user_type {/* Type "none" for the start of stack walk iteration. */ UNWIND_USER_TYPE_NONE =0 , +UNWIND_USER_TYPE_ARCH =BIT (UNWIND_USER_TYPE_ARCH_BIT ),UNWIND_USER_TYPE_FP =BIT (UNWIND_USER_TYPE_FP_BIT ), };
kernel/unwind/user.c +7 -0
@@ -79,6 +79,10 @@ static int unwind_user_next(struct unwin state->current_type = type;switch (type) { +case UNWIND_USER_TYPE_ARCH : +if (!unwind_user_next_arch (state)) +return 0 ; +continue ;case UNWIND_USER_TYPE_FP :if (!unwind_user_next_fp (state))return 0 ;
@@ -107,6 +111,9 @@ static int unwind_user_start(struct unwireturn -EINVAL ; } +if (HAVE_UNWIND_USER_ARCH ) + state->available_types |=UNWIND_USER_TYPE_ARCH ; +if (IS_ENABLED (CONFIG_HAVE_UNWIND_USER_FP )) state->available_types |=UNWIND_USER_TYPE_FP ;
Hello Peter!
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Peter Zijlstra
Trouble is, pretty much every unwind is 510 entries long -- this cannot be right. I'm sure there's a silly mistake in
unwind/user.cbut I'm too tired to find it just now. I'll try again tomorrow.PEBKAC
Quoted 7 lines
Anyway, while staring at this, I noted that the perf userspace unwind code has a few bits that are missing from the new shiny thing.
How about something like so? This add an optional arch specific unwinder at the very highest priority (bit 0) and uses that to do a few extra bits before disabling itself and falling back to whatever lower prio unwinder to do the actual unwinding.
unwind user sframe does not need any of this special handling, because it knows for each IP whether the SP or FP is the CFA base register and whether the FP and RA have been saved.
Isn't this actually specific to unwind user fp? If the IP is at function entry, then the FP has not been setup yet. I think unwind user fp could handle this using an arch specific is_uprobe_at_func_entry() to determine whether to use a new frame_fp_entry instead of frame_fp. For x86 the following frame_fp_entry should work, if I am not wrong:
#define ARCH_INIT_USER_FP_ENTRY_FRAME(ws) \ .cfa_off = 1*(ws), \ .ra_off = -1*(ws), \ .fp_off = 0, \ .use_fp = false,
Following roughly outlines the required changes:
kernel/unwind/user.c +0 -0
-static int unwind_user_next_fp(struct unwind_user_state *state) +static int unwind_user_next_common(struct unwind_user_state *state,
+ const struct unwind_user_frame *frame,
+ struct pt_regs *regs)
@@ -71,6 +83,7 @@ static int unwind_user_next_common(struct unwind_user_state *state,
state->sp = sp;
if (frame->fp_off)
state->fp = fp;
+ state->topmost = false;
return 0;
}
@@ -154,6 +167,7 @@ static int unwind_user_start(struct unwind_user_state *state)
state->sp = user_stack_pointer(regs);
state->fp = frame_pointer(regs);
state->ws = compat_user_mode(regs) ? sizeof(int) : sizeof(long);+ state->topmost = true;
return 0;
}
static int unwind_user_next_fp(struct unwind_user_state *state)
{
const struct unwind_user_frame fp_frame = {
ARCH_INIT_USER_FP_FRAME(state->ws)
};
const struct unwind_user_frame fp_entry_frame = {
ARCH_INIT_USER_FP_ENTRY_FRAME(state->ws)
};
struct pt_regs *regs = task_pt_regs(current);
if (state->topmost && is_uprobe_at_func_entry(regs))
return unwind_user_next_common(state, &fp_entry_frame, regs);
else
return unwind_user_next_common(state, &fp_frame, regs);
}include/linux/unwind_user_types.h +1 -0
@@ -43,6 +43,7 @@ struct unwind_user_state {unsigned int ws;enum unwind_user_type current_type;unsigned int available_types; +bool topmost;bool done; };
What do you think?
Quoted 44 lines
+++
b/arch/x86/kernel/unwind_user.c(quoted diff) +42
-0@@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include <linux/unwind_user.h> +#include <linux/uprobes.h> +#include <linux/uaccess.h> +#include <linux/sched/task_stack.h> +#include <asm/processor.h> +#include <asm/tlbflush.h> + +int unwind_user_next_arch(struct unwind_user_state *state) +{ + struct pt_regs *regs = task_pt_regs(current); + + /* only once, on the first iteration */ + state->available_types &= ~UNWIND_USER_TYPE_ARCH; + + /* We don't know how to unwind VM86 stacks. */ + if (regs->flags & X86_VM_MASK) { + state->done = true; + return 0; + } + + /* + * If we are called from uprobe handler, and we are indeed at the very + * entry to user function (which is normally a `push %rbp` instruction, + * under assumption of application being compiled with frame pointers), + * we should read return address from *regs->sp before proceeding + * to follow frame pointers, otherwise we'll skip immediate caller + * as %rbp is not yet setup. + */ + if (!is_uprobe_at_func_entry(regs)) + return -EINVAL; + +#ifdef CONFIG_COMPAT + if (state->ws == sizeof(int)) { + unsigned int retaddr; + int ret = get_user(retaddr, (unsigned int __user *)regs->sp); + if (ret) + return ret; + + state->ip = retaddr; + return 0;
Review line 52
Quoted 10 lines
(quoted diff) +10
-0+ } +#endif + unsigned long retaddr; + int ret = get_user(retaddr, (unsigned long __user *)regs->sp); + if (ret) + return ret; + + state->ip = retaddr; + return 0; +}
Above would then not be needed, as the unwind_user_next_fp() logic would do the right thing.
Quoted 27 lines
(quoted diff) +23
-0+++ b/arch/x86/kernel/uprobes.c@@ -1791,3 +1791,35 @@ bool arch_uretprobe_is_alive(struct retu else return regs->sp <= ret->stack; } + +/* + * Heuristic-based check if uprobe is installed at the function entry. + * + * Under assumption of user code being compiled with frame pointers, + * `push %rbp/%ebp` is a good indicator that we indeed are. + * + * Similarly, `endbr64` (assuming 64-bit mode) is also a common pattern. + * If we get this wrong, captured stack trace might have one extra bogus + * entry, but the rest of stack trace will still be meaningful. + */ +bool is_uprobe_at_func_entry(struct pt_regs *regs) +{ + struct arch_uprobe *auprobe; + + if (!current->utask) + return false; + + auprobe = current->utask->auprobe; + if (!auprobe) + return false; +
Review bool arch_uretprobe_is_alive(struct retu · line 1825
Quoted 10 lines
(quoted diff) +10
-0+ /* push %rbp/%ebp */ + if (auprobe->insn[0] == 0x55) + return true; + + /* endbr64 (64-bit only) */ + if (user_64bit_mode(regs) && is_endbr((u32 *)auprobe->insn)) + return true; + + return false; +}
Regards,
Jens
Signature · 9 lines
Jens Remus Linux on Z Development (D3303) +49-7031-16-1128 Office [email protected] IBM IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Geschäftsführung: David Faller; Sitz der Gesellschaft: Böblingen; Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM Data Privacy Statement: https://www.ibm.com/privacy/
Peter Zijlstra
maintainerQuoted 15 lines · Jens Remus
Hello Peter!
Quoted 12 lines · Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Peter Zijlstra
Trouble is, pretty much every unwind is 510 entries long -- this cannot be right. I'm sure there's a silly mistake in
unwind/user.cbut I'm too tired to find it just now. I'll try again tomorrow.PEBKAC
Anyway, while staring at this, I noted that the perf userspace unwind code has a few bits that are missing from the new shiny thing.
Quoted 8 lines
Quoted 4 lines
How about something like so? This add an optional arch specific unwinder at the very highest priority (bit 0) and uses that to do a few extra bits before disabling itself and falling back to whatever lower prio unwinder to do the actual unwinding.
unwind user sframe does not need any of this special handling, because it knows for each IP whether the SP or FP is the CFA base register and whether the FP and RA have been saved.
It still can't unwind VM86 stacks. But yes, it should do lots better with that start of function hack.
Quoted 62 lines
Isn't this actually specific to unwind user fp? If the IP is at function entry, then the FP has not been setup yet. I think unwind user fp could handle this using an arch specific
is_uprobe_at_func_entry()to determine whether to use a newframe_fp_entryinstead offrame_fp. For x86 the followingframe_fp_entryshould work, if I am not wrong:#define ARCH_INIT_USER_FP_ENTRY_FRAME(ws) \ .cfa_off = 1*(ws), \ .ra_off = -1*(ws), \ .fp_off = 0, \ .use_fp = false,Following roughly outlines the required changes:
kernel/unwind/user.c +5
-1-static int unwind_user_next_fp (struct unwind_user_state *state) +static int unwind_user_next_common (struct unwind_user_state *state, +const struct unwind_user_frame *frame, +struct pt_regs *regs)@@ -71,6 +83,7 @@ static int unwind_user_next_common(struct unwind_user_state *state, state->sp = sp;if (frame->fp_off) state->fp = fp; + state->topmost = false;return 0 ; }@@ -154,6 +167,7 @@ static int unwind_user_start(struct unwind_user_state *state) state->sp =user_stack_pointer (regs); state->fp =frame_pointer (regs); state->ws =compat_user_mode (regs) ?sizeof (int ) :sizeof (long ); + state->topmost = true;return 0 ; }static int
unwind_user_next_fp(struct unwind_user_state *state){ const struct unwind_user_frame fp_frame = { ARCH_INIT_USER_FP_FRAME(state->ws) }; const struct unwind_user_frame fp_entry_frame = { ARCH_INIT_USER_FP_ENTRY_FRAME(state->ws) }; struct pt_regs *regs = task_pt_regs(current); if (state->topmost && is_uprobe_at_func_entry(regs)) return unwind_user_next_common(state, &fp_entry_frame, regs); else return unwind_user_next_common(state, &fp_frame, regs); }include/linux/unwind_user_types.h +1
-0@@ -43,6 +43,7 @@ struct unwind_user_state {unsigned int ws;enum unwind_user_type current_type;unsigned int available_types; +bool topmost;bool done; };
Quoted 1 line
What do you think?
Yeah, I suppose that should work. Let me rework things accordingly.
Peter Zijlstra
maintainerQuoted 1 line · Peter Zijlstra
Yeah, I suppose that should work. Let me rework things accordingly.
---
Subject: unwind_user/x86: Teach FP unwind about start of function
From: Peter Zijlstra <[email protected]>
Date: Fri Oct 24 12:31:10 CEST 2025
When userspace is interrupted at the start of a function, before we get a chance to complete the frame, unwind will miss one caller.
X86 has a uprobe specific fixup for this, add bits to the generic unwinder to support this.
--- arch/x86/events/core.c | 40 ------------------------------------- arch/x86/include/asm/unwind_user.h | 12 +++++++++++ arch/x86/include/asm/uprobes.h | 9 ++++++++ arch/x86/kernel/uprobes.c | 32 +++++++++++++++++++++++++++++ include/linux/unwind_user_types.h | 1 kernel/unwind/user.c | 35 ++++++++++++++++++++++++-------- 6 files changed, 80 insertions(+), 49 deletions(-)
arch/x86/events/core.c +0 -40
@@ -2845,46 +2845,6 @@ static unsigned long get_segment_base(unreturn get_desc_base (desc); } -#ifdef CONFIG_UPROBES -/* -* Heuristic-based check if uprobe is installed at the function entry. -* -* Under assumption of user code being compiled with frame pointers, -* `push %rbp/%ebp` is a good indicator that we indeed are. -* -* Similarly, `endbr64` (assuming 64-bit mode) is also a common pattern. -* If we get this wrong, captured stack trace might have one extra bogus -* entry, but the rest of stack trace will still be meaningful. -*/ -static bool is_uprobe_at_func_entry (struct pt_regs *regs) -{ -struct arch_uprobe *auprobe; - -if (!current->utask) -return false; - - auprobe = current->utask->auprobe; -if (!auprobe) -return false; - -/* push %rbp/%ebp */ -if (auprobe->insn[0 ] ==0x55 ) -return true; - -/* endbr64 (64-bit only) */ -if (user_64bit_mode (regs) &&is_endbr ((u32 *)auprobe->insn)) -return true; - -return false; -} - -#else -static bool is_uprobe_at_func_entry (struct pt_regs *regs) -{ -return false; -} -#endif /* CONFIG_UPROBES */ -#ifdef CONFIG_IA32_EMULATION #include <linux/compat.h>
arch/x86/include/asm/unwind_user.h +12 -0
@@ -3,6 +3,7 @@#define _ASM_X86_UNWIND_USER_H#include <asm/ptrace.h> +#include <asm/uprobes.h> #define ARCH_INIT_USER_FP_FRAME (ws) \ .cfa_off = 2*(ws), \
@@ -10,6 +11,12 @@ .fp_off = -2*(ws), \ .use_fp = true, +#define ARCH_INIT_USER_FP_ENTRY_FRAME (ws) \ + .cfa_off = 1*(ws), \ + .ra_off = -1*(ws), \ + .fp_off = 0, \ + .use_fp = false, +static inline int unwind_user_word_size (struct pt_regs *regs) {/* We can't unwind VM86 stacks */
@@ -22,4 +29,9 @@ static inline int unwind_user_word_size( return sizeof(long); } +static inline bool unwind_user_at_function_start (struct pt_regs *regs) +{ +return is_uprobe_at_func_entry (regs); +} +#endif /* _ASM_X86_UNWIND_USER_H */
arch/x86/include/asm/uprobes.h +9 -0
@@ -62,4 +62,13 @@ struct arch_uprobe_task { unsigned int saved_tf; }; +#ifdef CONFIG_UPROBES +extern bool is_uprobe_at_func_entry (struct pt_regs *regs); +#else +static bool is_uprobe_at_func_entry (struct pt_regs *regs) +{ +return false; +} +#endif /* CONFIG_UPROBES */ +#endif /* _ASM_UPROBES_H */
arch/x86/kernel/uprobes.c +32 -0
@@ -1791,3 +1791,35 @@ bool arch_uretprobe_is_alive(struct retuelse return regs->sp <= ret->stack; } + +/* +* Heuristic-based check if uprobe is installed at the function entry. +* +* Under assumption of user code being compiled with frame pointers, +* `push %rbp/%ebp` is a good indicator that we indeed are. +* +* Similarly, `endbr64` (assuming 64-bit mode) is also a common pattern. +* If we get this wrong, captured stack trace might have one extra bogus +* entry, but the rest of stack trace will still be meaningful. +*/ +bool is_uprobe_at_func_entry (struct pt_regs *regs) +{ +struct arch_uprobe *auprobe; + +if (!current->utask) +return false; + + auprobe = current->utask->auprobe; +if (!auprobe) +return false; + +/* push %rbp/%ebp */ +if (auprobe->insn[0 ] ==0x55 ) +return true; + +/* endbr64 (64-bit only) */ +if (user_64bit_mode (regs) &&is_endbr ((u32 *)auprobe->insn)) +return true; + +return false; +}
include/linux/unwind_user_types.h +1 -0
@@ -39,6 +39,7 @@ struct unwind_user_state {unsigned int ws;enum unwind_user_type current_type;unsigned int available_types; +bool topmost;bool done; };
kernel/unwind/user.c +26 -9
@@ -26,14 +26,12 @@ get_user_word(unsigned long *word, unsigreturn get_user (*word, addr); } -static int unwind_user_next_fp (struct unwind_user_state *state) +static int unwind_user_next_common (struct unwind_user_state *state, +const struct unwind_user_frame *frame) { -const struct unwind_user_frame frame = { -ARCH_INIT_USER_FP_FRAME (state->ws) - };unsigned long cfa, fp, ra; -if (frame.use_fp) { +if (frame->use_fp) {if (state->fp < state->sp)return -EINVAL ; cfa = state->fp;
@@ -42,7 +40,7 @@ static int unwind_user_next_fp(struct un }/* Get the Canonical Frame Address (CFA) */ - cfa += frame.cfa_off; + cfa += frame->cfa_off;/* stack going in wrong direction? */ if (cfa <= state->sp)
@@ -53,19 +51,37 @@ static int unwind_user_next_fp(struct unreturn -EINVAL ;/* Find the Return Address (RA) */ -if (get_user_word (&ra, cfa, frame.ra_off, state->ws)) +if (get_user_word (&ra, cfa, frame->ra_off, state->ws))return -EINVAL ; -if (frame.fp_off &&get_user_word (&fp, cfa, frame.fp_off, state->ws)) +if (frame->fp_off &&get_user_word (&fp, cfa, frame->fp_off, state->ws))return -EINVAL ; state->ip = ra; state->sp = cfa; -if (frame.fp_off) +if (frame->fp_off) state->fp = fp; + state->topmost = false;return 0 ; } +static int unwind_user_next_fp (struct unwind_user_state *state) +{ +struct pt_regs *regs =task_pt_regs (current); + +const struct unwind_user_frame fp_frame = { +ARCH_INIT_USER_FP_FRAME (state->ws) + }; +const struct unwind_user_frame fp_entry_frame = { +ARCH_INIT_USER_FP_ENTRY_FRAME (state->ws) + }; + +if (state->topmost &&unwind_user_at_function_start (regs)) +return unwind_user_next_common (state, &fp_entry_frame); + +return unwind_user_next_common (state, &fp_frame); +} +static int unwind_user_next (struct unwind_user_state *state) {unsigned long iter_mask = state->available_types;
@@ -118,6 +134,7 @@ static int unwind_user_start(struct unwi state->done = true;return -EINVAL ; } + state->topmost = true;return 0 ; }
Peter Zijlstra
maintainerQuoted 69 lines · Peter Zijlstra
arch/x86/include/asm/unwind_user.h +12
-0@@ -3,6 +3,7 @@#define _ASM_X86_UNWIND_USER_H#include <asm/ptrace.h> +#include <asm/uprobes.h> #define ARCH_INIT_USER_FP_FRAME (ws) \ .cfa_off = 2*(ws), \@@ -10,6 +11,12 @@ .fp_off = -2*(ws), \ .use_fp = true, +#define ARCH_INIT_USER_FP_ENTRY_FRAME (ws) \ + .cfa_off = 1*(ws), \ + .ra_off = -1*(ws), \ + .fp_off = 0, \ + .use_fp = false, +static inline int unwind_user_word_size (struct pt_regs *regs) {/* We can't unwind VM86 stacks */ @@ -22,4 +29,9 @@ static inline int unwind_user_word_size( return sizeof(long); } +static inline bool unwind_user_at_function_start (struct pt_regs *regs) +{ +return is_uprobe_at_func_entry (regs); +} +#endif /* _ASM_X86_UNWIND_USER_H */ include/linux/unwind_user_types.h +1
-0@@ -39,6 +39,7 @@ struct unwind_user_state {unsigned int ws;enum unwind_user_type current_type;unsigned int available_types; +bool topmost;bool done; };kernel/unwind/user.c +17
-0+static int unwind_user_next_fp (struct unwind_user_state *state) +{ +struct pt_regs *regs =task_pt_regs (current); + +const struct unwind_user_frame fp_frame = { +ARCH_INIT_USER_FP_FRAME (state->ws) + }; +const struct unwind_user_frame fp_entry_frame = { +ARCH_INIT_USER_FP_ENTRY_FRAME (state->ws) + }; + +if (state->topmost &&unwind_user_at_function_start (regs)) +return unwind_user_next_common (state, &fp_entry_frame); + +return unwind_user_next_common (state, &fp_frame); +} +static int unwind_user_next (struct unwind_user_state *state) {unsigned long iter_mask = state->available_types;
kernel/unwind/user.c static int unwind_user_start(struct unwi · line 140
Quoted 7 lines
kernel/unwind/user.c +1
-0@@ -118,6 +134,7 @@ static int unwind_user_start(struct unwi state->done = true;return -EINVAL ; } + state->topmost = true;return 0 ; }
And right before sending this; I realized we could do the unwind_user_at_function_start() in unwind_user_start() and set something like state->entry = true instead of topmost.
That saves having to do task_pt_regs() in unwind_user_next_fp().
Does that make sense?
Peter Zijlstra
maintainerQuoted 83 lines · Peter Zijlstra
Quoted 69 lines · Peter Zijlstra
arch/x86/include/asm/unwind_user.h +12
-0@@ -3,6 +3,7 @@#define _ASM_X86_UNWIND_USER_H#include <asm/ptrace.h> +#include <asm/uprobes.h> #define ARCH_INIT_USER_FP_FRAME (ws) \ .cfa_off = 2*(ws), \@@ -10,6 +11,12 @@ .fp_off = -2*(ws), \ .use_fp = true, +#define ARCH_INIT_USER_FP_ENTRY_FRAME (ws) \ + .cfa_off = 1*(ws), \ + .ra_off = -1*(ws), \ + .fp_off = 0, \ + .use_fp = false, +static inline int unwind_user_word_size (struct pt_regs *regs) {/* We can't unwind VM86 stacks */ @@ -22,4 +29,9 @@ static inline int unwind_user_word_size( return sizeof(long); } +static inline bool unwind_user_at_function_start (struct pt_regs *regs) +{ +return is_uprobe_at_func_entry (regs); +} +#endif /* _ASM_X86_UNWIND_USER_H */ include/linux/unwind_user_types.h +1
-0@@ -39,6 +39,7 @@ struct unwind_user_state {unsigned int ws;enum unwind_user_type current_type;unsigned int available_types; +bool topmost;bool done; };kernel/unwind/user.c +17
-0+static int unwind_user_next_fp (struct unwind_user_state *state) +{ +struct pt_regs *regs =task_pt_regs (current); + +const struct unwind_user_frame fp_frame = { +ARCH_INIT_USER_FP_FRAME (state->ws) + }; +const struct unwind_user_frame fp_entry_frame = { +ARCH_INIT_USER_FP_ENTRY_FRAME (state->ws) + }; + +if (state->topmost &&unwind_user_at_function_start (regs)) +return unwind_user_next_common (state, &fp_entry_frame); + +return unwind_user_next_common (state, &fp_frame); +} +static int unwind_user_next (struct unwind_user_state *state) {unsigned long iter_mask = state->available_types;kernel/unwind/user.c static int unwind_user_start(struct unwi · line 140
Quoted 7 lines
kernel/unwind/user.c +1
-0@@ -118,6 +134,7 @@ static int unwind_user_start(struct unwi state->done = true;return -EINVAL ; } + state->topmost = true;return 0 ; }And right before sending this; I realized we could do the
unwind_user_at_function_start()inunwind_user_start()and set something likestate->entry= true instead of topmost.
Quoted 3 lines
That saves having to do
task_pt_regs()inunwind_user_next_fp().Does that make sense?
Urgh, that makes us call that weird hack for sframe too, which isn't needed. Oh well, ignore this.
Peter Zijlstra
maintainerQuoted 2 lines · Peter Zijlstra
Urgh, that makes us call that weird hack for sframe too, which isn't needed. Oh well, ignore this.
I've decided to stop tinkering for today and pushed out the lot into:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git perf/core
It seems to build and work on the one test build I did, so fingers crossed.
If there is anything you want changed, please holler, I'll not push to tip until at least Monday anyway.
Hello Peter,
very nice!
Quoted 18 lines · Peter Zijlstra
Subject: unwind_user/x86: Teach FP unwind about start of function
From: Peter Zijlstra <[email protected]>
Date: Fri Oct 24 12:31:10 CEST 2025When userspace is interrupted at the start of a function, before we get a chance to complete the frame, unwind will miss one caller.
X86 has a uprobe specific fixup for this, add bits to the generic unwinder to support this.
Suggested-by Jens Remus <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>+++
b/kernel/unwind/user.ckernel/unwind/user.c +3
-0+static int unwind_user_next_fp (struct unwind_user_state *state) +{ +struct pt_regs *regs =task_pt_regs (current);
kernel/unwind/user.c static int unwind_user_next_fp(struct un · line 80
Quoted 10 lines
kernel/unwind/user.c +10
-0+ +const struct unwind_user_frame fp_frame = { +ARCH_INIT_USER_FP_FRAME (state->ws) + }; +const struct unwind_user_frame fp_entry_frame = { +ARCH_INIT_USER_FP_ENTRY_FRAME (state->ws) + }; + +if (state->topmost &&unwind_user_at_function_start (regs)) +return unwind_user_next_common (state, &fp_entry_frame);
IIUC this will cause kernel/unwind/user.c to fail compile on architectures that will support HAVE_UNWIND_USER_SFRAME but not HAVE_UNWIND_USER_FP (such as s390), and thus do not need to implement unwind_user_at_function_start().
Either s390 would need to supply a dummy unwind_user_at_function_start() or the unwind user sframe series needs to address this and supply a dummy one if FP is not enabled, so that the code compiles with only SFRAME enabled.
What do you think?
kernel/unwind/user.c static int unwind_user_next_fp(struct un · line 83
Quoted 3 lines
kernel/unwind/user.c +3
-0+ +return unwind_user_next_common (state, &fp_frame); +}
Thanks and regards,
Jens
Signature · 9 lines
Jens Remus Linux on Z Development (D3303) +49-7031-16-1128 Office [email protected] IBM IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Geschäftsführung: David Faller; Sitz der Gesellschaft: Böblingen; Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM Data Privacy Statement: https://www.ibm.com/privacy/
Peter Zijlstra
maintainerQuoted 40 lines · Jens Remus
Hello Peter,
very nice!
Quoted 18 lines · Peter Zijlstra
Subject: unwind_user/x86: Teach FP unwind about start of function
From: Peter Zijlstra <[email protected]>
Date: Fri Oct 24 12:31:10 CEST 2025When userspace is interrupted at the start of a function, before we get a chance to complete the frame, unwind will miss one caller.
X86 has a uprobe specific fixup for this, add bits to the generic unwinder to support this.
Suggested-by Jens Remus <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>+++
b/kernel/unwind/user.ckernel/unwind/user.c +3
-0+static int unwind_user_next_fp (struct unwind_user_state *state) +{ +struct pt_regs *regs =task_pt_regs (current);kernel/unwind/user.c static int unwind_user_next_fp(struct un · line 80
Quoted 10 lines
kernel/unwind/user.c +10
-0+ +const struct unwind_user_frame fp_frame = { +ARCH_INIT_USER_FP_FRAME (state->ws) + }; +const struct unwind_user_frame fp_entry_frame = { +ARCH_INIT_USER_FP_ENTRY_FRAME (state->ws) + }; + +if (state->topmost &&unwind_user_at_function_start (regs)) +return unwind_user_next_common (state, &fp_entry_frame);IIUC this will cause
kernel/unwind/user.cto fail compile on architectures that will supportHAVE_UNWIND_USER_SFRAMEbut notHAVE_UNWIND_USER_FP(such as s390), and thus do not need to implementunwind_user_at_function_start().
Quoted 6 lines
Either s390 would need to supply a dummy
unwind_user_at_function_start()or the unwind user sframe series needs to address this and supply a dummy one if FP is not enabled, so that the code compiles with only SFRAME enabled.What do you think?
I'll make it conditional on HAVE_UNWIND_USER_FP -- but tomorrow or so.
[tip: perf/core] unwind_user/x86: Teach FP unwind about start of function
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: ae25884ad749e7f6e0c3565513bdc8aa2554a425 Gitweb: https://git.kernel.org/tip/ae25884ad749e7f6e0c3565513bdc8aa2554a425 Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 24 Oct 2025 12:31:10 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 29 Oct 2025 10:29:58 +01:00
unwind_user/x86: Teach FP unwind about start of function
When userspace is interrupted at the start of a function, before we get a chance to complete the frame, unwind will miss one caller.
X86 has a uprobe specific fixup for this, add bits to the generic unwinder to support this.
--- arch/x86/events/core.c | 40 +----------------------------- arch/x86/include/asm/unwind_user.h | 12 +++++++++- arch/x86/include/asm/uprobes.h | 9 +++++++- arch/x86/kernel/uprobes.c | 32 +++++++++++++++++++++++- include/linux/unwind_user_types.h | 1 +- kernel/unwind/user.c | 39 +++++++++++++++++++++------- 6 files changed, 84 insertions(+), 49 deletions(-)
arch/x86/events/core.c +0 -40
@@ -2845,46 +2845,6 @@ static unsigned long get_segment_base(unsigned int segment)return get_desc_base (desc); } -#ifdef CONFIG_UPROBES -/* -* Heuristic-based check if uprobe is installed at the function entry. -* -* Under assumption of user code being compiled with frame pointers, -* `push %rbp/%ebp` is a good indicator that we indeed are. -* -* Similarly, `endbr64` (assuming 64-bit mode) is also a common pattern. -* If we get this wrong, captured stack trace might have one extra bogus -* entry, but the rest of stack trace will still be meaningful. -*/ -static bool is_uprobe_at_func_entry (struct pt_regs *regs) -{ -struct arch_uprobe *auprobe; - -if (!current->utask) -return false; - - auprobe = current->utask->auprobe; -if (!auprobe) -return false; - -/* push %rbp/%ebp */ -if (auprobe->insn[0 ] ==0x55 ) -return true; - -/* endbr64 (64-bit only) */ -if (user_64bit_mode (regs) &&is_endbr ((u32 *)auprobe->insn)) -return true; - -return false; -} - -#else -static bool is_uprobe_at_func_entry (struct pt_regs *regs) -{ -return false; -} -#endif /* CONFIG_UPROBES */ -#ifdef CONFIG_IA32_EMULATION #include <linux/compat.h>
arch/x86/include/asm/unwind_user.h +12 -0
@@ -3,6 +3,7 @@#define _ASM_X86_UNWIND_USER_H#include <asm/ptrace.h> +#include <asm/uprobes.h> #define ARCH_INIT_USER_FP_FRAME (ws) \ .cfa_off = 2*(ws), \
@@ -10,6 +11,12 @@ .fp_off = -2*(ws), \ .use_fp = true, +#define ARCH_INIT_USER_FP_ENTRY_FRAME (ws) \ + .cfa_off = 1*(ws), \ + .ra_off = -1*(ws), \ + .fp_off = 0, \ + .use_fp = false, +static inline int unwind_user_word_size (struct pt_regs *regs) {/* We can't unwind VM86 stacks */
@@ -22,4 +29,9 @@ static inline int unwind_user_word_size(struct pt_regs *regs) return sizeof(long); } +static inline bool unwind_user_at_function_start (struct pt_regs *regs) +{ +return is_uprobe_at_func_entry (regs); +} +#endif /* _ASM_X86_UNWIND_USER_H */
arch/x86/include/asm/uprobes.h +9 -0
@@ -62,4 +62,13 @@ struct arch_uprobe_task { unsigned int saved_tf; }; +#ifdef CONFIG_UPROBES +extern bool is_uprobe_at_func_entry (struct pt_regs *regs); +#else +static bool is_uprobe_at_func_entry (struct pt_regs *regs) +{ +return false; +} +#endif /* CONFIG_UPROBES */ +#endif /* _ASM_UPROBES_H */
arch/x86/kernel/uprobes.c +32 -0
@@ -1791,3 +1791,35 @@ bool arch_uretprobe_is_alive(struct return_instance *ret, enum rp_check ctx,else return regs->sp <= ret->stack; } + +/* +* Heuristic-based check if uprobe is installed at the function entry. +* +* Under assumption of user code being compiled with frame pointers, +* `push %rbp/%ebp` is a good indicator that we indeed are. +* +* Similarly, `endbr64` (assuming 64-bit mode) is also a common pattern. +* If we get this wrong, captured stack trace might have one extra bogus +* entry, but the rest of stack trace will still be meaningful. +*/ +bool is_uprobe_at_func_entry (struct pt_regs *regs) +{ +struct arch_uprobe *auprobe; + +if (!current->utask) +return false; + + auprobe = current->utask->auprobe; +if (!auprobe) +return false; + +/* push %rbp/%ebp */ +if (auprobe->insn[0 ] ==0x55 ) +return true; + +/* endbr64 (64-bit only) */ +if (user_64bit_mode (regs) &&is_endbr ((u32 *)auprobe->insn)) +return true; + +return false; +}
include/linux/unwind_user_types.h +1 -0
@@ -39,6 +39,7 @@ struct unwind_user_state {unsigned int ws;enum unwind_user_type current_type;unsigned int available_types; +bool topmost;bool done; };
kernel/unwind/user.c +30 -9
@@ -26,14 +26,12 @@ get_user_word(unsigned long *word, unsigned long base, int off, unsigned int ws)return get_user (*word, addr); } -static int unwind_user_next_fp (struct unwind_user_state *state) +static int unwind_user_next_common (struct unwind_user_state *state, +const struct unwind_user_frame *frame) { -const struct unwind_user_frame frame = { -ARCH_INIT_USER_FP_FRAME (state->ws) - };unsigned long cfa, fp, ra; -if (frame.use_fp) { +if (frame->use_fp) {if (state->fp < state->sp)return -EINVAL ; cfa = state->fp;
@@ -42,7 +40,7 @@ static int unwind_user_next_fp(struct unwind_user_state *state) }/* Get the Canonical Frame Address (CFA) */ - cfa += frame.cfa_off; + cfa += frame->cfa_off;/* stack going in wrong direction? */ if (cfa <= state->sp)
@@ -53,19 +51,41 @@ static int unwind_user_next_fp(struct unwind_user_state *state)return -EINVAL ;/* Find the Return Address (RA) */ -if (get_user_word (&ra, cfa, frame.ra_off, state->ws)) +if (get_user_word (&ra, cfa, frame->ra_off, state->ws))return -EINVAL ; -if (frame.fp_off &&get_user_word (&fp, cfa, frame.fp_off, state->ws)) +if (frame->fp_off &&get_user_word (&fp, cfa, frame->fp_off, state->ws))return -EINVAL ; state->ip = ra; state->sp = cfa; -if (frame.fp_off) +if (frame->fp_off) state->fp = fp; + state->topmost = false;return 0 ; } +static int unwind_user_next_fp (struct unwind_user_state *state) +{ +#ifdef CONFIG_HAVE_UNWIND_USER_FP +struct pt_regs *regs =task_pt_regs (current); + +if (state->topmost &&unwind_user_at_function_start (regs)) { +const struct unwind_user_frame fp_entry_frame = { +ARCH_INIT_USER_FP_ENTRY_FRAME (state->ws) + }; +return unwind_user_next_common (state, &fp_entry_frame); + } + +const struct unwind_user_frame fp_frame = { +ARCH_INIT_USER_FP_FRAME (state->ws) + }; +return unwind_user_next_common (state, &fp_frame); +#else +return -EINVAL ; +#endif +} +static int unwind_user_next (struct unwind_user_state *state) {unsigned long iter_mask = state->available_types;
@@ -118,6 +138,7 @@ static int unwind_user_start(struct unwind_user_state *state) state->done = true;return -EINVAL ; } + state->topmost = true;return 0 ; }
* Peter Zijlstra:
arch/x86/kernel/uprobes.c bool arch_uretprobe_is_alive(struct retu · line 1805
Quoted 11 lines
arch/x86/kernel/uprobes.c +11
-0+/* +* Heuristic-based check if uprobe is installed at the function entry. +* +* Under assumption of user code being compiled with frame pointers, +* `push %rbp/%ebp` is a good indicator that we indeed are. +* +* Similarly, `endbr64` (assuming 64-bit mode) is also a common pattern. +* If we get this wrong, captured stack trace might have one extra bogus +* entry, but the rest of stack trace will still be meaningful. +*/ +bool is_uprobe_at_func_entry (struct pt_regs *regs)
Is this specifically for uprobes? Wouldn't it make sense to tell the kernel when the uprobe is installed whether the frame pointer has been set up at this point? Userspace can typically figure this out easily enough (it's not much more difficult to find the address of the function).
Thanks,
Florian
Peter Zijlstra
maintainerQuoted 11 lines · Florian Weimer
* Peter Zijlstra:
Quoted 9 lines
+/*
+ * Heuristic-based check if uprobe is installed at the function entry.
+ *
+ * Under assumption of user code being compiled with frame pointers,
+ *push %rbp/%ebpis a good indicator that we indeed are.
+ *
+ * Similarly,endbr64(assuming 64-bit mode) is also a common pattern.
+ * If we get this wrong, captured stack trace might have one extra bogus
+ * entry, but the rest of stack trace will still be meaningful.
Quoted 8 lines
Quoted 2 lines
+ */
+boolis_uprobe_at_func_entry(struct pt_regs *regs)Is this specifically for uprobes? Wouldn't it make sense to tell the kernel when the uprobe is installed whether the frame pointer has been set up at this point? Userspace can typically figure this out easily enough (it's not much more difficult to find the address of the function).
Yeah, I suppose so. Not sure the actual user interface for this allows for that. Someone would have to dig into that a bit.
[tip: perf/core] perf: Support deferred user unwind
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: c69993ecdd4dfde2b7da08b022052a33b203da07 Gitweb: https://git.kernel.org/tip/c69993ecdd4dfde2b7da08b022052a33b203da07 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Oct 2025 15:17:05 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 29 Oct 2025 10:29:58 +01:00
perf: Support deferred user unwind
Add support for deferred userspace unwind to perf.
Where perf currently relies on in-place stack unwinding; from NMI context and all that. This moves the userspace part of the unwind to right before the return-to-userspace.
This has two distinct benefits, the biggest is that it moves the unwind to a faultable context. It becomes possible to fault in debug info (.eh_frame, SFrame etc.) that might not otherwise be readily available. And secondly, it de-duplicates the user callchain where multiple samples happen during the same kernel entry.
To facilitate this the perf interface is extended with a new record type:
PERF_RECORD_CALLCHAIN_DEFERRED
and two new attribute flags:
perf_event_attr::defer_callchain - to request the user unwind be deferred
perf_event_attr::defer_output - to request PERF_RECORD_CALLCHAIN_DEFERRED records
The existing PERF_RECORD_SAMPLE callchain section gets a new context type:
PERF_CONTEXT_USER_DEFERRED
After which will come a single entry, denoting the 'cookie' of the deferred callchain that should be attached here, matching the 'cookie' field of the above mentioned PERF_RECORD_CALLCHAIN_DEFERRED.
The 'defer_callchain' flag is expected on all events with PERF_SAMPLE_CALLCHAIN. The 'defer_output' flag is expect on the event responsible for collecting side-band events (like mmap, comm etc.). Setting 'defer_output' on multiple events will get you duplicated PERF_RECORD_CALLCHAIN_DEFERRED records.
Based on earlier patches by Josh and Steven.
--- include/linux/perf_event.h | 2 +- include/linux/unwind_deferred.h | 12 +---- include/linux/unwind_deferred_types.h | 13 ++++- include/uapi/linux/perf_event.h | 21 ++++++- kernel/bpf/stackmap.c | 4 +- kernel/events/callchain.c | 14 ++++- kernel/events/core.c | 78 +++++++++++++++++++++++++- tools/include/uapi/linux/perf_event.h | 21 ++++++- 8 files changed, 145 insertions(+), 20 deletions(-)
include/linux/perf_event.h +1 -1
@@ -1720,7 +1720,7 @@ extern void perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pextern void perf_callchain_kernel (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs);extern struct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user, -u32 max_stack,bool crosstask,bool add_mark); +u32 max_stack,bool crosstask,bool add_mark,u64 defer_cookie);extern int get_callchain_buffers (int max_stack);extern void put_callchain_buffers (void );extern struct perf_callchain_entry *get_callchain_entry (int *rctx);
include/linux/unwind_deferred.h +0 -12
@@ -6,18 +6,6 @@#include <linux/unwind_user.h> #include <linux/unwind_deferred_types.h> -struct unwind_work ; - -typedef void (*unwind_callback_t )(struct unwind_work *work, -struct unwind_stacktrace *trace, -u64 cookie); - -struct unwind_work { -struct list_head list; -unwind_callback_t func; -int bit; -}; -#ifdef CONFIG_UNWIND_USER enum {
include/linux/unwind_deferred_types.h +13 -0
@@ -39,4 +39,17 @@ struct unwind_task_info { union unwind_task_id id; }; +struct unwind_work ; +struct unwind_stacktrace ; + +typedef void (*unwind_callback_t )(struct unwind_work *work, +struct unwind_stacktrace *trace, +u64 cookie); + +struct unwind_work { +struct list_head list; +unwind_callback_t func; +int bit; +}; +#endif /* _LINUX_UNWIND_USER_DEFERRED_TYPES_H */
include/uapi/linux/perf_event.h +20 -1
@@ -463,7 +463,9 @@ struct perf_event_attr { inherit_thread : 1, /* children only inherit if cloned with CLONE_THREAD */ remove_on_exec : 1, /* event is removed from task on exec */ sigtrap : 1, /* send synchronous SIGTRAP on event */ - __reserved_1 : 26; + defer_callchain: 1, /* request PERF_RECORD_CALLCHAIN_DEFERRED records */ + defer_output : 1, /* output PERF_RECORD_CALLCHAIN_DEFERRED records */ + __reserved_1 : 24;union {__u32 wakeup_events;/* wake up every n events */
@@ -1239,6 +1241,22 @@ enum perf_event_type {*/ PERF_RECORD_AUX_OUTPUT_HW_ID =21 , +/* +* This user callchain capture was deferred until shortly before +* returning to user space. Previous samples would have kernel +* callchains only and they need to be stitched with this to make full +* callchains. +* +* struct { +* struct perf_event_header header; +* u64 cookie; +* u64 nr; +* u64 ips[nr]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_CALLCHAIN_DEFERRED =22 , + PERF_RECORD_MAX, /* non-ABI */ };
@@ -1269,6 +1287,7 @@ enum perf_callchain_context {PERF_CONTEXT_HV = (__u64)-32 ,PERF_CONTEXT_KERNEL = (__u64)-128 ,PERF_CONTEXT_USER = (__u64)-512 , +PERF_CONTEXT_USER_DEFERRED = (__u64)-640 ,PERF_CONTEXT_GUEST = (__u64)-2048 , PERF_CONTEXT_GUEST_KERNEL = (__u64)-2176,
kernel/bpf/stackmap.c +2 -2
@@ -315,7 +315,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, max_depth = sysctl_perf_event_max_stack; trace =get_perf_callchain (regs, kernel, user, max_depth, - false, false); + false, false,0 );if (unlikely (!trace))/* couldn't fetch the stack trace */
@@ -452,7 +452,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, trace =get_callchain_entry_for_task (task, max_depth); else trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask, false); + crosstask, false,0 );if (unlikely (!trace) || trace->nr < skip) {if (may_fault)
kernel/events/callchain.c +13 -1
@@ -218,7 +218,7 @@ static void fixup_uretprobe_trampoline_entries(struct perf_callchain_entry *entrstruct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user, -u32 max_stack,bool crosstask,bool add_mark) +u32 max_stack,bool crosstask,bool add_mark,u64 defer_cookie) {struct perf_callchain_entry *entry;struct perf_callchain_entry_ctx ctx;
@@ -251,6 +251,18 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, regs =task_pt_regs (current); } +if (defer_cookie) { +/* +* Foretell the coming of PERF_RECORD_CALLCHAIN_DEFERRED +* which can be stitched to this one, and add +* the cookie after it (it will be cut off when the +* user stack is copied to the callchain). +*/ +perf_callchain_store_context (&ctx,PERF_CONTEXT_USER_DEFERRED ); +perf_callchain_store_context (&ctx, defer_cookie); + goto exit_put; + } +if (add_mark)perf_callchain_store_context (&ctx,PERF_CONTEXT_USER );
kernel/events/core.c +76 -2
116 lines not on this page · raw message
tools/include/uapi/linux/perf_event.h +20 -1
@@ -463,7 +463,9 @@ struct perf_event_attr { inherit_thread : 1, /* children only inherit if cloned with CLONE_THREAD */ remove_on_exec : 1, /* event is removed from task on exec */ sigtrap : 1, /* send synchronous SIGTRAP on event */ - __reserved_1 : 26; + defer_callchain: 1, /* request PERF_RECORD_CALLCHAIN_DEFERRED records */ + defer_output : 1, /* output PERF_RECORD_CALLCHAIN_DEFERRED records */ + __reserved_1 : 24;union {__u32 wakeup_events;/* wake up every n events */
@@ -1239,6 +1241,22 @@ enum perf_event_type {*/ PERF_RECORD_AUX_OUTPUT_HW_ID =21 , +/* +* This user callchain capture was deferred until shortly before +* returning to user space. Previous samples would have kernel +* callchains only and they need to be stitched with this to make full +* callchains. +* +* struct { +* struct perf_event_header header; +* u64 cookie; +* u64 nr; +* u64 ips[nr]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_CALLCHAIN_DEFERRED =22 , + PERF_RECORD_MAX, /* non-ABI */ };
@@ -1269,6 +1287,7 @@ enum perf_callchain_context {PERF_CONTEXT_HV = (__u64)-32 ,PERF_CONTEXT_KERNEL = (__u64)-128 ,PERF_CONTEXT_USER = (__u64)-512 , +PERF_CONTEXT_USER_DEFERRED = (__u64)-640 ,PERF_CONTEXT_GUEST = (__u64)-2048 , PERF_CONTEXT_GUEST_KERNEL = (__u64)-2176,