perf: Deferred unwinding of user space stack traces
Subsystems (13)
Steven Rostedt
maintainerv5 cover perf: Deferred unwinding of user space stack traces
I'm currently working on getting sframe support from the kernel. Josh Poimboeuf did a lot of the hard work already, but he told me he doesn't have time to continue it so I'm picking it up where he left off.
His last series of v4 is here:
https://lore.kernel.org/all/[email protected]/
It covers a lot of topics as he found issues with other aspects of the kernel that needed to be fixed for sframes to work properly.
This series focuses only on implementing the deferred user space stack tracing that can be used by perf. It does not implement sframe support nor does it add an interface that can be used by ftrace or LTTng (that will come later).
I based this off of the latest tip/master:
commit 7ab869c799fc0fb22f9b4c2f36aaa603d9c7cc9d
A new generic interface is created for user space stack tracing. It does not replace the current stack_trace_save_user() interface, but creates a new interface that can handle more complex tasks. For instance, deferred stack tracing.
Josh's code had an interface with:
unwind_deferred_init()
unwind_deferred_request()
unwind_deferred_cancel()
That is not in this series.
That code made it possible for a tracer to register that it will be wanting to have user space stacktraces and give it a callback to call when the task goes back to user space. An interface to request a stacktrace and is given a "cookie" that will be used to map it to the stacktrace that will be retrieved when the task exits. And an interface to unregister (cancel).
But that is not in this series. This adds a new interface that is called by the tracer from a faultable context and gets the user space stack trace immediately. It also has the cache logic where if the same stacktrace is requested more than once (possibly by different requesters) it doesn't have to regenerate the trace.
The new function is:
unwind_deferred_trace()
Since perf is unique from other tracers where it has a separate event for basically every task and/or every event and can have hundreds of requesters for a task stack trace, I found it was just better to have perf use its own setup with a task_work per event. This is very similar to what Josh did with his last version within the above three functions. But since that was not very useful for ftrace or LTTng, and perf didn't need the cookie concept, I decided to just implement that directly within perf. That made the unwinder code simpler for this version.
Josh Poimboeuf (12):
unwind_user: Add user space unwinding API
unwind_user: Add frame pointer support
unwind_user/x86: Enable frame pointer unwinding on x86
perf/x86: Rename and move get_segment_base() and make it global
unwind_user: Add compat mode frame pointer support unwind_user/x86: Enable compat mode frame pointer unwinding on x86 unwind_user/deferred: Add unwind cache
perf: Remove get_perf_callchain() 'init_nr' argument
perf: Have get_perf_callchain() return NULL if 'crosstask' and 'user' are set
perf: Simplify get_perf_callchain() user logic
perf: Skip user unwind if !current->mm
perf: Support deferred user callchains
Namhyung Kim (4):
perf tools: Minimal CALLCHAIN_DEFERRED support perf record: Enable defer_callchain for user callchains perf script: Display PERF_RECORD_CALLCHAIN_DEFERRED perf tools: Merge deferred user callchains
Steven Rostedt (1):
unwind_user/deferred: Add unwind_deferred_trace()
----
MAINTAINERS | 8 ++ arch/Kconfig | 14 +++ arch/x86/Kconfig | 2 + arch/x86/events/core.c | 44 +------- arch/x86/include/asm/ptrace.h | 2 + arch/x86/include/asm/unwind_user.h | 61 +++++++++++ arch/x86/include/asm/unwind_user_types.h | 17 +++ arch/x86/kernel/ptrace.c | 38 +++++++ include/asm-generic/Kbuild | 2 + include/asm-generic/unwind_user.h | 24 +++++ include/asm-generic/unwind_user_types.h | 9 ++ include/linux/entry-common.h | 2 + include/linux/perf_event.h | 15 ++- include/linux/sched.h | 5 + include/linux/unwind_deferred.h | 31 ++++++ include/linux/unwind_deferred_types.h | 14 +++ include/linux/unwind_user.h | 15 +++ include/linux/unwind_user_types.h | 35 +++++++ include/uapi/linux/perf_event.h | 19 +++- kernel/Makefile | 1 + kernel/bpf/stackmap.c | 8 +- kernel/events/callchain.c | 47 +++++---- kernel/events/core.c | 165 +++++++++++++++++++++++++++++- kernel/fork.c | 4 + kernel/unwind/Makefile | 1 + kernel/unwind/deferred.c | 62 +++++++++++ kernel/unwind/user.c | 130 +++++++++++++++++++++++ tools/include/uapi/linux/perf_event.h | 19 +++- tools/lib/perf/include/perf/event.h | 7 ++ tools/perf/Documentation/perf-script.txt | 5 + tools/perf/builtin-script.c | 92 +++++++++++++++++ tools/perf/util/callchain.c | 24 +++++ tools/perf/util/callchain.h | 3 + tools/perf/util/event.c | 1 + tools/perf/util/evlist.c | 1 + tools/perf/util/evlist.h | 1 + tools/perf/util/evsel.c | 39 +++++++ tools/perf/util/evsel.h | 1 + tools/perf/util/machine.c | 1 + tools/perf/util/perf_event_attr_fprintf.c | 1 + tools/perf/util/sample.h | 3 +- tools/perf/util/session.c | 78 ++++++++++++++ tools/perf/util/tool.c | 2 + tools/perf/util/tool.h | 4 +- 44 files changed, 985 insertions(+), 72 deletions(-) create mode 100644 arch/x86/include/asm/unwind_user.h create mode 100644 arch/x86/include/asm/unwind_user_types.h create mode 100644 include/asm-generic/unwind_user.h create mode 100644 include/asm-generic/unwind_user_types.h create mode 100644 include/linux/unwind_deferred.h create mode 100644 include/linux/unwind_deferred_types.h create mode 100644 include/linux/unwind_user.h create mode 100644 include/linux/unwind_user_types.h create mode 100644 kernel/unwind/Makefile create mode 100644 kernel/unwind/deferred.c create mode 100644 kernel/unwind/user.c
Steven Rostedt
maintainerPATCH v5 2/17 unwind_user: Add frame pointer support
From: Josh Poimboeuf <[email protected]>
Add optional support for user space frame pointer unwinding. If supported, the arch needs to enable CONFIG_HAVE_UNWIND_USER_FP and define ARCH_INIT_USER_FP_FRAME.
By encoding the frame offsets in struct unwind_user_frame, much of this code can also be reused for future unwinder implementations like sframe.
--- arch/Kconfig | 4 +++ include/asm-generic/unwind_user.h | 9 ++++++ include/linux/unwind_user_types.h | 1 + kernel/unwind/user.c | 51 +++++++++++++++++++++++++++++-- 4 files changed, 63 insertions(+), 2 deletions(-) create mode 100644 include/asm-generic/unwind_user.h
arch/Kconfig +4 -0
@@ -438,6 +438,10 @@ config HAVE_HARDLOCKUP_DETECTOR_ARCH config UNWIND_USER bool +config HAVE_UNWIND_USER_FP + bool + select UNWIND_USER + config HAVE_PERF_REGS bool help
include/asm-generic/unwind_user.h new file +9 -0
@@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_GENERIC_UNWIND_USER_H +#define _ASM_GENERIC_UNWIND_USER_H + +#ifndef ARCH_INIT_USER_FP_FRAME +#define ARCH_INIT_USER_FP_FRAME +#endif + +#endif /* _ASM_GENERIC_UNWIND_USER_H */
include/linux/unwind_user_types.h +1 -0
@@ -6,6 +6,7 @@enum unwind_user_type {UNWIND_USER_TYPE_NONE , +UNWIND_USER_TYPE_FP , };struct unwind_stacktrace {
kernel/unwind/user.c +49 -2
66 lines not on this page · raw message
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 1/17 unwind_user: Add user space unwinding API
In reply to Steven RostedtFrom: Josh Poimboeuf <[email protected]>
Introduce a generic API for unwinding user stacks.
In order to expand user space unwinding to be able to handle more complex scenarios, such as deferred unwinding and reading user space information, create a generic interface that all architectures can use that support the various unwinding methods.
This is an alternative method for handling user space stack traces from the simple stack_trace_save_user() API. This does not replace that interface, but this interface will be used to expand the functionality of user space stack walking.
---
Changes since v4: https://lore.kernel.org/all/cd54347d26967427bc831f473d76e424ba3eba95.1737511963.git.jpoimboe@kernel.org/
- Removed unused variables from unwind_user_next() (Jens Remus)
- Added "return" to unwind_user_next() (Andrii Nakryiko)
MAINTAINERS | 8 +++++ arch/Kconfig | 3 ++ include/linux/unwind_user.h | 15 +++++++++ include/linux/unwind_user_types.h | 31 +++++++++++++++++ kernel/Makefile | 1 + kernel/unwind/Makefile | 1 + kernel/unwind/user.c | 55 +++++++++++++++++++++++++++++++ 7 files changed, 114 insertions(+) create mode 100644 include/linux/unwind_user.h create mode 100644 include/linux/unwind_user_types.h create mode 100644 kernel/unwind/Makefile create mode 100644 kernel/unwind/user.c
MAINTAINERS +8 -0
@@ -25278,6 +25278,14 @@ F: Documentation/driver-api/uio-howto.rst F: drivers/uio/ F: include/linux/uio_driver.h +USERSPACE STACK UNWINDING +M: Josh Poimboeuf <[email protected]> +M: Steven Rostedt <[email protected]> +S: Maintained +F: include/linux/unwind*.h +F: kernel/unwind/ + + UTIL-LINUX PACKAGE M: Karel Zak <[email protected]> L: [email protected]
arch/Kconfig +3 -0
@@ -435,6 +435,9 @@ config HAVE_HARDLOCKUP_DETECTOR_ARCH It uses the same command line parameters, and sysctl interface, as the generic hardlockup detectors. +config UNWIND_USER + bool + config HAVE_PERF_REGS bool help
include/linux/unwind_user.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_UNWIND_USER_H +#define _LINUX_UNWIND_USER_H + +#include <linux/unwind_user_types.h> + +int unwind_user_start (struct unwind_user_state *state); +int unwind_user_next (struct unwind_user_state *state); + +int unwind_user (struct unwind_stacktrace *trace,unsigned int max_entries); + +#define for_each_user_frame (state) \ + for (unwind_user_start((state)); !(state)->done; unwind_user_next((state))) + +#endif /* _LINUX_UNWIND_USER_H */
include/linux/unwind_user_types.h new file +31 -0
@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_UNWIND_USER_TYPES_H +#define _LINUX_UNWIND_USER_TYPES_H + +#include <linux/types.h> + +enum unwind_user_type { +UNWIND_USER_TYPE_NONE , +}; + +struct unwind_stacktrace { +unsigned int nr; +unsigned long *entries; +}; + +struct unwind_user_frame { +s32 cfa_off; +s32 ra_off; +s32 fp_off; +bool use_fp; +}; + +struct unwind_user_state { +unsigned long ip; +unsigned long sp; +unsigned long fp; +enum unwind_user_type type; +bool done; +}; + +#endif /* _LINUX_UNWIND_USER_TYPES_H */
kernel/Makefile +1 -0
@@ -55,6 +55,7 @@ obj-y += rcu/ obj-y += livepatch/ obj-y += dma/ obj-y += entry/ +obj-y += unwind/ obj-$(CONFIG_MODULES) += module/ obj-$(CONFIG_KCMP) += kcmp.o
kernel/unwind/Makefile new file +1 -0
@@ -0,0 +1 @@ + obj-$(CONFIG_UNWIND_USER) += user.o
kernel/unwind/user.c new file +55 -0
55 lines not on this page · raw message
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 3/17 unwind_user/x86: Enable frame pointer unwinding on x86
In reply to Steven RostedtFrom: Josh Poimboeuf <[email protected]>
Use ARCH_INIT_USER_FP_FRAME to describe how frame pointers are unwound on x86, and enable CONFIG_HAVE_UNWIND_USER_FP accordingly so the unwind_user interfaces can be used.
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/unwind_user.h | 11 +++++++++++ 2 files changed, 12 insertions(+) create mode 100644 arch/x86/include/asm/unwind_user.h
arch/x86/Kconfig +1 -0
@@ -301,6 +301,7 @@ config X86 select HAVE_SYSCALL_TRACEPOINTS select HAVE_UACCESS_VALIDATION if HAVE_OBJTOOL select HAVE_UNSTABLE_SCHED_CLOCK + select HAVE_UNWIND_USER_FP if X86_64 select HAVE_USER_RETURN_NOTIFIER select HAVE_GENERIC_VDSO select VDSO_GETRANDOM if X86_64
arch/x86/include/asm/unwind_user.h new file +11 -0
@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_UNWIND_USER_H +#define _ASM_X86_UNWIND_USER_H + +#define ARCH_INIT_USER_FP_FRAME \ + .cfa_off = (s32)sizeof(long) * 2, \ + .ra_off = (s32)sizeof(long) * -1, \ + .fp_off = (s32)sizeof(long) * -2, \ + .use_fp = true, + +#endif /* _ASM_X86_UNWIND_USER_H */
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 4/17 perf/x86: Rename and move get_segment_base() and make it global
In reply to Steven RostedtFrom: Josh Poimboeuf <[email protected]>
get_segment_base() will be used by the unwind_user code, so make it global and rename it so it doesn't conflict with a KVM function of the same name.
As the function is no longer specific to perf, move it to ptrace.c as that seems to be a better location for a generic function like this.
---
Changes since v4: https://lore.kernel.org/all/e54e48ab228aa386fb4c5f64ee02ef5291f2d3da.1737511963.git.jpoimboe@kernel.org/
- Moved function out of perf and to ptrace.c (Peter Zijlstra)
- Added lockdep_assert_irqs_disabled() as it needs interrupts disabled (Steven Rostedt)
arch/x86/events/core.c | 44 ++++------------------------------- arch/x86/include/asm/ptrace.h | 2 ++ arch/x86/kernel/ptrace.c | 38 ++++++++++++++++++++++++++++++ 3 files changed, 45 insertions(+), 39 deletions(-)
arch/x86/events/core.c +5 -39
@@ -42,6 +42,7 @@#include <asm/ldt.h> #include <asm/unwind.h> #include <asm/uprobes.h> +#include <asm/ptrace.h> #include <asm/ibt.h> #include "perf_event.h"
@@ -2807,41 +2808,6 @@ valid_user_frame(const void __user *fp, unsigned long size)return __access_ok (fp, size); } -static unsigned long get_segment_base (unsigned int segment) -{ -struct desc_struct *desc; -unsigned int idx = segment >>3 ; - -if ((segment &SEGMENT_TI_MASK ) ==SEGMENT_LDT ) { -#ifdef CONFIG_MODIFY_LDT_SYSCALL -struct ldt_struct *ldt; - -/* -* If we're not in a valid context with a real (not just lazy) -* user mm, then don't even try. -*/ -if (!nmi_uaccess_okay ()) -return 0 ; - -/* IRQs are off, so this synchronizes with smp_store_release */ - ldt =smp_load_acquire (¤t->mm->context.ldt); -if (!ldt || idx >= ldt->nr_entries) -return 0 ; - - desc = &ldt->entries[idx]; -#else -return 0 ; -#endif - }else { -if (idx >=GDT_ENTRIES ) -return 0 ; - - desc =raw_cpu_ptr (gdt_page.gdt) + idx; - } - -return get_desc_base (desc); -} -#ifdef CONFIG_UPROBES /* * Heuristic-based check if uprobe is installed at the function entry.
@@ -2898,8 +2864,8 @@ perf_callchain_user32(struct pt_regs *regs, struct perf_callchain_entry_ctx *entif (user_64bit_mode (regs))return 0 ; - cs_base =get_segment_base (regs->cs); - ss_base =get_segment_base (regs->ss); + cs_base =segment_base_address (regs->cs); + ss_base =segment_base_address (regs->ss); fp =compat_ptr (ss_base + regs->bp);pagefault_disable ();
@@ -3018,11 +2984,11 @@ static unsigned long code_segment_base(struct pt_regs *regs)return 0x10 * regs->cs;if (user_mode (regs) && regs->cs != __USER_CS) -return get_segment_base (regs->cs); +return segment_base_address (regs->cs);#else if (user_mode (regs) && !user_64bit_mode (regs) && regs->cs != __USER32_CS) -return get_segment_base (regs->cs); +return segment_base_address (regs->cs);#endif return 0 ; }
arch/x86/include/asm/ptrace.h +2 -0
@@ -314,6 +314,8 @@ static __always_inline bool regs_irqs_disabled(struct pt_regs *regs)return !(regs->flags &X86_EFLAGS_IF ); } +unsigned long segment_base_address (unsigned int segment); +/* Query offset/name of register from its name/offset */ extern int regs_query_register_offset (const char *name);extern const char *regs_query_register_name (unsigned int offset);
arch/x86/kernel/ptrace.c +38 -0
@@ -41,6 +41,7 @@#include <asm/syscall.h> #include <asm/fsgsbase.h> #include <asm/io_bitmap.h> +#include <asm/mmu_context.h> #include "tls.h"
@@ -339,6 +340,43 @@ static int set_segment_reg(struct task_struct *task,#endif /* CONFIG_X86_32 */ +unsigned long segment_base_address (unsigned int segment) +{ +struct desc_struct *desc; +unsigned int idx = segment >>3 ; + +lockdep_assert_irqs_disabled (); + +if ((segment &SEGMENT_TI_MASK ) ==SEGMENT_LDT ) { +#ifdef CONFIG_MODIFY_LDT_SYSCALL +struct ldt_struct *ldt; + +/* +* If we're not in a valid context with a real (not just lazy) +* user mm, then don't even try. +*/ +if (!nmi_uaccess_okay ()) +return 0 ; + +/* IRQs are off, so this synchronizes with smp_store_release */ + ldt =smp_load_acquire (¤t->mm->context.ldt); +if (!ldt || idx >= ldt->nr_entries) +return 0 ; + + desc = &ldt->entries[idx]; +#else +return 0 ; +#endif + }else { +if (idx >=GDT_ENTRIES ) +return 0 ; + + desc =raw_cpu_ptr (gdt_page.gdt) + idx; + } + +return get_desc_base (desc); +} +static unsigned long get_flags (struct task_struct *task) {unsigned long retval =task_pt_regs (task)->flags;
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 5/17 unwind_user: Add compat mode frame pointer support
In reply to Steven RostedtFrom: Josh Poimboeuf <[email protected]>
Add optional support for user space compat mode frame pointer unwinding. If supported, the arch needs to enable CONFIG_HAVE_UNWIND_USER_COMPAT_FP and define ARCH_INIT_USER_COMPAT_FP_FRAME.
--- arch/Kconfig | 4 +++ include/asm-generic/Kbuild | 2 ++ include/asm-generic/unwind_user.h | 15 +++++++++++ include/asm-generic/unwind_user_types.h | 9 +++++++ include/linux/unwind_user_types.h | 3 +++ kernel/unwind/user.c | 36 ++++++++++++++++++++++--- 6 files changed, 65 insertions(+), 4 deletions(-) create mode 100644 include/asm-generic/unwind_user_types.h
arch/Kconfig +4 -0
@@ -442,6 +442,10 @@ config HAVE_UNWIND_USER_FP bool select UNWIND_USER +config HAVE_UNWIND_USER_COMPAT_FP + bool + depends on HAVE_UNWIND_USER_FP + config HAVE_PERF_REGS bool help
include/asm-generic/Kbuild +2 -0
@@ -59,6 +59,8 @@ mandatory-y += tlbflush.h mandatory-y += topology.h mandatory-y += trace_clock.h mandatory-y += uaccess.h +mandatory-y += unwind_user.h +mandatory-y += unwind_user_types.h mandatory-y += vermagic.h mandatory-y += vga.h mandatory-y += video.h
include/asm-generic/unwind_user.h +15 -0
@@ -2,8 +2,23 @@#ifndef _ASM_GENERIC_UNWIND_USER_H#define _ASM_GENERIC_UNWIND_USER_H +#include <asm/unwind_user_types.h> +#ifndef ARCH_INIT_USER_FP_FRAME #define ARCH_INIT_USER_FP_FRAME #endif +#ifndef ARCH_INIT_USER_COMPAT_FP_FRAME +#define ARCH_INIT_USER_COMPAT_FP_FRAME +#define in_compat_mode (regs) false +#endif + +#ifndef arch_unwind_user_init +static inline void arch_unwind_user_init (struct unwind_user_state *state,struct pt_regs *reg) {} +#endif + +#ifndef arch_unwind_user_next +static inline void arch_unwind_user_next (struct unwind_user_state *state) {} +#endif +#endif /* _ASM_GENERIC_UNWIND_USER_H */
include/asm-generic/unwind_user_types.h new file +9 -0
@@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_GENERIC_UNWIND_USER_TYPES_H +#define _ASM_GENERIC_UNWIND_USER_TYPES_H + +#ifndef arch_unwind_user_state +struct arch_unwind_user_state {}; +#endif + +#endif /* _ASM_GENERIC_UNWIND_USER_TYPES_H */
include/linux/unwind_user_types.h +3 -0
@@ -3,10 +3,12 @@#define _LINUX_UNWIND_USER_TYPES_H#include <linux/types.h> +#include <asm/unwind_user_types.h> enum unwind_user_type {UNWIND_USER_TYPE_NONE ,UNWIND_USER_TYPE_FP , +UNWIND_USER_TYPE_COMPAT_FP , };struct unwind_stacktrace {
@@ -25,6 +27,7 @@ struct unwind_user_state {unsigned long ip;unsigned long sp;unsigned long fp; +struct arch_unwind_user_state arch;enum unwind_user_type type;bool done; };
kernel/unwind/user.c +32 -4
@@ -13,12 +13,32 @@ static struct unwind_user_frame fp_frame = { ARCH_INIT_USER_FP_FRAME }; +static struct unwind_user_frame compat_fp_frame = { +ARCH_INIT_USER_COMPAT_FP_FRAME +}; +static inline bool fp_state (struct unwind_user_state *state) {return IS_ENABLED (CONFIG_HAVE_UNWIND_USER_FP ) && state->type ==UNWIND_USER_TYPE_FP ; } +static inline bool compat_state (struct unwind_user_state *state) +{ +return IS_ENABLED (CONFIG_HAVE_UNWIND_USER_COMPAT_FP ) && + state->type ==UNWIND_USER_TYPE_COMPAT_FP ; +} + +#define UNWIND_GET_USER_LONG (to, from, state) \ +({ \ + int __ret; \ + if (compat_state(state)) \ + __ret = get_user(to, (u32 __user *)(from)); \ + else \ + __ret = get_user(to, (u64 __user *)(from)); \ + __ret; \ +}) +int unwind_user_next (struct unwind_user_state *state) {struct unwind_user_frame _frame;
@@ -28,7 +48,9 @@ int unwind_user_next(struct unwind_user_state *state)if (state->done)return -EINVAL ; -if (fp_state (state)) +if (compat_state (state)) + frame = &compat_fp_frame; +else if (fp_state (state)) frame = &fp_frame;else goto the_end;
@@ -39,10 +61,10 @@ int unwind_user_next(struct unwind_user_state *state)if (cfa <= state->sp) goto the_end; -if (get_user (ra, (unsigned long *)(cfa + frame->ra_off))) +if (UNWIND_GET_USER_LONG (ra, cfa + frame->ra_off, state)) goto the_end; -if (frame->fp_off &&get_user (fp, (unsigned long __user *)(cfa + frame->fp_off))) +if (frame->fp_off &&UNWIND_GET_USER_LONG (fp, cfa + frame->fp_off, state)) goto the_end; state->ip = ra;
@@ -50,6 +72,8 @@ int unwind_user_next(struct unwind_user_state *state)if (frame->fp_off) state->fp = fp; +arch_unwind_user_next (state); +return 0 ; the_end:
@@ -68,7 +92,9 @@ int unwind_user_start(struct unwind_user_state *state)return -EINVAL ; } -if (IS_ENABLED (CONFIG_HAVE_UNWIND_USER_FP )) +if (IS_ENABLED (CONFIG_HAVE_UNWIND_USER_COMPAT_FP ) &&in_compat_mode (regs)) + state->type =UNWIND_USER_TYPE_COMPAT_FP ; +else if (IS_ENABLED (CONFIG_HAVE_UNWIND_USER_FP )) state->type =UNWIND_USER_TYPE_FP ;else state->type =UNWIND_USER_TYPE_NONE ;
@@ -77,6 +103,8 @@ int unwind_user_start(struct unwind_user_state *state) state->sp =user_stack_pointer (regs); state->fp =frame_pointer (regs); +arch_unwind_user_init (state, regs); +return 0 ; }
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 6/17 unwind_user/x86: Enable compat mode frame pointer unwinding on x86
In reply to Steven RostedtFrom: Josh Poimboeuf <[email protected]>
Use ARCH_INIT_USER_COMPAT_FP_FRAME to describe how frame pointers are unwound on x86, and implement the hooks needed to add the segment base addresses. Enable HAVE_UNWIND_USER_COMPAT_FP if the system has compat mode compiled in.
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/unwind_user.h | 50 ++++++++++++++++++++++++ arch/x86/include/asm/unwind_user_types.h | 17 ++++++++ 3 files changed, 68 insertions(+) create mode 100644 arch/x86/include/asm/unwind_user_types.h
arch/x86/Kconfig +1 -0
@@ -301,6 +301,7 @@ config X86 select HAVE_SYSCALL_TRACEPOINTS select HAVE_UACCESS_VALIDATION if HAVE_OBJTOOL select HAVE_UNSTABLE_SCHED_CLOCK + select HAVE_UNWIND_USER_COMPAT_FP if IA32_EMULATION select HAVE_UNWIND_USER_FP if X86_64 select HAVE_USER_RETURN_NOTIFIER select HAVE_GENERIC_VDSO
arch/x86/include/asm/unwind_user.h +50 -0
60 lines not on this page · raw message
arch/x86/include/asm/unwind_user_types.h new file +17 -0
@@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_UNWIND_USER_TYPES_H +#define _ASM_UNWIND_USER_TYPES_H + +#ifdef CONFIG_IA32_EMULATION + +struct arch_unwind_user_state { +unsigned long ss_base; +unsigned long cs_base; +}; +#define arch_unwind_user_state arch_unwind_user_state + +#endif /* CONFIG_IA32_EMULATION */ + +#include <asm-generic/unwind_user_types.h> + +#endif /* _ASM_UNWIND_USER_TYPES_H */
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 7/17 unwind_user/deferred: Add unwind_deferred_trace()
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
Add a function that must be called inside a faultable context that will retrieve a user space stack trace. The function unwind_deferred_trace() can be called by a tracer when a task is about to enter user space, or has just come back from user space and has interrupts enabled.
This code is based on work by Josh Poimboeuf's deferred unwinding code:
--- include/linux/sched.h | 5 +++ include/linux/unwind_deferred.h | 24 +++++++++++++ include/linux/unwind_deferred_types.h | 9 +++++ kernel/fork.c | 4 +++ kernel/unwind/Makefile | 2 +- kernel/unwind/deferred.c | 49 +++++++++++++++++++++++++++ 6 files changed, 92 insertions(+), 1 deletion(-) create mode 100644 include/linux/unwind_deferred.h create mode 100644 include/linux/unwind_deferred_types.h create mode 100644 kernel/unwind/deferred.c
include/linux/sched.h +5 -0
@@ -47,6 +47,7 @@#include <linux/livepatch_sched.h> #include <linux/uidgid_types.h> #include <linux/tracepoint-defs.h> +#include <linux/unwind_deferred_types.h> #include <asm/kmap_size.h> /* task_struct member predeclarations (sorted alphabetically): */
@@ -1646,6 +1647,10 @@ struct task_struct {struct user_event_mm *user_event_mm;#endif +#ifdef CONFIG_UNWIND_USER +struct unwind_task_info unwind_info; +#endif +/* CPU-specific state of this task: */ struct thread_struct thread;
include/linux/unwind_deferred.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_UNWIND_USER_DEFERRED_H +#define _LINUX_UNWIND_USER_DEFERRED_H + +#include <linux/unwind_user.h> +#include <linux/unwind_deferred_types.h> + +#ifdef CONFIG_UNWIND_USER + +void unwind_task_init (struct task_struct *task); +void unwind_task_free (struct task_struct *task); + +int unwind_deferred_trace (struct unwind_stacktrace *trace); + +#else /* !CONFIG_UNWIND_USER */ + +static inline void unwind_task_init (struct task_struct *task) {} +static inline void unwind_task_free (struct task_struct *task) {} + +static inline int unwind_deferred_trace (struct unwind_stacktrace *trace) {return -ENOSYS ; } + +#endif /* !CONFIG_UNWIND_USER */ + +#endif /* _LINUX_UNWIND_USER_DEFERRED_H */
include/linux/unwind_deferred_types.h new file +9 -0
@@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_UNWIND_USER_DEFERRED_TYPES_H +#define _LINUX_UNWIND_USER_DEFERRED_TYPES_H + +struct unwind_task_info { +unsigned long *entries; +}; + +#endif /* _LINUX_UNWIND_USER_DEFERRED_TYPES_H */
kernel/fork.c +4 -0
@@ -105,6 +105,7 @@#include <uapi/linux/pidfd.h> #include <linux/pidfs.h> #include <linux/tick.h> +#include <linux/unwind_deferred.h> #include <asm/pgalloc.h> #include <linux/uaccess.h>
@@ -991,6 +992,7 @@ void __put_task_struct(struct task_struct *tsk)WARN_ON (refcount_read (&tsk->usage));WARN_ON (tsk == current); +unwind_task_free (tsk);sched_ext_free (tsk);io_uring_free (tsk);cgroup_free (tsk);
@@ -2395,6 +2397,8 @@ __latent_entropy struct task_struct *copy_process( p->bpf_ctx =NULL ;#endif +unwind_task_init (p); +/* Perform scheduler related setup. Assign this task to a CPU. */ retval =sched_fork (clone_flags, p);if (retval)
kernel/unwind/Makefile +1 -1
@@ -1 +1 @@ - obj-$(CONFIG_UNWIND_USER) += user.o + obj-$(CONFIG_UNWIND_USER) += user.o deferred.o
kernel/unwind/deferred.c new file +49 -0
49 lines not on this page · raw message
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 8/17 unwind_user/deferred: Add unwind cache
In reply to Steven RostedtFrom: Josh Poimboeuf <[email protected]>
Cache the results of the unwind to ensure the unwind is only performed once, even when called by multiple tracers.
The cache nr_entries gets cleared every time the task enters the kernel. When a stacktrace is requested, nr_entries gets set to the number of entries in the stacktrace. If another stacktrace is requested, if nr_entries is not zero, then it contains the same stacktrace that would be retrieved so it is not processed again and the entries is given to the caller.
---
Changes since v4: https://lore.kernel.org/all/51855c0902486060cd6e1ccc6b22fd092a2e676d.1737511963.git.jpoimboe@kernel.org/
- Instead of using a cookie to determine if the cached stacktrace is the same, use nr_entries. By clearing it every time the task enters the kernel, if it's not zero, then that means the stacktrace is still in the cache. (Peter Zijlstra)
include/linux/entry-common.h | 2 ++ include/linux/unwind_deferred.h | 7 +++++++ include/linux/unwind_deferred_types.h | 7 ++++++- kernel/unwind/deferred.c | 27 ++++++++++++++++++++------- 4 files changed, 35 insertions(+), 8 deletions(-)
include/linux/entry-common.h +2 -0
@@ -12,6 +12,7 @@#include <linux/resume_user_mode.h> #include <linux/tick.h> #include <linux/kmsan.h> +#include <linux/unwind_deferred.h> #include <asm/entry-common.h>
@@ -111,6 +112,7 @@ static __always_inline void enter_from_user_mode(struct pt_regs *regs)CT_WARN_ON (__ct_state () !=CT_STATE_USER );user_exit_irqoff (); +unwind_enter_from_user_mode ();instrumentation_begin ();kmsan_unpoison_entry_regs (regs);
include/linux/unwind_deferred.h +7 -0
@@ -12,6 +12,11 @@ void unwind_task_free(struct task_struct *task);int unwind_deferred_trace (struct unwind_stacktrace *trace); +static __always_inline void unwind_enter_from_user_mode(void) +{ + current->unwind_info.cache.nr_entries =0 ; +} +#else /* !CONFIG_UNWIND_USER */ static inline void unwind_task_init (struct task_struct *task) {}
@@ -19,6 +24,8 @@ static inline void unwind_task_free(struct task_struct *task) {}static inline int unwind_deferred_trace (struct unwind_stacktrace *trace) {return -ENOSYS ; } +static inline void unwind_enter_from_user_mode (void ) {} +#endif /* !CONFIG_UNWIND_USER */ #endif /* _LINUX_UNWIND_USER_DEFERRED_H */
include/linux/unwind_deferred_types.h +6 -1
@@ -2,8 +2,13 @@#ifndef _LINUX_UNWIND_USER_DEFERRED_TYPES_H#define _LINUX_UNWIND_USER_DEFERRED_TYPES_H -struct unwind_task_info { +struct unwind_cache {unsigned long *entries; +unsigned int nr_entries; +}; + +struct unwind_task_info { +struct unwind_cache cache; };#endif /* _LINUX_UNWIND_USER_DEFERRED_TYPES_H */
kernel/unwind/deferred.c +20 -7
@@ -12,6 +12,7 @@int unwind_deferred_trace (struct unwind_stacktrace *trace) {struct unwind_task_info *info = ¤t->unwind_info; +struct unwind_cache *cache = &info->cache;/* Should always be called from faultable context */ might_fault ();
@@ -20,17 +21,29 @@ int unwind_deferred_trace(struct unwind_stacktrace *trace)if (!current->mm)return -EINVAL ; -if (!info->entries) { - info->entries =kmalloc_array (UNWIND_MAX_ENTRIES ,sizeof (long ), -GFP_KERNEL ); -if (!info->entries) -return -ENOMEM ; +if (!cache->entries) { + cache->entries =kmalloc_array (UNWIND_MAX_ENTRIES ,sizeof (long ), +GFP_KERNEL ); +if (!cache->entries) +return -ENOMEM ; + } + + trace->entries = cache->entries; + +if (cache->nr_entries) { +/* +* The user stack has already been previously unwound in this +* entry context. Skip the unwind and use the cache. +*/ + trace->nr = cache->nr_entries; +return 0 ; } trace->nr =0 ; - trace->entries = info->entries;unwind_user (trace,UNWIND_MAX_ENTRIES ); + cache->nr_entries = trace->nr; +return 0 ; }
@@ -45,5 +58,5 @@ void unwind_task_free(struct task_struct *task) {struct unwind_task_info *info = &task->unwind_info; -kfree (info->entries); +kfree (info->cache.entries); }
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 9/17 perf: Remove get_perf_callchain() init_nr argument
In reply to Steven RostedtFrom: Josh Poimboeuf <[email protected]>
The 'init_nr' argument has double duty: it's used to initialize both the number of contexts and the number of stack entries. That's confusing and the callers always pass zero anyway. Hard code the zero.
--- include/linux/perf_event.h | 2 +- kernel/bpf/stackmap.c | 4 ++-- kernel/events/callchain.c | 12 ++++++------ kernel/events/core.c | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-)
include/linux/perf_event.h +1 -1
@@ -1651,7 +1651,7 @@ DECLARE_PER_CPU(struct perf_callchain_entry, perf_callchain_entry);extern void perf_callchain_user (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs);extern void perf_callchain_kernel (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs);extern struct perf_callchain_entry * -get_perf_callchain (struct pt_regs *regs,u32 init_nr,bool kernel,bool user, +get_perf_callchain (struct pt_regs *regs,bool kernel,bool user,u32 max_stack,bool crosstask,bool add_mark);extern int get_callchain_buffers (int max_stack);extern void put_callchain_buffers (void );
kernel/bpf/stackmap.c +2 -2
@@ -314,7 +314,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map,if (max_depth > sysctl_perf_event_max_stack) max_depth = sysctl_perf_event_max_stack; - trace =get_perf_callchain (regs,0 , kernel, user, max_depth, + trace =get_perf_callchain (regs, kernel, user, max_depth, false, false);if (unlikely (!trace))
@@ -451,7 +451,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,else if (kernel && task) trace =get_callchain_entry_for_task (task, max_depth);else - trace =get_perf_callchain (regs,0 , kernel, user, max_depth, + trace =get_perf_callchain (regs, kernel, user, max_depth, crosstask, false);if (unlikely (!trace) || trace->nr < skip) {
kernel/events/callchain.c +6 -6
@@ -217,7 +217,7 @@ static void fixup_uretprobe_trampoline_entries(struct perf_callchain_entry *entr }struct perf_callchain_entry * -get_perf_callchain (struct pt_regs *regs,u32 init_nr,bool kernel,bool user, +get_perf_callchain (struct pt_regs *regs,bool kernel,bool user,u32 max_stack,bool crosstask,bool add_mark) {struct perf_callchain_entry *entry;
@@ -228,11 +228,11 @@ get_perf_callchain(struct pt_regs *regs, u32 init_nr, bool kernel, bool user,if (!entry)return NULL ; - ctx.entry = entry; - ctx.max_stack = max_stack; - ctx.nr = entry->nr = init_nr; - ctx.contexts =0 ; - ctx.contexts_maxed = false; + ctx.entry = entry; + ctx.max_stack = max_stack; + ctx.nr = entry->nr =0 ; + ctx.contexts =0 ; + ctx.contexts_maxed = false;if (kernel && !user_mode (regs)) {if (add_mark)
kernel/events/core.c +1 -1
@@ -8110,7 +8110,7 @@ perf_callchain(struct perf_event *event, struct pt_regs *regs)if (!kernel && !user)return &__empty_callchain; - callchain =get_perf_callchain (regs,0 , kernel, user, + callchain =get_perf_callchain (regs, kernel, user, max_stack, crosstask, true);return callchain ?: &__empty_callchain; }
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 10/17 perf: Have get_perf_callchain() return NULL if crosstask and user are set
In reply to Steven RostedtFrom: Josh Poimboeuf <[email protected]>
get_perf_callchain() doesn't support cross-task unwinding for user space stacks, have it return NULL if both the crosstask and user arguments are set.
---
Changes since v4: https://lore.kernel.org/all/73229250c0279667f617c3d6da121c6621164e4d.1737511963.git.jpoimboe@kernel.org/
- The crosstask is still used for kernel stacks, so that use case is still need. (Andrii Nakryiko)
kernel/events/callchain.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/events/callchain.c +4 -4
@@ -224,6 +224,10 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user,struct perf_callchain_entry_ctx ctx;int rctx, start_entry_idx; +/* crosstask is not supported for user stacks */ +if (crosstask && user) +return NULL ; + entry =get_callchain_entry (&rctx);if (!entry)return NULL ;
@@ -249,9 +253,6 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, }if (regs) { -if (crosstask) - goto exit_put; -if (add_mark)perf_callchain_store_context (&ctx,PERF_CONTEXT_USER );
@@ -261,7 +262,6 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, } } -exit_put:put_callchain_entry (rctx);return entry;
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 11/17 perf: Simplify get_perf_callchain() user logic
In reply to Steven RostedtFrom: Josh Poimboeuf <[email protected]>
Simplify the get_perf_callchain() user logic a bit. task_pt_regs() should never be NULL.
--- kernel/events/callchain.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-)
kernel/events/callchain.c +9 -11
@@ -246,22 +246,20 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user,if (user) {if (!user_mode (regs)) { -if (current->mm) - regs =task_pt_regs (current); -else - regs =NULL ; +if (!current->mm) + goto exit_put; + regs =task_pt_regs (current); } -if (regs) { -if (add_mark) -perf_callchain_store_context (&ctx,PERF_CONTEXT_USER ); +if (add_mark) +perf_callchain_store_context (&ctx,PERF_CONTEXT_USER ); - start_entry_idx = entry->nr; -perf_callchain_user (&ctx, regs); -fixup_uretprobe_trampoline_entries (entry, start_entry_idx); - } + start_entry_idx = entry->nr; +perf_callchain_user (&ctx, regs); +fixup_uretprobe_trampoline_entries (entry, start_entry_idx); } +exit_put:put_callchain_entry (rctx);return entry;
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 12/17 perf: Skip user unwind if !current->mm
In reply to Steven RostedtFrom: Josh Poimboeuf <[email protected]>
If the task doesn't have any memory, there's no stack to unwind.
---
Note from v4: https://lore.kernel.org/all/24cde1381495cd637f0d5663ab7629572bc0eca6.1737511963.git.jpoimboe@kernel.org/
- From the previous version, Peter Zijlstra brought up concern about io_uring helpers, but this was never resolved. No update was made.
kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/events/core.c +1 -1
@@ -8101,7 +8101,7 @@ struct perf_callchain_entry * perf_callchain(struct perf_event *event, struct pt_regs *regs) { bool kernel = !event->attr.exclude_callchain_kernel; -bool user = !event->attr.exclude_callchain_user; +bool user = !event->attr.exclude_callchain_user && current->mm;/* Disallow cross-task user callchains. */ bool crosstask = event->ctx->task && event->ctx->task != current;const u32 max_stack = event->attr.sample_max_stack;
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 13/17 perf: Support deferred user callchains
In reply to Steven RostedtFrom: Josh Poimboeuf <[email protected]>
Use the new unwind_deferred_trace() interface (if available) to defer unwinds to task context. This will allow the use of .sframe (when it becomes available) and also prevents duplicate userspace unwinds.
--- arch/Kconfig | 3 + include/linux/perf_event.h | 13 +- include/uapi/linux/perf_event.h | 19 ++- kernel/bpf/stackmap.c | 4 +- kernel/events/callchain.c | 11 +- kernel/events/core.c | 163 +++++++++++++++++++++++++- tools/include/uapi/linux/perf_event.h | 19 ++- 7 files changed, 224 insertions(+), 8 deletions(-)
arch/Kconfig +3 -0
@@ -446,6 +446,9 @@ config HAVE_UNWIND_USER_COMPAT_FP bool depends on HAVE_UNWIND_USER_FP +config HAVE_PERF_CALLCHAIN_DEFERRED + bool + config HAVE_PERF_REGS bool help
include/linux/perf_event.h +12 -1
@@ -62,6 +62,7 @@ struct perf_guest_info_callbacks {#include <linux/security.h> #include <linux/static_call.h> #include <linux/lockdep.h> +#include <linux/unwind_deferred.h> #include <asm/local.h> struct perf_callchain_entry {
@@ -830,6 +831,10 @@ struct perf_event {struct callback_head pending_task;unsigned int pending_work; +unsigned int pending_unwind_callback; +struct callback_head pending_unwind_work; +struct rcuwait pending_unwind_wait; +atomic_t event_limit;/* address range filters */
@@ -1652,12 +1657,18 @@ extern void perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pextern void perf_callchain_kernel (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs);extern struct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user, -u32 max_stack,bool crosstask,bool add_mark); +u32 max_stack,bool crosstask,bool add_mark,bool defer_user);extern int get_callchain_buffers (int max_stack);extern void put_callchain_buffers (void );extern struct perf_callchain_entry *get_callchain_entry (int *rctx);extern void put_callchain_entry (int rctx); +#ifdef CONFIG_HAVE_PERF_CALLCHAIN_DEFERRED +extern void perf_callchain_user_deferred (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs); +#else +static inline void perf_callchain_user_deferred (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs) {} +#endif +extern int sysctl_perf_event_max_stack;extern int sysctl_perf_event_max_contexts_per_stack;
include/uapi/linux/perf_event.h +18 -1
@@ -462,7 +462,8 @@ struct perf_event_attr { inherit_thread : 1, /* children only inherit if cloned with CLONE_THREAD */ remove_on_exec : 1, /* event is removed from task on exec */ sigtrap : 1, /* send synchronous SIGTRAP on event */ - __reserved_1 : 26; + defer_callchain: 1, /* generate PERF_RECORD_CALLCHAIN_DEFERRED records */ + __reserved_1 : 25;union {__u32 wakeup_events;/* wakeup every n events */
@@ -1228,6 +1229,21 @@ enum perf_event_type {*/ PERF_RECORD_AUX_OUTPUT_HW_ID =21 , +/* +* This user callchain capture was deferred until shortly before +* returning to user space. Previous samples would have kernel +* callchains only and they need to be stitched with this to make full +* callchains. +* +* struct { +* struct perf_event_header header; +* u64 nr; +* u64 ips[nr]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_CALLCHAIN_DEFERRED =22 , + PERF_RECORD_MAX, /* non-ABI */ };
@@ -1258,6 +1274,7 @@ enum perf_callchain_context {PERF_CONTEXT_HV = (__u64)-32 ,PERF_CONTEXT_KERNEL = (__u64)-128 ,PERF_CONTEXT_USER = (__u64)-512 , +PERF_CONTEXT_USER_DEFERRED = (__u64)-640 ,PERF_CONTEXT_GUEST = (__u64)-2048 , PERF_CONTEXT_GUEST_KERNEL = (__u64)-2176,
kernel/bpf/stackmap.c +2 -2
@@ -315,7 +315,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, max_depth = sysctl_perf_event_max_stack; trace =get_perf_callchain (regs, kernel, user, max_depth, - false, false); + false, false, false);if (unlikely (!trace))/* couldn't fetch the stack trace */
@@ -452,7 +452,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, trace =get_callchain_entry_for_task (task, max_depth); else trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask, false); + crosstask, false, false);if (unlikely (!trace) || trace->nr < skip) {if (may_fault)
kernel/events/callchain.c +10 -1
@@ -218,7 +218,7 @@ static void fixup_uretprobe_trampoline_entries(struct perf_callchain_entry *entrstruct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user, -u32 max_stack,bool crosstask,bool add_mark) +u32 max_stack,bool crosstask,bool add_mark,bool defer_user) {struct perf_callchain_entry *entry;struct perf_callchain_entry_ctx ctx;
@@ -251,6 +251,15 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, regs =task_pt_regs (current); } +if (defer_user) { +/* +* Foretell the coming of PERF_RECORD_CALLCHAIN_DEFERRED +* which can be stitched to this one. +*/ +perf_callchain_store_context (&ctx,PERF_CONTEXT_USER_DEFERRED ); + goto exit_put; + } +if (add_mark)perf_callchain_store_context (&ctx,PERF_CONTEXT_USER );
kernel/events/core.c +161 -2
203 lines not on this page · raw message
tools/include/uapi/linux/perf_event.h +18 -1
@@ -462,7 +462,8 @@ struct perf_event_attr { inherit_thread : 1, /* children only inherit if cloned with CLONE_THREAD */ remove_on_exec : 1, /* event is removed from task on exec */ sigtrap : 1, /* send synchronous SIGTRAP on event */ - __reserved_1 : 26; + defer_callchain: 1, /* generate PERF_RECORD_CALLCHAIN_DEFERRED records */ + __reserved_1 : 25;union {__u32 wakeup_events;/* wakeup every n events */
@@ -1228,6 +1229,21 @@ enum perf_event_type {*/ PERF_RECORD_AUX_OUTPUT_HW_ID =21 , +/* +* This user callchain capture was deferred until shortly before +* returning to user space. Previous samples would have kernel +* callchains only and they need to be stitched with this to make full +* callchains. +* +* struct { +* struct perf_event_header header; +* u64 nr; +* u64 ips[nr]; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_CALLCHAIN_DEFERRED =22 , + PERF_RECORD_MAX, /* non-ABI */ };
@@ -1258,6 +1274,7 @@ enum perf_callchain_context {PERF_CONTEXT_HV = (__u64)-32 ,PERF_CONTEXT_KERNEL = (__u64)-128 ,PERF_CONTEXT_USER = (__u64)-512 , +PERF_CONTEXT_USER_DEFERRED = (__u64)-640 ,PERF_CONTEXT_GUEST = (__u64)-2048 , PERF_CONTEXT_GUEST_KERNEL = (__u64)-2176,
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 14/17 perf tools: Minimal CALLCHAIN_DEFERRED support
In reply to Steven RostedtFrom: Namhyung Kim <[email protected]>
Add a new event type for deferred callchains and a new callback for the struct perf_tool. For now it doesn't actually handle the deferred callchains but it just marks the sample if it has the PERF_CONTEXT_ USER_DEFFERED in the callchain array.
At least, perf report can dump the raw data with this change. Actually this requires the next commit to enable attr.defer_callchain, but if you already have a data file, it'll show the following result.
$ perf report -D ... [email protected] [0x40]: event: 22 . . ... raw event: size 64 bytes . 0000: 16 00 00 00 02 00 40 00 02 00 00 00 00 00 00 00 ......@......... . 0010: 00 fe ff ff ff ff ff ff 4b d3 3f 25 45 7f 00 00 ........K.?%E... . 0020: 21 03 00 00 21 03 00 00 43 02 12 ab 05 00 00 00 !...!...C....... . 0030: 00 00 00 00 00 00 00 00 09 00 00 00 00 00 00 00 ................
0 24344920643 0x5fe0 [0x40]: PERF_RECORD_CALLCHAIN_DEFERRED(IP, 0x2): 801/801: 0 ... FP chain: nr:2
..... 0: fffffffffffffe00
..... 1: 00007f45253fd34b
: unhandled!
--- tools/lib/perf/include/perf/event.h | 7 +++++++ tools/perf/util/event.c | 1 + tools/perf/util/evsel.c | 15 +++++++++++++++ tools/perf/util/machine.c | 1 + tools/perf/util/perf_event_attr_fprintf.c | 1 + tools/perf/util/sample.h | 3 ++- tools/perf/util/session.c | 17 +++++++++++++++++ tools/perf/util/tool.c | 1 + tools/perf/util/tool.h | 3 ++- 9 files changed, 47 insertions(+), 2 deletions(-)
tools/lib/perf/include/perf/event.h +7 -0
@@ -151,6 +151,12 @@ struct perf_record_switch {__u32 next_prev_tid; }; +struct perf_record_callchain_deferred { +struct perf_event_header header; +__u64 nr; +__u64 ips[]; +}; +struct perf_record_header_attr {struct perf_event_header header;struct perf_event_attr attr;
@@ -494,6 +500,7 @@ union perf_event {struct perf_record_read read;struct perf_record_throttle throttle;struct perf_record_sample sample; +struct perf_record_callchain_deferred callchain_deferred;struct perf_record_bpf_event bpf;struct perf_record_ksymbol ksymbol;struct perf_record_text_poke_event text_poke;
tools/perf/util/event.c +1 -0
@@ -58,6 +58,7 @@ static const char *perf_event__names[] = { [PERF_RECORD_CGROUP] = "CGROUP", [PERF_RECORD_TEXT_POKE] = "TEXT_POKE", [PERF_RECORD_AUX_OUTPUT_HW_ID] = "AUX_OUTPUT_HW_ID", + [PERF_RECORD_CALLCHAIN_DEFERRED] = "CALLCHAIN_DEFERRED", [PERF_RECORD_HEADER_ATTR] = "ATTR", [PERF_RECORD_HEADER_EVENT_TYPE] = "EVENT_TYPE", [PERF_RECORD_HEADER_TRACING_DATA] = "TRACING_DATA",
tools/perf/util/evsel.c +15 -0
@@ -2948,6 +2948,18 @@ int evsel__parse_sample(struct evsel *evsel, union perf_event *event, data->data_src =PERF_MEM_DATA_SRC_NONE ; data->vcpu =-1 ; +if (event->header.type ==PERF_RECORD_CALLCHAIN_DEFERRED ) { +const u64 max_callchain_nr =UINT64_MAX /sizeof (u64); + + data->callchain = (struct ip_callchain *)&event->callchain_deferred.nr; +if (data->callchain->nr > max_callchain_nr) +return -EFAULT ; + +if (evsel->core.attr.sample_id_all) +perf_evsel__parse_id_sample (evsel, event, data); +return 0 ; + } +if (event->header.type !=PERF_RECORD_SAMPLE ) {if (!evsel->core.attr.sample_id_all)return 0 ;
@@ -3078,6 +3090,9 @@ int evsel__parse_sample(struct evsel *evsel, union perf_event *event,if (data->callchain->nr > max_callchain_nr)return -EFAULT ; sz = data->callchain->nr *sizeof (u64); +if (evsel->core.attr.defer_callchain && data->callchain->nr >=1 && + data->callchain->ips[data->callchain->nr -1 ] ==PERF_CONTEXT_USER_DEFERRED ) + data->deferred_callchain = true;OVERFLOW_CHECK (array, sz, max_size); array = (void *)array + sz; }
tools/perf/util/machine.c +1 -0
@@ -2089,6 +2089,7 @@ static int add_callchain_ip(struct thread *thread, *cpumode =PERF_RECORD_MISC_KERNEL ;break ;case PERF_CONTEXT_USER : +case PERF_CONTEXT_USER_DEFERRED : *cpumode =PERF_RECORD_MISC_USER ;break ;default :
tools/perf/util/perf_event_attr_fprintf.c +1 -0
@@ -343,6 +343,7 @@ int perf_event_attr__fprintf(FILE *fp, struct perf_event_attr *attr,PRINT_ATTRf (inherit_thread, p_unsigned);PRINT_ATTRf (remove_on_exec, p_unsigned);PRINT_ATTRf (sigtrap, p_unsigned); +PRINT_ATTRf (defer_callchain, p_unsigned);PRINT_ATTRn ("{ wakeup_events, wakeup_watermark }" , wakeup_events, p_unsigned, false);PRINT_ATTRf (bp_type, p_unsigned);
tools/perf/util/sample.h +2 -1
@@ -108,7 +108,8 @@ struct perf_sample {u16 p_stage_cyc;u16 retire_lat; }; -bool no_hw_idx;/* No hw_idx collected in branch_stack */ +bool no_hw_idx;/* No hw_idx collected in branch_stack */ +bool deferred_callchain;/* Has deferred user callchains */ char insn[MAX_INSN ];void *raw_data;struct ip_callchain *callchain;
tools/perf/util/session.c +17 -0
@@ -715,6 +715,7 @@ static perf_event__swap_op perf_event__swap_ops[] = { [PERF_RECORD_CGROUP] = perf_event__cgroup_swap, [PERF_RECORD_TEXT_POKE] = perf_event__text_poke_swap, [PERF_RECORD_AUX_OUTPUT_HW_ID] = perf_event__all64_swap, + [PERF_RECORD_CALLCHAIN_DEFERRED] = perf_event__all64_swap, [PERF_RECORD_HEADER_ATTR] = perf_event__hdr_attr_swap, [PERF_RECORD_HEADER_EVENT_TYPE] = perf_event__event_type_swap, [PERF_RECORD_HEADER_TRACING_DATA] = perf_event__tracing_data_swap,
@@ -1118,6 +1119,19 @@ static void dump_sample(struct evsel *evsel, union perf_event *event,sample_read__printf (sample, evsel->core.attr.read_format); } +static void dump_deferred_callchain (struct evsel *evsel,union perf_event *event, +struct perf_sample *sample) +{ +if (!dump_trace) +return ; + +printf ("(IP, 0x%x): %d/%d: %#" PRIx64"\n" , + event->header.misc, sample->pid, sample->tid, sample->ip); + +if (evsel__has_callchain (evsel)) +callchain__printf (evsel, sample); +} +static void dump_read (struct evsel *evsel,union perf_event *event) {struct perf_record_read *read_event = &event->read;
@@ -1348,6 +1362,9 @@ static int machines__deliver_event(struct machines *machines,return tool->text_poke (tool, event, sample, machine);case PERF_RECORD_AUX_OUTPUT_HW_ID :return tool->aux_output_hw_id (tool, event, sample, machine); +case PERF_RECORD_CALLCHAIN_DEFERRED : +dump_deferred_callchain (evsel, event, sample); +return tool->callchain_deferred (tool, event, sample, evsel, machine);default : ++evlist->stats.nr_unknown_events;return -1 ;
tools/perf/util/tool.c +1 -0
@@ -259,6 +259,7 @@ void perf_tool__init(struct perf_tool *tool, bool ordered_events) tool->read = process_event_sample_stub; tool->throttle = process_event_stub; tool->unthrottle = process_event_stub; + tool->callchain_deferred = process_event_sample_stub; tool->attr = process_event_synth_attr_stub; tool->event_update = process_event_synth_event_update_stub; tool->tracing_data = process_event_synth_tracing_data_stub;
tools/perf/util/tool.h +2 -1
@@ -42,7 +42,8 @@ enum show_feature_header {struct perf_tool {event_sample sample, - read; + read, + callchain_deferred;event_op mmap, mmap2, comm,
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 15/17 perf record: Enable defer_callchain for user callchains
In reply to Steven RostedtFrom: Namhyung Kim <[email protected]>
And add the missing feature detection logic to clear the flag on old kernels.
$ perf record -g -vv true
...
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 136
config 0 (PERF_COUNT_HW_CPU_CYCLES)
{ sample_period, sample_freq } 4000
sample_type IP|TID|TIME|CALLCHAIN|PERIOD
read_format ID|LOST
disabled 1
inherit 1
mmap 1
comm 1
freq 1
enable_on_exec 1
task 1
sample_id_all 1
mmap2 1
comm_exec 1
ksymbol 1
bpf_event 1
defer_callchain 1
------------------------------------------------------------
sys_perf_event_open: pid 162755 cpu 0 group_fd -1 flags 0x8
sys_perf_event_open failed, error -22
switching off deferred callchain support--- tools/perf/util/evsel.c | 24 ++++++++++++++++++++++++ tools/perf/util/evsel.h | 1 + 2 files changed, 25 insertions(+)
tools/perf/util/evsel.c +24 -0
@@ -1076,6 +1076,14 @@ static void __evsel__config_callchain(struct evsel *evsel, struct record_opts *o } } +if (param->record_mode ==CALLCHAIN_FP && !attr->exclude_callchain_user) { +/* +* Enable deferred callchains optimistically. It'll be switched +* off later if the kernel doesn't support it. +*/ + attr->defer_callchain =1 ; + } +if (function) {pr_info ("Disabling user space callchains for function trace event.\n" ); attr->exclude_callchain_user =1 ;
@@ -2123,6 +2131,8 @@ static int __evsel__prepare_open(struct evsel *evsel, struct perf_cpu_map *cpus,static void evsel__disable_missing_features (struct evsel *evsel) { +if (perf_missing_features.defer_callchain) + evsel->core.attr.defer_callchain =0 ;if (perf_missing_features.inherit_sample_read && evsel->core.attr.inherit && (evsel->core.attr.sample_type &PERF_SAMPLE_READ )) evsel->core.attr.inherit =0 ;
@@ -2397,6 +2407,15 @@ static bool evsel__detect_missing_features(struct evsel *evsel, struct perf_cpu/* Please add new feature detection here. */ + attr.defer_callchain = true; + attr.sample_type =PERF_SAMPLE_CALLCHAIN ; +if (has_attr_feature (&attr,/*flags=*/ 0 )) + goto found; + perf_missing_features.defer_callchain = true; +pr_debug2 ("switching off deferred callchain support\n" ); + attr.defer_callchain = false; + attr.sample_type =0 ; + attr.inherit = true; attr.sample_type =PERF_SAMPLE_READ ;if (has_attr_feature (&attr,/*flags=*/ 0 ))
@@ -2508,6 +2527,11 @@ static bool evsel__detect_missing_features(struct evsel *evsel, struct perf_cpu errno = old_errno; check: +if (evsel->core.attr.defer_callchain && + evsel->core.attr.sample_type &PERF_SAMPLE_CALLCHAIN && + perf_missing_features.defer_callchain) +return true; +if (evsel->core.attr.inherit && (evsel->core.attr.sample_type &PERF_SAMPLE_READ ) && perf_missing_features.inherit_sample_read)
tools/perf/util/evsel.h +1 -0
@@ -211,6 +211,7 @@ struct perf_missing_features {bool branch_counters;bool aux_action;bool inherit_sample_read; +bool defer_callchain; };extern struct perf_missing_features perf_missing_features;
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 16/17 perf script: Display PERF_RECORD_CALLCHAIN_DEFERRED
In reply to Steven RostedtFrom: Namhyung Kim <[email protected]>
Handle the deferred callchains in the script output.
$ perf script
perf 801 [000] 18.031793: 1 cycles:P:
ffffffff91a14c36 __intel_pmu_enable_all.isra.0+0x56 ([kernel.kallsyms])
ffffffff91d373e9 perf_ctx_enable+0x39 ([kernel.kallsyms])
ffffffff91d36af7 event_function+0xd7 ([kernel.kallsyms])
ffffffff91d34222 remote_function+0x42 ([kernel.kallsyms])
ffffffff91c1ebe1 generic_exec_single+0x61 ([kernel.kallsyms])
ffffffff91c1edac smp_call_function_single+0xec ([kernel.kallsyms])
ffffffff91d37a9d event_function_call+0x10d ([kernel.kallsyms])
ffffffff91d33557 perf_event_for_each_child+0x37 ([kernel.kallsyms])
ffffffff91d47324 _perf_ioctl+0x204 ([kernel.kallsyms])
ffffffff91d47c43 perf_ioctl+0x33 ([kernel.kallsyms])
ffffffff91e2f216 __x64_sys_ioctl+0x96 ([kernel.kallsyms])
ffffffff9265f1ae do_syscall_64+0x9e ([kernel.kallsyms])
ffffffff92800130 entry_SYSCALL_64+0xb0 ([kernel.kallsyms])perf 801 [000] 18.031814: DEFERRED CALLCHAIN
7fb5fc22034b __GI___ioctl+0x3b (/usr/lib/x86_64-linux-gnu/libc.so.6)
--- tools/perf/builtin-script.c | 89 +++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+)
tools/perf/builtin-script.c +89 -0
@@ -2483,6 +2483,93 @@ static int process_sample_event(const struct perf_tool *tool,return ret; } +static int process_deferred_sample_event (const struct perf_tool *tool, +union perf_event *event, +struct perf_sample *sample, +struct evsel *evsel, +struct machine *machine) +{ + struct perf_script *scr = container_of(tool, struct perf_script, tool); +struct perf_event_attr *attr = &evsel->core.attr; +struct evsel_script *es = evsel->priv; +unsigned int type =output_type (attr->type); +struct addr_location al; +FILE *fp = es->fp; +int ret =0 ; + +if (output[type].fields ==0 ) +return 0 ; + +/* Set thread to NULL to indicate addr_al and al are not initialized */ +addr_location__init (&al); + +if (perf_time__ranges_skip_sample (scr->ptime_range, scr->range_num, + sample->time)) { + goto out_put; + } + +if (debug_mode) { +if (sample->time < last_timestamp) { +pr_err ("Samples misordered, previous: %" PRIu64 +" this: %" PRIu64"\n" , last_timestamp, + sample->time); + nr_unordered++; + } + last_timestamp = sample->time; + goto out_put; + } + +if (filter_cpu (sample)) + goto out_put; + +if (machine__resolve (machine, &al, sample) <0 ) { +pr_err ("problem processing %d event, skipping it.\n" , + event->header.type); + ret =-1 ; + goto out_put; + } + +if (al.filtered) + goto out_put; + +if (!show_event (sample, evsel, al.thread, &al,NULL )) + goto out_put; + +if (evswitch__discard (&scr->evswitch, evsel)) + goto out_put; + +perf_sample__fprintf_start (scr, sample, al.thread, evsel, +PERF_RECORD_CALLCHAIN_DEFERRED , fp); +fprintf (fp,"DEFERRED CALLCHAIN" ); + +if (PRINT_FIELD (IP )) { +struct callchain_cursor *cursor =NULL ; + +if (symbol_conf.use_callchain && sample->callchain) { + cursor =get_tls_callchain_cursor (); +if (thread__resolve_callchain (al.thread, cursor, evsel, + sample,NULL ,NULL , + scripting_max_stack)) { +pr_info ("cannot resolve deferred callchains\n" ); + cursor =NULL ; + } + } + +fputc (cursor ?'\n' :' ' , fp); +sample__fprintf_sym (sample, &al,0 , output[type].print_ip_opts, + cursor, symbol_conf.bt_stop_list, fp); + } + +fprintf (fp,"\n" ); + +if (verbose >0 ) +fflush (fp); + +out_put: +addr_location__exit (&al); +return ret; +} +// Used when scr->per_event_dump is not set static struct evsel_script es_stdout;
@@ -4069,6 +4156,7 @@ int cmd_script(int argc, const char **argv)perf_tool__init (&script.tool, !unsorted_dump); script.tool.sample = process_sample_event; + script.tool.callchain_deferred = process_deferred_sample_event; script.tool.mmap = perf_event__process_mmap; script.tool.mmap2 = perf_event__process_mmap2; script.tool.comm = perf_event__process_comm;
@@ -4095,6 +4183,7 @@ int cmd_script(int argc, const char **argv) script.tool.throttle = process_throttle_event; script.tool.unthrottle = process_throttle_event; script.tool.ordering_requires_timestamps = true; + script.tool.merge_deferred_callchains = false; session =perf_session__new (&data, &script.tool);if (IS_ERR (session))return PTR_ERR (session);
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v5 17/17 perf tools: Merge deferred user callchains
In reply to Steven RostedtFrom: Namhyung Kim <[email protected]>
Save samples with deferred callchains in a separate list and deliver them after merging the user callchains. If users don't want to merge they can set tool->merge_deferred_callchains to false to prevent the behavior.
With previous result, now perf script will show the merged callchains.
$ perf script
perf 801 [000] 18.031793: 1 cycles:P:
ffffffff91a14c36 __intel_pmu_enable_all.isra.0+0x56 ([kernel.kallsyms])
ffffffff91d373e9 perf_ctx_enable+0x39 ([kernel.kallsyms])
ffffffff91d36af7 event_function+0xd7 ([kernel.kallsyms])
ffffffff91d34222 remote_function+0x42 ([kernel.kallsyms])
ffffffff91c1ebe1 generic_exec_single+0x61 ([kernel.kallsyms])
ffffffff91c1edac smp_call_function_single+0xec ([kernel.kallsyms])
ffffffff91d37a9d event_function_call+0x10d ([kernel.kallsyms])
ffffffff91d33557 perf_event_for_each_child+0x37 ([kernel.kallsyms])
ffffffff91d47324 _perf_ioctl+0x204 ([kernel.kallsyms])
ffffffff91d47c43 perf_ioctl+0x33 ([kernel.kallsyms])
ffffffff91e2f216 __x64_sys_ioctl+0x96 ([kernel.kallsyms])
ffffffff9265f1ae do_syscall_64+0x9e ([kernel.kallsyms])
ffffffff92800130 entry_SYSCALL_64+0xb0 ([kernel.kallsyms])
7fb5fc22034b __GI___ioctl+0x3b (/usr/lib/x86_64-linux-gnu/libc.so.6)
...The old output can be get using --no-merge-callchain option. Also perf report can get the user callchain entry at the end.
$ perf report --no-children --percent-limit=0 --stdio -q -S __intel_pmu_enable_all.isra.0
# symbol: __intel_pmu_enable_all.isra.0
0.00% perf [kernel.kallsyms]
|
---__intel_pmu_enable_all.isra.0
perf_ctx_enable
event_function
remote_function
generic_exec_single
smp_call_function_single
event_function_call
perf_event_for_each_child
_perf_ioctl
perf_ioctl
__x64_sys_ioctl
do_syscall_64
entry_SYSCALL_64
__GI___ioctl--- tools/perf/Documentation/perf-script.txt | 5 ++ tools/perf/builtin-script.c | 5 +- tools/perf/util/callchain.c | 24 +++++++++ tools/perf/util/callchain.h | 3 ++ tools/perf/util/evlist.c | 1 + tools/perf/util/evlist.h | 1 + tools/perf/util/session.c | 63 +++++++++++++++++++++++- tools/perf/util/tool.c | 1 + tools/perf/util/tool.h | 1 + 9 files changed, 102 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-script.txt +5 -0
@@ -527,6 +527,11 @@ include::itrace.txt[] The known limitations include exception handing such as setjmp/longjmp will have calls/returns not match. +--merge-callchains:: + Enable merging deferred user callchains if available. This is the + default behavior. If you want to see separate CALLCHAIN_DEFERRED + records for some reason, use --no-merge-callchains explicitly. + :GMEXAMPLECMD: script :GMEXAMPLESUBCMD: include::guest-files.txt[]
tools/perf/builtin-script.c +4 -1
@@ -3775,6 +3775,7 @@ int cmd_script(int argc, const char **argv)bool header_only = false;bool script_started = false;bool unsorted_dump = false; +bool merge_deferred_callchains = true;char *rec_script_path =NULL ;char *rep_script_path =NULL ;struct perf_session *session;
@@ -3928,6 +3929,8 @@ int cmd_script(int argc, const char **argv) "Guest code can be found in hypervisor process"),OPT_BOOLEAN ('\0' ,"stitch-lbr" , &script.stitch_lbr,"Enable LBR callgraph stitching approach" ), +OPT_BOOLEAN ('\0' ,"merge-callchains" , &merge_deferred_callchains, +"Enable merge deferred user callchains" ),OPTS_EVSWITCH (&script.evswitch), OPT_END() };
@@ -4183,7 +4186,7 @@ int cmd_script(int argc, const char **argv) script.tool.throttle = process_throttle_event; script.tool.unthrottle = process_throttle_event; script.tool.ordering_requires_timestamps = true; - script.tool.merge_deferred_callchains = false; + script.tool.merge_deferred_callchains = merge_deferred_callchains; session =perf_session__new (&data, &script.tool);if (IS_ERR (session))return PTR_ERR (session);
tools/perf/util/callchain.c +24 -0
@@ -1828,3 +1828,27 @@ int sample__for_each_callchain_node(struct thread *thread, struct evsel *evsel, }return 0 ; } + +int sample__merge_deferred_callchain (struct perf_sample *sample_orig, +struct perf_sample *sample_callchain) +{ +u64 nr_orig = sample_orig->callchain->nr -1 ; +u64 nr_deferred = sample_callchain->callchain->nr; +struct ip_callchain *callchain; + + callchain =calloc (1 + nr_orig + nr_deferred,sizeof (u64)); +if (callchain ==NULL ) { + sample_orig->deferred_callchain = false; +return -ENOMEM ; + } + + callchain->nr = nr_orig + nr_deferred; +/* copy except for the last PERF_CONTEXT_USER_DEFERRED */ +memcpy (callchain->ips, sample_orig->callchain->ips, nr_orig *sizeof (u64)); +/* copy deferred use callchains */ +memcpy (&callchain->ips[nr_orig], sample_callchain->callchain->ips, + nr_deferred *sizeof (u64)); + + sample_orig->callchain = callchain; +return 0 ; +}
tools/perf/util/callchain.h +3 -0
@@ -317,4 +317,7 @@ int sample__for_each_callchain_node(struct thread *thread, struct evsel *evsel, struct perf_sample *sample, int max_stack, bool symbols, callchain_iter_fn cb, void *data); +int sample__merge_deferred_callchain (struct perf_sample *sample_orig, +struct perf_sample *sample_callchain); +#endif /* __PERF_CALLCHAIN_H */
tools/perf/util/evlist.c +1 -0
@@ -82,6 +82,7 @@ void evlist__init(struct evlist *evlist, struct perf_cpu_map *cpus, evlist->ctl_fd.ack =-1 ; evlist->ctl_fd.pos =-1 ; evlist->nr_br_cntr =-1 ; +INIT_LIST_HEAD (&evlist->deferred_samples); }struct evlist *evlist__new (void )
tools/perf/util/evlist.h +1 -0
@@ -84,6 +84,7 @@ struct evlist {int pos;/* index at evlist core object to check signals */ } ctl_fd;struct event_enable_timer *eet; +struct list_head deferred_samples; };struct evsel_str_handler {
tools/perf/util/session.c +62 -1
81 lines not on this page · raw message
tools/perf/util/tool.c +1 -0
@@ -238,6 +238,7 @@ void perf_tool__init(struct perf_tool *tool, bool ordered_events) tool->cgroup_events = false; tool->no_warn = false; tool->show_feat_hdr =SHOW_FEAT_NO_HEADER ; + tool->merge_deferred_callchains = true; tool->sample = process_event_sample_stub; tool->mmap = process_event_stub;
tools/perf/util/tool.h +1 -0
@@ -87,6 +87,7 @@ struct perf_tool {bool cgroup_events;bool no_warn;bool dont_split_sample_group; +bool merge_deferred_callchains;enum show_feature_header show_feat_hdr; };
Signature · 1 line
2.47.2
Peter Zijlstra
maintainerRe: [PATCH v5 11/17] perf: Simplify get_perf_callchain() user logic
In reply to Steven RostedtQuoted 17 lines · Steven Rostedt
From: Josh Poimboeuf <[email protected]>
Simplify the
get_perf_callchain()user logic a bit.task_pt_regs()should never be NULL.Acked-by Namhyung Kim <[email protected]>Signed-off-by Josh Poimboeuf <[email protected]>Signed-off-by Steven Rostedt (Google) <[email protected]>--- kernel/events/callchain.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-)kernel/events/callchain.c +0
-0
kernel/events/callchain.c get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, · line 251
Quoted 10 lines
kernel/events/callchain.c +3
-4@@ -246,22 +246,20 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user,if (user) {if (!user_mode (regs)) { -if (current->mm) - regs =task_pt_regs (current); -else - regs =NULL ; +if (!current->mm) + goto exit_put; + regs =task_pt_regs (current);
I'm thinking this might be one of those is-kthread test written as has-mm, and they're broken.
Notably things like the io-uring kthreads do have mm.
Quoted 25 lines
kernel/events/callchain.c +6
-7} -if (regs) { -if (add_mark) -perf_callchain_store_context (&ctx,PERF_CONTEXT_USER ); +if (add_mark) +perf_callchain_store_context (&ctx,PERF_CONTEXT_USER ); - start_entry_idx = entry->nr; -perf_callchain_user (&ctx, regs); -fixup_uretprobe_trampoline_entries (entry, start_entry_idx); - } + start_entry_idx = entry->nr; +perf_callchain_user (&ctx, regs); +fixup_uretprobe_trampoline_entries (entry, start_entry_idx); } +exit_put:put_callchain_entry (rctx);return entry;Signature · 1 line
2.47.2
Peter Zijlstra
maintainerRe: [PATCH v5 12/17] perf: Skip user unwind if !current->mm
In reply to Steven RostedtQuoted 11 lines · Steven Rostedt
From: Josh Poimboeuf <[email protected]>
If the task doesn't have any memory, there's no stack to unwind.
Signed-off-by Josh Poimboeuf <[email protected]>Signed-off-by Steven Rostedt (Google) <[email protected]>---
Note from v4: https://lore.kernel.org/all/24cde1381495cd637f0d5663ab7629572bc0eca6.1737511963.git.jpoimboe@kernel.org/- From the previous version, Peter Zijlstra brought up concern about
io_uringhelpers, but this was never resolved. No update was made.
What is there to resolve? If you want to check for kthreads, don't check for ->mm
Quoted 21 lines
kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/events/core.c +1
-1@@ -8101,7 +8101,7 @@ struct perf_callchain_entry * perf_callchain(struct perf_event *event, struct pt_regs *regs) { bool kernel = !event->attr.exclude_callchain_kernel; -bool user = !event->attr.exclude_callchain_user; +bool user = !event->attr.exclude_callchain_user && current->mm;/* Disallow cross-task user callchains. */ bool crosstask = event->ctx->task && event->ctx->task != current;const u32 max_stack = event->attr.sample_max_stack;Signature · 1 line
2.47.2
Peter Zijlstra
maintainerRe: [PATCH v5 13/17] perf: Support deferred user callchains
In reply to Steven RostedtQuoted 21 lines · Steven Rostedt
kernel/events/core.c +21
-0+static int deferred_request (struct perf_event *event) +{ +struct callback_head *work = &event->pending_unwind_work; +int pending; +int ret; + +if (!current->mm || !user_mode (task_pt_regs (current))) +return -EINVAL ; + +if (in_nmi ()) +return deferred_request_nmi (event); + +guard (irqsave)(); + +/* callback already pending? */ + pending =READ_ONCE (event->pending_unwind_callback); +if (pending) +return 1 ; + +/* Claim the work unless an NMI just now swooped in to do so. */ +if (!try_cmpxchg (&event->pending_unwind_callback, &pending,1 ))
kernel/events/core.c static u64 perf_get_page_size(unsigned long addr) · line 8238
Quoted 10 lines
kernel/events/core.c +10
-0+return 1 ; + +/* The work has been claimed, now schedule it. */ + ret =task_work_add (current, work,TWA_RESUME ); +if (WARN_ON_ONCE (ret)) { +WRITE_ONCE (event->pending_unwind_callback,0 ); +return ret; + } +return 0 ; +}
You have whitespace issues ...
Steven Rostedt
maintainerRe: [PATCH v5 12/17] perf: Skip user unwind if !current->mm
In reply to Peter ZijlstraQuoted 7 lines · On Thu, 24 Apr 2025 18:37:18 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 4 lines
Note from v4: https://lore.kernel.org/all/24cde1381495cd637f0d5663ab7629572bc0eca6.1737511963.git.jpoimboe@kernel.org/
- From the previous version, Peter Zijlstra brought up concern about
io_uringhelpers, but this was never resolved. No update was made.What is there to resolve? If you want to check for kthreads, don't check for ->mm
I misunderstood what the issue was then ;-)
OK, so the check should be task->flags & PF_KTHREAD ?
-- Steve
Steven Rostedt
maintainerLovely, gmail doesn't like my email :-p
-- Steve
--------
This is the mail system at host dfw.source.kernel.org.
I'm sorry to have to inform you that your message could not be delivered to one or more recipients. It's attached below.
For further assistance, please send mail to postmaster.
If you do so, please include this problem report. You can delete your own text from the attached returned message.
The mail system
<[email protected]>: host gmail-smtp-in.l.google.com[142.251.116.27] said:
550-5.7.1 [139.178.84.217 12] Gmail has detected that this message is
550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to Gmail, 550-5.7.1 this message has been blocked. For more information, go to 550 5.7.1 https://support.google.com/mail/?p=UnsolicitedMessageError
46e09a7af769-7304f39134fsi989826a34.281 - gsmtp (in reply to end of DATA command)
<[email protected]>: host gmail-smtp-in.l.google.com[142.251.116.27] said:
550-5.7.1 [139.178.84.217 12] Gmail has detected that this message is
550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to Gmail, 550-5.7.1 this message has been blocked. For more information, go to 550 5.7.1 https://support.google.com/mail/?p=UnsolicitedMessageError
006d021491bc7-606469381a0si978892eaf.76 - gsmtp (in reply to end of DATA command)
<[email protected]>: host gmail-smtp-in.l.google.com[74.125.20.26]
said: 550-5.7.1 [172.234.252.31 12] Gmail has detected that this
message is 550-5.7.1 likely unsolicited mail. To reduce the amount of spam
sent to Gmail, 550-5.7.1 this message has been blocked. For more
information, go to 550 5.7.1
https://support.google.com/mail/?p=UnsolicitedMessageError
41be03b00d2f7-b15fb7b1684si2312872a12.593 - gsmtp (in reply to end of DATA
command)<[email protected]>: host gmail-smtp-in.l.google.com[74.125.20.26]
said: 550-5.7.1 [172.234.252.31 12] Gmail has detected that this
message is 550-5.7.1 likely unsolicited mail. To reduce the amount of spam
sent to Gmail, 550-5.7.1 this message has been blocked. For more
information, go to 550 5.7.1
https://support.google.com/mail/?p=UnsolicitedMessageError
d9443c01a7336-22db4fdfc07si22432465ad.243 - gsmtp (in reply to end of DATA
command)Steven Rostedt
maintainerRe: [PATCH v5 13/17] perf: Support deferred user callchains
In reply to Peter ZijlstraQuoted 13 lines · On Thu, 24 Apr 2025 18:38:45 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 13 lines
kernel/events/core.c +13
-0+guard (irqsave)(); + +/* callback already pending? */ + pending =READ_ONCE (event->pending_unwind_callback); +if (pending) +return 1 ; + +/* Claim the work unless an NMI just now swooped in to do so. */ +if (!try_cmpxchg (&event->pending_unwind_callback, &pending,1 )) +return 1 ; + +/* The work has been claimed, now schedule it. */ + ret =task_work_add (current, work,TWA_RESUME );
Quoted 8 lines
kernel/events/core.c static u64 perf_get_page_size(unsigned long addr) · line 8238
Quoted 6 lines
kernel/events/core.c +6
-0+if (WARN_ON_ONCE (ret)) { +WRITE_ONCE (event->pending_unwind_callback,0 ); +return ret; + } +return 0 ; +}You have whitespace issues ...
Bah, that may have been due to cut and paste to fix up things during the multiple rebases I did.
Thanks!
-- Steve
Steven Rostedt
maintainerRe: [PATCH v5 11/17] perf: Simplify get_perf_callchain() user logic
In reply to Peter ZijlstraQuoted 16 lines · On Thu, 24 Apr 2025 18:36:07 +0200 Peter Zijlstra <[email protected]> wrote:
kernel/events/callchain.c get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, · line 251
Quoted 12 lines
+++
b/kernel/events/callchain.ckernel/events/callchain.c +3
-4@@ -246,22 +246,20 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user,if (user) {if (!user_mode (regs)) { -if (current->mm) - regs =task_pt_regs (current); -else - regs =NULL ; +if (!current->mm) + goto exit_put; + regs =task_pt_regs (current);I'm thinking this might be one of those is-kthread test written as has-mm, and they're broken.
Quoted 1 line
Notably things like the io-uring kthreads do have mm.
Would there ever be a case where:
current->mm == NULL && !(current->flags & PF_KTHREAD)
?
That is, do we still need to check for current->mm if it's not a kernel thread, or can we assume it exists?
-- Steve
Mathieu Desnoyers
reviewerRe: [PATCH v5 11/17] perf: Simplify get_perf_callchain() user logic
Quoted 19 lines · Steven Rostedt
Quoted 16 lines · On Thu, 24 Apr 2025 18:36:07 +0200 Peter Zijlstra <[email protected]> wrote:
kernel/events/callchain.c get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, · line 251
Quoted 12 lines
+++
b/kernel/events/callchain.ckernel/events/callchain.c +3
-4@@ -246,22 +246,20 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user,if (user) {if (!user_mode (regs)) { -if (current->mm) - regs =task_pt_regs (current); -else - regs =NULL ; +if (!current->mm) + goto exit_put; + regs =task_pt_regs (current);I'm thinking this might be one of those is-kthread test written as has-mm, and they're broken.
Quoted 1 line
Quoted 1 line
Notably things like the io-uring kthreads do have mm.
Can a kthread such as io-uring have user_mode(regs) == true ?
Quoted 9 lines
Would there ever be a case where:
current->mm == NULL && !(current->flags & PF_KTHREAD)?
That is, do we still need to check for
current->mmif it's not a kernel thread, or can we assume it exists?
The case I'd be careful about is if the code can nest over exit_mm() (e.g. interrupt) after it sets current->mm = NULL.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Steven Rostedt
maintainerRe: [PATCH v5 11/17] perf: Simplify get_perf_callchain() user logic
Quoted 2 lines · On Thu, 24 Apr 2025 13:42:16 -0400 Mathieu Desnoyers…
The case I'd be careful about is if the code can nest over
exit_mm()(e.g. interrupt) after it setscurrent->mm= NULL.
Then probably we should always check if current->mm is non-NULL before using it then, and not rely on just task->flags.
-- Steve
Quoted 2 lines · Steven Rostedt
Lovely, gmail doesn't like my email :-p
Yeah, I had a moment ago 3 of your emails in my spam, so it seems it flagged them also for those getting it from a list.
Cheers,
Miguel
Steven Rostedt
maintainerQuoted 6 lines · On Thu, 24 Apr 2025 20:32:10 +0200 Miguel Ojeda…
Quoted 2 lines · Steven Rostedt
Lovely, gmail doesn't like my email :-p
Yeah, I had a moment ago 3 of your emails in my spam, so it seems it flagged them also for those getting it from a list.
Probably because I use kernel.org as my mail server but my address is "[email protected]". Maybe I'll need to post patches as [email protected] :-p
-- Steve
Steven Rostedt
maintainerRe: [PATCH v5 08/17] unwind_user/deferred: Add unwind cache
In reply to Steven Rostedtinclude/linux/unwind_deferred.h void unwind_task_free(struct task_struct *task); · line 19
Quoted 11 lines · On Thu, 24 Apr 2025 12:25:37 -0400 Steven Rostedt <[email protected]> wrote:
include/linux/unwind_deferred.h +5
-0@@ -12,6 +12,11 @@ void unwind_task_free(struct task_struct *task);int unwind_deferred_trace (struct unwind_stacktrace *trace); +static __always_inline void unwind_enter_from_user_mode(void) +{ + current->unwind_info.cache.nr_entries =0 ; +} +
Hmm, the first patches just had unwind_enter_from_user_mode(), but the later patches (that I'm currently working on) have unwind_exit_to_user_mode().
I think the exit is better than the entry, and I will change this to just use the exit_to_user_mode and removed the enter_from_user_mode.
-- Steve
Peter Zijlstra
maintainerRe: [PATCH v5 11/17] perf: Simplify get_perf_callchain() user logic
In reply to Steven RostedtQuoted 21 lines · Steven Rostedt
Quoted 16 lines · On Thu, 24 Apr 2025 18:36:07 +0200 Peter Zijlstra <[email protected]> wrote:
kernel/events/callchain.c get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, · line 251
Quoted 12 lines
+++
b/kernel/events/callchain.ckernel/events/callchain.c +3
-4@@ -246,22 +246,20 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user,if (user) {if (!user_mode (regs)) { -if (current->mm) - regs =task_pt_regs (current); -else - regs =NULL ; +if (!current->mm) + goto exit_put; + regs =task_pt_regs (current);I'm thinking this might be one of those is-kthread test written as has-mm, and they're broken.
Quoted 1 line
Notably things like the io-uring kthreads do have mm.
Quoted 8 lines
Would there ever be a case where:
current->mm == NULL && !(current->flags & PF_KTHREAD)?
That is, do we still need to check for
current->mmif it's not a kernel thread, or can we assume it exists?
IIRC just checking PF_KTHREAD should be sufficient.
Namhyung Kim
maintainerRe: [PATCH v5 13/17] perf: Support deferred user callchains
In reply to Steven RostedtHello,
Quoted 11 lines · Steven Rostedt
From: Josh Poimboeuf <[email protected]>
Use the new
unwind_deferred_trace()interface (if available) to defer unwinds to task context. This will allow the use of .sframe (when it becomes available) and also prevents duplicate userspace unwinds.Suggested-by Peter Zijlstra <[email protected]>Co-developed-by Steven Rostedt (Google) <[email protected]>Signed-off-by Josh Poimboeuf <[email protected]>Signed-off-by Steven Rostedt (Google) <[email protected]>---
[SNIP]
kernel/events/core.c static u64 perf_get_page_size(unsigned long addr) · line 8212
Quoted 11 lines
kernel/events/core.c +11
-0+/* +* Returns: +* > 0 : if already queued. +* 0 : if it performed the queuing +* < 0 : if it did not get queued. +*/ +static int deferred_request (struct perf_event *event) +{ +struct callback_head *work = &event->pending_unwind_work; +int pending; +int ret;
I'm not sure if it works for per-CPU events. The event is shared so any task can request the deferred callchains. Does it handle if task A requests one and scheduled out before going to the user mode, and task B on the CPU also requests another after that? I'm afraid not..
kernel/events/core.c static u64 perf_get_page_size(unsigned long addr) · line 8215
Quoted 3 lines
kernel/events/core.c +3
-0+ +if (!current->mm || !user_mode (task_pt_regs (current))) +return -EINVAL ;
Does it mean it cannot use deferred callstack when it's in the kernel mode like during a syscall?
Thanks,
Namhyung
Quoted 23 lines
kernel/events/core.c +23
-0+ +if (in_nmi ()) +return deferred_request_nmi (event); + +guard (irqsave)(); + +/* callback already pending? */ + pending =READ_ONCE (event->pending_unwind_callback); +if (pending) +return 1 ; + +/* Claim the work unless an NMI just now swooped in to do so. */ +if (!try_cmpxchg (&event->pending_unwind_callback, &pending,1 )) +return 1 ; + +/* The work has been claimed, now schedule it. */ + ret =task_work_add (current, work,TWA_RESUME ); +if (WARN_ON_ONCE (ret)) { +WRITE_ONCE (event->pending_unwind_callback,0 ); +return ret; + } +return 0 ; +}
Steven Rostedt
maintainerRe: [PATCH v5 13/17] perf: Support deferred user callchains
Quoted 8 lines · On Fri, 25 Apr 2025 08:24:47 -0700 Namhyung Kim <[email protected]> wrote:
Quoted 8 lines
kernel/events/core.c +8
-0+/* +* Returns: +* > 0 : if already queued. +* 0 : if it performed the queuing +* < 0 : if it did not get queued. +*/ +static int deferred_request (struct perf_event *event) +{
Quoted 8 lines
kernel/events/core.c static u64 perf_get_page_size(unsigned long addr) · line 8212
Quoted 3 lines
kernel/events/core.c +3
-0+struct callback_head *work = &event->pending_unwind_work; +int pending; +int ret;I'm not sure if it works for per-CPU events. The event is shared so any task can request the deferred callchains. Does it handle if task A requests one and scheduled out before going to the user mode, and task B on the CPU also requests another after that? I'm afraid not..
I was afraid of that.
This is no different that what Josh did in his last set in v4. I'm guessing the issue is running with "-a", correct?
Could we just not use deferred when running with "-a" for now? Or could we possibly just make the deferred stacktrace its own event? Have it be possible that perf just registers a signal instance with the deferred unwinding logic, and then perf can handle where to write the information. I don't know perf well enough to implement that.
Josh's code had it call the unwind_deferred_init() and just used its own event to callback to and that was called on hundreds of events when I ran:
perf record -g <whatever>
Same if I added the "-a" option.
The above function return values came from Josh's code:
I just moved it out of deferred.c and into perf itself, and removed the cookie logic.
Quoted 7 lines
kernel/events/core.c static u64 perf_get_page_size(unsigned long addr) · line 8215
Quoted 4 lines
kernel/events/core.c +3
-0+ +if (!current->mm || !user_mode (task_pt_regs (current))) +return -EINVAL ;Does it mean it cannot use deferred callstack when it's in the kernel mode like during a syscall?
task_pt_regs(current) will return the regs from when the task entered the kernel. So the answer is no, it will still trace if an interrupt happened while a task is in a system call.
-- Steve
Namhyung Kim
maintainerRe: [PATCH v5 13/17] perf: Support deferred user callchains
Quoted 19 lines · Steven Rostedt
Quoted 8 lines · On Fri, 25 Apr 2025 08:24:47 -0700 Namhyung Kim <[email protected]> wrote:
Quoted 8 lines
kernel/events/core.c +8
-0+/* +* Returns: +* > 0 : if already queued. +* 0 : if it performed the queuing +* < 0 : if it did not get queued. +*/ +static int deferred_request (struct perf_event *event) +{Quoted 8 lines
kernel/events/core.c static u64 perf_get_page_size(unsigned long addr) · line 8212
Quoted 3 lines
kernel/events/core.c +3
-0+struct callback_head *work = &event->pending_unwind_work; +int pending; +int ret;I'm not sure if it works for per-CPU events. The event is shared so any task can request the deferred callchains. Does it handle if task A requests one and scheduled out before going to the user mode, and task B on the CPU also requests another after that? I'm afraid not..
Quoted 4 lines
I was afraid of that.
This is no different that what Josh did in his last set in v4. I'm guessing the issue is running with "-a", correct?
Yes.
Quoted 6 lines
Could we just not use deferred when running with "-a" for now? Or could we possibly just make the deferred stacktrace its own event? Have it be possible that perf just registers a signal instance with the deferred unwinding logic, and then perf can handle where to write the information. I don't know perf well enough to implement that.
Even if it excludes per-CPU events, per-task events also can attach to a CPU and that's the default behavior of the perf record IIRC. In that case, it needs to be careful when it accesses the event since the task can migrate to another CPU. So I'm not sure if it's a good idea to track event that requested the deferred callchains.
Also it doesn't need to emit duplicate deferred callchains if a task has multiple events and they are requesting callchains. Unfortunately, the kernel cannot know which events are related or profiled together.
Hmm.. maybe we can add a cookie to the event itself (by ioctl or something) in order to group events in a profiling session and then use that for deferred callchains? Task should maintain a list of active cookies (or sessions) somehow but then perf can check if the current CPU has events with matching cookies and emit a deferred callchain.
Quoted 23 lines
Josh's code had it call the
unwind_deferred_init()and just used its own event to callback to and that was called on hundreds of events when I ran:perf record -g <whatever>
Same if I added the "-a" option.
The above function return values came from Josh's code:
I just moved it out of
deferred.cand into perf itself, and removed the cookie logic.Quoted 7 lines
kernel/events/core.c static u64 perf_get_page_size(unsigned long addr) · line 8215
Quoted 4 lines
kernel/events/core.c +3
-0+ +if (!current->mm || !user_mode (task_pt_regs (current))) +return -EINVAL ;Does it mean it cannot use deferred callstack when it's in the kernel mode like during a syscall?
Quoted 3 lines
task_pt_regs(current)will return the regs from when the task entered the kernel. So the answer is no, it will still trace if an interrupt happened while a task is in a system call.
Ok, thanks for the explanation. The the user_mode check was for kernel threads, right?
Thanks,
Namhyung
Steven Rostedt
maintainerRe: [PATCH v5 13/17] perf: Support deferred user callchains
Quoted 7 lines · On Mon, 28 Apr 2025 13:42:15 -0700 Namhyung Kim <[email protected]> wrote:
Quoted 6 lines
Could we just not use deferred when running with "-a" for now? Or could we possibly just make the deferred stacktrace its own event? Have it be possible that perf just registers a signal instance with the deferred unwinding logic, and then perf can handle where to write the information. I don't know perf well enough to implement that.
Quoted 5 lines
Even if it excludes per-CPU events, per-task events also can attach to a CPU and that's the default behavior of the perf record IIRC. In that case, it needs to be careful when it accesses the event since the task can migrate to another CPU. So I'm not sure if it's a good idea to track event that requested the deferred callchains.
Wait? Even for per task, it uses per cpu?
Quoted 10 lines
Also it doesn't need to emit duplicate deferred callchains if a task has multiple events and they are requesting callchains. Unfortunately, the kernel cannot know which events are related or profiled together.
Hmm.. maybe we can add a cookie to the event itself (by ioctl or something) in order to group events in a profiling session and then use that for deferred callchains? Task should maintain a list of active cookies (or sessions) somehow but then perf can check if the current CPU has events with matching cookies and emit a deferred callchain.
Could we add a callchain event? It gets woken at any request but not triggered until the task returns. A way that there would be only a single event for every perf instance, but it can trace any task.
It could use the cookie method that ftrace uses, where the request gets a cookie, and can be recorded to the perf event in the interrupt. Then the callchain would record the cookie along with the stack trace, and then perf tool could just match up the kernel stacks with their cookies to the user stack with its cookie.
-- Steve
Namhyung Kim
maintainerRe: [PATCH v5 13/17] perf: Support deferred user callchains
Quoted 11 lines · Steven Rostedt
Quoted 7 lines · On Mon, 28 Apr 2025 13:42:15 -0700 Namhyung Kim <[email protected]> wrote:
Quoted 6 lines
Could we just not use deferred when running with "-a" for now? Or could we possibly just make the deferred stacktrace its own event? Have it be possible that perf just registers a signal instance with the deferred unwinding logic, and then perf can handle where to write the information. I don't know perf well enough to implement that.
Quoted 7 lines
Quoted 5 lines
Even if it excludes per-CPU events, per-task events also can attach to a CPU and that's the default behavior of the perf record IIRC. In that case, it needs to be careful when it accesses the event since the task can migrate to another CPU. So I'm not sure if it's a good idea to track event that requested the deferred callchains.
Wait? Even for per task, it uses per cpu?
Yep, it has per-task + per-CPU events.
Quoted 12 lines
Quoted 11 lines
Also it doesn't need to emit duplicate deferred callchains if a task has multiple events and they are requesting callchains. Unfortunately, the kernel cannot know which events are related or profiled together.
Hmm.. maybe we can add a cookie to the event itself (by ioctl or something) in order to group events in a profiling session and then use that for deferred callchains? Task should maintain a list of active cookies (or sessions) somehow but then perf can check if the current CPU has events with matching cookies and emit a deferred callchain.
Quoted 3 lines
Could we add a callchain event? It gets woken at any request but not triggered until the task returns. A way that there would be only a single event for every perf instance, but it can trace any task.
Thing is that the kernel doesn't know the relationship between events. For example, if I run this command on a machine with 100 CPUs:
$ perf record -e cycles,instructions -- $MYPROG
it would open 200 events and they don't know each other. Later other process can start a new perf profiling for the same task. IIUC there's no way to identify which one is related in the kernel.
So I think we need a way to share some informaiton for those 200 events and then emits deferred callchain records with the shared info.
Quoted 6 lines
It could use the cookie method that ftrace uses, where the request gets a cookie, and can be recorded to the perf event in the interrupt. Then the callchain would record the cookie along with the stack trace, and then perf tool could just match up the kernel stacks with their cookies to the user stack with its cookie.
Yep, but the kernel should know which events (or ring buffer) it should emit the deferred callchains. I don't think it needs to include the cookie in the perf data, but it can be used to find which event or ring buffer for the session is related to this request.
Thanks,
Namhyung
Steven Rostedt
maintainerRe: [PATCH v5 13/17] perf: Support deferred user callchains
Quoted 11 lines · On Mon, 28 Apr 2025 17:29:53 -0700 Namhyung Kim <[email protected]> wrote:
Thing is that the kernel doesn't know the relationship between events. For example, if I run this command on a machine with 100 CPUs:
$ perf record -e cycles,instructions -- $MYPROGit would open 200 events and they don't know each other. Later other process can start a new perf profiling for the same task. IIUC there's no way to identify which one is related in the kernel.
So I think we need a way to share some informaiton for those 200 events and then emits deferred callchain records with the shared info.
Hmm, I'm thinking of creating an internal perf descriptor that would join events by who created them. That is, the first event created will take the thread leader (pid of the task) and check if an entity exists for it. If one doesn't exist it will create it and add itself to that event if it has a deferred trace attribute set. If it already exists, it will just add itself to it. This deferred descriptor will register itself with the deferred unwinder like ftrace does (one per process), and then use it to defer callbacks. When the callback happens, it will look for the thread event or CPU event that matches the current thread or current CPU and record the backtrace there.
Quoted 8 lines
Quoted 7 lines
It could use the cookie method that ftrace uses, where the request gets a cookie, and can be recorded to the perf event in the interrupt. Then the callchain would record the cookie along with the stack trace, and then perf tool could just match up the kernel stacks with their cookies to the user stack with its cookie.
Quoted 4 lines
Yep, but the kernel should know which events (or ring buffer) it should emit the deferred callchains. I don't think it needs to include the cookie in the perf data, but it can be used to find which event or ring buffer for the session is related to this request.
Let me see if my suggestion would work or not. I'll try it out and see what happens. And post patches later.
-- Steve
Steven Rostedt
maintainerRe: [PATCH v5 13/17] perf: Support deferred user callchains
In reply to Steven RostedtQuoted 17 lines · On Thu, 24 Apr 2025 12:25:42 -0400 Steven Rostedt <[email protected]> wrote:
kernel/events/core.c +17
-0+static void perf_event_callchain_deferred (struct callback_head *work) +{ + struct perf_event *event = container_of(work, struct perf_event, pending_unwind_work); +struct perf_callchain_deferred_event deferred_event; +u64 callchain_context =PERF_CONTEXT_USER ; +struct unwind_stacktrace trace; +struct perf_output_handle handle; +struct perf_sample_data data; +u64 nr; + +if (!event->pending_unwind_callback) +return ; + +if (unwind_deferred_trace (&trace) <0 ) + goto out; + +/*
kernel/events/core.c static void perf_pending_irq(struct irq_work *entry) · line 7310
Quoted 10 lines
kernel/events/core.c +10
-0+* All accesses to the event must belong to the same implicit RCU +* read-side critical section as the ->pending_unwind_callback reset. +* See comment in perf_pending_unwind_sync(). +*/ +guard (rcu)(); + +if (!current->mm) + goto out; + + nr = trace.nr +1 ;/* '+1' == callchain_context */
Hi Namhyung,
Talking with Beau about how Microsoft does their own deferred tracing, I wonder if the timestamp approach would be useful.
This is where a timestamp is taken at the first request for a deferred trace, and this is recorded in the trace when it happens. It basically states that "this trace is good up until the given timestamp".
The rationale for this is for lost events. Let's say you have:
<task enters kernel>
Request deferred trace
<buffer fills up and events start to get lost>
Deferred trace happens (but is dropped due to buffer being full)
<task exits kernel>
<task enters kernel again>
Request deferred trace (Still dropped due to buffer being full)
<Reader catches up and buffer is free again>
Deferred trace happens (this time it is recorded>
<task exits kernel>
How would user space know that the deferred trace that was recorded doesn't go with the request (and kernel stack trace) that was done initially)?
If we add a timestamp, then it would look like:
<task enters kernel>
Request deferred trace
[Record timestamp]
<buffer fills up and events start to get lost>
Deferred trace happens with timestamp (but is dropped due to buffer being full)
<task exits kernel>
<task enters kernel again>
Request deferred trace (Still dropped due to buffer being full) [Record timestamp]
<Reader catches up and buffer is free again>
Deferred trace happens with timestamp (this time it is recorded>
<task exits kernel>
Then user space will look at the timestamp that was recorded and know that it's not for the initial request because the timestamp of the kernel stack trace done was before the timestamp of the user space stacktrace and therefore is not valid for the kernel stacktrace.
The timestamp would become zero when exiting to user space. The first request will add it but would need a cmpxchg to do so, and if the cmpxchg fails, it then needs to check if the one recorded is before the current one, and if it isn't it still needs to update the timestamp (this is to handle races with NMIs).
Basically, the timestamp would replace the cookie method.
Thoughts?
-- Steve
Quoted 25 lines
kernel/events/core.c +25
-0+ + deferred_event.header.type =PERF_RECORD_CALLCHAIN_DEFERRED ; + deferred_event.header.misc =PERF_RECORD_MISC_USER ; + deferred_event.header.size =sizeof (deferred_event) + (nr *sizeof (u64)); + + deferred_event.nr = nr; + +perf_event_header__init_id (&deferred_event.header, &data, event); + +if (perf_output_begin (&handle, &data, event, deferred_event.header.size)) + goto out; + +perf_output_put (&handle, deferred_event); +perf_output_put (&handle, callchain_context); +perf_output_copy (&handle, trace.entries, trace.nr *sizeof (u64)); +perf_event__output_id_sample (event, &handle, &data); + +perf_output_end (&handle); + +out: + event->pending_unwind_callback =0 ; +local_dec (&event->ctx->nr_no_switch_fast); +rcuwait_wake_up (&event->pending_unwind_wait); +} +
Namhyung Kim
maintainerRe: [PATCH v5 13/17] perf: Support deferred user callchains
Hi Steve,
Quoted 54 lines · Steven Rostedt
Quoted 17 lines · On Thu, 24 Apr 2025 12:25:42 -0400 Steven Rostedt <[email protected]> wrote:
kernel/events/core.c +17
-0+static void perf_event_callchain_deferred (struct callback_head *work) +{ + struct perf_event *event = container_of(work, struct perf_event, pending_unwind_work); +struct perf_callchain_deferred_event deferred_event; +u64 callchain_context =PERF_CONTEXT_USER ; +struct unwind_stacktrace trace; +struct perf_output_handle handle; +struct perf_sample_data data; +u64 nr; + +if (!event->pending_unwind_callback) +return ; + +if (unwind_deferred_trace (&trace) <0 ) + goto out; + +/* kernel/events/core.c static void perf_pending_irq(struct irq_work *entry) · line 7310
Quoted 10 lines
kernel/events/core.c +10
-0+* All accesses to the event must belong to the same implicit RCU +* read-side critical section as the ->pending_unwind_callback reset. +* See comment in perf_pending_unwind_sync(). +*/ +guard (rcu)(); + +if (!current->mm) + goto out; + + nr = trace.nr +1 ;/* '+1' == callchain_context */ Hi Namhyung,
Talking with Beau about how Microsoft does their own deferred tracing, I wonder if the timestamp approach would be useful.
This is where a timestamp is taken at the first request for a deferred trace, and this is recorded in the trace when it happens. It basically states that "this trace is good up until the given timestamp".
The rationale for this is for lost events. Let's say you have:
<task enters kernel>
Request deferred trace<buffer fills up and events start to get lost>
Deferred trace happens (but is dropped due to buffer being full)
<task exits kernel>
<task enters kernel again>
Request deferred trace (Still dropped due to buffer being full)
Quoted 7 lines
<Reader catches up and buffer is free again>
Deferred trace happens (this time it is recorded>
<task exits kernel>How would user space know that the deferred trace that was recorded doesn't go with the request (and kernel stack trace) that was done initially)?
Right, this is a problem.
Quoted 19 lines
If we add a timestamp, then it would look like:
<task enters kernel>
Request deferred trace
[Record timestamp]<buffer fills up and events start to get lost>
Deferred trace happens with timestamp (but is dropped due to buffer being full)
<task exits kernel>
<task enters kernel again>
Request deferred trace (Still dropped due to buffer being full) [Record timestamp]<Reader catches up and buffer is free again>
Quoted 7 lines
Deferred trace happens with timestamp (this time it is recorded>
<task exits kernel>Then user space will look at the timestamp that was recorded and know that it's not for the initial request because the timestamp of the kernel stack trace done was before the timestamp of the user space stacktrace and therefore is not valid for the kernel stacktrace.
IIUC the deferred stacktrace will have the timestamp of the first request, right?
Quoted 6 lines
The timestamp would become zero when exiting to user space. The first request will add it but would need a cmpxchg to do so, and if the cmpxchg fails, it then needs to check if the one recorded is before the current one, and if it isn't it still needs to update the timestamp (this is to handle races with NMIs).
Yep, it needs to maintain an accurate first timestamp.
Quoted 4 lines
Basically, the timestamp would replace the cookie method.
Thoughts?
Sounds good to me. You'll need to add it to the PERF_RECORD_DEFERRED_CALLCHAIN. Probably it should check if sample_type has PERF_SAMPLE_TIME. It'd work along with PERF_SAMPLE_TID (which will be added by the perf tools anyway).
Thanks,
Namhyung
Quoted 26 lines
Quoted 26 lines
kernel/events/core.c +25
-0+ + deferred_event.header.type =PERF_RECORD_CALLCHAIN_DEFERRED ; + deferred_event.header.misc =PERF_RECORD_MISC_USER ; + deferred_event.header.size =sizeof (deferred_event) + (nr *sizeof (u64)); + + deferred_event.nr = nr; + +perf_event_header__init_id (&deferred_event.header, &data, event); + +if (perf_output_begin (&handle, &data, event, deferred_event.header.size)) + goto out; + +perf_output_put (&handle, deferred_event); +perf_output_put (&handle, callchain_context); +perf_output_copy (&handle, trace.entries, trace.nr *sizeof (u64)); +perf_event__output_id_sample (event, &handle, &data); + +perf_output_end (&handle); + +out: + event->pending_unwind_callback =0 ; +local_dec (&event->ctx->nr_no_switch_fast); +rcuwait_wake_up (&event->pending_unwind_wait); +} +
Mathieu Desnoyers
reviewerRe: [PATCH v5 13/17] perf: Support deferred user callchains
In reply to Steven RostedtQuoted 94 lines · Steven Rostedt
Quoted 17 lines · On Thu, 24 Apr 2025 12:25:42 -0400 Steven Rostedt <[email protected]> wrote:
kernel/events/core.c +17
-0+static void perf_event_callchain_deferred (struct callback_head *work) +{ + struct perf_event *event = container_of(work, struct perf_event, pending_unwind_work); +struct perf_callchain_deferred_event deferred_event; +u64 callchain_context =PERF_CONTEXT_USER ; +struct unwind_stacktrace trace; +struct perf_output_handle handle; +struct perf_sample_data data; +u64 nr; + +if (!event->pending_unwind_callback) +return ; + +if (unwind_deferred_trace (&trace) <0 ) + goto out; + +/* kernel/events/core.c static void perf_pending_irq(struct irq_work *entry) · line 7310
Quoted 10 lines
kernel/events/core.c +10
-0+* All accesses to the event must belong to the same implicit RCU +* read-side critical section as the ->pending_unwind_callback reset. +* See comment in perf_pending_unwind_sync(). +*/ +guard (rcu)(); + +if (!current->mm) + goto out; + + nr = trace.nr +1 ;/* '+1' == callchain_context */ Hi Namhyung,
Talking with Beau about how Microsoft does their own deferred tracing, I wonder if the timestamp approach would be useful.
This is where a timestamp is taken at the first request for a deferred trace, and this is recorded in the trace when it happens. It basically states that "this trace is good up until the given timestamp".
The rationale for this is for lost events. Let's say you have:
<task enters kernel>
Request deferred trace<buffer fills up and events start to get lost>
Deferred trace happens (but is dropped due to buffer being full)
<task exits kernel>
<task enters kernel again>
Request deferred trace (Still dropped due to buffer being full)<Reader catches up and buffer is free again>
Deferred trace happens (this time it is recorded>
<task exits kernel>How would user space know that the deferred trace that was recorded doesn't go with the request (and kernel stack trace) that was done initially)?
If we add a timestamp, then it would look like:
<task enters kernel>
Request deferred trace
[Record timestamp]<buffer fills up and events start to get lost>
Deferred trace happens with timestamp (but is dropped due to buffer being full)
<task exits kernel>
<task enters kernel again>
Request deferred trace (Still dropped due to buffer being full) [Record timestamp]<Reader catches up and buffer is free again>
Deferred trace happens with timestamp (this time it is recorded>
<task exits kernel>Then user space will look at the timestamp that was recorded and know that it's not for the initial request because the timestamp of the kernel stack trace done was before the timestamp of the user space stacktrace and therefore is not valid for the kernel stacktrace.
The timestamp would become zero when exiting to user space. The first request will add it but would need a cmpxchg to do so, and if the cmpxchg fails, it then needs to check if the one recorded is before the current one, and if it isn't it still needs to update the timestamp (this is to handle races with NMIs).
Quoted 3 lines
Basically, the timestamp would replace the cookie method.
Thoughts?
AFAIR, the cookie method generates the cookie by combining the cpu number with a per-cpu count.
This ensures that there are not two cookies emitted at the same time from two CPUs that have the same value by accident.
How would the timestamp method prevent this ?
Thanks,
Mathieu
Quoted 29 lines
-- Steve
Quoted 25 lines
kernel/events/core.c +25
-0+ + deferred_event.header.type =PERF_RECORD_CALLCHAIN_DEFERRED ; + deferred_event.header.misc =PERF_RECORD_MISC_USER ; + deferred_event.header.size =sizeof (deferred_event) + (nr *sizeof (u64)); + + deferred_event.nr = nr; + +perf_event_header__init_id (&deferred_event.header, &data, event); + +if (perf_output_begin (&handle, &data, event, deferred_event.header.size)) + goto out; + +perf_output_put (&handle, deferred_event); +perf_output_put (&handle, callchain_context); +perf_output_copy (&handle, trace.entries, trace.nr *sizeof (u64)); +perf_event__output_id_sample (event, &handle, &data); + +perf_output_end (&handle); + +out: + event->pending_unwind_callback =0 ; +local_dec (&event->ctx->nr_no_switch_fast); +rcuwait_wake_up (&event->pending_unwind_wait); +} +
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Steven Rostedt
maintainerRe: [PATCH v5 13/17] perf: Support deferred user callchains
Quoted 7 lines · On Thu, 8 May 2025 14:49:59 -0400 Mathieu Desnoyers…
AFAIR, the cookie method generates the cookie by combining the cpu number with a per-cpu count.
This ensures that there are not two cookies emitted at the same time from two CPUs that have the same value by accident.
How would the timestamp method prevent this ?
Do we care? It only needs to be unique per pid doesn't it?
-- Steve
Mathieu Desnoyers
reviewerRe: [PATCH v5 13/17] perf: Support deferred user callchains
Quoted 6 lines · Steven Rostedt
Quoted 3 lines · On Thu, 8 May 2025 14:49:59 -0400 Mathieu Desnoyers…
AFAIR, the cookie method generates the cookie by combining the cpu number with a per-cpu count.
Quoted 6 lines
Quoted 4 lines
This ensures that there are not two cookies emitted at the same time from two CPUs that have the same value by accident.
How would the timestamp method prevent this ?
Do we care? It only needs to be unique per pid doesn't it?
Is it possible to have many threads writing into the same ring buffer in that scenario ? Are all event records stamped with their associated PID ? As long as we have enough information to know which thread was associated with the timestamp cookie on both ends (request for callchain and saving the user callchain on return to userspace), we should be OK.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Namhyung Kim
maintainerRe: [PATCH v5 13/17] perf: Support deferred user callchains
Hi Mathieu,
Quoted 12 lines · Mathieu Desnoyers
Quoted 11 lines · Steven Rostedt
Quoted 7 lines · On Thu, 8 May 2025 14:49:59 -0400 Mathieu Desnoyers…
AFAIR, the cookie method generates the cookie by combining the cpu number with a per-cpu count.
This ensures that there are not two cookies emitted at the same time from two CPUs that have the same value by accident.
How would the timestamp method prevent this ?
Quoted 8 lines
Quoted 1 line
Do we care? It only needs to be unique per pid doesn't it?
Is it possible to have many threads writing into the same ring buffer in that scenario ? Are all event records stamped with their associated PID ? As long as we have enough information to know which thread was associated with the timestamp cookie on both ends (request for callchain and saving the user callchain on return to userspace), we should be OK.
Yep, basically perf sets PERF_SAMPLE_TID (and sample_id_all) which makes every records come with PID/TID..
Thanks,
Namhyung
Steven Rostedt
maintainerRe: [PATCH v5 13/17] perf: Support deferred user callchains
Quoted 9 lines · On Fri, 9 May 2025 08:45:46 -0700 Namhyung Kim <[email protected]> wrote:
Quoted 6 lines
Is it possible to have many threads writing into the same ring buffer in that scenario ? Are all event records stamped with their associated PID ? As long as we have enough information to know which thread was associated with the timestamp cookie on both ends (request for callchain and saving the user callchain on return to userspace), we should be OK.
Yep, basically perf sets
PERF_SAMPLE_TID(andsample_id_all) which makes every records come with PID/TID..
If LTTng is fine with this too, then we should be OK as perf and ftrace both record TID in the events. I'm assuming that BPF would be OK too.
-- Steve