Skip to contentSkip to conversation

unwind_user: x86: Deferred unwinding infrastructure

PATCH Superseded Steven Rostedt 4 participants 23 messages May 13, 2025 – May 21, 2025

Subsystems (12)
x86 BPF perf events MM core +8 more subsystems

v9 of 12 Superseded Latest is v12 →

lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

v9 cover unwind_user: x86: Deferred unwinding infrastructure

This series does not make any user space visible changes. It only adds the necessary infrastructure of the deferred unwinder.

Based off of tip/master: d60119b82d8871c66563f8657bb3e550a80234de

This has modifications in x86 and I would like it to go through the x86 tree. Preferably it can go into this merge window so we can focus on getting perf and ftrace to work on top of this.

Perf exposes a lot of the interface to user space as the perf tool needs to handle the merging of the stacks, I figured it would be better to just get the kernel side mostly done and then work out the kinks of the code between user and kernel.

As there is no exposure to user space at this time, if something is found wrong with it it can be fixed without worrying about breaking API.

I ran all these patches through my tests and they passed (had to add the fix for the CONFIG_MODULE in tip/master:
https://lore.kernel.org/all/[email protected]/)

Changes since v8: https://lore.kernel.org/all/[email protected]/

- The patches posted here have not been updated since v8.

- Removed updates to perf proper

This is only to create the unwind infrastructure so that perf and ftrace can be worked on simultaneously without dependencies for each.

- Discussion about using guard for SRCU

Andrii Nakryiko brought up using guard for SRCU around the list iteration, but it was decided that just using the normal methods were fine for this use case.

Josh Poimboeuf (9):
unwind_user: Add user space unwinding API
unwind_user: Add frame pointer support unwind_user/x86: Enable frame pointer unwinding on x86 perf/x86: Rename and move get_segment_base() and make it global
unwind_user: Add compat mode frame pointer support unwind_user/x86: Enable compat mode frame pointer unwinding on x86 unwind_user/deferred: Add unwind cache unwind_user/deferred: Add deferred unwinding interface unwind_user/deferred: Make unwind deferral requests NMI-safe

Steven Rostedt (4):
unwind_user/deferred: Add unwind_deferred_trace() unwind deferred: Use bitmask to determine which callbacks to call unwind deferred: Use SRCU unwind_deferred_task_work()
unwind: Clear unwind_mask on exit back to user space

----

 MAINTAINERS                              |   8 +
 arch/Kconfig                             |  11 +
 arch/x86/Kconfig                         |   2 +
 arch/x86/events/core.c                   |  44 +---
 arch/x86/include/asm/ptrace.h            |   2 +
 arch/x86/include/asm/unwind_user.h       |  61 +++++
 arch/x86/include/asm/unwind_user_types.h |  17 ++
 arch/x86/kernel/ptrace.c                 |  38 ++++
 include/asm-generic/Kbuild               |   2 +
 include/asm-generic/unwind_user.h        |  24 ++
 include/asm-generic/unwind_user_types.h  |   9 +
 include/linux/entry-common.h             |   2 +
 include/linux/sched.h                    |   6 +
 include/linux/unwind_deferred.h          |  72 ++++++
 include/linux/unwind_deferred_types.h    |  17 ++
 include/linux/unwind_user.h              |  15 ++
 include/linux/unwind_user_types.h        |  35 +++
 kernel/Makefile                          |   1 +
 kernel/fork.c                            |   4 +
 kernel/unwind/Makefile                   |   1 +
 kernel/unwind/deferred.c                 | 367 +++++++++++++++++++++++++++++++
 kernel/unwind/user.c                     | 130 +++++++++++
 22 files changed, 829 insertions(+), 39 deletions(-)
 create mode 100644 arch/x86/include/asm/unwind_user.h
 create mode 100644 arch/x86/include/asm/unwind_user_types.h
 create mode 100644 include/asm-generic/unwind_user.h
 create mode 100644 include/asm-generic/unwind_user_types.h
 create mode 100644 include/linux/unwind_deferred.h
 create mode 100644 include/linux/unwind_deferred_types.h
 create mode 100644 include/linux/unwind_user.h
 create mode 100644 include/linux/unwind_user_types.h
 create mode 100644 kernel/unwind/Makefile
 create mode 100644 kernel/unwind/deferred.c
 create mode 100644 kernel/unwind/user.c
author maintainer
#1 lore raw

PATCH v9 1/13 unwind_user: Add user space unwinding API

From: Josh Poimboeuf <[email protected]>

Introduce a generic API for unwinding user stacks.

In order to expand user space unwinding to be able to handle more complex scenarios, such as deferred unwinding and reading user space information, create a generic interface that all architectures can use that support the various unwinding methods.

This is an alternative method for handling user space stack traces from the simple stack_trace_save_user() API. This does not replace that interface, but this interface will be used to expand the functionality of user space stack walking.

None of the structures introduced will be exposed to user space tooling.

---
 MAINTAINERS                       |  8 +++++
 arch/Kconfig                      |  3 ++
 include/linux/unwind_user.h       | 15 +++++++++
 include/linux/unwind_user_types.h | 31 +++++++++++++++++
 kernel/Makefile                   |  1 +
 kernel/unwind/Makefile            |  1 +
 kernel/unwind/user.c              | 55 +++++++++++++++++++++++++++++++
 7 files changed, 114 insertions(+)
 create mode 100644 include/linux/unwind_user.h
 create mode 100644 include/linux/unwind_user_types.h
 create mode 100644 kernel/unwind/Makefile
 create mode 100644 kernel/unwind/user.c
MAINTAINERS +8 -0
index d978af532430..57e86405a8af 100644
@@ -25373,6 +25373,14 @@ F:	Documentation/driver-api/uio-howto.rst
 F:	drivers/uio/
 F:	include/linux/uio_driver.h
 
+USERSPACE STACK UNWINDING
+M:	Josh Poimboeuf <[email protected]>
+M:	Steven Rostedt <[email protected]>
+S:	Maintained
+F:	include/linux/unwind*.h
+F:	kernel/unwind/
+
+
 UTIL-LINUX PACKAGE
 M:	Karel Zak <[email protected]>
 L:	[email protected]
arch/Kconfig +3 -0
index b0adb665041f..ccbcead9fac0 100644
@@ -435,6 +435,9 @@ config HAVE_HARDLOCKUP_DETECTOR_ARCH
 	  It uses the same command line parameters, and sysctl interface,
 	  as the generic hardlockup detectors.
 
+config UNWIND_USER
+	bool
+
 config HAVE_PERF_REGS
 	bool
 	help
include/linux/unwind_user.h new file +15 -0
new file mode 100644
index 000000000000..aa7923c1384f
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_UNWIND_USER_H
+#define _LINUX_UNWIND_USER_H
+
+#include <linux/unwind_user_types.h>
+
+int unwind_user_start(struct unwind_user_state *state);
+int unwind_user_next(struct unwind_user_state *state);
+
+int unwind_user(struct unwind_stacktrace *trace, unsigned int max_entries);
+
+#define for_each_user_frame(state) \
+	for (unwind_user_start((state)); !(state)->done; unwind_user_next((state)))
+
+#endif /* _LINUX_UNWIND_USER_H */
include/linux/unwind_user_types.h new file +31 -0
new file mode 100644
index 000000000000..6ed1b4ae74e1
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_UNWIND_USER_TYPES_H
+#define _LINUX_UNWIND_USER_TYPES_H
+
+#include <linux/types.h>
+
+enum unwind_user_type {
+	UNWIND_USER_TYPE_NONE,
+};
+
+struct unwind_stacktrace {
+	unsigned int	nr;
+	unsigned long	*entries;
+};
+
+struct unwind_user_frame {
+	s32 cfa_off;
+	s32 ra_off;
+	s32 fp_off;
+	bool use_fp;
+};
+
+struct unwind_user_state {
+	unsigned long ip;
+	unsigned long sp;
+	unsigned long fp;
+	enum unwind_user_type type;
+	bool done;
+};
+
+#endif /* _LINUX_UNWIND_USER_TYPES_H */
kernel/Makefile +1 -0
index 434929de17ef..5a2b2be2a32d 100644
@@ -55,6 +55,7 @@ obj-y += rcu/
 obj-y += livepatch/
 obj-y += dma/
 obj-y += entry/
+obj-y += unwind/
 obj-$(CONFIG_MODULES) += module/
 
 obj-$(CONFIG_KCMP) += kcmp.o
kernel/unwind/Makefile new file +1 -0
new file mode 100644
index 000000000000..349ce3677526
@@ -0,0 +1 @@
+ obj-$(CONFIG_UNWIND_USER) += user.o
kernel/unwind/user.c new file +55 -0

55 lines not on this page · raw message

Signature · 1 line
2.47.2
author maintainer
#2 lore raw

PATCH v9 2/13 unwind_user: Add frame pointer support

In reply to Steven Rostedt

From: Josh Poimboeuf <[email protected]>

Add optional support for user space frame pointer unwinding. If supported, the arch needs to enable CONFIG_HAVE_UNWIND_USER_FP and define ARCH_INIT_USER_FP_FRAME.

By encoding the frame offsets in struct unwind_user_frame, much of this code can also be reused for future unwinder implementations like sframe.

---
 arch/Kconfig                      |  4 +++
 include/asm-generic/unwind_user.h |  9 ++++++
 include/linux/unwind_user_types.h |  1 +
 kernel/unwind/user.c              | 51 +++++++++++++++++++++++++++++--
 4 files changed, 63 insertions(+), 2 deletions(-)
 create mode 100644 include/asm-generic/unwind_user.h
arch/Kconfig +4 -0
index ccbcead9fac0..0e3844c0e200 100644
@@ -438,6 +438,10 @@ config HAVE_HARDLOCKUP_DETECTOR_ARCH
 config UNWIND_USER
 	bool
 
+config HAVE_UNWIND_USER_FP
+	bool
+	select UNWIND_USER
+
 config HAVE_PERF_REGS
 	bool
 	help
include/asm-generic/unwind_user.h new file +9 -0
new file mode 100644
index 000000000000..832425502fb3
@@ -0,0 +1,9 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_GENERIC_UNWIND_USER_H
+#define _ASM_GENERIC_UNWIND_USER_H
+
+#ifndef ARCH_INIT_USER_FP_FRAME
+ #define ARCH_INIT_USER_FP_FRAME
+#endif
+
+#endif /* _ASM_GENERIC_UNWIND_USER_H */
include/linux/unwind_user_types.h +1 -0
index 6ed1b4ae74e1..65bd070eb6b0 100644
@@ -6,6 +6,7 @@
 
 enum unwind_user_type {
 	UNWIND_USER_TYPE_NONE,
+	UNWIND_USER_TYPE_FP,
 };
 
 struct unwind_stacktrace {
kernel/unwind/user.c +49 -2

66 lines not on this page · raw message

Signature · 1 line
2.47.2
author maintainer
#3 lore raw

PATCH v9 3/13 unwind_user/x86: Enable frame pointer unwinding on x86

In reply to Steven Rostedt

From: Josh Poimboeuf <[email protected]>

Use ARCH_INIT_USER_FP_FRAME to describe how frame pointers are unwound on x86, and enable CONFIG_HAVE_UNWIND_USER_FP accordingly so the unwind_user interfaces can be used.

---
 arch/x86/Kconfig                   |  1 +
 arch/x86/include/asm/unwind_user.h | 11 +++++++++++
 2 files changed, 12 insertions(+)
 create mode 100644 arch/x86/include/asm/unwind_user.h
arch/x86/Kconfig +1 -0
index 891a69b308cb..456e085263e1 100644
@@ -301,6 +301,7 @@ config X86
 	select HAVE_SYSCALL_TRACEPOINTS
 	select HAVE_UACCESS_VALIDATION		if HAVE_OBJTOOL
 	select HAVE_UNSTABLE_SCHED_CLOCK
+	select HAVE_UNWIND_USER_FP		if X86_64
 	select HAVE_USER_RETURN_NOTIFIER
 	select HAVE_GENERIC_VDSO
 	select VDSO_GETRANDOM			if X86_64
arch/x86/include/asm/unwind_user.h new file +11 -0
new file mode 100644
index 000000000000..8597857bf896
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_UNWIND_USER_H
+#define _ASM_X86_UNWIND_USER_H
+
+#define ARCH_INIT_USER_FP_FRAME							\
+	.cfa_off	= (s32)sizeof(long) *  2,				\
+	.ra_off		= (s32)sizeof(long) * -1,				\
+	.fp_off		= (s32)sizeof(long) * -2,				\
+	.use_fp		= true,
+
+#endif /* _ASM_X86_UNWIND_USER_H */
Signature · 1 line
2.47.2
author maintainer
#4 lore raw

PATCH v9 4/13 perf/x86: Rename and move get_segment_base() and make it global

In reply to Steven Rostedt

From: Josh Poimboeuf <[email protected]>

get_segment_base() will be used by the unwind_user code, so make it global and rename it so it doesn't conflict with a KVM function of the same name.

As the function is no longer specific to perf, move it to ptrace.c as that seems to be a better location for a generic function like this.

---
 arch/x86/events/core.c        | 44 ++++-------------------------------
 arch/x86/include/asm/ptrace.h |  2 ++
 arch/x86/kernel/ptrace.c      | 38 ++++++++++++++++++++++++++++++
 3 files changed, 45 insertions(+), 39 deletions(-)
arch/x86/events/core.c +5 -39
index 2e10dcf897c5..cc6329235b68 100644
@@ -43,6 +43,7 @@
 #include <asm/ldt.h>
 #include <asm/unwind.h>
 #include <asm/uprobes.h>
+#include <asm/ptrace.h>
 #include <asm/ibt.h>
 
 #include "perf_event.h"
@@ -2809,41 +2810,6 @@ valid_user_frame(const void __user *fp, unsigned long size)
 	return __access_ok(fp, size);
 }
 
-static unsigned long get_segment_base(unsigned int segment)
-{
-	struct desc_struct *desc;
-	unsigned int idx = segment >> 3;
-
-	if ((segment & SEGMENT_TI_MASK) == SEGMENT_LDT) {
-#ifdef CONFIG_MODIFY_LDT_SYSCALL
-		struct ldt_struct *ldt;
-
-		/*
-		 * If we're not in a valid context with a real (not just lazy)
-		 * user mm, then don't even try.
-		 */
-		if (!nmi_uaccess_okay())
-			return 0;
-
-		/* IRQs are off, so this synchronizes with smp_store_release */
-		ldt = smp_load_acquire(&current->mm->context.ldt);
-		if (!ldt || idx >= ldt->nr_entries)
-			return 0;
-
-		desc = &ldt->entries[idx];
-#else
-		return 0;
-#endif
-	} else {
-		if (idx >= GDT_ENTRIES)
-			return 0;
-
-		desc = raw_cpu_ptr(gdt_page.gdt) + idx;
-	}
-
-	return get_desc_base(desc);
-}
-
 #ifdef CONFIG_UPROBES
 /*
  * Heuristic-based check if uprobe is installed at the function entry.
@@ -2900,8 +2866,8 @@ perf_callchain_user32(struct pt_regs *regs, struct perf_callchain_entry_ctx *ent
 	if (user_64bit_mode(regs))
 		return 0;
 
-	cs_base = get_segment_base(regs->cs);
-	ss_base = get_segment_base(regs->ss);
+	cs_base = segment_base_address(regs->cs);
+	ss_base = segment_base_address(regs->ss);
 
 	fp = compat_ptr(ss_base + regs->bp);
 	pagefault_disable();
@@ -3020,11 +2986,11 @@ static unsigned long code_segment_base(struct pt_regs *regs)
 		return 0x10 * regs->cs;
 
 	if (user_mode(regs) && regs->cs != __USER_CS)
-		return get_segment_base(regs->cs);
+		return segment_base_address(regs->cs);
 #else
 	if (user_mode(regs) && !user_64bit_mode(regs) &&
 	    regs->cs != __USER32_CS)
-		return get_segment_base(regs->cs);
+		return segment_base_address(regs->cs);
 #endif
 	return 0;
 }
arch/x86/include/asm/ptrace.h +2 -0
index 50f75467f73d..59357ec98e52 100644
@@ -314,6 +314,8 @@ static __always_inline bool regs_irqs_disabled(struct pt_regs *regs)
 	return !(regs->flags & X86_EFLAGS_IF);
 }
 
+unsigned long segment_base_address(unsigned int segment);
+
 /* Query offset/name of register from its name/offset */
 extern int regs_query_register_offset(const char *name);
 extern const char *regs_query_register_name(unsigned int offset);
arch/x86/kernel/ptrace.c +38 -0
index 095f04bdabdc..81353a09701b 100644
@@ -41,6 +41,7 @@
 #include <asm/syscall.h>
 #include <asm/fsgsbase.h>
 #include <asm/io_bitmap.h>
+#include <asm/mmu_context.h>
 
 #include "tls.h"
 
@@ -339,6 +340,43 @@ static int set_segment_reg(struct task_struct *task,
 
 #endif	/* CONFIG_X86_32 */
 
+unsigned long segment_base_address(unsigned int segment)
+{
+	struct desc_struct *desc;
+	unsigned int idx = segment >> 3;
+
+	lockdep_assert_irqs_disabled();
+
+	if ((segment & SEGMENT_TI_MASK) == SEGMENT_LDT) {
+#ifdef CONFIG_MODIFY_LDT_SYSCALL
+		struct ldt_struct *ldt;
+
+		/*
+		 * If we're not in a valid context with a real (not just lazy)
+		 * user mm, then don't even try.
+		 */
+		if (!nmi_uaccess_okay())
+			return 0;
+
+		/* IRQs are off, so this synchronizes with smp_store_release */
+		ldt = smp_load_acquire(&current->mm->context.ldt);
+		if (!ldt || idx >= ldt->nr_entries)
+			return 0;
+
+		desc = &ldt->entries[idx];
+#else
+		return 0;
+#endif
+	} else {
+		if (idx >= GDT_ENTRIES)
+			return 0;
+
+		desc = raw_cpu_ptr(gdt_page.gdt) + idx;
+	}
+
+	return get_desc_base(desc);
+}
+
 static unsigned long get_flags(struct task_struct *task)
 {
 	unsigned long retval = task_pt_regs(task)->flags;
Signature · 1 line
2.47.2
author maintainer
#5 lore raw

PATCH v9 5/13 unwind_user: Add compat mode frame pointer support

In reply to Steven Rostedt

From: Josh Poimboeuf <[email protected]>

Add optional support for user space compat mode frame pointer unwinding. If supported, the arch needs to enable CONFIG_HAVE_UNWIND_USER_COMPAT_FP and define ARCH_INIT_USER_COMPAT_FP_FRAME.

---
 arch/Kconfig                            |  4 +++
 include/asm-generic/Kbuild              |  2 ++
 include/asm-generic/unwind_user.h       | 15 +++++++++++
 include/asm-generic/unwind_user_types.h |  9 +++++++
 include/linux/unwind_user_types.h       |  3 +++
 kernel/unwind/user.c                    | 36 ++++++++++++++++++++++---
 6 files changed, 65 insertions(+), 4 deletions(-)
 create mode 100644 include/asm-generic/unwind_user_types.h
arch/Kconfig +4 -0
index 0e3844c0e200..dbb1cc89e040 100644
@@ -442,6 +442,10 @@ config HAVE_UNWIND_USER_FP
 	bool
 	select UNWIND_USER
 
+config HAVE_UNWIND_USER_COMPAT_FP
+	bool
+	depends on HAVE_UNWIND_USER_FP
+
 config HAVE_PERF_REGS
 	bool
 	help
include/asm-generic/Kbuild +2 -0
index 8675b7b4ad23..b797a2434396 100644
@@ -59,6 +59,8 @@ mandatory-y += tlbflush.h
 mandatory-y += topology.h
 mandatory-y += trace_clock.h
 mandatory-y += uaccess.h
+mandatory-y += unwind_user.h
+mandatory-y += unwind_user_types.h
 mandatory-y += vermagic.h
 mandatory-y += vga.h
 mandatory-y += video.h
include/asm-generic/unwind_user.h +15 -0
index 832425502fb3..385638ce4aec 100644
@@ -2,8 +2,23 @@
 #ifndef _ASM_GENERIC_UNWIND_USER_H
 #define _ASM_GENERIC_UNWIND_USER_H
 
+#include <asm/unwind_user_types.h>
+
 #ifndef ARCH_INIT_USER_FP_FRAME
  #define ARCH_INIT_USER_FP_FRAME
 #endif
 
+#ifndef ARCH_INIT_USER_COMPAT_FP_FRAME
+ #define ARCH_INIT_USER_COMPAT_FP_FRAME
+ #define in_compat_mode(regs) false
+#endif
+
+#ifndef arch_unwind_user_init
+static inline void arch_unwind_user_init(struct unwind_user_state *state, struct pt_regs *reg) {}
+#endif
+
+#ifndef arch_unwind_user_next
+static inline void arch_unwind_user_next(struct unwind_user_state *state) {}
+#endif
+
 #endif /* _ASM_GENERIC_UNWIND_USER_H */
include/asm-generic/unwind_user_types.h new file +9 -0
new file mode 100644
index 000000000000..ee803de7c998
@@ -0,0 +1,9 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_GENERIC_UNWIND_USER_TYPES_H
+#define _ASM_GENERIC_UNWIND_USER_TYPES_H
+
+#ifndef arch_unwind_user_state
+struct arch_unwind_user_state {};
+#endif
+
+#endif /* _ASM_GENERIC_UNWIND_USER_TYPES_H */
include/linux/unwind_user_types.h +3 -0
index 65bd070eb6b0..3ec4a097a3dd 100644
@@ -3,10 +3,12 @@
 #define _LINUX_UNWIND_USER_TYPES_H
 
 #include <linux/types.h>
+#include <asm/unwind_user_types.h>
 
 enum unwind_user_type {
 	UNWIND_USER_TYPE_NONE,
 	UNWIND_USER_TYPE_FP,
+	UNWIND_USER_TYPE_COMPAT_FP,
 };
 
 struct unwind_stacktrace {
@@ -25,6 +27,7 @@ struct unwind_user_state {
 	unsigned long ip;
 	unsigned long sp;
 	unsigned long fp;
+	struct arch_unwind_user_state arch;
 	enum unwind_user_type type;
 	bool done;
 };
kernel/unwind/user.c +32 -4
index 0671a81494d3..635cc04bb299 100644
@@ -13,12 +13,32 @@ static struct unwind_user_frame fp_frame = {
 	ARCH_INIT_USER_FP_FRAME
 };
 
+static struct unwind_user_frame compat_fp_frame = {
+	ARCH_INIT_USER_COMPAT_FP_FRAME
+};
+
 static inline bool fp_state(struct unwind_user_state *state)
 {
 	return IS_ENABLED(CONFIG_HAVE_UNWIND_USER_FP) &&
 	       state->type == UNWIND_USER_TYPE_FP;
 }
 
+static inline bool compat_state(struct unwind_user_state *state)
+{
+	return IS_ENABLED(CONFIG_HAVE_UNWIND_USER_COMPAT_FP) &&
+	       state->type == UNWIND_USER_TYPE_COMPAT_FP;
+}
+
+#define UNWIND_GET_USER_LONG(to, from, state)				\
+({									\
+	int __ret;							\
+	if (compat_state(state))					\
+		__ret = get_user(to, (u32 __user *)(from));		\
+	else								\
+		__ret = get_user(to, (u64 __user *)(from));		\
+	__ret;								\
+})
+
 int unwind_user_next(struct unwind_user_state *state)
 {
 	struct unwind_user_frame _frame;
@@ -28,7 +48,9 @@ int unwind_user_next(struct unwind_user_state *state)
 	if (state->done)
 		return -EINVAL;
 
-	if (fp_state(state))
+	if (compat_state(state))
+		frame = &compat_fp_frame;
+	else if (fp_state(state))
 		frame = &fp_frame;
 	else
 		goto the_end;
@@ -39,10 +61,10 @@ int unwind_user_next(struct unwind_user_state *state)
 	if (cfa <= state->sp)
 		goto the_end;
 
-	if (get_user(ra, (unsigned long *)(cfa + frame->ra_off)))
+	if (UNWIND_GET_USER_LONG(ra, cfa + frame->ra_off, state))
 		goto the_end;
 
-	if (frame->fp_off && get_user(fp, (unsigned long __user *)(cfa + frame->fp_off)))
+	if (frame->fp_off && UNWIND_GET_USER_LONG(fp, cfa + frame->fp_off, state))
 		goto the_end;
 
 	state->ip = ra;
@@ -50,6 +72,8 @@ int unwind_user_next(struct unwind_user_state *state)
 	if (frame->fp_off)
 		state->fp = fp;
 
+	arch_unwind_user_next(state);
+
 	return 0;
 
 the_end:
@@ -68,7 +92,9 @@ int unwind_user_start(struct unwind_user_state *state)
 		return -EINVAL;
 	}
 
-	if (IS_ENABLED(CONFIG_HAVE_UNWIND_USER_FP))
+	if (IS_ENABLED(CONFIG_HAVE_UNWIND_USER_COMPAT_FP) && in_compat_mode(regs))
+		state->type = UNWIND_USER_TYPE_COMPAT_FP;
+	else if (IS_ENABLED(CONFIG_HAVE_UNWIND_USER_FP))
 		state->type = UNWIND_USER_TYPE_FP;
 	else
 		state->type = UNWIND_USER_TYPE_NONE;
@@ -77,6 +103,8 @@ int unwind_user_start(struct unwind_user_state *state)
 	state->sp = user_stack_pointer(regs);
 	state->fp = frame_pointer(regs);
 
+	arch_unwind_user_init(state, regs);
+
 	return 0;
 }
 
Signature · 1 line
2.47.2
author maintainer
#6 lore raw

PATCH v9 6/13 unwind_user/x86: Enable compat mode frame pointer unwinding on x86

In reply to Steven Rostedt

From: Josh Poimboeuf <[email protected]>

Use ARCH_INIT_USER_COMPAT_FP_FRAME to describe how frame pointers are unwound on x86, and implement the hooks needed to add the segment base addresses. Enable HAVE_UNWIND_USER_COMPAT_FP if the system has compat mode compiled in.

---
 arch/x86/Kconfig                         |  1 +
 arch/x86/include/asm/unwind_user.h       | 50 ++++++++++++++++++++++++
 arch/x86/include/asm/unwind_user_types.h | 17 ++++++++
 3 files changed, 68 insertions(+)
 create mode 100644 arch/x86/include/asm/unwind_user_types.h
arch/x86/Kconfig +1 -0
index 456e085263e1..90f055956fce 100644
@@ -301,6 +301,7 @@ config X86
 	select HAVE_SYSCALL_TRACEPOINTS
 	select HAVE_UACCESS_VALIDATION		if HAVE_OBJTOOL
 	select HAVE_UNSTABLE_SCHED_CLOCK
+	select HAVE_UNWIND_USER_COMPAT_FP	if IA32_EMULATION
 	select HAVE_UNWIND_USER_FP		if X86_64
 	select HAVE_USER_RETURN_NOTIFIER
 	select HAVE_GENERIC_VDSO
arch/x86/include/asm/unwind_user.h +50 -0

60 lines not on this page · raw message

arch/x86/include/asm/unwind_user_types.h new file +17 -0
new file mode 100644
index 000000000000..d7074dc5f0ce
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_UNWIND_USER_TYPES_H
+#define _ASM_UNWIND_USER_TYPES_H
+
+#ifdef CONFIG_IA32_EMULATION
+
+struct arch_unwind_user_state {
+	unsigned long ss_base;
+	unsigned long cs_base;
+};
+#define arch_unwind_user_state arch_unwind_user_state
+
+#endif /* CONFIG_IA32_EMULATION */
+
+#include <asm-generic/unwind_user_types.h>
+
+#endif /* _ASM_UNWIND_USER_TYPES_H */
Signature · 1 line
2.47.2
author maintainer
#7 lore raw

PATCH v9 7/13 unwind_user/deferred: Add unwind_deferred_trace()

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

Add a function that must be called inside a faultable context that will retrieve a user space stack trace. The function unwind_deferred_trace() can be called by a tracer when a task is about to enter user space, or has just come back from user space and has interrupts enabled.

This code is based on work by Josh Poimboeuf's deferred unwinding code:

---
 include/linux/sched.h                 |  5 +++
 include/linux/unwind_deferred.h       | 24 +++++++++++
 include/linux/unwind_deferred_types.h |  9 ++++
 kernel/fork.c                         |  4 ++
 kernel/unwind/Makefile                |  2 +-
 kernel/unwind/deferred.c              | 60 +++++++++++++++++++++++++++
 6 files changed, 103 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/unwind_deferred.h
 create mode 100644 include/linux/unwind_deferred_types.h
 create mode 100644 kernel/unwind/deferred.c
include/linux/sched.h +5 -0
index 4ecc0c6b1cb0..a1e1c07cadfb 100644
@@ -47,6 +47,7 @@
 #include <linux/livepatch_sched.h>
 #include <linux/uidgid_types.h>
 #include <linux/tracepoint-defs.h>
+#include <linux/unwind_deferred_types.h>
 #include <asm/kmap_size.h>
 
 /* task_struct member predeclarations (sorted alphabetically): */
@@ -1646,6 +1647,10 @@ struct task_struct {
 	struct user_event_mm		*user_event_mm;
 #endif
 
+#ifdef CONFIG_UNWIND_USER
+	struct unwind_task_info		unwind_info;
+#endif
+
 	/* CPU-specific state of this task: */
 	struct thread_struct		thread;
 
include/linux/unwind_deferred.h new file +24 -0
new file mode 100644
index 000000000000..5064ebe38c4f
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_UNWIND_USER_DEFERRED_H
+#define _LINUX_UNWIND_USER_DEFERRED_H
+
+#include <linux/unwind_user.h>
+#include <linux/unwind_deferred_types.h>
+
+#ifdef CONFIG_UNWIND_USER
+
+void unwind_task_init(struct task_struct *task);
+void unwind_task_free(struct task_struct *task);
+
+int unwind_deferred_trace(struct unwind_stacktrace *trace);
+
+#else /* !CONFIG_UNWIND_USER */
+
+static inline void unwind_task_init(struct task_struct *task) {}
+static inline void unwind_task_free(struct task_struct *task) {}
+
+static inline int unwind_deferred_trace(struct unwind_stacktrace *trace) { return -ENOSYS; }
+
+#endif /* !CONFIG_UNWIND_USER */
+
+#endif /* _LINUX_UNWIND_USER_DEFERRED_H */
include/linux/unwind_deferred_types.h new file +9 -0
new file mode 100644
index 000000000000..aa32db574e43
@@ -0,0 +1,9 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_UNWIND_USER_DEFERRED_TYPES_H
+#define _LINUX_UNWIND_USER_DEFERRED_TYPES_H
+
+struct unwind_task_info {
+	unsigned long		*entries;
+};
+
+#endif /* _LINUX_UNWIND_USER_DEFERRED_TYPES_H */
kernel/fork.c +4 -0
index 1f5d8083eeb2..b767727c6c06 100644
@@ -105,6 +105,7 @@
 #include <uapi/linux/pidfd.h>
 #include <linux/pidfs.h>
 #include <linux/tick.h>
+#include <linux/unwind_deferred.h>
 
 #include <asm/pgalloc.h>
 #include <linux/uaccess.h>
@@ -991,6 +992,7 @@ void __put_task_struct(struct task_struct *tsk)
 	WARN_ON(refcount_read(&tsk->usage));
 	WARN_ON(tsk == current);
 
+	unwind_task_free(tsk);
 	sched_ext_free(tsk);
 	io_uring_free(tsk);
 	cgroup_free(tsk);
@@ -2404,6 +2406,8 @@ __latent_entropy struct task_struct *copy_process(
 	p->bpf_ctx = NULL;
 #endif
 
+	unwind_task_init(p);
+
 	/* Perform scheduler related setup. Assign this task to a CPU. */
 	retval = sched_fork(clone_flags, p);
 	if (retval)
kernel/unwind/Makefile +1 -1
index 349ce3677526..6752ac96d7e2 100644
@@ -1 +1 @@
- obj-$(CONFIG_UNWIND_USER) += user.o
+ obj-$(CONFIG_UNWIND_USER)		+= user.o deferred.o
kernel/unwind/deferred.c new file +60 -0

60 lines not on this page · raw message

Signature · 1 line
2.47.2
author maintainer
#8 lore raw

PATCH v9 8/13 unwind_user/deferred: Add unwind cache

In reply to Steven Rostedt

From: Josh Poimboeuf <[email protected]>

Cache the results of the unwind to ensure the unwind is only performed once, even when called by multiple tracers.

The cache nr_entries gets cleared every time the task exits the kernel. When a stacktrace is requested, nr_entries gets set to the number of entries in the stacktrace. If another stacktrace is requested, if nr_entries is not zero, then it contains the same stacktrace that would be retrieved so it is not processed again and the entries is given to the caller.

---
 include/linux/entry-common.h          |  2 ++
 include/linux/unwind_deferred.h       |  8 ++++++++
 include/linux/unwind_deferred_types.h |  7 ++++++-
 kernel/unwind/deferred.c              | 26 ++++++++++++++++++++------
 4 files changed, 36 insertions(+), 7 deletions(-)
include/linux/entry-common.h +2 -0
index f94f3fdf15fc..6e850c9d3f0c 100644
@@ -12,6 +12,7 @@
 #include <linux/resume_user_mode.h>
 #include <linux/tick.h>
 #include <linux/kmsan.h>
+#include <linux/unwind_deferred.h>
 
 #include <asm/entry-common.h>
 #include <asm/syscall.h>
@@ -362,6 +363,7 @@ static __always_inline void exit_to_user_mode(void)
 	lockdep_hardirqs_on_prepare();
 	instrumentation_end();
 
+	unwind_exit_to_user_mode();
 	user_enter_irqoff();
 	arch_exit_to_user_mode();
 	lockdep_hardirqs_on(CALLER_ADDR0);
include/linux/unwind_deferred.h +8 -0
index 5064ebe38c4f..7d6cb2ffd084 100644
@@ -12,6 +12,12 @@ void unwind_task_free(struct task_struct *task);
 
 int unwind_deferred_trace(struct unwind_stacktrace *trace);
 
+static __always_inline void unwind_exit_to_user_mode(void)
+{
+	if (unlikely(current->unwind_info.cache))
+		current->unwind_info.cache->nr_entries = 0;
+}
+
 #else /* !CONFIG_UNWIND_USER */
 
 static inline void unwind_task_init(struct task_struct *task) {}
@@ -19,6 +25,8 @@ static inline void unwind_task_free(struct task_struct *task) {}
 
 static inline int unwind_deferred_trace(struct unwind_stacktrace *trace) { return -ENOSYS; }
 
+static inline void unwind_exit_to_user_mode(void) {}
+
 #endif /* !CONFIG_UNWIND_USER */
 
 #endif /* _LINUX_UNWIND_USER_DEFERRED_H */
include/linux/unwind_deferred_types.h +6 -1
index aa32db574e43..db5b54b18828 100644
@@ -2,8 +2,13 @@
 #ifndef _LINUX_UNWIND_USER_DEFERRED_TYPES_H
 #define _LINUX_UNWIND_USER_DEFERRED_TYPES_H
 
+struct unwind_cache {
+	unsigned int		nr_entries;
+	unsigned long		entries[];
+};
+
 struct unwind_task_info {
-	unsigned long		*entries;
+	struct unwind_cache	*cache;
 };
 
 #endif /* _LINUX_UNWIND_USER_DEFERRED_TYPES_H */
kernel/unwind/deferred.c +20 -6
index 0bafb95e6336..e3913781c8c6 100644
@@ -24,6 +24,7 @@
 int unwind_deferred_trace(struct unwind_stacktrace *trace)
 {
 	struct unwind_task_info *info = &current->unwind_info;
+	struct unwind_cache *cache;
 
 	/* Should always be called from faultable context */
 	might_fault();
@@ -31,17 +32,30 @@ int unwind_deferred_trace(struct unwind_stacktrace *trace)
 	if (current->flags & PF_EXITING)
 		return -EINVAL;
 
-	if (!info->entries) {
-		info->entries = kmalloc_array(UNWIND_MAX_ENTRIES, sizeof(long),
-					      GFP_KERNEL);
-		if (!info->entries)
+	if (!info->cache) {
+		info->cache = kzalloc(struct_size(cache, entries, UNWIND_MAX_ENTRIES),
+				      GFP_KERNEL);
+		if (!info->cache)
 			return -ENOMEM;
 	}
 
+	cache = info->cache;
+	trace->entries = cache->entries;
+
+	if (cache->nr_entries) {
+		/*
+		 * The user stack has already been previously unwound in this
+		 * entry context.  Skip the unwind and use the cache.
+		 */
+		trace->nr = cache->nr_entries;
+		return 0;
+	}
+
 	trace->nr = 0;
-	trace->entries = info->entries;
 	unwind_user(trace, UNWIND_MAX_ENTRIES);
 
+	cache->nr_entries = trace->nr;
+
 	return 0;
 }
 
@@ -56,5 +70,5 @@ void unwind_task_free(struct task_struct *task)
 {
 	struct unwind_task_info *info = &task->unwind_info;
 
-	kfree(info->entries);
+	kfree(info->cache);
 }
Signature · 1 line
2.47.2
author maintainer
#9 lore raw

PATCH v9 9/13 unwind_user/deferred: Add deferred unwinding interface

In reply to Steven Rostedt

From: Josh Poimboeuf <[email protected]>

Add an interface for scheduling task work to unwind the user space stack before returning to user space. This solves several problems for its callers:

- Ensure the unwind happens in task context even if the caller may be running in NMI or interrupt context.

- Avoid duplicate unwinds, whether called multiple times by the same caller or by different callers.

- Take a timestamp when the first request comes in since the task entered the kernel. This will be returned to the calling function along with the stack trace when the task leaves the kernel. This timestamp can be used to correlate kernel unwinds/traces with the user unwind.

The timestamp is created to detect when the stacktrace is the same. It is generated the first time a user space stacktrace is requested after the task enters the kernel.

The timestamp is passed to the caller on request, and when the stacktrace is generated upon returning to user space, it call the requester's callback with the timestamp as well as the stacktrace.

---
 include/linux/unwind_deferred.h       |  18 ++++
 include/linux/unwind_deferred_types.h |   3 +
 kernel/unwind/deferred.c              | 131 +++++++++++++++++++++++++-
 3 files changed, 151 insertions(+), 1 deletion(-)
include/linux/unwind_deferred.h +18 -0
index 7d6cb2ffd084..a384eef719a3 100644
@@ -2,9 +2,19 @@
 #ifndef _LINUX_UNWIND_USER_DEFERRED_H
 #define _LINUX_UNWIND_USER_DEFERRED_H
 
+#include <linux/task_work.h>
 #include <linux/unwind_user.h>
 #include <linux/unwind_deferred_types.h>
 
+struct unwind_work;
+
+typedef void (*unwind_callback_t)(struct unwind_work *work, struct unwind_stacktrace *trace, u64 timestamp);
+
+struct unwind_work {
+	struct list_head		list;
+	unwind_callback_t		func;
+};
+
 #ifdef CONFIG_UNWIND_USER
 
 void unwind_task_init(struct task_struct *task);
@@ -12,10 +22,15 @@ void unwind_task_free(struct task_struct *task);
 
 int unwind_deferred_trace(struct unwind_stacktrace *trace);
 
+int unwind_deferred_init(struct unwind_work *work, unwind_callback_t func);
+int unwind_deferred_request(struct unwind_work *work, u64 *timestamp);
+void unwind_deferred_cancel(struct unwind_work *work);
+
 static __always_inline void unwind_exit_to_user_mode(void)
 {
 	if (unlikely(current->unwind_info.cache))
 		current->unwind_info.cache->nr_entries = 0;
+	current->unwind_info.timestamp = 0;
 }
 
 #else /* !CONFIG_UNWIND_USER */
@@ -24,6 +39,9 @@ static inline void unwind_task_init(struct task_struct *task) {}
 static inline void unwind_task_free(struct task_struct *task) {}
 
 static inline int unwind_deferred_trace(struct unwind_stacktrace *trace) { return -ENOSYS; }
+static inline int unwind_deferred_init(struct unwind_work *work, unwind_callback_t func) { return -ENOSYS; }
+static inline int unwind_deferred_request(struct unwind_work *work, u64 *timestamp) { return -ENOSYS; }
+static inline void unwind_deferred_cancel(struct unwind_work *work) {}
 
 static inline void unwind_exit_to_user_mode(void) {}
 
include/linux/unwind_deferred_types.h +3 -0
index db5b54b18828..5df264cf81ad 100644
@@ -9,6 +9,9 @@ struct unwind_cache {
 
 struct unwind_task_info {
 	struct unwind_cache	*cache;
+	struct callback_head	work;
+	u64			timestamp;
+	int			pending;
 };
 
 #endif /* _LINUX_UNWIND_USER_DEFERRED_TYPES_H */
kernel/unwind/deferred.c +130 -1

158 lines not on this page · raw message

Signature · 1 line
2.47.2
author maintainer
#10 lore raw

PATCH v9 10/13 unwind_user/deferred: Make unwind deferral requests NMI-safe

In reply to Steven Rostedt

From: Josh Poimboeuf <[email protected]>

Make unwind_deferred_request() NMI-safe so tracers in NMI context can call it and safely request a user space stacktrace when the task exits.

A "nmi_timestamp" is added to the unwind_task_info that gets updated by NMIs to not race with setting the info->timestamp.

---
 include/linux/unwind_deferred_types.h |  1 +
 kernel/unwind/deferred.c              | 91 ++++++++++++++++++++++++---
 2 files changed, 84 insertions(+), 8 deletions(-)
include/linux/unwind_deferred_types.h +1 -0
index 5df264cf81ad..ae27a02234b8 100644
@@ -11,6 +11,7 @@ struct unwind_task_info {
 	struct unwind_cache	*cache;
 	struct callback_head	work;
 	u64			timestamp;
+	u64			nmi_timestamp;
 	int			pending;
 };
 
kernel/unwind/deferred.c +83 -8

134 lines not on this page · raw message

Signature · 1 line
2.47.2
author maintainer
#11 lore raw

PATCH v9 11/13 unwind deferred: Use bitmask to determine which callbacks to call

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

In order to know which registered callback requested a stacktrace for when the task goes back to user space, add a bitmask for all registered tracers. The bitmask is the size of log, which means that on a 32 bit machine, it can have at most 32 registered tracers, and on 64 bit, it can have at most 64 registered tracers. This should not be an issue as there should not be more than 10 (unless BPF can abuse this?).

When a tracer registers with unwind_deferred_init() it will get a bit number assigned to it. When a tracer requests a stacktrace, it will have its bit set within the task_struct. When the task returns back to user space, it will call the callbacks for all the registered tracers where their bits are set in the task's mask.

When a tracer is removed by the unwind_deferred_cancel() all current tasks will clear the associated bit, just in case another tracer gets registered immediately afterward and then gets their callback called unexpectedly.

---
 include/linux/sched.h           |  1 +
 include/linux/unwind_deferred.h |  1 +
 kernel/unwind/deferred.c        | 46 ++++++++++++++++++++++++++++-----
 3 files changed, 41 insertions(+), 7 deletions(-)
include/linux/sched.h +1 -0
index a1e1c07cadfb..d3ee0c5405d6 100644
@@ -1649,6 +1649,7 @@ struct task_struct {
 
 #ifdef CONFIG_UNWIND_USER
 	struct unwind_task_info		unwind_info;
+	unsigned long			unwind_mask;
 #endif
 
 	/* CPU-specific state of this task: */
include/linux/unwind_deferred.h +1 -0
index a384eef719a3..1789c3624723 100644
@@ -13,6 +13,7 @@ typedef void (*unwind_callback_t)(struct unwind_work *work, struct unwind_stackt
 struct unwind_work {
 	struct list_head		list;
 	unwind_callback_t		func;
+	int				bit;
 };
 
 #ifdef CONFIG_UNWIND_USER
kernel/unwind/deferred.c +39 -7

117 lines not on this page · raw message

Signature · 1 line
2.47.2
author maintainer
#12 lore raw

PATCH v9 12/13 unwind deferred: Use SRCU unwind_deferred_task_work()

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

Instead of using the callback_mutex to protect the link list of callbacks in unwind_deferred_task_work(), use SRCU instead. This gets called every time a task exits that has to record a stack trace that was requested. This can happen for many tasks on several CPUs at the same time. A mutex is a bottleneck and can cause a bit of contention and slow down performance.

As the callbacks themselves are allowed to sleep, regular RCU can not be used to protect the list. Instead use SRCU, as that still allows the callbacks to sleep and the list can be read without needing to hold the callback_mutex.

---
Discussion about using guard for SRCU:
https://lore.kernel.org/[email protected]

Andrii Nakryiko brought up using guard for SRCU around the list iteration, but it was decided that just using the normal methods were fine for this use case.

 kernel/unwind/deferred.c | 33 +++++++++++++++++++++++++--------
 1 file changed, 25 insertions(+), 8 deletions(-)
kernel/unwind/deferred.c +25 -8
index 7ae0bec5b36a..5d6976ee648f 100644
@@ -13,10 +13,11 @@
 
 #define UNWIND_MAX_ENTRIES 512
 
-/* Guards adding to and reading the list of callbacks */
+/* Guards adding to or removing from the list of callbacks */
 static DEFINE_MUTEX(callback_mutex);
 static LIST_HEAD(callbacks);
 static unsigned long unwind_mask;
+DEFINE_STATIC_SRCU(unwind_srcu);
 
 /*
  * Read the task context timestamp, if this is the first caller then
@@ -108,6 +109,7 @@ static void unwind_deferred_task_work(struct callback_head *head)
 	struct unwind_work *work;
 	u64 timestamp;
 	struct task_struct *task = current;
+	int idx;
 
 	if (WARN_ON_ONCE(!info->pending))
 		return;
@@ -133,13 +135,15 @@ static void unwind_deferred_task_work(struct callback_head *head)
 
 	timestamp = info->timestamp;
 
-	guard(mutex)(&callback_mutex);
-	list_for_each_entry(work, &callbacks, list) {
+	idx = srcu_read_lock(&unwind_srcu);
+	list_for_each_entry_srcu(work, &callbacks, list,
+				 srcu_read_lock_held(&unwind_srcu)) {
 		if (task->unwind_mask & (1UL << work->bit)) {
 			work->func(work, &trace, timestamp);
 			clear_bit(work->bit, &current->unwind_mask);
 		}
 	}
+	srcu_read_unlock(&unwind_srcu, idx);
 }
 
 static int unwind_deferred_request_nmi(struct unwind_work *work, u64 *timestamp)
@@ -216,6 +220,7 @@ int unwind_deferred_request(struct unwind_work *work, u64 *timestamp)
 {
 	struct unwind_task_info *info = &current->unwind_info;
 	int pending;
+	int bit;
 	int ret;
 
 	*timestamp = 0;
@@ -227,12 +232,17 @@ int unwind_deferred_request(struct unwind_work *work, u64 *timestamp)
 	if (in_nmi())
 		return unwind_deferred_request_nmi(work, timestamp);
 
+	/* Do not allow cancelled works to request again */
+	bit = READ_ONCE(work->bit);
+	if (WARN_ON_ONCE(bit < 0))
+		return -EINVAL;
+
 	guard(irqsave)();
 
 	*timestamp = get_timestamp(info);
 
 	/* This is already queued */
-	if (current->unwind_mask & (1UL << work->bit))
+	if (current->unwind_mask & (1UL << bit))
 		return 1;
 
 	/* callback already pending? */
@@ -258,19 +268,26 @@ int unwind_deferred_request(struct unwind_work *work, u64 *timestamp)
 void unwind_deferred_cancel(struct unwind_work *work)
 {
 	struct task_struct *g, *t;
+	int bit;
 
 	if (!work)
 		return;
 
 	guard(mutex)(&callback_mutex);
-	list_del(&work->list);
+	list_del_rcu(&work->list);
+	bit = work->bit;
+
+	/* Do not allow any more requests and prevent callbacks */
+	work->bit = -1;
+
+	clear_bit(bit, &unwind_mask);
 
-	clear_bit(work->bit, &unwind_mask);
+	synchronize_srcu(&unwind_srcu);
 
 	guard(rcu)();
 	/* Clear this bit from all threads */
 	for_each_process_thread(g, t) {
-		clear_bit(work->bit, &t->unwind_mask);
+		clear_bit(bit, &t->unwind_mask);
 	}
 }
 
@@ -287,7 +304,7 @@ int unwind_deferred_init(struct unwind_work *work, unwind_callback_t func)
 	work->bit = ffz(unwind_mask);
 	unwind_mask |= 1UL << work->bit;
 
-	list_add(&work->list, &callbacks);
+	list_add_rcu(&work->list, &callbacks);
 	work->func = func;
 	return 0;
 }
Signature · 1 line
2.47.2
author maintainer
#13 lore raw

PATCH v9 13/13 unwind: Clear unwind_mask on exit back to user space

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

When testing the deferred unwinder by attaching deferred user space stacktraces to events, a live lock happened. This was when the deferred unwinding was added to the irqs_disabled event, which happens after the task_work callbacks are called and before the task goes back to user space.

The event callback would be registered when irqs were disabled, the task_work would trigger, call the callback for this work and clear the work's bit. Then before getting back to user space, irqs would be disabled again, the event triggered again, and a new task_work registered. This caused an infinite loop and the system hung.

To prevent this, clear the bits at the very last moment before going back to user space and when instrumentation is disabled. That is in unwind_exit_to_user_mode().

Move the pending bit from a value on the task_struct to the most significant bit of the unwind_mask (saves space on the task_struct). This will allow modifying the pending bit along with the work bits atomically.

Instead of clearing a work's bit after its callback is called, it is delayed until exit. If the work is requested again, the task_work is not queued again and the work will be notified that the task has already been called (via UNWIND_ALREADY_EXECUTED return value).

The pending bit is cleared before calling the callback functions but the current work bits remain. If one of the called works registers again, it will not trigger a task_work if its bit is still present in the task's unwind_mask.

If a new work registers, then it will set both the pending bit and its own bit but clear the other work bits so that their callbacks do not get called again.

---
 include/linux/unwind_deferred.h       | 23 ++++++-
 include/linux/unwind_deferred_types.h |  1 -
 kernel/unwind/deferred.c              | 96 +++++++++++++++++++--------
 3 files changed, 90 insertions(+), 30 deletions(-)
include/linux/unwind_deferred.h +22 -1
index 1789c3624723..b3c8703fcc22 100644
@@ -18,6 +18,14 @@ struct unwind_work {
 
 #ifdef CONFIG_UNWIND_USER
 
+#define UNWIND_PENDING_BIT	(BITS_PER_LONG - 1)
+#define UNWIND_PENDING		(1UL << UNWIND_PENDING_BIT)
+
+enum {
+	UNWIND_ALREADY_PENDING	= 1,
+	UNWIND_ALREADY_EXECUTED	= 2,
+};
+
 void unwind_task_init(struct task_struct *task);
 void unwind_task_free(struct task_struct *task);
 
@@ -29,7 +37,20 @@ void unwind_deferred_cancel(struct unwind_work *work);
 
 static __always_inline void unwind_exit_to_user_mode(void)
 {
-	if (unlikely(current->unwind_info.cache))
+	unsigned long bits;
+
+	/* Was there any unwinding? */
+	if (likely(!current->unwind_mask))
+		return;
+
+	bits = current->unwind_mask;
+	do {
+		/* Is a task_work going to run again before going back */
+		if (bits & UNWIND_PENDING)
+			return;
+	} while (!try_cmpxchg(&current->unwind_mask, &bits, 0UL));
+
+	if (likely(current->unwind_info.cache))
 		current->unwind_info.cache->nr_entries = 0;
 	current->unwind_info.timestamp = 0;
 }
include/linux/unwind_deferred_types.h +0 -1
index ae27a02234b8..28811a9d4262 100644
@@ -12,7 +12,6 @@ struct unwind_task_info {
 	struct callback_head	work;
 	u64			timestamp;
 	u64			nmi_timestamp;
-	int			pending;
 };
 
 #endif /* _LINUX_UNWIND_USER_DEFERRED_TYPES_H */
kernel/unwind/deferred.c +68 -28

168 lines not on this page · raw message

Signature · 1 line
2.47.2
author maintainer
#14 lore raw
In reply to Steven Rostedt
Quoted 3 lines · On Tue, 13 May 2025 18:34:35 -0400 Steven Rostedt <[email protected]> wrote:

This has modifications in x86 and I would like it to go through the x86 tree. Preferably it can go into this merge window so we can focus on getting perf and ftrace to work on top of this.

I think it may be best for me to remove the two x86 specific patches, and rebuild the ftrace work on top of it. For testing, I'll just keep those two patches in my tree locally, but then I can get this moving for this merge window.

Next merge window, we can spend more time on getting the perf API working properly.

-- Steve

maintainer
#15 lore raw

Hi Steve,

Quoted 11 lines · Steven Rostedt
Quoted 3 lines · On Tue, 13 May 2025 18:34:35 -0400 Steven Rostedt <[email protected]> wrote:

This has modifications in x86 and I would like it to go through the x86 tree. Preferably it can go into this merge window so we can focus on getting perf and ftrace to work on top of this.

I think it may be best for me to remove the two x86 specific patches, and rebuild the ftrace work on top of it. For testing, I'll just keep those two patches in my tree locally, but then I can get this moving for this merge window.

Maybe I asked this before but I don't remember if I got the answer. :) How does it handle task exits as it won't go to userspace? I guess it'll lose user callstacks for exit syscalls and other termination paths.

Similarly, it will miss user callstacks in the samples at the end of profiling if the target tasks remain in the kernel (or they sleep). It looks like a fundamental limitation of the deferred callchains.

Thanks,
Namhyung

Quoted 5 lines

Next merge window, we can spend more time on getting the perf API working properly.

-- Steve

author maintainer
#16 lore raw
Quoted 15 lines · On Fri, 16 May 2025 16:39:56 -0700 Namhyung Kim <[email protected]> wrote:

Hi Steve,

Quoted 11 lines · Steven Rostedt
Quoted 3 lines · On Tue, 13 May 2025 18:34:35 -0400 Steven Rostedt <[email protected]> wrote:

This has modifications in x86 and I would like it to go through the x86 tree. Preferably it can go into this merge window so we can focus on getting perf and ftrace to work on top of this.

I think it may be best for me to remove the two x86 specific patches, and rebuild the ftrace work on top of it. For testing, I'll just keep those two patches in my tree locally, but then I can get this moving for this merge window.

Quoted 8 lines

Maybe I asked this before but I don't remember if I got the answer. :) How does it handle task exits as it won't go to userspace? I guess it'll lose user callstacks for exit syscalls and other termination paths.

Similarly, it will miss user callstacks in the samples at the end of profiling if the target tasks remain in the kernel (or they sleep). It looks like a fundamental limitation of the deferred callchains.

Ah, I think I forgot about that. I believe the exit path can also be a faultable path. All it needs is a hook to do the exit. Is there any "task work" clean up on exit? I need to take a look.

-- Steve

#17 lore raw
Quoted 22 lines · Steven Rostedt
Quoted 19 lines · On Fri, 16 May 2025 16:39:56 -0700 Namhyung Kim <[email protected]> wrote:

Hi Steve,

Quoted 11 lines · Steven Rostedt
Quoted 3 lines · On Tue, 13 May 2025 18:34:35 -0400 Steven Rostedt <[email protected]> wrote:

This has modifications in x86 and I would like it to go through the x86 tree. Preferably it can go into this merge window so we can focus on getting perf and ftrace to work on top of this.

I think it may be best for me to remove the two x86 specific patches, and rebuild the ftrace work on top of it. For testing, I'll just keep those two patches in my tree locally, but then I can get this moving for this merge window.

Maybe I asked this before but I don't remember if I got the answer. :) How does it handle task exits as it won't go to userspace? I guess it'll lose user callstacks for exit syscalls and other termination paths.

Quoted 8 lines
Quoted 4 lines

Similarly, it will miss user callstacks in the samples at the end of profiling if the target tasks remain in the kernel (or they sleep). It looks like a fundamental limitation of the deferred callchains.

Ah, I think I forgot about that. I believe the exit path can also be a faultable path. All it needs is a hook to do the exit. Is there any "task work" clean up on exit? I need to take a look.

Could you please not rush this facility into v6.16? It barely had any design review so far, and I'm still not entirely sure about the approach.

Thanks,

Ingo

author maintainer
#18 lore raw
Quoted 7 lines · On Tue, 20 May 2025 11:35:56 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 3 lines

Ah, I think I forgot about that. I believe the exit path can also be a faultable path. All it needs is a hook to do the exit. Is there any "task work" clean up on exit? I need to take a look.

Could you please not rush this facility into v6.16? It barely had any design review so far, and I'm still not entirely sure about the approach.

Hi Ingo,

Note, there has been a lot of discussion on this approach, although it's been mostly at conferences and in meetings. At GNU Cauldron in September 2024 (before Plumbers) Josh, Mathieu and myself discussed it in quite detail. I've then been hosting a monthly meeting with engineers from Google, Red Hat, EffiOS, Oracle, Microsoft and others (I can invite you to it if you would like). There's actually two meetings (one that is in a Asia friendly timezone and another in a European friendly timezone). The first patches from this went out in October, 2024:

https://lore.kernel.org/all/[email protected]/

There wasn't much discussion on it, although Peter did reply, and I believe that we did address all of his concerns.

Josh then changed the approach from what we originally discussed, which was to just have each tracer attach a task_work to the task it wants a trace thinking that would be good enough, but Mathieu and I found that it doesn't work because even a perf event can not handle this because it would need to keep track of several tasks that may migrate.

Let me state the goal of this work, as it started from a session at the 2022 Tracing Summit in London. We needed a way to get reliable user space stack traces without using frame pointers. We discussed various methods, including using eh_frame but they all had issues. We concluded that it would be nice to have an ORC unwinder (technically it's called a stack walker), for user space.

Then in Nov 2022, I read about "sframes" which was exactly what we were looking for:

https://www.phoronix.com/news/GNU-Binutils-SFrame

At FOSDEM 2023, I asked Jose Marchesi (a GCC maintainer) about sframes, and it just happened to be one of his employees that created it (Indu Bhagat). At Kernel Recipes 2023, Brendan Gregg, during his talk, asked if it would be great to have user space stack walking without needing to run everything with frame pointers. I raised my hand and asked if he had heard about "sframes", which he did not. I explained what it was and he was very interested. After that talk, Josh Poimboeuf came up to me and asked if he could implement this, which I agreed.

One thing that is needed for sframes is that it has to be done in a faultable context. The sframe sections are like ORC, where it has lookup tables, but they live in the user space address, and because they can be large, they can't be locked into memory. This brings up the deferred unwinding aspect.

At the LSFMMBPF 2023 conference, Indu and myself ran a session on sframes to get a better idea on how to implement this. The perf user space stack tracing was mentioned, where if frame pointers are not there, perf may copy thousands of bytes of the user space stack into the perf buffer. If there's a long system call, it may do this several times, and because the user space stack does not change while the task is in the kernel, this is thousands of bytes of duplicate data. I asked Jiri Olsa "why not just defer the stack trace and only make a single entry", and I believe he replied "we are thinking about it", but nothing further came about it.

Now, there's a serious push to get sframes upstream, and that will take a few steps. These are:

1) Create a new user unwind stack call that is expected to be called in faultable context. If a tracer (perf, ftrace, BPF or whatever) knows it's in a faultable context and wants a trace, it can simply ask for it.

It was also asked to have a user space system call that can get this trace so that a function in a user space applications can see what is calling it.

2) Create a deferred stack unwinding infrastructure that can be used by many clients (perf, ftrace, BPF or whatever) and called in any context (interrupt or NMI).

As the unwind stack call needs to be in a faultable context, and it is very common to want both a kernel stack trace along with a user space stack trace and this can happen in an NMI, allow the kernel stack trace to be executed and delay the user stack trace.

The accounting for this is not easy, as it has a many to many relationship. You could have perf, ftrace and BPF all asking for a delayed stack trace and they all need to be called. But each could be wanting a stack trace from a different set of tasks. Keeping track of which tracer gets a callback for which task is where this patch set comes in. Or at least just the infrastructure part.

3) Add sframes.

The final part is to get this working with sframes. Where a distro builds all its applications with sframes enabled and then perf, ftrace, BPF or whatever gets access to it through this interface.

There's quite a momentum of work happening today that is being built expecting us to get to step 3. There's no use to adding sframes to applications if the kernel can't read them. The only way to read them is to have this deferred infrastructure.

We've discussed the current design quiet a bit, but until there's actual users starting to build on top of it, all the corner cases may not come out. That's why I'm suggesting if we can just get the basic infrastructure in this merge window, where it's not fully enabled (there are no users of it), we can then have several users build on top of it in the next merge window to see if it finds anything that breaks.

As perf has the biggest user space ABI, where the delayed stack trace may be in a different event buffer than where the kernel stack trace occurred (due to migration), it's the one I'm most concern about getting this right. As once it is exposed to user space, it can never change. That's the one I do want to focus on the most. But it shouldn't delay getting the non user space visible aspect of the kernel side moving forward. If we find an issue, it can always be changed because it doesn't affect user space.

I've been testing this on the ftrace side and so far, everything works fine. But the ftrace side has the deferred trace always in the same instance buffer as where it was triggered, as it doesn't depend on user space API for that.

I've also been running this on perf and it too is working well. But I don't know the perf interface well enough to make sure there isn't other corner cases that I may be missing.

For 6.16, I would just like to get the common infrastructure in the kernel so there's no dependency on different tracers. This patch set adds the changes but does not enable it yet. This way, perf, ftrace and BPF can all work to build on top of the changes without needing to share code.

There's only two patches that touch x86 here. I have another patch series that removes them and implements this for ftrace, but because no architecture has it enabled, it's just dead code. But it would also allow to build on top of the infrastructure where any architecture could enable it. Kind of like what PREEMPT_RT did.

-- Steve

author maintainer
#19 lore raw
Quoted 4 lines · On Tue, 20 May 2025 11:57:21 -0400 Steven Rostedt <[email protected]> wrote:

For 6.16, I would just like to get the common infrastructure in the kernel so there's no dependency on different tracers. This patch set adds the changes but does not enable it yet. This way, perf, ftrace and BPF can all work to build on top of the changes without needing to share code.

Another thing that would work for me is not to push it this merge window, but if you could make a separate branch based on top of v6.15 when it is released, that has this code, where other subsystems could work on top of that, would be great!

-- Steve

#20 lore raw
In reply to Namhyung Kim
Quoted 15 lines · On Fri, 16 May 2025 16:39:56 -0700 Namhyung Kim <[email protected]> wrote:

Hi Steve,

Quoted 11 lines · Steven Rostedt
Quoted 3 lines · On Tue, 13 May 2025 18:34:35 -0400 Steven Rostedt <[email protected]> wrote:

This has modifications in x86 and I would like it to go through the x86 tree. Preferably it can go into this merge window so we can focus on getting perf and ftrace to work on top of this.

I think it may be best for me to remove the two x86 specific patches, and rebuild the ftrace work on top of it. For testing, I'll just keep those two patches in my tree locally, but then I can get this moving for this merge window.

Quoted 7 lines

Maybe I asked this before but I don't remember if I got the answer. :) How does it handle task exits as it won't go to userspace? I guess it'll lose user callstacks for exit syscalls and other termination paths.

Similarly, it will miss user callstacks in the samples at the end of profiling if the target tasks remain in the kernel (or they sleep). It looks like a fundamental limitation of the deferred callchains.

Can we use a hybrid approach for this case?
It might be more balanced (from the performance point of view) to save the full stack in a classic way only in this case, rather than faulting on process exit or doing file access just to load the sframe.

Thanks,

Quoted 9 lines

Thanks,
Namhyung

Quoted 5 lines

Next merge window, we can spend more time on getting the perf API working properly.

-- Steve

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
author maintainer
#21 lore raw
Quoted 3 lines · On Wed, 21 May 2025 08:26:05 +0900 Masami Hiramatsu (Google)…
Quoted 3 lines

Maybe I asked this before but I don't remember if I got the answer. :) How does it handle task exits as it won't go to userspace? I guess it'll lose user callstacks for exit syscalls and other termination paths.

I just checked, and the good news is that task_work does indeed get called when a task exits. The bad news is that it happens after do_exit() cleans up the task's "mm" structure via exit_mm(). Which means that current->mm is NULL :-p

There's a proposal to move trace_sched_process_exit() to before exit_mm(). If that happens, we could make that tracepoint a "faultable" tracepoint and then the unwind infrastructure could attach to it and do the unwinding from that tracepoint.

Quoted 4 lines
Quoted 4 lines

Similarly, it will miss user callstacks in the samples at the end of profiling if the target tasks remain in the kernel (or they sleep). It looks like a fundamental limitation of the deferred callchains.

Yes that is a limitation.

Quoted 5 lines

Can we use a hybrid approach for this case?
It might be more balanced (from the performance point of view) to save the full stack in a classic way only in this case, rather than faulting on process exit or doing file access just to load the sframe.

Another approach is that the tool (like perf) could request to take the user space stack trace every time a task enters the kernel via a system call.

-- Steve

author maintainer
#22 lore raw
Quoted 4 lines · On Tue, 20 May 2025 19:55:49 -0400 Steven Rostedt <[email protected]> wrote:

There's a proposal to move trace_sched_process_exit() to before exit_mm(). If that happens, we could make that tracepoint a "faultable" tracepoint and then the unwind infrastructure could attach to it and do the unwinding from that tracepoint.

The below patch does work. It's just a PoC and would need to be broken up and also cleaned up.

I created a TRACE_EVENT_FAULTABLE() that is basically just a TRACE_EVENT_SYSCALL(), and used that for the sched_process_exit tracepoint.

I then had the unwinder attach to that tracepoint when the first unwind callback is registered.

I had to change the check in the trace from testing PF_EXITING to just current->mm is NULL.

But this does work for the exiting of a task:

-- Steve

include/linux/tracepoint.h +2 -0
index a351763e6965..eb98bb61126e 100644
@@ -617,6 +617,8 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)
 #define TRACE_EVENT_SYSCALL(name, proto, args, struct, assign,	\
 			    print, reg, unreg)			\
 	DECLARE_TRACE_SYSCALL(name, PARAMS(proto), PARAMS(args))
+#define TRACE_EVENT_FAULTABLE(name, proto, args, struct, assign, print)	\
+	DECLARE_TRACE_SYSCALL(name, PARAMS(proto), PARAMS(args))
 
 #define TRACE_EVENT_FLAGS(event, flag)
 
include/trace/define_trace.h +5 -0
index ed52d0506c69..b228424744fd 100644
@@ -50,6 +50,10 @@
 #define TRACE_EVENT_SYSCALL(name, proto, args, struct, assign, print, reg, unreg) \
 	DEFINE_TRACE_SYSCALL(name, reg, unreg, PARAMS(proto), PARAMS(args))
 
+#undef TRACE_EVENT_FAULTABLE
+#define TRACE_EVENT_FAULTABLE(name, proto, args, struct, assign, print) \
+	DEFINE_TRACE_SYSCALL(name, NULL, NULL, PARAMS(proto), PARAMS(args))
+
 #undef TRACE_EVENT_NOP
 #define TRACE_EVENT_NOP(name, proto, args, struct, assign, print)
 
@@ -125,6 +129,7 @@
 #undef TRACE_EVENT_FN
 #undef TRACE_EVENT_FN_COND
 #undef TRACE_EVENT_SYSCALL
+#undef TRACE_EVENT_FAULTABLE
 #undef TRACE_EVENT_CONDITION
 #undef TRACE_EVENT_NOP
 #undef DEFINE_EVENT_NOP
include/trace/events/sched.h +2 -2
index 3bec9fb73a36..c6d7894970e3 100644
@@ -326,13 +326,13 @@ DEFINE_EVENT(sched_process_template, sched_process_free,
 	     TP_ARGS(p));
 
 /*
- * Tracepoint for a task exiting.
+ * Tracepoint for a task exiting (allows faulting)
  * Note, it's a superset of sched_process_template and should be kept
  * compatible as much as possible. sched_process_exits has an extra
  * `group_dead` argument, so sched_process_template can't be used,
  * unfortunately, just like sched_migrate_task above.
  */
-TRACE_EVENT(sched_process_exit,
+TRACE_EVENT_FAULTABLE(sched_process_exit,
 
 	TP_PROTO(struct task_struct *p, bool group_dead),
 
include/trace/trace_events.h +10 -0
index 4f22136fd465..0ed57e7906d1 100644
@@ -55,6 +55,16 @@
 			     PARAMS(print));		       \
 	DEFINE_EVENT(name, name, PARAMS(proto), PARAMS(args));
 
+#undef TRACE_EVENT_FAULTABLE
+#define TRACE_EVENT_FAULTABLE(name, proto, args, tstruct, assign, print) \
+	DECLARE_EVENT_SYSCALL_CLASS(name,		       \
+			     PARAMS(proto),		       \
+			     PARAMS(args),		       \
+			     PARAMS(tstruct),		       \
+			     PARAMS(assign),		       \
+			     PARAMS(print));		       \
+	DEFINE_EVENT(name, name, PARAMS(proto), PARAMS(args));
+
 #include "stages/stage1_struct_define.h"
 
 #undef DECLARE_EVENT_CLASS
kernel/unwind/deferred.c +34 -4
index 63d0237bad3e..7aad471f2887 100644
@@ -11,6 +11,8 @@
 #include <linux/slab.h>
 #include <linux/mm.h>
 
+#include <trace/events/sched.h>
+
 #define UNWIND_MAX_ENTRIES 512
 
 /* Guards adding to or removing from the list of callbacks */
@@ -77,7 +79,7 @@ int unwind_deferred_trace(struct unwind_stacktrace *trace)
 	/* Should always be called from faultable context */
 	might_fault();
 
-	if (current->flags & PF_EXITING)
+	if (!current->mm)
 		return -EINVAL;
 
 	if (!info->cache) {
@@ -107,14 +109,14 @@ int unwind_deferred_trace(struct unwind_stacktrace *trace)
 	return 0;
 }
 
-static void unwind_deferred_task_work(struct callback_head *head)
+static void process_unwind_deferred(void)
 {
-	struct unwind_task_info *info = container_of(head, struct unwind_task_info, work);
+	struct task_struct *task = current;
+	struct unwind_task_info *info = &task->unwind_info;
 	struct unwind_stacktrace trace;
 	struct unwind_work *work;
 	unsigned long bits;
 	u64 timestamp;
-	struct task_struct *task = current;
 	int idx;
 
 	if (WARN_ON_ONCE(!unwind_pending(task)))
@@ -152,6 +155,21 @@ static void unwind_deferred_task_work(struct callback_head *head)
 	srcu_read_unlock(&unwind_srcu, idx);
 }
 
+static void unwind_deferred_task_work(struct callback_head *head)
+{
+	process_unwind_deferred();
+}
+
+static void unwind_deferred_callback(void *data, struct task_struct *p, bool group_dead)
+{
+	if (!unwind_pending(p))
+		return;
+
+	process_unwind_deferred();
+
+	task_work_cancel(p, &p->unwind_info.work);
+}
+
 static int unwind_deferred_request_nmi(struct unwind_work *work, u64 *timestamp)
 {
 	struct unwind_task_info *info = &current->unwind_info;
@@ -329,6 +347,10 @@ void unwind_deferred_cancel(struct unwind_work *work)
 	for_each_process_thread(g, t) {
 		clear_bit(bit, &t->unwind_mask);
 	}
+
+	/* Is this the last registered unwinding? */
+	if (!unwind_mask)
+		unregister_trace_sched_process_exit(unwind_deferred_callback, NULL);
 }
 
 int unwind_deferred_init(struct unwind_work *work, unwind_callback_t func)
@@ -341,6 +363,15 @@ int unwind_deferred_init(struct unwind_work *work, unwind_callback_t func)
 	if (unwind_mask == ~(UNWIND_PENDING))
 		return -EBUSY;
 
+	/* Is this the first registered unwinding? */
+	if (!unwind_mask) {
+		int ret;
+
+		ret = register_trace_sched_process_exit(unwind_deferred_callback, NULL);
+		if (ret < 0)
+			return ret;
+	}
+
 	work->bit = ffz(unwind_mask);
 	unwind_mask |= 1UL << work->bit;