Skip to contentSkip to conversation

bpf, x86/unwind/orc: Support reliable unwinding through BPF stack frames

PATCH Superseded Josh Poimboeuf 3 participants 8 messages Dec 2, 2025

Subsystems (6)
x86 BPF BPF core Live patching +2 more subsystems

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Josh Poimboeuf

author maintainer
#0 lore raw

cover bpf, x86/unwind/orc: Support reliable unwinding through BPF stack frames

Fix livepatch stalls which may be seen when a task is blocked with BPF JIT on its kernel stack.

Josh Poimboeuf (2):
bpf: Add bpf_has_frame_pointer()
x86/unwind/orc: Support reliable unwinding through BPF stack frames

 arch/x86/kernel/unwind_orc.c | 39 +++++++++++++++++++++++++-----------
 arch/x86/net/bpf_jit_comp.c  | 10 +++++++++
 include/linux/bpf.h          |  3 +++
 kernel/bpf/core.c            | 16 +++++++++++++++
 4 files changed, 56 insertions(+), 12 deletions(-)
Signature · 1 line
2.51.1
author maintainer
#1 lore raw

PATCH 1/2 bpf: Add bpf_has_frame_pointer()

Introduce a bpf_has_frame_pointer() helper that uwninders can call to determine whether a given instruction pointer is within the valid frame pointer region of a BPF JIT program or trampoline (i.e., after the prologue, before the epilogue).

This will enable livepatch (with the ORC unwinder) to reliably unwind through BPF JIT frames.

Signed-off-by Josh Poimboeuf <[email protected]>
---
 arch/x86/net/bpf_jit_comp.c | 10 ++++++++++
 include/linux/bpf.h         |  3 +++
 kernel/bpf/core.c           | 16 ++++++++++++++++
 3 files changed, 29 insertions(+)
arch/x86/net/bpf_jit_comp.c +10 -0
index de5083cb1d37..510e3e62fd2f 100644
@@ -1661,6 +1661,9 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 	emit_prologue(&prog, image, stack_depth,
 		      bpf_prog_was_classic(bpf_prog), tail_call_reachable,
 		      bpf_is_subprog(bpf_prog), bpf_prog->aux->exception_cb);
+
+	bpf_prog->aux->ksym.fp_start = prog - temp;
+
 	/* Exception callback will clobber callee regs for its own use, and
 	 * restore the original callee regs from main prog's stack frame.
 	 */

@@ -2716,6 +2719,8 @@ st:			if (is_imm8(insn->off))
 					pop_r12(&prog);
 			}
 			EMIT1(0xC9);         /* leave */
+			bpf_prog->aux->ksym.fp_end = prog - temp;
+
 			emit_return(&prog, image + addrs[i - 1] + (prog - temp));
 			break;
 

@@ -3299,6 +3304,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	}
 	EMIT1(0x55);		 /* push rbp */
 	EMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */
+	im->ksym.fp_start = prog - (u8 *)rw_image;
+
 	if (!is_imm8(stack_size)) {
 		/* sub rsp, stack_size */
 		EMIT3_off32(0x48, 0x81, 0xEC, stack_size);

@@ -3436,7 +3443,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8);
 
 	emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off);
+
 	EMIT1(0xC9); /* leave */
+	im->ksym.fp_end = prog - (u8 *)rw_image;
+
 	if (flags & BPF_TRAMP_F_SKIP_FRAME) {
 		/* skip our return address and return to parent */
 		EMIT4(0x48, 0x83, 0xC4, 8); /* add rsp, 8 */
include/linux/bpf.h +3 -0
index d808253f2e94..e3f56e8443da 100644
@@ -1257,6 +1257,8 @@ struct bpf_ksym {
 	struct list_head	 lnode;
 	struct latch_tree_node	 tnode;
 	bool			 prog;
+	u32			 fp_start;
+	u32			 fp_end;
 };
 
 enum bpf_tramp_prog_type {

@@ -1483,6 +1485,7 @@ void bpf_image_ksym_add(struct bpf_ksym *ksym);
 void bpf_image_ksym_del(struct bpf_ksym *ksym);
 void bpf_ksym_add(struct bpf_ksym *ksym);
 void bpf_ksym_del(struct bpf_ksym *ksym);
+bool bpf_has_frame_pointer(unsigned long ip);
 int bpf_jit_charge_modmem(u32 size);
 void bpf_jit_uncharge_modmem(u32 size);
 bool bpf_prog_has_trampoline(const struct bpf_prog *prog);
kernel/bpf/core.c +16 -0
index d595fe512498..7cd8382d1152 100644
@@ -760,6 +760,22 @@ struct bpf_prog *bpf_prog_ksym_find(unsigned long addr)
 	       NULL;
 }
 
+bool bpf_has_frame_pointer(unsigned long ip)
+{
+	struct bpf_ksym *ksym;
+	unsigned long offset;
+
+	guard(rcu)();
+
+	ksym = bpf_ksym_find(ip);
+	if (!ksym || !ksym->fp_start || !ksym->fp_end)
+		return false;
+
+	offset = ip - ksym->start;
+
+	return offset >= ksym->fp_start && offset < ksym->fp_end;
+}
+
 const struct exception_table_entry *search_bpf_extables(unsigned long addr)
 {
 	const struct exception_table_entry *e = NULL;
Signature · 1 line
2.51.1
maintainer Acked-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] bpf: Add bpf_has_frame_pointer()

Quoted 10 lines · Josh Poimboeuf

Introduce a bpf_has_frame_pointer() helper that uwninders can call to determine whether a given instruction pointer is within the valid frame pointer region of a BPF JIT program or trampoline (i.e., after the prologue, before the epilogue).

This will enable livepatch (with the ORC unwinder) to reliably unwind through BPF JIT frames.

Signed-off-by Josh Poimboeuf <[email protected]>

LGTM!

Acked-by Song Liu <[email protected]>
author maintainer
#3 lore raw

Re: [PATCH 1/2] bpf: Add bpf_has_frame_pointer()

In reply to Josh Poimboeuf

arch/x86/net/bpf_jit_comp.c static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · line 3307

Quoted 5 lines · Josh Poimboeuf
arch/x86/net/bpf_jit_comp.c +1 -0
@@ -3299,6 +3304,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	}
 	EMIT1(0x55);		 /* push rbp */
 	EMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */
+	im->ksym.fp_start = prog - (u8 *)rw_image;

CI is reporting NULL pointer deref here, @im can be NULL. Will fix.

Signature · 1 line
Josh
author maintainer
#4 lore raw

PATCH 2/2 x86/unwind/orc: Support reliable unwinding through BPF stack frames

In reply to Josh Poimboeuf

BPF JIT programs and trampolines use a frame pointer, so the current ORC unwinder strategy of falling back to frame pointers (when an ORC entry is missing) usually works in practice when unwinding through BPF JIT stack frames.

However, that frame pointer fallback is just a guess, so the unwind gets marked unreliable for live patching, which can cause livepatch transition stalls.

Make the common case reliable by calling the bpf_has_frame_pointer() helper to detect the valid frame pointer region of BPF JIT programs and trampolines.

Fixes ee9f8fce9964 ("x86/unwind: Add the ORC unwinder")
Reported-by Andrey Grodzovsky <[email protected]>
Closes https://lore.kernel.org/[email protected] open here
Signed-off-by Josh Poimboeuf <[email protected]>
---
 arch/x86/kernel/unwind_orc.c | 39 +++++++++++++++++++++++++-----------
 1 file changed, 27 insertions(+), 12 deletions(-)
arch/x86/kernel/unwind_orc.c +27 -12
index 977ee75e047c..f610fde2d5c4 100644
@@ -2,6 +2,7 @@
 #include <linux/objtool.h>
 #include <linux/module.h>
 #include <linux/sort.h>
+#include <linux/bpf.h>
 #include <asm/ptrace.h>
 #include <asm/stacktrace.h>
 #include <asm/unwind.h>

@@ -172,6 +173,25 @@ static struct orc_entry *orc_ftrace_find(unsigned long ip)
 }
 #endif
 
+/* Fake frame pointer entry -- used as a fallback for generated code */
+static struct orc_entry orc_fp_entry = {
+	.type		= ORC_TYPE_CALL,
+	.sp_reg		= ORC_REG_BP,
+	.sp_offset	= 16,
+	.bp_reg		= ORC_REG_PREV_SP,
+	.bp_offset	= -16,
+};
+
+static struct orc_entry *orc_bpf_find(unsigned long ip)
+{
+#ifdef CONFIG_BPF_JIT
+	if (bpf_has_frame_pointer(ip))
+		return &orc_fp_entry;
+#endif
+
+	return NULL;
+}
+
 /*
  * If we crash with IP==0, the last successfully executed instruction
  * was probably an indirect function call with a NULL function pointer,

@@ -186,15 +206,6 @@ static struct orc_entry null_orc_entry = {
 	.type = ORC_TYPE_CALL
 };
 
-/* Fake frame pointer entry -- used as a fallback for generated code */
-static struct orc_entry orc_fp_entry = {
-	.type		= ORC_TYPE_CALL,
-	.sp_reg		= ORC_REG_BP,
-	.sp_offset	= 16,
-	.bp_reg		= ORC_REG_PREV_SP,
-	.bp_offset	= -16,
-};
-
 static struct orc_entry *orc_find(unsigned long ip)
 {
 	static struct orc_entry *orc;

@@ -238,6 +249,11 @@ static struct orc_entry *orc_find(unsigned long ip)
 	if (orc)
 		return orc;
 
+	/* BPF lookup: */
+	orc = orc_bpf_find(ip);
+	if (orc)
+		return orc;
+
 	return orc_ftrace_find(ip);
 }
 

@@ -495,9 +511,8 @@ bool unwind_next_frame(struct unwind_state *state)
 	if (!orc) {
 		/*
 		 * As a fallback, try to assume this code uses a frame pointer.
-		 * This is useful for generated code, like BPF, which ORC
-		 * doesn't know about.  This is just a guess, so the rest of
-		 * the unwind is no longer considered reliable.
+		 * This is just a guess, so the rest of the unwind is no longer
+		 * considered reliable.
 		 */
 		orc = &orc_fp_entry;
 		state->error = true;
Signature · 1 line
2.51.1
maintainer Acked-by on v1 2/2
#5 lore raw

Re: [PATCH 2/2] x86/unwind/orc: Support reliable unwinding through BPF stack frames

Quoted 10 lines · Josh Poimboeuf

BPF JIT programs and trampolines use a frame pointer, so the current ORC unwinder strategy of falling back to frame pointers (when an ORC entry is missing) usually works in practice when unwinding through BPF JIT stack frames.

However, that frame pointer fallback is just a guess, so the unwind gets marked unreliable for live patching, which can cause livepatch transition stalls.

Quoted 8 lines

Make the common case reliable by calling the bpf_has_frame_pointer() helper to detect the valid frame pointer region of BPF JIT programs and trampolines.

Fixes ee9f8fce9964 ("x86/unwind: Add the ORC unwinder")
Reported-by Andrey Grodzovsky <[email protected]>
Closes https://lore.kernel.org/[email protected] open here
Signed-off-by Josh Poimboeuf <[email protected]>
Acked-by Song Liu <[email protected]>
#6 lore raw
In reply to Josh Poimboeuf
Quoted 7 lines · Josh Poimboeuf

Fix livepatch stalls which may be seen when a task is blocked with BPF JIT on its kernel stack.

Josh Poimboeuf (2):
bpf: Add bpf_has_frame_pointer()
x86/unwind/orc: Support reliable unwinding through BPF stack frames

Quoted 6 lines
  arch/x86/kernel/unwind_orc.c | 39 +++++++++++++++++++++++++-----------
  arch/x86/net/bpf_jit_comp.c  | 10 +++++++++
  include/linux/bpf.h          |  3 +++
  kernel/bpf/core.c            | 16 +++++++++++++++
  4 files changed, 56 insertions(+), 12 deletions(-)
Acked-and-tested-by Andrey Grodzovsky<[email protected]>

Question - This looks to be x86 specific issue since ORC unwinding is x86 specific and as such this has no impact on ARM, correct ?

Andrey

author maintainer
#7 lore raw
Quoted 16 lines · Andrey Grodzovsky
Quoted 7 lines · Josh Poimboeuf

Fix livepatch stalls which may be seen when a task is blocked with BPF JIT on its kernel stack.

Josh Poimboeuf (2):
bpf: Add bpf_has_frame_pointer()
x86/unwind/orc: Support reliable unwinding through BPF stack frames

Quoted 6 lines
  arch/x86/kernel/unwind_orc.c | 39 +++++++++++++++++++++++++-----------
  arch/x86/net/bpf_jit_comp.c  | 10 +++++++++
  include/linux/bpf.h          |  3 +++
  kernel/bpf/core.c            | 16 +++++++++++++++
  4 files changed, 56 insertions(+), 12 deletions(-)
Quoted 4 lines
Acked-and-tested-by Andrey Grodzovsky<[email protected]>

Question - This looks to be x86 specific issue since ORC unwinding is x86 specific and as such this has no impact on ARM, correct ?

Correct, though if ARM ever switches to sframe for in-kernel unwinding, it might also want to use this interface.

Signature · 1 line
Josh