Skip to contentSkip to conversation

bpf: guard uprobes against private-stack corruption

PATCH In review Christian Simon 4 participants 8 messages Aug 18 – Aug 25

Subsystems (8)
Stable branch Kselftest BPF Tracing +4 more subsystems
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 bpf: guard uprobes against private-stack corruption

Eligible BPF programs use one private stack per program and CPU. Both bpf_prog_run_array_uprobe() and uprobe_prog_run() use migrate_disable() to keep an invocation on one CPU, but another task can still preempt it and run the same program on that CPU. The second invocation then reuses and can overwrite the first invocation's private stack.

Protect each real program invocation with the existing per-program recursion context. When the program is already active on this CPU, account for the missed invocation and skip it. Return zero when skipping an uprobe-multi invocation so session handling does not suppress its return probe.

Skip dummy_bpf_prog in the classic array path before acquiring the recursion context because its active pointer is NULL.

Add a regression test that pins two threads to one CPU and overlaps classic and multi uprobe invocations while preserving a sentinel in a private stack frame. Without the guards, the second invocation executes and corrupts the first invocation's sentinel.

Fixes 7d1cd70d4b16 ("bpf, x86: Support private stack in jit")
Fixes 6c17a882d380 ("bpf, arm64: JIT support for private stack")

---
Changes in v2:
- Address review comments from sashiko-bot
- Guard the uprobe-multi path as well.
- Remove const from correct line
- Add a regression selftest for uprobe-classic/multi paths.
- Add the arm64 Fixes tag.

 include/linux/bpf.h                           |  33 +++--
 kernel/trace/bpf_trace.c                      |  11 +-
 .../bpf/prog_tests/uprobe_private_stack.c     | 118 ++++++++++++++++++
 .../bpf/progs/uprobe_private_stack.c          |  54 ++++++++
 4 files changed, 206 insertions(+), 10 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/uprobe_private_stack.c
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_private_stack.c
include/linux/bpf.h +25 -8
index 7719f6528445..a94fc9898ece 100644
@@ -2572,6 +2572,14 @@ static inline void bpf_reset_run_ctx(struct bpf_run_ctx *old_ctx)
 
 typedef u32 (*bpf_prog_run_fn)(const struct bpf_prog *prog, const void *ctx);
 
+#ifdef CONFIG_BPF_SYSCALL
+void notrace bpf_prog_inc_misses_counter(struct bpf_prog *prog);
+#else
+static inline void bpf_prog_inc_misses_counter(struct bpf_prog *prog)
+{
+}
+#endif
+
 static __always_inline u32
 bpf_prog_run_array(const struct bpf_prog_array *array,
 		   const void *ctx, bpf_prog_run_fn run_prog)
@@ -2617,7 +2625,7 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
 			  const void *ctx, bpf_prog_run_fn run_prog)
 {
 	const struct bpf_prog_array_item *item;
-	const struct bpf_prog *prog;
+	struct bpf_prog *prog;
 	struct bpf_run_ctx *old_run_ctx;
 	struct bpf_trace_run_ctx run_ctx;
 	u32 ret = 1;
@@ -2635,15 +2643,30 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
 	old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
 	item = &array->items[0];
 	while ((prog = READ_ONCE(item->prog))) {
+		/* dummy_bpf_prog has no recursion state. */
+		if (unlikely(!prog->len)) {
+			item++;
+			continue;
+		}
+
+		if (unlikely(!bpf_prog_get_recursion_context(prog))) {
+			bpf_prog_inc_misses_counter(prog);
+			bpf_prog_put_recursion_context(prog);
+			item++;
+			continue;
+		}
+
 		if (!prog->sleepable)
 			rcu_read_lock();
 
 		run_ctx.bpf_cookie = item->bpf_cookie;
 		ret &= run_prog(prog, ctx);
-		item++;
 
 		if (!prog->sleepable)
 			rcu_read_unlock();
+
+		bpf_prog_put_recursion_context(prog);
+		item++;
 	}
 	bpf_reset_run_ctx(old_run_ctx);
 	migrate_enable();
@@ -3208,8 +3231,6 @@ static inline bool has_current_bpf_ctx(void)
 	return !!current->bpf_ctx;
 }
 
-void notrace bpf_prog_inc_misses_counter(struct bpf_prog *prog);
-
 void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
 		     enum bpf_dynptr_type type, u32 offset, u32 size);
 void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr);
@@ -3538,10 +3559,6 @@ static inline bool has_current_bpf_ctx(void)
 	return false;
 }
 
-static inline void bpf_prog_inc_misses_counter(struct bpf_prog *prog)
-{
-}
-
 static inline void bpf_cgrp_storage_free(struct cgroup *cgroup)
 {
 }
kernel/trace/bpf_trace.c +9 -2
index 75495a5c3507..379d472590ec 100644
@@ -3139,7 +3139,7 @@ static int uprobe_prog_run(struct bpf_uprobe *uprobe,
 	struct bpf_prog *prog = link->link.prog;
 	bool sleepable = prog->sleepable;
 	struct bpf_run_ctx *old_run_ctx;
-	int err;
+	int err = 0;
 
 	if (link->task && !same_thread_group(current, link->task))
 		return 0;
@@ -3151,10 +3151,17 @@ static int uprobe_prog_run(struct bpf_uprobe *uprobe,
 
 	migrate_disable();
 
+	if (unlikely(!bpf_prog_get_recursion_context(prog))) {
+		bpf_prog_inc_misses_counter(prog);
+		goto out;
+	}
+
 	old_run_ctx = bpf_set_run_ctx(&run_ctx.session_ctx.run_ctx);
-	err = bpf_prog_run(link->link.prog, regs);
+	err = bpf_prog_run(prog, regs);
 	bpf_reset_run_ctx(old_run_ctx);
 
+out:
+	bpf_prog_put_recursion_context(prog);
 	migrate_enable();
 
 	if (sleepable)
tools/testing/selftests/bpf/prog_tests/uprobe_private_stack.c new file +118 -0

118 lines not on this page · raw message

tools/testing/selftests/bpf/progs/uprobe_private_stack.c new file +54 -0

54 lines not on this page · raw message

Signature · 1 line
2.54.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Incomplete fix for private stack corruption: BPF programs attached to both uprobes and kprobes/tracepoints bypass recursion protection. --

--- Patch [1]: [PATCH bpf v2] bpf: guard uprobes against private-stack corruption --- commit 91e05311e6d3a25ea1742394d64f625138f2c862
Author: Christian Simon <[email protected]>

bpf: guard uprobes against private-stack corruption

Eligible BPF programs use one private stack per program and CPU. This commit adds per-program recursion protection to the uprobe execution paths to prevent a second invocation from overwriting the first invocation's private stack.

Quoted 4 lines
include/linux/bpf.h +0 -0
index 7719f65284456..a94fc9898ecef 100644

[ ... ]

include/linux/bpf.h bpf_prog_run_array_uprobe(const struct bpf_prog_array *array, · line 2652

Quoted 11 lines
include/linux/bpf.h +7 -0
@@ -2635,15 +2643,30 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
 	old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
 	item = &array->items[0];
 	while ((prog = READ_ONCE(item->prog))) {
+		/* dummy_bpf_prog has no recursion state. */
+		if (unlikely(!prog->len)) {
+			item++;
+			continue;
+		}
+
+		if (unlikely(!bpf_prog_get_recursion_context(prog))) {

[Severity: High]
This is a pre-existing issue, but does this leave the private stack vulnerable to corruption if a program is attached to both an uprobe and a kprobe?

The uprobe path increments the per-program recursion counter here via bpf_prog_get_recursion_context(), but it does not increment the global bpf_prog_active counter.

If an interrupt triggers a kprobe while the uprobe is executing on the same CPU, the kprobe dispatcher only checks the global counter:

kernel/trace/bpf_trace.c:trace_call_bpf() {
    if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1)) {
        ...
        ret = bpf_prog_run_array(..., bpf_prog_run);
    ...
}

Because the uprobe didn't increment the global bpf_prog_active counter, trace_call_bpf() will proceed and execute the same BPF program again, bypassing the per-program recursion protection and potentially overwriting the first invocation's stack frames.

Does kprobe_multi_link_prog_run() also have this same vulnerability since it relies solely on bpf_prog_active before executing bpf_prog_run()?

Quoted 21 lines
include/linux/bpf.h +9 -1
+			bpf_prog_inc_misses_counter(prog);
+			bpf_prog_put_recursion_context(prog);
+			item++;
+			continue;
+		}
+
 		if (!prog->sleepable)
 			rcu_read_lock();
 
 		run_ctx.bpf_cookie = item->bpf_cookie;
 		ret &= run_prog(prog, ctx);
-		item++;
 
 		if (!prog->sleepable)
 			rcu_read_unlock();
+
+		bpf_prog_put_recursion_context(prog);
+		item++;
 	}
 	bpf_reset_run_ctx(old_run_ctx);
 	migrate_enable();
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Christian Simon
Quoted 33 lines · Christian Simon

Eligible BPF programs use one private stack per program and CPU. Both bpf_prog_run_array_uprobe() and uprobe_prog_run() use migrate_disable() to keep an invocation on one CPU, but another task can still preempt it and run the same program on that CPU. The second invocation then reuses and can overwrite the first invocation's private stack.

Protect each real program invocation with the existing per-program recursion context. When the program is already active on this CPU, account for the missed invocation and skip it. Return zero when skipping an uprobe-multi invocation so session handling does not suppress its return probe.

Skip dummy_bpf_prog in the classic array path before acquiring the recursion context because its active pointer is NULL.

Add a regression test that pins two threads to one CPU and overlaps classic and multi uprobe invocations while preserving a sentinel in a private stack frame. Without the guards, the second invocation executes and corrupts the first invocation's sentinel.

Fixes 7d1cd70d4b16 ("bpf, x86: Support private stack in jit")
Fixes 6c17a882d380 ("bpf, arm64: JIT support for private stack")

---
Changes in v2:
- Address review comments from sashiko-bot
- Guard the uprobe-multi path as well.
- Remove const from correct line
- Add a regression selftest for uprobe-classic/multi paths.
- Add the arm64 Fixes tag.

Quoted 7 lines
 include/linux/bpf.h                           |  33 +++--
 kernel/trace/bpf_trace.c                      |  11 +-
 .../bpf/prog_tests/uprobe_private_stack.c     | 118 ++++++++++++++++++
 .../bpf/progs/uprobe_private_stack.c          |  54 ++++++++
 4 files changed, 206 insertions(+), 10 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/uprobe_private_stack.c
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_private_stack.c

please split the change into 3 patches - uprobe, uprobe_multi, selftest

Quoted 34 lines
include/linux/bpf.h +10 -1
index 7719f6528445..a94fc9898ece 100644
@@ -2572,6 +2572,14 @@ static inline void bpf_reset_run_ctx(struct bpf_run_ctx *old_ctx)
 
 typedef u32 (*bpf_prog_run_fn)(const struct bpf_prog *prog, const void *ctx);
 
+#ifdef CONFIG_BPF_SYSCALL
+void notrace bpf_prog_inc_misses_counter(struct bpf_prog *prog);
+#else
+static inline void bpf_prog_inc_misses_counter(struct bpf_prog *prog)
+{
+}
+#endif
+
 static __always_inline u32
 bpf_prog_run_array(const struct bpf_prog_array *array,
 		   const void *ctx, bpf_prog_run_fn run_prog)
@@ -2617,7 +2625,7 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
 			  const void *ctx, bpf_prog_run_fn run_prog)
 {
 	const struct bpf_prog_array_item *item;
-	const struct bpf_prog *prog;
+	struct bpf_prog *prog;
 	struct bpf_run_ctx *old_run_ctx;
 	struct bpf_trace_run_ctx run_ctx;
 	u32 ret = 1;
@@ -2635,15 +2643,30 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
 	old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
 	item = &array->items[0];
 	while ((prog = READ_ONCE(item->prog))) {
+		/* dummy_bpf_prog has no recursion state. */

include/linux/bpf.h bpf_prog_run_array_uprobe(const struct bpf_prog_array *array, · line 2656

Quoted 10 lines
include/linux/bpf.h +10 -0
+		if (unlikely(!prog->len)) {
+			item++;
+			continue;
+		}
+
+		if (unlikely(!bpf_prog_get_recursion_context(prog))) {
+			bpf_prog_inc_misses_counter(prog);
+			bpf_prog_put_recursion_context(prog);
+			item++;
+			continue;

so uprobe program has this problem only when using private stack, right? but we exclude all the programs, I think we should check program has private stack as well

jirka

Quoted 14 lines
include/linux/bpf.h +5 -1
+		}
+
 		if (!prog->sleepable)
 			rcu_read_lock();
 
 		run_ctx.bpf_cookie = item->bpf_cookie;
 		ret &= run_prog(prog, ctx);
-		item++;
 
 		if (!prog->sleepable)
 			rcu_read_unlock();
+
+		bpf_prog_put_recursion_context(prog);
+		item++;

SNIP

#3 lore raw
In reply to Christian Simon
Quoted 77 lines · Christian Simon

Eligible BPF programs use one private stack per program and CPU. Both bpf_prog_run_array_uprobe() and uprobe_prog_run() use migrate_disable() to keep an invocation on one CPU, but another task can still preempt it and run the same program on that CPU. The second invocation then reuses and can overwrite the first invocation's private stack.

Protect each real program invocation with the existing per-program recursion context. When the program is already active on this CPU, account for the missed invocation and skip it. Return zero when skipping an uprobe-multi invocation so session handling does not suppress its return probe.

Skip dummy_bpf_prog in the classic array path before acquiring the recursion context because its active pointer is NULL.

Add a regression test that pins two threads to one CPU and overlaps classic and multi uprobe invocations while preserving a sentinel in a private stack frame. Without the guards, the second invocation executes and corrupts the first invocation's sentinel.

Fixes 7d1cd70d4b16 ("bpf, x86: Support private stack in jit")
Fixes 6c17a882d380 ("bpf, arm64: JIT support for private stack")

---
Changes in v2:
- Address review comments from sashiko-bot
- Guard the uprobe-multi path as well.
- Remove const from correct line
- Add a regression selftest for uprobe-classic/multi paths.
- Add the arm64 Fixes tag.

 include/linux/bpf.h                           |  33 +++--
 kernel/trace/bpf_trace.c                      |  11 +-
 .../bpf/prog_tests/uprobe_private_stack.c     | 118 ++++++++++++++++++
 .../bpf/progs/uprobe_private_stack.c          |  54 ++++++++
 4 files changed, 206 insertions(+), 10 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/uprobe_private_stack.c
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_private_stack.c
include/linux/bpf.h +12 -1
index 7719f6528445..a94fc9898ece 100644
@@ -2572,6 +2572,14 @@ static inline void bpf_reset_run_ctx(struct bpf_run_ctx *old_ctx)
 
 typedef u32 (*bpf_prog_run_fn)(const struct bpf_prog *prog, const void *ctx);
 
+#ifdef CONFIG_BPF_SYSCALL
+void notrace bpf_prog_inc_misses_counter(struct bpf_prog *prog);
+#else
+static inline void bpf_prog_inc_misses_counter(struct bpf_prog *prog)
+{
+}
+#endif
+
 static __always_inline u32
 bpf_prog_run_array(const struct bpf_prog_array *array,
                   const void *ctx, bpf_prog_run_fn run_prog)
@@ -2617,7 +2625,7 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
                          const void *ctx, bpf_prog_run_fn run_prog)
 {
        const struct bpf_prog_array_item *item;
-       const struct bpf_prog *prog;
+       struct bpf_prog *prog;
        struct bpf_run_ctx *old_run_ctx;
        struct bpf_trace_run_ctx run_ctx;
        u32 ret = 1;
@@ -2635,15 +2643,30 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
        old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
        item = &array->items[0];
        while ((prog = READ_ONCE(item->prog))) {
+               /* dummy_bpf_prog has no recursion state. */
+               if (unlikely(!prog->len)) {
+                       item++;

include/linux/bpf.h bpf_prog_run_array_uprobe(const struct bpf_prog_array *array, · line 2658

Quoted 10 lines
include/linux/bpf.h +10 -0
+                       continue;
+               }
+
+               if (unlikely(!bpf_prog_get_recursion_context(prog))) {
+                       bpf_prog_inc_misses_counter(prog);
+                       bpf_prog_put_recursion_context(prog);
+                       item++;
+                       continue;
+               }
+

I think it's unacceptable to skip sleepable uprobe execution just because there is the same BPF program attached to a *different* uprobe (and all due to a private stack that no one asked for or needs for uprobes, really).

As a short-term fix, we should probably disable private stack for sleepable uprobe/kprobe program (and tracepoint/raw_tracepoint), and think how we can make private stack less per-CPU dependent.

pw-bot: cr

Quoted 24 lines
include/linux/bpf.h +3 -3
                if (!prog->sleepable)
                        rcu_read_lock();
 
                run_ctx.bpf_cookie = item->bpf_cookie;
                ret &= run_prog(prog, ctx);
-               item++;
 
                if (!prog->sleepable)
                        rcu_read_unlock();
+
+               bpf_prog_put_recursion_context(prog);
+               item++;
        }
        bpf_reset_run_ctx(old_run_ctx);
        migrate_enable();
@@ -3208,8 +3231,6 @@ static inline bool has_current_bpf_ctx(void)
        return !!current->bpf_ctx;
 }
 
-void notrace bpf_prog_inc_misses_counter(struct bpf_prog *prog);
-
 void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
                     enum bpf_dynptr_type type, u32 offset, u32 size);
 void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr);

[...]

#4 lore raw
Quoted 94 lines · Andrii Nakryiko
Quoted 77 lines · Christian Simon

Eligible BPF programs use one private stack per program and CPU. Both bpf_prog_run_array_uprobe() and uprobe_prog_run() use migrate_disable() to keep an invocation on one CPU, but another task can still preempt it and run the same program on that CPU. The second invocation then reuses and can overwrite the first invocation's private stack.

Protect each real program invocation with the existing per-program recursion context. When the program is already active on this CPU, account for the missed invocation and skip it. Return zero when skipping an uprobe-multi invocation so session handling does not suppress its return probe.

Skip dummy_bpf_prog in the classic array path before acquiring the recursion context because its active pointer is NULL.

Add a regression test that pins two threads to one CPU and overlaps classic and multi uprobe invocations while preserving a sentinel in a private stack frame. Without the guards, the second invocation executes and corrupts the first invocation's sentinel.

Fixes 7d1cd70d4b16 ("bpf, x86: Support private stack in jit")
Fixes 6c17a882d380 ("bpf, arm64: JIT support for private stack")

---
Changes in v2:
- Address review comments from sashiko-bot
- Guard the uprobe-multi path as well.
- Remove const from correct line
- Add a regression selftest for uprobe-classic/multi paths.
- Add the arm64 Fixes tag.

 include/linux/bpf.h                           |  33 +++--
 kernel/trace/bpf_trace.c                      |  11 +-
 .../bpf/prog_tests/uprobe_private_stack.c     | 118 ++++++++++++++++++
 .../bpf/progs/uprobe_private_stack.c          |  54 ++++++++
 4 files changed, 206 insertions(+), 10 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/uprobe_private_stack.c
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_private_stack.c
include/linux/bpf.h +12 -1
index 7719f6528445..a94fc9898ece 100644
@@ -2572,6 +2572,14 @@ static inline void bpf_reset_run_ctx(struct bpf_run_ctx *old_ctx)
 
 typedef u32 (*bpf_prog_run_fn)(const struct bpf_prog *prog, const void *ctx);
 
+#ifdef CONFIG_BPF_SYSCALL
+void notrace bpf_prog_inc_misses_counter(struct bpf_prog *prog);
+#else
+static inline void bpf_prog_inc_misses_counter(struct bpf_prog *prog)
+{
+}
+#endif
+
 static __always_inline u32
 bpf_prog_run_array(const struct bpf_prog_array *array,
                   const void *ctx, bpf_prog_run_fn run_prog)
@@ -2617,7 +2625,7 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
                          const void *ctx, bpf_prog_run_fn run_prog)
 {
        const struct bpf_prog_array_item *item;
-       const struct bpf_prog *prog;
+       struct bpf_prog *prog;
        struct bpf_run_ctx *old_run_ctx;
        struct bpf_trace_run_ctx run_ctx;
        u32 ret = 1;
@@ -2635,15 +2643,30 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
        old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
        item = &array->items[0];
        while ((prog = READ_ONCE(item->prog))) {
+               /* dummy_bpf_prog has no recursion state. */
+               if (unlikely(!prog->len)) {
+                       item++;

include/linux/bpf.h bpf_prog_run_array_uprobe(const struct bpf_prog_array *array, · line 2658

Quoted 10 lines
include/linux/bpf.h +10 -0
+                       continue;
+               }
+
+               if (unlikely(!bpf_prog_get_recursion_context(prog))) {
+                       bpf_prog_inc_misses_counter(prog);
+                       bpf_prog_put_recursion_context(prog);
+                       item++;
+                       continue;
+               }
+

I think it's unacceptable to skip sleepable uprobe execution just because there is the same BPF program attached to a *different* uprobe (and all due to a private stack that no one asked for or needs for uprobes, really).

Quoted 3 lines

As a short-term fix, we should probably disable private stack for sleepable uprobe/kprobe program (and tracepoint/raw_tracepoint), and think how we can make private stack less per-CPU dependent.

+1 for disabling private stack for uprobes

thanks,
jirka

#5 lore raw
Quoted 96 lines · Jiri Olsa
Quoted 94 lines · Andrii Nakryiko
Quoted 77 lines · Christian Simon

Eligible BPF programs use one private stack per program and CPU. Both bpf_prog_run_array_uprobe() and uprobe_prog_run() use migrate_disable() to keep an invocation on one CPU, but another task can still preempt it and run the same program on that CPU. The second invocation then reuses and can overwrite the first invocation's private stack.

Protect each real program invocation with the existing per-program recursion context. When the program is already active on this CPU, account for the missed invocation and skip it. Return zero when skipping an uprobe-multi invocation so session handling does not suppress its return probe.

Skip dummy_bpf_prog in the classic array path before acquiring the recursion context because its active pointer is NULL.

Add a regression test that pins two threads to one CPU and overlaps classic and multi uprobe invocations while preserving a sentinel in a private stack frame. Without the guards, the second invocation executes and corrupts the first invocation's sentinel.

Fixes 7d1cd70d4b16 ("bpf, x86: Support private stack in jit")
Fixes 6c17a882d380 ("bpf, arm64: JIT support for private stack")

---
Changes in v2:
- Address review comments from sashiko-bot
- Guard the uprobe-multi path as well.
- Remove const from correct line
- Add a regression selftest for uprobe-classic/multi paths.
- Add the arm64 Fixes tag.

 include/linux/bpf.h                           |  33 +++--
 kernel/trace/bpf_trace.c                      |  11 +-
 .../bpf/prog_tests/uprobe_private_stack.c     | 118 ++++++++++++++++++
 .../bpf/progs/uprobe_private_stack.c          |  54 ++++++++
 4 files changed, 206 insertions(+), 10 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/uprobe_private_stack.c
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_private_stack.c
include/linux/bpf.h +12 -1
index 7719f6528445..a94fc9898ece 100644
@@ -2572,6 +2572,14 @@ static inline void bpf_reset_run_ctx(struct bpf_run_ctx *old_ctx)
 
 typedef u32 (*bpf_prog_run_fn)(const struct bpf_prog *prog, const void *ctx);
 
+#ifdef CONFIG_BPF_SYSCALL
+void notrace bpf_prog_inc_misses_counter(struct bpf_prog *prog);
+#else
+static inline void bpf_prog_inc_misses_counter(struct bpf_prog *prog)
+{
+}
+#endif
+
 static __always_inline u32
 bpf_prog_run_array(const struct bpf_prog_array *array,
                   const void *ctx, bpf_prog_run_fn run_prog)
@@ -2617,7 +2625,7 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
                          const void *ctx, bpf_prog_run_fn run_prog)
 {
        const struct bpf_prog_array_item *item;
-       const struct bpf_prog *prog;
+       struct bpf_prog *prog;
        struct bpf_run_ctx *old_run_ctx;
        struct bpf_trace_run_ctx run_ctx;
        u32 ret = 1;
@@ -2635,15 +2643,30 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
        old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
        item = &array->items[0];
        while ((prog = READ_ONCE(item->prog))) {
+               /* dummy_bpf_prog has no recursion state. */
+               if (unlikely(!prog->len)) {
+                       item++;

include/linux/bpf.h bpf_prog_run_array_uprobe(const struct bpf_prog_array *array, · line 2658

Quoted 10 lines
include/linux/bpf.h +10 -0
+                       continue;
+               }
+
+               if (unlikely(!bpf_prog_get_recursion_context(prog))) {
+                       bpf_prog_inc_misses_counter(prog);
+                       bpf_prog_put_recursion_context(prog);
+                       item++;
+                       continue;
+               }
+

I think it's unacceptable to skip sleepable uprobe execution just because there is the same BPF program attached to a *different* uprobe (and all due to a private stack that no one asked for or needs for uprobes, really).

Quoted 8 lines
Quoted 3 lines

As a short-term fix, we should probably disable private stack for sleepable uprobe/kprobe program (and tracepoint/raw_tracepoint), and think how we can make private stack less per-CPU dependent.

+1 for disabling private stack for uprobes

thanks,
jirka

Thanks for your guidance. I have just posted v3 of the patch set, which disables private stacks for sleepable programs.

I am still unsure why non-sleepable uprobes would be unaffected. On a preemptible kernel, their execution paths use migrate_disable() rather than preempt_disable(), and rcu_read_lock() does not prevent preemption with preemptible RCU. It therefore seems possible for a task to be scheduled out while running a non-sleepable uprobe, after which another task could invoke the same program on the same CPU and reuse its per-CPU, per-program private stack. Am I missing another mechanism that prevents this?

Cheers,
Christian

---
v3: https://lore.kernel.org/bpf/[email protected]/

#6 lore raw
Quoted 103 lines · Christian Simon
Quoted 102 lines · Jiri Olsa
Quoted 94 lines · Andrii Nakryiko
Quoted 77 lines · Christian Simon

Eligible BPF programs use one private stack per program and CPU. Both bpf_prog_run_array_uprobe() and uprobe_prog_run() use migrate_disable() to keep an invocation on one CPU, but another task can still preempt it and run the same program on that CPU. The second invocation then reuses and can overwrite the first invocation's private stack.

Protect each real program invocation with the existing per-program recursion context. When the program is already active on this CPU, account for the missed invocation and skip it. Return zero when skipping an uprobe-multi invocation so session handling does not suppress its return probe.

Skip dummy_bpf_prog in the classic array path before acquiring the recursion context because its active pointer is NULL.

Add a regression test that pins two threads to one CPU and overlaps classic and multi uprobe invocations while preserving a sentinel in a private stack frame. Without the guards, the second invocation executes and corrupts the first invocation's sentinel.

Fixes 7d1cd70d4b16 ("bpf, x86: Support private stack in jit")
Fixes 6c17a882d380 ("bpf, arm64: JIT support for private stack")

---
Changes in v2:
- Address review comments from sashiko-bot
- Guard the uprobe-multi path as well.
- Remove const from correct line
- Add a regression selftest for uprobe-classic/multi paths.
- Add the arm64 Fixes tag.

 include/linux/bpf.h                           |  33 +++--
 kernel/trace/bpf_trace.c                      |  11 +-
 .../bpf/prog_tests/uprobe_private_stack.c     | 118 ++++++++++++++++++
 .../bpf/progs/uprobe_private_stack.c          |  54 ++++++++
 4 files changed, 206 insertions(+), 10 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/uprobe_private_stack.c
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_private_stack.c
include/linux/bpf.h +12 -1
index 7719f6528445..a94fc9898ece 100644
@@ -2572,6 +2572,14 @@ static inline void bpf_reset_run_ctx(struct bpf_run_ctx *old_ctx)
 
 typedef u32 (*bpf_prog_run_fn)(const struct bpf_prog *prog, const void *ctx);
 
+#ifdef CONFIG_BPF_SYSCALL
+void notrace bpf_prog_inc_misses_counter(struct bpf_prog *prog);
+#else
+static inline void bpf_prog_inc_misses_counter(struct bpf_prog *prog)
+{
+}
+#endif
+
 static __always_inline u32
 bpf_prog_run_array(const struct bpf_prog_array *array,
                   const void *ctx, bpf_prog_run_fn run_prog)
@@ -2617,7 +2625,7 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
                          const void *ctx, bpf_prog_run_fn run_prog)
 {
        const struct bpf_prog_array_item *item;
-       const struct bpf_prog *prog;
+       struct bpf_prog *prog;
        struct bpf_run_ctx *old_run_ctx;
        struct bpf_trace_run_ctx run_ctx;
        u32 ret = 1;
@@ -2635,15 +2643,30 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
        old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
        item = &array->items[0];
        while ((prog = READ_ONCE(item->prog))) {
+               /* dummy_bpf_prog has no recursion state. */
+               if (unlikely(!prog->len)) {
+                       item++;

include/linux/bpf.h bpf_prog_run_array_uprobe(const struct bpf_prog_array *array, · line 2658

Quoted 10 lines
include/linux/bpf.h +10 -0
+                       continue;
+               }
+
+               if (unlikely(!bpf_prog_get_recursion_context(prog))) {
+                       bpf_prog_inc_misses_counter(prog);
+                       bpf_prog_put_recursion_context(prog);
+                       item++;
+                       continue;
+               }
+

I think it's unacceptable to skip sleepable uprobe execution just because there is the same BPF program attached to a *different* uprobe (and all due to a private stack that no one asked for or needs for uprobes, really).

Quoted 3 lines

As a short-term fix, we should probably disable private stack for sleepable uprobe/kprobe program (and tracepoint/raw_tracepoint), and think how we can make private stack less per-CPU dependent.

+1 for disabling private stack for uprobes

Quoted 7 lines
Quoted 2 lines

thanks,
jirka

Thanks for your guidance. I have just posted v3 of the patch set, which disables private stacks for sleepable programs.

I am still unsure why non-sleepable uprobes would be unaffected. On a

I think we need to disable both sleepable and non-sleepable uprobes

problem is that we can't say if kprobe program is going to be attached as uprobe or kprobe, so we'd need to disable both kprobe/uprobes, which I'm not sure is a problem

I don't see how tracepoint/raw_tracepoint are affected by this problem, because they have either retursion context or bpf_prog_active check

jirka

Quoted 12 lines

preemptible kernel, their execution paths use migrate_disable() rather than preempt_disable(), and rcu_read_lock() does not prevent preemption with preemptible RCU. It therefore seems possible for a task to be scheduled out while running a non-sleepable uprobe, after which another task could invoke the same program on the same CPU and reuse its per-CPU, per-program private stack. Am I missing another mechanism that prevents this?

Cheers,
Christian

---
v3: https://lore.kernel.org/bpf/[email protected]/

#7 lore raw
Quoted 111 lines · Jiri Olsa
Quoted 109 lines · Christian Simon
Quoted 96 lines · Jiri Olsa
Quoted 94 lines · Andrii Nakryiko
Quoted 77 lines · Christian Simon

Eligible BPF programs use one private stack per program and CPU. Both bpf_prog_run_array_uprobe() and uprobe_prog_run() use migrate_disable() to keep an invocation on one CPU, but another task can still preempt it and run the same program on that CPU. The second invocation then reuses and can overwrite the first invocation's private stack.

Protect each real program invocation with the existing per-program recursion context. When the program is already active on this CPU, account for the missed invocation and skip it. Return zero when skipping an uprobe-multi invocation so session handling does not suppress its return probe.

Skip dummy_bpf_prog in the classic array path before acquiring the recursion context because its active pointer is NULL.

Add a regression test that pins two threads to one CPU and overlaps classic and multi uprobe invocations while preserving a sentinel in a private stack frame. Without the guards, the second invocation executes and corrupts the first invocation's sentinel.

Fixes 7d1cd70d4b16 ("bpf, x86: Support private stack in jit")
Fixes 6c17a882d380 ("bpf, arm64: JIT support for private stack")

---
Changes in v2:
- Address review comments from sashiko-bot
- Guard the uprobe-multi path as well.
- Remove const from correct line
- Add a regression selftest for uprobe-classic/multi paths.
- Add the arm64 Fixes tag.

 include/linux/bpf.h                           |  33 +++--
 kernel/trace/bpf_trace.c                      |  11 +-
 .../bpf/prog_tests/uprobe_private_stack.c     | 118 ++++++++++++++++++
 .../bpf/progs/uprobe_private_stack.c          |  54 ++++++++
 4 files changed, 206 insertions(+), 10 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/uprobe_private_stack.c
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_private_stack.c
include/linux/bpf.h +12 -1
index 7719f6528445..a94fc9898ece 100644
@@ -2572,6 +2572,14 @@ static inline void bpf_reset_run_ctx(struct bpf_run_ctx *old_ctx)
 
 typedef u32 (*bpf_prog_run_fn)(const struct bpf_prog *prog, const void *ctx);
 
+#ifdef CONFIG_BPF_SYSCALL
+void notrace bpf_prog_inc_misses_counter(struct bpf_prog *prog);
+#else
+static inline void bpf_prog_inc_misses_counter(struct bpf_prog *prog)
+{
+}
+#endif
+
 static __always_inline u32
 bpf_prog_run_array(const struct bpf_prog_array *array,
                   const void *ctx, bpf_prog_run_fn run_prog)
@@ -2617,7 +2625,7 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
                          const void *ctx, bpf_prog_run_fn run_prog)
 {
        const struct bpf_prog_array_item *item;
-       const struct bpf_prog *prog;
+       struct bpf_prog *prog;
        struct bpf_run_ctx *old_run_ctx;
        struct bpf_trace_run_ctx run_ctx;
        u32 ret = 1;
@@ -2635,15 +2643,30 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
        old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
        item = &array->items[0];
        while ((prog = READ_ONCE(item->prog))) {
+               /* dummy_bpf_prog has no recursion state. */
+               if (unlikely(!prog->len)) {
+                       item++;

include/linux/bpf.h bpf_prog_run_array_uprobe(const struct bpf_prog_array *array, · line 2658

Quoted 10 lines
include/linux/bpf.h +10 -0
+                       continue;
+               }
+
+               if (unlikely(!bpf_prog_get_recursion_context(prog))) {
+                       bpf_prog_inc_misses_counter(prog);
+                       bpf_prog_put_recursion_context(prog);
+                       item++;
+                       continue;
+               }
+

I think it's unacceptable to skip sleepable uprobe execution just because there is the same BPF program attached to a *different* uprobe (and all due to a private stack that no one asked for or needs for uprobes, really).

Quoted 8 lines
Quoted 3 lines

As a short-term fix, we should probably disable private stack for sleepable uprobe/kprobe program (and tracepoint/raw_tracepoint), and think how we can make private stack less per-CPU dependent.

+1 for disabling private stack for uprobes

thanks,
jirka

Thanks for your guidance. I have just posted v3 of the patch set, which disables private stacks for sleepable programs.

Quoted 8 lines
Quoted 1 line

I am still unsure why non-sleepable uprobes would be unaffected. On a

I think we need to disable both sleepable and non-sleepable uprobes

problem is that we can't say if kprobe program is going to be attached as uprobe or kprobe, so we'd need to disable both kprobe/uprobes, which I'm not sure is a problem

I don't think it is, private stack is most critical for struct_ops where we can have intentionally recursive calls (sched-ext had/has such use case, I believe), I think it's much less of a problem and risk for normal uprobe/kprobe (and other tracing programs for that matter)

Quoted 18 lines

I don't see how tracepoint/raw_tracepoint are affected by this problem, because they have either retursion context or bpf_prog_active check

jirka

Quoted 12 lines

preemptible kernel, their execution paths use migrate_disable() rather than preempt_disable(), and rcu_read_lock() does not prevent preemption with preemptible RCU. It therefore seems possible for a task to be scheduled out while running a non-sleepable uprobe, after which another task could invoke the same program on the same CPU and reuse its per-CPU, per-program private stack. Am I missing another mechanism that prevents this?

Cheers,
Christian

---
v3: https://lore.kernel.org/bpf/[email protected]/