disable private stack for sleepable programs
Subsystems (10)
v3 cover disable private stack for sleepable programs
A JITed BPF program can use one private stack per program and CPU. Sleepable programs can be preempted, allowing another task to run the same program on the same CPU. The second invocation then reuses and can overwrite the first invocation's private stack.
This patch series disables private stacks for sleepable programs and adds a selftest covering that stack corruption.
---
Changes in v3:
- Target sleepable only.
- Add the powerpc Fixes tag.
- Split fix and selftests
v2: https://lore.kernel.org/bpf/[email protected]/
Christian Simon (2):
bpf: disable private stack for sleepable programs selftests/bpf: verify preemptible uprobes avoid private stack
kernel/bpf/verifier.c | 9 + .../bpf/prog_tests/uprobe_sleepable_stack.c | 165 ++++++++++++++++++ .../bpf/progs/uprobe_sleepable_stack.c | 89 ++++++++++ 3 files changed, 263 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/uprobe_sleepable_stack.c create mode 100644 tools/testing/selftests/bpf/progs/uprobe_sleepable_stack.c
base-commit: 75b0a6db4300e4c2c9e97a0848deaa7acfb42fb7
Signature · 1 line
2.54.0
PATCH v3 1/2 bpf: disable private stack for sleepable programs
A JITed BPF program can use one private stack per program and CPU. Sleepable programs can be preempted, allowing another task to run the same program on the same CPU. The second invocation then reuses and can overwrite the first invocation's private stack.
Disable private stack for sleepable programs so they use the regular kernel stack, which handles preemption correctly. This change is intentionally limited to programs marked sleepable; preemptible non-sleepable dispatch paths require separate protection.
--- kernel/bpf/verifier.c | 9 +++++++++ 1 file changed, 9 insertions(+)
kernel/bpf/verifier.c +9 -0
@@ -5237,6 +5237,15 @@ static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog)if (!bpf_jit_supports_private_stack ())return NO_PRIV_STACK ; +/* +* Sleepable programs can be preempted, allowing another task to run +* the same program on the same CPU. Since private stack is per-CPU +* and per-program, the second invocation would corrupt the first's +* stack. Disable private stack for sleepable programs. +*/ +if (prog->sleepable) +return NO_PRIV_STACK ; +/* bpf_prog_check_recur() checks all prog types that use bpf trampoline * while kprobe/tp/perf_event/raw_tp don't use trampoline hence checked * explicitly.
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Non-sleepable BPF uprobe programs remain vulnerable to per-CPU private stack corruption. --
--- Patch [1]: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs --- commit 5b27c96c855b5a6df31f1d2543a92540c69a5d98
Author: Christian Simon <[email protected]>
bpf: disable private stack for sleepable programs
This commit disables the use of private stacks for sleepable BPF programs since they can be preempted, which would allow a second invocation of the program on the same CPU to corrupt the first invocation's stack. The fix forces sleepable programs to use the regular kernel stack which handles preemption correctly.
Quoted 10 lines
kernel/bpf/verifier.c +2
-0@@ -5237,6 +5237,15 @@ static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog)if (!bpf_jit_supports_private_stack ())return NO_PRIV_STACK ; +/* +* Sleepable programs can be preempted, allowing another task to run
kernel/bpf/verifier.c static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog) · line 5251
Quoted 10 lines
kernel/bpf/verifier.c +7
-0+* the same program on the same CPU. Since private stack is per-CPU +* and per-program, the second invocation would corrupt the first's +* stack. Disable private stack for sleepable programs. +*/ +if (prog->sleepable) +return NO_PRIV_STACK ; +/* bpf_prog_check_recur() checks all prog types that use bpf trampoline * while kprobe/tp/perf_event/raw_tp don't use trampoline hence checked * explicitly.
[Severity: High]
This is a pre-existing issue and was not introduced by this patch, but does this leave non-sleepable uprobes vulnerable to the same stack corruption?
The patch correctly fixes sleepable programs, but the commit message mentions that preemptible non-sleepable dispatch paths require separate protection. However, non-sleepable uprobes are granted private stacks because they fall under BPF_PROG_TYPE_KPROBE in the switch statement immediately below this code.
If we look at bpf_prog_run_array_uprobe() in include/linux/bpf.h, non-sleepable uprobes are executed with only rcu_read_lock():
if (!prog->sleepable)
rcu_read_lock();
run_ctx.bpf_cookie = item->bpf_cookie;
ret &= run_prog(prog, ctx);
item++;
if (!prog->sleepable)
rcu_read_unlock();Under CONFIG_PREEMPT_RCU and CONFIG_PREEMPT, rcu_read_lock() does not disable preemption. Could a task executing this non-sleepable uprobe be preempted, allowing another task on the same CPU to trigger the same uprobe and overwrite its per-CPU private stack?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewRe: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs
In reply to Christian SimonIs this comment accurate for all the program types below?
Quoted 5 lines
kernel/bpf/verifier.c +0
-0
kernel/bpf/verifier.c static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog) · line 5245
Quoted 9 lines
kernel/bpf/verifier.c +6
-0@@ -5237,6 +5237,15 @@ static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog)if (!bpf_jit_supports_private_stack ())return NO_PRIV_STACK ; +/* +* Sleepable programs can be preempted, allowing another task to run +* the same program on the same CPU. Since private stack is per-CPU +* and per-program, the second invocation would corrupt the first's +* stack. Disable private stack for sleepable programs. +*/
Is this comment accurate for all the program types below?
For BPF_PROG_TYPE_TRACING / LSM / STRUCT_OPS, private stack is only granted when bpf_prog_check_recur(prog) holds, and that same predicate makes bpf_trampoline_enter() install __bpf_prog_enter_sleepable_recur(), which bumps the per-CPU prog->active counter under migrate_disable() and skips the nested invocation.
Similarly, the sleepable raw-tracepoint runner __bpf_trace_run() calls bpf_prog_get_recursion_context().
The statement "the second invocation would corrupt the first's stack" would be true for sleepable uprobe programs run via uprobe_prog_run(), which takes no recursion context. Could the comment explain why the check is placed before the switch rather than in the case arm that actually needs it?
kernel/bpf/verifier.c static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog) · line 5248
Quoted 3 lines
kernel/bpf/verifier.c +3
-0+if (prog->sleepable) +return NO_PRIV_STACK ; +
Does placing this check before the prog-type switch have unintended consequences?
The check strips the private stack from BPF_PROG_TYPE_TRACING / LSM / STRUCT_OPS even though those paths are already protected. For these types, bpf_trampoline_enter() consults bpf_prog_check_recur(prog):
kernel/bpf/trampoline.c:bpf_trampoline_enter() {
if (bpf_prog_check_recur(prog))
return sleepable ? __bpf_prog_enter_sleepable_recur :
__bpf_prog_enter_recur;
and __bpf_prog_enter_sleepable_recur() does migrate_disable() and then bpf_prog_get_recursion_context(prog), which uses a per-CPU prog->active counter. A second task entering the same program on the same CPU sees active != 1 and is skipped, so it can never reach the private stack.
The real hole appears to be below the new check: BPF_PROG_TYPE_KPROBE / TRACEPOINT / PERF_EVENT / RAW_TRACEPOINT return PRIV_STACK_ADAPTIVE unconditionally. For sleepable uprobes (the only sleepable case allowed by can_be_sleepable()), the uprobe runner takes no recursion context:
kernel/trace/bpf_trace.c:uprobe_prog_run() {
if (sleepable)
rcu_read_lock_trace();
...
migrate_disable();
err = bpf_prog_run(link->link.prog, regs);so a sleepable uprobe can genuinely be re-entered on the same CPU.
Moving the check into the case arm (BPF_PROG_TYPE_KPROBE through BPF_PROG_TYPE_RAW_TRACEPOINT) would close the uprobe hole without taking the private stack away from the trampoline types that already handle re-entry.
There's also a user-visible side effect: priv_stack_mode decides how check_max_stack_depth_subprog() budgets stack. PRIV_STACK_ADAPTIVE checks each subprog on its own against MAX_BPF_STACK; NO_PRIV_STACK accumulates depth += subprog_depth across bpf2bpf frames and checks the combined total against MAX_BPF_STACK (512).
Any existing sleepable program whose bpf2bpf subtree exceeds 512 bytes total - SEC("fentry.s/..."), SEC("fexit.s/..."), SEC("fmod_ret.s/..."), SEC("uprobe.s/...") - now fails to load with -EACCES and "combined stack size of %d calls is %d. Too large", where it verified fine before. The selftest at tools/testing/selftests/bpf/progs/verifier_private_stack.c:86 ("Private stack, subtree > MAX_BPF_STACK", 512-byte main + 32-byte subprog, __success) has exactly this shape; it only survives because it uses the non-sleepable SEC("kprobe"). With Cc: [email protected], this would break working programs on a stable update.
kernel/bpf/verifier.c static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog) · line 5251
Quoted 3 lines
kernel/bpf/verifier.c +0
-0/* bpf_prog_check_recur() checks all prog types that use bpf trampoline * while kprobe/tp/perf_event/raw_tp don't use trampoline hence checked * explicitly.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32604492082
Re: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs
In reply to Christian SimonQuoted 4 lines · Christian Simon
A JITed BPF program can use one private stack per program and CPU. Sleepable programs can be preempted, allowing another task to run the same program on the same CPU. The second invocation then reuses and can overwrite the first invocation's private stack.
I'm confused by this. sleepable progs go through __bpf_prog_enter_sleepable_recur() which has per-prog recurison counter. So preemption of the prog doesn't break private stack.
If the same prog attemps to execute on the same cpu it will be skipped.
syscall prog types go via bpf_prog_run_array_sleepable() that have per prog recursions counter.
Looks like we're not doing it for bpf_prog_run_array_uprobe(). I'm not sure what the right trade off here.
I feel universally checking for recursion is better then selectively disabling private stack for uprobe.
Quoted 20 lines
Disable private stack for sleepable programs so they use the regular kernel stack, which handles preemption correctly. This change is intentionally limited to programs marked sleepable; preemptible non-sleepable dispatch paths require separate protection.
Fixes 7d1cd70d4b16 ("bpf, x86: Support private stack in jit")Fixes 6c17a882d380 ("bpf, arm64: JIT support for private stack")Fixes 156d985123b6 ("powerpc64/bpf: Implement JIT support for private stack")Signed-off-by Christian Simon <[email protected]>--- kernel/bpf/verifier.c | 9 +++++++++ 1 file changed, 9 insertions(+)kernel/bpf/verifier.c +0
-0@@ -5237,6 +5237,15 @@ static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog)if (!bpf_jit_supports_private_stack ())
kernel/bpf/verifier.c static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog) · line 5247
Quoted 10 lines
kernel/bpf/verifier.c +8
-0return NO_PRIV_STACK ; +/* +* Sleepable programs can be preempted, allowing another task to run +* the same program on the same CPU. Since private stack is per-CPU +* and per-program, the second invocation would corrupt the first's +* stack. Disable private stack for sleepable programs. +*/ +if (prog->sleepable) +return NO_PRIV_STACK ;
This is not true in genreal. sleepable progs can use priv stack.
pw-bot: cr
Re: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs
Quoted 11 lines · Alexei Starovoitov
Quoted 4 lines · Christian Simon
A JITed BPF program can use one private stack per program and CPU. Sleepable programs can be preempted, allowing another task to run the same program on the same CPU. The second invocation then reuses and can overwrite the first invocation's private stack.
I'm confused by this. sleepable progs go through
__bpf_prog_enter_sleepable_recur()which has per-prog recurison counter. So preemption of the prog doesn't break private stack.
If the same prog attemps to execute on the same cpu it will be skipped.
Quoted 7 lines
syscall prog types go via
bpf_prog_run_array_sleepable()that have per prog recursions counter.Looks like we're not doing it for
bpf_prog_run_array_uprobe(). I'm not sure what the right trade off here.
I feel universally checking for recursion is better then selectively disabling private stack for uprobe.
also during the load we can't tell if kprobe program will be attached as kprobe or uprobe [1] so having a way to disable private stack for a program would solve this as well
jirka
[1] https://lore.kernel.org/bpf/ao2f-rBgT0SqX6Pw@krava/
Quoted 35 lines
Quoted 21 lines
Disable private stack for sleepable programs so they use the regular kernel stack, which handles preemption correctly. This change is intentionally limited to programs marked sleepable; preemptible non-sleepable dispatch paths require separate protection.
Fixes 7d1cd70d4b16 ("bpf, x86: Support private stack in jit")Fixes 6c17a882d380 ("bpf, arm64: JIT support for private stack")Fixes 156d985123b6 ("powerpc64/bpf: Implement JIT support for private stack")Signed-off-by Christian Simon <[email protected]>--- kernel/bpf/verifier.c | 9 +++++++++ 1 file changed, 9 insertions(+)kernel/bpf/verifier.c +0
-0@@ -5237,6 +5237,15 @@ static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog)if (!bpf_jit_supports_private_stack ())kernel/bpf/verifier.c static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog) · line 5247
Quoted 10 lines
kernel/bpf/verifier.c +8
-0return NO_PRIV_STACK ; +/* +* Sleepable programs can be preempted, allowing another task to run +* the same program on the same CPU. Since private stack is per-CPU +* and per-program, the second invocation would corrupt the first's +* stack. Disable private stack for sleepable programs. +*/ +if (prog->sleepable) +return NO_PRIV_STACK ;This is not true in genreal. sleepable progs can use priv stack.
pw-bot: cr
Re: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs
In reply to Alexei StarovoitovQuoted 12 lines · Alexei Starovoitov
Quoted 4 lines · Christian Simon
A JITed BPF program can use one private stack per program and CPU. Sleepable programs can be preempted, allowing another task to run the same program on the same CPU. The second invocation then reuses and can overwrite the first invocation's private stack.
I'm confused by this. sleepable progs go through
__bpf_prog_enter_sleepable_recur()which has per-prog recurison counter. So preemption of the prog doesn't break private stack.
If the same prog attemps to execute on the same cpu it will be skipped.
Quoted 7 lines
syscall prog types go via
bpf_prog_run_array_sleepable()that have per prog recursions counter.Looks like we're not doing it for
bpf_prog_run_array_uprobe(). I'm not sure what the right trade off here.
I feel universally checking for recursion is better then selectively disabling private stack for uprobe.
I'd really like to avoid adding this "recursion protection" to uprobe. With uprobes, there is no recursion, it's called from well defined context in the kernel and you can't have recursive uprobe BPF programs.
All you can have is a very valid and possible sleepable uprobe interleaving, which the user cannot prevent or work around, they have no control over this and it's just a fact of life.
E.g., a simple scenario, we attach one bpf program (let's call it U)
to some USDT. BPF program U is sleepable and actually can sleep due to page faults (e.g., unwinding Python stack trace requires sleepable mode for reliably getting filename strings from Python runtime, which are not always paged in).
In such a case, you can have thread A and thread B both hitting the same USDT (e.g., somewhere in memory allocator or whatnot). Let's say thread A hits it first on CPU X, BPF program U starts executing and unwinding Python stack, does bpf_copy_from_user() for string contents and causes page fault, is taken off CPU X. Meanwhile thread B hits USDT on the same CPU X, kernel runs program U, and it is supposed to work completely independently and concurrently (no shared state or whatever) from U's execution in thread A.
Yet, if we add this per-CPU "recursion check", we'll just skip U's execution for thread B. This is data loss, and it's very bad in practice because it frequently just invalidates the entire data collection trustworthiness.
So I think we should disable private stack for uprobes (sleepable or not) instead. I'm not sure private stack buys us anything for uprobe cases.
As Jiri mentioned, at verification/jitting time we can't tell kprobe from uprobe, though, so that's a bit of a problem, but it's a separate discussion we should have. Perhaps having an extra expected attach type, or type, or some program flag to designate uprobe program as such would be appropriate, not sure. Let's discuss that.
But please, let's not make uprobes unreliable for no good reason.
Quoted 35 lines
Quoted 21 lines
Disable private stack for sleepable programs so they use the regular kernel stack, which handles preemption correctly. This change is intentionally limited to programs marked sleepable; preemptible non-sleepable dispatch paths require separate protection.
Fixes 7d1cd70d4b16 ("bpf, x86: Support private stack in jit")Fixes 6c17a882d380 ("bpf, arm64: JIT support for private stack")Fixes 156d985123b6 ("powerpc64/bpf: Implement JIT support for private stack")Signed-off-by Christian Simon <[email protected]>--- kernel/bpf/verifier.c | 9 +++++++++ 1 file changed, 9 insertions(+)kernel/bpf/verifier.c +0
-0@@ -5237,6 +5237,15 @@ static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog)if (!bpf_jit_supports_private_stack ())kernel/bpf/verifier.c static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog) · line 5247
Quoted 10 lines
kernel/bpf/verifier.c +8
-0return NO_PRIV_STACK ; +/* +* Sleepable programs can be preempted, allowing another task to run +* the same program on the same CPU. Since private stack is per-CPU +* and per-program, the second invocation would corrupt the first's +* stack. Disable private stack for sleepable programs. +*/ +if (prog->sleepable) +return NO_PRIV_STACK ;This is not true in genreal. sleepable progs can use priv stack.
pw-bot: cr
Re: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs
Quoted 47 lines · Andrii Nakryiko
Quoted 12 lines · Alexei Starovoitov
Quoted 4 lines · Christian Simon
A JITed BPF program can use one private stack per program and CPU. Sleepable programs can be preempted, allowing another task to run the same program on the same CPU. The second invocation then reuses and can overwrite the first invocation's private stack.
I'm confused by this. sleepable progs go through
__bpf_prog_enter_sleepable_recur()which has per-prog recurison counter. So preemption of the prog doesn't break private stack.
If the same prog attemps to execute on the same cpu it will be skipped.Quoted 7 lines
syscall prog types go via
bpf_prog_run_array_sleepable()that have per prog recursions counter.Looks like we're not doing it for
bpf_prog_run_array_uprobe(). I'm not sure what the right trade off here.
I feel universally checking for recursion is better then selectively disabling private stack for uprobe.I'd really like to avoid adding this "recursion protection" to uprobe. With uprobes, there is no recursion, it's called from well defined context in the kernel and you can't have recursive uprobe BPF programs.
All you can have is a very valid and possible sleepable uprobe interleaving, which the user cannot prevent or work around, they have no control over this and it's just a fact of life.
E.g., a simple scenario, we attach one bpf program (let's call it U)
to some USDT. BPF program U is sleepable and actually can sleep due to page faults (e.g., unwinding Python stack trace requires sleepable mode for reliably getting filename strings from Python runtime, which are not always paged in).In such a case, you can have thread A and thread B both hitting the same USDT (e.g., somewhere in memory allocator or whatnot). Let's say thread A hits it first on CPU X, BPF program U starts executing and unwinding Python stack, does
bpf_copy_from_user()for string contents and causes page fault, is taken off CPU X. Meanwhile thread B hits USDT on the same CPU X, kernel runs program U, and it is supposed to work completely independently and concurrently (no shared state or whatever) from U's execution in thread A.
Quoted 4 lines
Yet, if we add this per-CPU "recursion check", we'll just skip U's execution for thread B. This is data loss, and it's very bad in practice because it frequently just invalidates the entire data collection trustworthiness.
ok. fair
Quoted 3 lines
So I think we should disable private stack for uprobes (sleepable or not) instead. I'm not sure private stack buys us anything for uprobe cases.
why disable priv stack for non-sleepable uprobes? While non-sleepable bpf prog is executing the same or different uprobe cannot execute on the same cpu.
So bpf prog can be preempted by kernel execution,
but a user task cannot start preempt bpf prog,
so 2nd uprobe cannot start running,
no?
Quoted 5 lines
As Jiri mentioned, at verification/jitting time we can't tell kprobe from uprobe, though, so that's a bit of a problem, but it's a separate discussion we should have. Perhaps having an extra expected attach type, or type, or some program flag to designate uprobe program as such would be appropriate, not sure. Let's discuss that.
we don't have sleepable kprobes.
So disable sleepable && [ku]probe type is enough. no?
Re: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs
Quoted 50 lines · Alexei Starovoitov
Quoted 47 lines · Andrii Nakryiko
Quoted 12 lines · Alexei Starovoitov
Quoted 4 lines · Christian Simon
A JITed BPF program can use one private stack per program and CPU. Sleepable programs can be preempted, allowing another task to run the same program on the same CPU. The second invocation then reuses and can overwrite the first invocation's private stack.
I'm confused by this. sleepable progs go through
__bpf_prog_enter_sleepable_recur()which has per-prog recurison counter. So preemption of the prog doesn't break private stack.
If the same prog attemps to execute on the same cpu it will be skipped.Quoted 7 lines
syscall prog types go via
bpf_prog_run_array_sleepable()that have per prog recursions counter.Looks like we're not doing it for
bpf_prog_run_array_uprobe(). I'm not sure what the right trade off here.
I feel universally checking for recursion is better then selectively disabling private stack for uprobe.I'd really like to avoid adding this "recursion protection" to uprobe. With uprobes, there is no recursion, it's called from well defined context in the kernel and you can't have recursive uprobe BPF programs.
All you can have is a very valid and possible sleepable uprobe interleaving, which the user cannot prevent or work around, they have no control over this and it's just a fact of life.
E.g., a simple scenario, we attach one bpf program (let's call it U)
to some USDT. BPF program U is sleepable and actually can sleep due to page faults (e.g., unwinding Python stack trace requires sleepable mode for reliably getting filename strings from Python runtime, which are not always paged in).In such a case, you can have thread A and thread B both hitting the same USDT (e.g., somewhere in memory allocator or whatnot). Let's say thread A hits it first on CPU X, BPF program U starts executing and unwinding Python stack, does
bpf_copy_from_user()for string contents and causes page fault, is taken off CPU X. Meanwhile thread B hits USDT on the same CPU X, kernel runs program U, and it is supposed to work completely independently and concurrently (no shared state or whatever) from U's execution in thread A.
Quoted 7 lines
Quoted 4 lines
Yet, if we add this per-CPU "recursion check", we'll just skip U's execution for thread B. This is data loss, and it's very bad in practice because it frequently just invalidates the entire data collection trustworthiness.
ok. fair
great, thanks!
Quoted 11 lines
Quoted 3 lines
So I think we should disable private stack for uprobes (sleepable or not) instead. I'm not sure private stack buys us anything for uprobe cases.
why disable priv stack for non-sleepable uprobes? While non-sleepable bpf prog is executing the same or different uprobe cannot execute on the same cpu.
So bpf prog can be preempted by kernel execution,
but a user task cannot start preempt bpf prog,
so 2nd uprobe cannot start running,
no?
I think that changes on preemptible kernels, this was called out in discussions on previous versions of this patch. So only for that reason.
For non-preemptible kernels yes, it's only sleepable that is a problem.
Quoted 9 lines
Quoted 6 lines
As Jiri mentioned, at verification/jitting time we can't tell kprobe from uprobe, though, so that's a bit of a problem, but it's a separate discussion we should have. Perhaps having an extra expected attach type, or type, or some program flag to designate uprobe program as such would be appropriate, not sure. Let's discuss that.
we don't have sleepable kprobes.
So disable sleepable && [ku]probe type is enough. no?
Yes, unless we want to also safeguard preemptible kernels. But we can do that as an extra #ifdef CONFIG_PREEMPT_RT or something?
Re: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs
Quoted 75 lines · Andrii Nakryiko
Quoted 50 lines · Alexei Starovoitov
Quoted 47 lines · Andrii Nakryiko
Quoted 12 lines · Alexei Starovoitov
Quoted 4 lines · Christian Simon
A JITed BPF program can use one private stack per program and CPU. Sleepable programs can be preempted, allowing another task to run the same program on the same CPU. The second invocation then reuses and can overwrite the first invocation's private stack.
I'm confused by this. sleepable progs go through
__bpf_prog_enter_sleepable_recur()which has per-prog recurison counter. So preemption of the prog doesn't break private stack.
If the same prog attemps to execute on the same cpu it will be skipped.Quoted 7 lines
syscall prog types go via
bpf_prog_run_array_sleepable()that have per prog recursions counter.Looks like we're not doing it for
bpf_prog_run_array_uprobe(). I'm not sure what the right trade off here.
I feel universally checking for recursion is better then selectively disabling private stack for uprobe.I'd really like to avoid adding this "recursion protection" to uprobe. With uprobes, there is no recursion, it's called from well defined context in the kernel and you can't have recursive uprobe BPF programs.
All you can have is a very valid and possible sleepable uprobe interleaving, which the user cannot prevent or work around, they have no control over this and it's just a fact of life.
E.g., a simple scenario, we attach one bpf program (let's call it U)
to some USDT. BPF program U is sleepable and actually can sleep due to page faults (e.g., unwinding Python stack trace requires sleepable mode for reliably getting filename strings from Python runtime, which are not always paged in).In such a case, you can have thread A and thread B both hitting the same USDT (e.g., somewhere in memory allocator or whatnot). Let's say thread A hits it first on CPU X, BPF program U starts executing and unwinding Python stack, does
bpf_copy_from_user()for string contents and causes page fault, is taken off CPU X. Meanwhile thread B hits USDT on the same CPU X, kernel runs program U, and it is supposed to work completely independently and concurrently (no shared state or whatever) from U's execution in thread A.Quoted 7 lines
Quoted 4 lines
Yet, if we add this per-CPU "recursion check", we'll just skip U's execution for thread B. This is data loss, and it's very bad in practice because it frequently just invalidates the entire data collection trustworthiness.
ok. fair
great, thanks!
Quoted 11 lines
Quoted 3 lines
So I think we should disable private stack for uprobes (sleepable or not) instead. I'm not sure private stack buys us anything for uprobe cases.
why disable priv stack for non-sleepable uprobes? While non-sleepable bpf prog is executing the same or different uprobe cannot execute on the same cpu.
So bpf prog can be preempted by kernel execution,
but a user task cannot start preempt bpf prog,
so 2nd uprobe cannot start running,
no?
Quoted 3 lines
I think that changes on preemptible kernels, this was called out in discussions on previous versions of this patch. So only for that reason.
my understanding is that preemptable kernel doesn't mean that bpf prog can be preempted by user space.
only by kernel.
I looked up earlier thread, but don't understand what Jiri meant.
Jiri,
please clarify what problem do you see with non-sleepable uprobes?
Re: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs
Quoted 81 lines · Alexei Starovoitov
Quoted 75 lines · Andrii Nakryiko
Quoted 50 lines · Alexei Starovoitov
Quoted 47 lines · Andrii Nakryiko
Quoted 12 lines · Alexei Starovoitov
Quoted 4 lines · Christian Simon
A JITed BPF program can use one private stack per program and CPU. Sleepable programs can be preempted, allowing another task to run the same program on the same CPU. The second invocation then reuses and can overwrite the first invocation's private stack.
I'm confused by this. sleepable progs go through
__bpf_prog_enter_sleepable_recur()which has per-prog recurison counter. So preemption of the prog doesn't break private stack.
If the same prog attemps to execute on the same cpu it will be skipped.Quoted 7 lines
syscall prog types go via
bpf_prog_run_array_sleepable()that have per prog recursions counter.Looks like we're not doing it for
bpf_prog_run_array_uprobe(). I'm not sure what the right trade off here.
I feel universally checking for recursion is better then selectively disabling private stack for uprobe.I'd really like to avoid adding this "recursion protection" to uprobe. With uprobes, there is no recursion, it's called from well defined context in the kernel and you can't have recursive uprobe BPF programs.
All you can have is a very valid and possible sleepable uprobe interleaving, which the user cannot prevent or work around, they have no control over this and it's just a fact of life.
E.g., a simple scenario, we attach one bpf program (let's call it U)
to some USDT. BPF program U is sleepable and actually can sleep due to page faults (e.g., unwinding Python stack trace requires sleepable mode for reliably getting filename strings from Python runtime, which are not always paged in).In such a case, you can have thread A and thread B both hitting the same USDT (e.g., somewhere in memory allocator or whatnot). Let's say thread A hits it first on CPU X, BPF program U starts executing and unwinding Python stack, does
bpf_copy_from_user()for string contents and causes page fault, is taken off CPU X. Meanwhile thread B hits USDT on the same CPU X, kernel runs program U, and it is supposed to work completely independently and concurrently (no shared state or whatever) from U's execution in thread A.Quoted 7 lines
Quoted 4 lines
Yet, if we add this per-CPU "recursion check", we'll just skip U's execution for thread B. This is data loss, and it's very bad in practice because it frequently just invalidates the entire data collection trustworthiness.
ok. fair
great, thanks!
Quoted 11 lines
Quoted 3 lines
So I think we should disable private stack for uprobes (sleepable or not) instead. I'm not sure private stack buys us anything for uprobe cases.
why disable priv stack for non-sleepable uprobes? While non-sleepable bpf prog is executing the same or different uprobe cannot execute on the same cpu.
So bpf prog can be preempted by kernel execution,
but a user task cannot start preempt bpf prog,
so 2nd uprobe cannot start running,
no?Quoted 3 lines
I think that changes on preemptible kernels, this was called out in discussions on previous versions of this patch. So only for that reason.
Quoted 8 lines
my understanding is that preemptable kernel doesn't mean that bpf prog can be preempted by user space.
only by kernel.I looked up earlier thread, but don't understand what Jiri meant.
Jiri,
please clarify what problem do you see with non-sleepable uprobes?
hum.. non-sleepable uprobe prog is run by bpf_prog_run_array_uprobe and it disables only task migration, preemption is not disabled and holds rcu_read_lock (which seems ok for preemption)
so I'm not sure why it wouldn't be preemptible by another task
jirka
Re: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs
Quoted 3 lines · Jiri Olsa
Quoted 3 lines
my understanding is that preemptable kernel doesn't mean that bpf prog can be preempted by user space.
only by kernel.
[...]
Quoted 4 lines
hum.. non-sleepable uprobe prog is run by
bpf_prog_run_array_uprobeand it disables only task migration, preemption is not disabled and holdsrcu_read_lock(which seems ok for preemption)
so I'm not sure why it wouldn't be preemptible by another task
It is preemptible, and the private stack gets corrupted when it happens. We hit this in a uprobe/uretprobe based tracer and reduced it to a small reproducer.
The second writer is not user space. The scheduler preempts the task while it is inside the program. migrate_disable() keeps that task on its CPU, but the CPU can still run another task. That task hits the same uprobe, runs the same program and writes its frame into the same per-CPU private stack. When the first task resumes, its frame holds the second task's values. Neither uprobe run path disables preemption:
bpf_prog_run_array_uprobe() migrate_disable(), plus rcu_read_lock()
for non-sleepable programs
uprobe_prog_run() (multi) migrate_disable(), plus rcu_read_lock()
or rcu_read_lock_trace()
With PREEMPT_RCU, rcu_read_lock() does not disable preemption.
bpf_enable_priv_stack() gives a private stack to every BPF_PROG_TYPE_KPROBE program with at least BPF_PRIV_STACK_MIN_SIZE of stack and no tail call. So classic uprobes and uprobe.multi are affected as well, not only sleepable programs.
The test: four processes pinned to one CPU call getppid() in a loop for 10 s. The program fills 16 stack slots (a 216-byte frame) with bpf_get_current_pid_tgid(), spins with 200 bpf_ktime_get_ns() calls, and counts runs where a slot no longer matches a fresh bpf_get_current_pid_tgid(). The JIT output confirms that this program uses the private stack. The controls do not: one adds a tail call into an empty PROG_ARRAY, the other uses a single slot.
Clobbered entry runs / total runs, one 10 s run per cell (7.0: one of two runs, the other was within a few percent), x86_64, 4-vCPU KVM guests:
kernel preempt uprobe uprobe.multi uprobe.s 6.12.112 voluntary 0/288000 0/296000 0/288000 * 6.18.56 none 0/272000 0/288000 0/288000 6.18.56 voluntary 0/272000 0/288000 0/288000 6.18.56 full 497/272000 481/280000 501/280000 6.18.56 lazy 0/280000 0/280000 0/288000 7.0 (Ubuntu) full 1221/856004 1202/864004 1158/856004 7.0 (Ubuntu) lazy 0/840004 0/848004 1/856004 7.2.10 full 649/296000 654/288000 642/288000 7.2.10 lazy 642/288000 622/280000 655/288000 bpf-next full 700/288000 595/280000 619/280000 bpf-next lazy 634/288000 670/288000 647/288000
* 6.12 has no private stack. Its JIT output uses the task stack.
The return probes gave numbers of the same size. Both controls stayed at 0 in every row. On 7.0 under lazy, each variant gave 0 to 2 clobbered runs per 10 s run across repeats. bpf-next was 7.3.0-rc5-g763187dbbf83. On 6.18, lazy stays at zero while 7.2 and bpf-next corrupt under lazy as often as under full. We did not look into why.
In the tracer this showed up as copies from another task's buffer pointer and length (wrong bytes or -EFAULT), and as return probes that found no entry state because the tid in the frame was not theirs. This was on the Ubuntu 7.0 kernel with its default lazy preemption, at one to four bad calls per million under steady load, and more when the host was busy. It went to zero once the programs stopped using the private stack.
On the fix: I agree with Andrii that skipping the second run is wrong for uprobes. It is a separate, legitimate event, and a tracer that counts calls would silently lose it. Disabling the private stack for uprobes, sleepable or not, as he suggested, avoids that. At load time a classic uprobe program cannot be told apart from a kprobe, so this needs either the load-time marker that was discussed or dropping the private stack for BPF_PROG_TYPE_KPROBE. For uprobe.multi, expected_attach_type already says BPF_TRACE_UPROBE_MULTI.
Until then, a single tail call into an empty PROG_ARRAY keeps a program off the private stack.
I can turn the reproducer into a selftest, or run any version of the fix on these kernels.
Thanks,
HyeongJun
Re: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs
Quoted 83 lines · HyeongJun An
Quoted 3 lines · Jiri Olsa
Quoted 3 lines
my understanding is that preemptable kernel doesn't mean that bpf prog can be preempted by user space.
only by kernel.[...]
Quoted 4 lines
hum.. non-sleepable uprobe prog is run by
bpf_prog_run_array_uprobeand it disables only task migration, preemption is not disabled and holdsrcu_read_lock(which seems ok for preemption)
so I'm not sure why it wouldn't be preemptible by another taskIt is preemptible, and the private stack gets corrupted when it happens. We hit this in a uprobe/uretprobe based tracer and reduced it to a small reproducer.
The second writer is not user space. The scheduler preempts the task while it is inside the program.
migrate_disable()keeps that task on its CPU, but the CPU can still run another task. That task hits the same uprobe, runs the same program and writes its frame into the same per-CPU private stack. When the first task resumes, its frame holds the second task's values. Neither uprobe run path disables preemption:
bpf_prog_run_array_uprobe()migrate_disable(), plusrcu_read_lock()
for non-sleepable programs
uprobe_prog_run()(multi)migrate_disable(), plusrcu_read_lock()
orrcu_read_lock_trace()With
PREEMPT_RCU,rcu_read_lock()does not disable preemption.
bpf_enable_priv_stack()gives a private stack to everyBPF_PROG_TYPE_KPROBEprogram with at leastBPF_PRIV_STACK_MIN_SIZEof stack and no tail call. So classic uprobes and uprobe.multi are affected as well, not only sleepable programs.The test: four processes pinned to one CPU call
getppid()in a loop for 10 s. The program fills 16 stack slots (a 216-byte frame) withbpf_get_current_pid_tgid(), spins with 200bpf_ktime_get_ns()calls, and counts runs where a slot no longer matches a freshbpf_get_current_pid_tgid(). The JIT output confirms that this program uses the private stack. The controls do not: one adds a tail call into an emptyPROG_ARRAY, the other uses a single slot.Clobbered entry runs / total runs, one 10 s run per cell (7.0: one of two runs, the other was within a few percent),
x86_64, 4-vCPU KVM guests:kernel preempt uprobe uprobe.multi uprobe.s 6.12.112 voluntary 0/288000 0/296000 0/288000 * 6.18.56 none 0/272000 0/288000 0/288000 6.18.56 voluntary 0/272000 0/288000 0/288000 6.18.56 full 497/272000 481/280000 501/280000 6.18.56 lazy 0/280000 0/280000 0/288000 7.0 (Ubuntu) full 1221/856004 1202/864004 1158/856004 7.0 (Ubuntu) lazy 0/840004 0/848004 1/856004 7.2.10 full 649/296000 654/288000 642/288000 7.2.10 lazy 642/288000 622/280000 655/288000 bpf-next full 700/288000 595/280000 619/280000 bpf-next lazy 634/288000 670/288000 647/288000* 6.12 has no private stack. Its JIT output uses the task stack.
The return probes gave numbers of the same size. Both controls stayed at 0 in every row. On 7.0 under lazy, each variant gave 0 to 2 clobbered runs per 10 s run across repeats. bpf-next was 7.3.0-rc5-g763187dbbf83. On 6.18, lazy stays at zero while 7.2 and bpf-next corrupt under lazy as often as under full. We did not look into why.
In the tracer this showed up as copies from another task's buffer pointer and length (wrong bytes or
-EFAULT), and as return probes that found no entry state because the tid in the frame was not theirs. This was on the Ubuntu 7.0 kernel with its default lazy preemption, at one to four bad calls per million under steady load, and more when the host was busy. It went to zero once the programs stopped using the private stack.On the fix: I agree with Andrii that skipping the second run is wrong for uprobes. It is a separate, legitimate event, and a tracer that counts calls would silently lose it. Disabling the private stack for uprobes, sleepable or not, as he suggested, avoids that. At load time a classic uprobe program cannot be told apart from a kprobe, so this needs either the load-time marker that was discussed or dropping the private stack for
BPF_PROG_TYPE_KPROBE. For uprobe.multi,expected_attach_typealready saysBPF_TRACE_UPROBE_MULTI.
Quoted 5 lines
Until then, a single tail call into an empty
PROG_ARRAYkeeps a program off the private stack.I can turn the reproducer into a selftest, or run any version of the fix on these kernels.
sorry.. seems this fell through cracks
could we add a BPF_TRACE_UPROBE expected attach type for classic uprobes?
but I think we can't reject attaching an unmarked BPF_PROG_TYPE_KPROBE program to a uprobe if it was JITed to use a private stack, that would break existing users
thoughts?
thanks,
jirka
---
include/uapi/linux/bpf.h +1 -0
@@ -1200,6 +1200,7 @@ enum bpf_attach_type {BPF_TRACE_FENTRY_MULTI ,BPF_TRACE_FEXIT_MULTI ,BPF_TRACE_FSESSION_MULTI , +BPF_TRACE_UPROBE , __MAX_BPF_ATTACH_TYPE };
kernel/bpf/syscall.c +6 -0
@@ -2800,6 +2800,12 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type,return -EINVAL ;switch (prog_type) { +case BPF_PROG_TYPE_KPROBE : +if (expected_attach_type ==BPF_TRACE_UPROBE || + expected_attach_type ==BPF_TRACE_UPROBE_MULTI || + expected_attach_type ==BPF_TRACE_UPROBE_SESSION ) +return 0 ; +return expected_attach_type ? -EINVAL :0 ;case BPF_PROG_TYPE_CGROUP_SOCK :switch (expected_attach_type) {case BPF_CGROUP_INET_SOCK_CREATE :
kernel/bpf/verifier.c +4 -0
@@ -5699,6 +5699,10 @@ static enum priv_stack_mode bpf_enable_priv_stack(struct bpf_prog *prog) {if (!bpf_jit_supports_private_stack ())return NO_PRIV_STACK ; +if (prog->expected_attach_type ==BPF_TRACE_UPROBE || + prog->expected_attach_type ==BPF_TRACE_UPROBE_MULTI || + prog->expected_attach_type ==BPF_TRACE_UPROBE_SESSION ) +return NO_PRIV_STACK ;/* bpf_prog_check_recur() checks all prog types that use bpf trampoline * while kprobe/tp/perf_event/raw_tp don't use trampoline hence checked
kernel/events/core.c +13 -0
@@ -11835,6 +11835,19 @@ static int __perf_event_set_bpf_prog(struct perf_event *event, (is_syscall_tp && prog->type != BPF_PROG_TYPE_TRACEPOINT))return -EINVAL ; +/* XXX leftover.. Multi and session uprobes must use their BPF link attachment path. */ +if (prog->expected_attach_type ==BPF_TRACE_UPROBE_MULTI || + prog->expected_attach_type ==BPF_TRACE_UPROBE_SESSION ) +return -EINVAL ; + +if (prog->expected_attach_type ==BPF_TRACE_UPROBE && !is_uprobe) +return -EINVAL ; + +/* XXX can we do this? */ +/* A preempted uprobe can be re-entered and overwrite its private stack. */ +if (is_uprobe && !prog->expected_attach_type && prog->aux->jits_use_priv_stack) +return -EINVAL ; +if (prog->type ==BPF_PROG_TYPE_KPROBE && prog->sleepable && !is_uprobe)/* only uprobe programs are allowed to be sleepable */ return -EINVAL ;
tools/include/uapi/linux/bpf.h +1 -0
@@ -1200,6 +1200,7 @@ enum bpf_attach_type {BPF_TRACE_FENTRY_MULTI ,BPF_TRACE_FEXIT_MULTI ,BPF_TRACE_FSESSION_MULTI , +BPF_TRACE_UPROBE , __MAX_BPF_ATTACH_TYPE };
tools/lib/bpf/libbpf.c +4 -4
@@ -10842,11 +10842,11 @@ static const struct bpf_sec_def section_defs[] = {SEC_DEF ("sk_reuseport/migrate" ,SK_REUSEPORT ,BPF_SK_REUSEPORT_SELECT_OR_MIGRATE ,SEC_ATTACHABLE ),SEC_DEF ("sk_reuseport" ,SK_REUSEPORT ,BPF_SK_REUSEPORT_SELECT ,SEC_ATTACHABLE ),SEC_DEF ("kprobe+" ,KPROBE ,0 ,SEC_NONE , attach_kprobe), -SEC_DEF ("uprobe+" ,KPROBE ,0 ,SEC_NONE , attach_uprobe), -SEC_DEF ("uprobe.s+" ,KPROBE ,0 ,SEC_SLEEPABLE , attach_uprobe), +SEC_DEF ("uprobe+" ,KPROBE ,BPF_TRACE_UPROBE ,SEC_NONE , attach_uprobe), +SEC_DEF ("uprobe.s+" ,KPROBE ,BPF_TRACE_UPROBE ,SEC_SLEEPABLE , attach_uprobe),SEC_DEF ("kretprobe+" ,KPROBE ,0 ,SEC_NONE , attach_kprobe), -SEC_DEF ("uretprobe+" ,KPROBE ,0 ,SEC_NONE , attach_uprobe), -SEC_DEF ("uretprobe.s+" ,KPROBE ,0 ,SEC_SLEEPABLE , attach_uprobe), +SEC_DEF ("uretprobe+" ,KPROBE ,BPF_TRACE_UPROBE ,SEC_NONE , attach_uprobe), +SEC_DEF ("uretprobe.s+" ,KPROBE ,BPF_TRACE_UPROBE ,SEC_SLEEPABLE , attach_uprobe),SEC_DEF ("kprobe.multi+" ,KPROBE ,BPF_TRACE_KPROBE_MULTI ,SEC_NONE , attach_kprobe_multi),SEC_DEF ("kretprobe.multi+" ,KPROBE ,BPF_TRACE_KPROBE_MULTI ,SEC_NONE , attach_kprobe_multi), SEC_DEF("kprobe.session+", KPROBE, BPF_TRACE_KPROBE_SESSION, SEC_NONE, attach_kprobe_session),
Re: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs
Quoted 7 lines · Jiri Olsa
could we add a
BPF_TRACE_UPROBEexpected attach type for classic uprobes?but I think we can't reject attaching an unmarked
BPF_PROG_TYPE_KPROBEprogram to a uprobe if it was JITed to use a private stack, that would break existing usersthoughts?
The marker plus the libbpf change looks right to me. I ran your draft against the reproducer from my last mail to see what it does for the programs that stay unmarked: every program built with an older libbpf, and every loader outside libbpf (aya, cilium/ebpf, bpftrace) until it learns the new attach type. Our tracer is one of those.
Kernels are bpf-next 7101b10cf281 with your draft (jiri), your draft without the attach-time rejection (jiri-noreject), and jiri-noreject plus the run-path change below and Christian's v3 1/2 (jiri-a). "unmarked" is bpftool with the current libbpf, "marked" with your libbpf change. Same reproducer as before, one 10 s run per cell:
kernel loader uprobe uprobe.s uprobe.multi base unmarked 672/288000 655/296000 661/304000 base marked 620/304000 669/304000 592/296000 jiri unmarked attach EINVAL attach EINVAL 0/296000 jiri marked 0/296000 0/288000 0/304000 jiri-noreject unmarked 573/272000 640/300000 0/296000 jiri-noreject marked 0/288000 0/304000 0/304000 jiri-a unmarked 0/304000 0/296000 0/296000 jiri-a marked 0/296000 0/296000 0/296000
(full preemption shown. Lazy gave the same picture on every kernel. The tail-call and small-stack controls were 0 everywhere, and the return probes matched the entry probes. uprobe.multi is always loaded with BPF_TRACE_UPROBE_MULTI, so it is marked with both loaders.)
So with the rejection, unmarked uprobes fail to attach with a bare -EINVAL. Without it, they keep corrupting.
For the unmarked case I'd suggest disabling preemption around the run, only for non-sleepable programs that were JITed with a private stack. In the v2 review of the sleepable uprobe series Alexei wrote:
disabled preemption was an artifact of the past. It became unnecessary when migrate_disable was introduced.
That held when no program kept per-CPU state across a preemption point. The private stack brought per-CPU state back, so migrate_disable() is not enough again, but only for programs that use it. Everything else keeps running preemptible as today, and sleepable programs get no private stack (Christian's v3 1/2). With this, jiri-a is 0 for both loaders.
The comment above bpf_prog_run_array_uprobe() already says, for a non-sleepable program, "we take the rcu read section and disable preemption for that program alone", but the code has never disabled preemption there. The run-path part of jiri-a, which also applies to bpf-next as is:
include/linux/bpf.h +11 -0
@@ -2825,13 +2825,24 @@ bpf_prog_run_array_uprobe(const struct bpf_prog_array *array, old_run_ctx =bpf_set_run_ctx (&run_ctx.run_ctx); item = &array->items[0 ];while ((prog =READ_ONCE (item->prog))) { +/* +* The private stack is per CPU. With only migration disabled, +* another task can run the same program on this CPU and +* overwrite the frame of a preempted run. +*/ +bool no_preempt = !prog->sleepable && prog->aux->jits_use_priv_stack; +if (!prog->sleepable)rcu_read_lock (); +if (no_preempt) +preempt_disable (); run_ctx.bpf_cookie = item->bpf_cookie; ret &=run_prog (prog, ctx); item++; +if (no_preempt) +preempt_enable ();if (!prog->sleepable)rcu_read_unlock (); }
kernel/trace/bpf_trace.c +5 -0
@@ -3166,11 +3166,16 @@ static int uprobe_prog_run(struct bpf_uprobe *uprobe,rcu_read_lock ();migrate_disable (); +/* Same reason as in bpf_prog_run_array_uprobe() */ +if (!sleepable && prog->aux->jits_use_priv_stack) +preempt_disable (); old_run_ctx =bpf_set_run_ctx (&run_ctx.session_ctx.run_ctx); err =bpf_prog_run (link->link.prog, regs);bpf_reset_run_ctx (old_run_ctx); +if (!sleepable && prog->aux->jits_use_priv_stack) +preempt_enable ();migrate_enable ();if (sleepable)
Two other notes from building your draft:
- The new case BPF_PROG_TYPE_KPROBE in bpf_prog_load_check_attach() returns -EINVAL for kprobe.multi and kprobe.session, since only the three uprobe types and 0 pass. jiri-a lists those two as well.
- Not covered by these runs: the same program attached to a kprobe and a uprobe, where a kprobe run from an interrupt could enter it during a uprobe run. The reproducer only attaches by section name, so I have no data on that case.
I can send the run-path change as a proper patch on top of your series, and turn the reproducer into a selftest for classic, multi and sleepable uprobes with the two controls.
The run-path change and the test kernels were prepared with Claude Code (Claude Opus 5.5).
Thanks,
HyeongJun
Re: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs
include/linux/bpf.h bpf_prog_run_array_uprobe(const struct bpf_prog_array *array, · line 2838
Quoted 4 lines · HyeongJun An
include/linux/bpf.h +2
-0if (!prog->sleepable)rcu_read_lock (); +if (no_preempt) +preempt_disable ();
nack.
The models are sending the same broken thing.
Pls teach them to read lore for nacks.
Re: [PATCH bpf v3 1/2] bpf: disable private stack for sleepable programs
Sorry, and thank you for the correction. You're right that this was already nacked on the list, and I should have found it first. I'll be more careful to check lore before sending next time. Sorry for the noise.
Thanks,
HyeongJun
PATCH v3 2/2 selftests/bpf: verify preemptible uprobes avoid private stack
In reply to Christian SimonPin two threads to one CPU and overlap invocations of the same uprobe program. Verify that both invocations execute and that the first invocation's stack frame is not corrupted by the second.
Cover sleepable classic and multi-uprobe programs with both 64-byte stack frames, which select private stack when eligible, and small stack frames as controls. Skip the test when preemption, JIT, or architecture private-stack support is unavailable.
---
I have noticed the "volatile" warning, I do think it is fine for this selftests purposes.
.../bpf/prog_tests/uprobe_sleepable_stack.c | 165 ++++++++++++++++++ .../bpf/progs/uprobe_sleepable_stack.c | 89 ++++++++++ 2 files changed, 254 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/uprobe_sleepable_stack.c create mode 100644 tools/testing/selftests/bpf/progs/uprobe_sleepable_stack.c
tools/testing/selftests/bpf/prog_tests/uprobe_sleepable_stack.c new file +165 -0
@@ -0,0 +1,165 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Test that preemptible uprobe programs don't use private stack. +* +* Private stack is per-CPU and per-program, so preemption during a program's +* execution would allow another task to corrupt the stack. The verifier must +* disable private stack for programs whose invocation can be preempted. +* +* This test overlaps two invocations of the same uprobe program on the same +* CPU and verifies: +* 1. Both invocations execute +* 2. No stack corruption occurs (each task has its own stack frame) +*/ +#include <pthread.h> +#include <sched.h> +#include <unistd.h> + +#include <test_progs.h> +#include "uprobe_sleepable_stack.skel.h" + +static noinline void uprobe_sleepable_stack_trigger(void) +{ + asmvolatile ("" ); +} + +static void *trigger_uprobe (void *arg) +{ +uprobe_sleepable_stack_trigger (); +return NULL ; +} + +static void reset_state (struct uprobe_sleepable_stack *skel) +{ + skel->bss->ready =0 ; + skel->bss->release =0 ; + skel->bss->executions =0 ; + skel->bss->corruptions =0 ; + skel->bss->loop_exhausted =0 ; +} + +static void assert_results (struct uprobe_sleepable_stack *skel) +{ +ASSERT_EQ (skel->bss->loop_exhausted,0 ,"loop_exhausted" ); +ASSERT_EQ (skel->bss->executions,2 ,"executions" ); +ASSERT_EQ (skel->bss->corruptions,0 ,"corruptions" ); +} + +static void run_test (struct uprobe_sleepable_stack *skel,bool multi, +bool large_stack) +{ +LIBBPF_OPTS (bpf_uprobe_opts, opts); +LIBBPF_OPTS (bpf_uprobe_multi_opts, multi_opts); +const char *binary ="/proc/self/exe" ; +struct bpf_program *prog; +struct bpf_link *link; +pthread_t thread; +long link_err; +int err, i; + +reset_state (skel); + +if (multi) { + prog = large_stack ? skel->progs.uprobe_multi_sleepable_large_stack : + skel->progs.uprobe_multi_sleepable_small_stack; + link =bpf_program__attach_uprobe_multi ( + prog,0 , binary,"uprobe_sleepable_stack_trigger" , + &multi_opts); + }else { + opts.func_name ="uprobe_sleepable_stack_trigger" ; + prog = large_stack ? skel->progs.uprobe_sleepable_large_stack : + skel->progs.uprobe_sleepable_small_stack; + link =bpf_program__attach_uprobe_opts (prog,0 , binary,0 , + &opts); + } + + link_err =libbpf_get_error (link); +if (link_err == -EOPNOTSUPP ) { +test__skip (); +return ; + } +if (!ASSERT_OK_PTR (link,"attach_uprobe" )) +return ; + + err =pthread_create (&thread,NULL , trigger_uprobe,NULL ); +if (!ASSERT_OK (err,"pthread_create" )) + goto cleanup; + +for (i =0 ; i <10000 ; i++) { +if (__atomic_load_n (&skel->bss->ready, __ATOMIC_ACQUIRE)) +break ; +usleep (1000 ); + } + +if (ASSERT_LT (i,10000 ,"first_uprobe_ready" )) +uprobe_sleepable_stack_trigger (); +__atomic_store_n (&skel->bss->release,1 , __ATOMIC_RELEASE); + + err =pthread_join (thread,NULL ); +if (!ASSERT_OK (err,"pthread_join" )) + goto cleanup; + +assert_results (skel); + +cleanup: +bpf_link__destroy (link); +} + +void test_uprobe_sleepable_stack (void ) +{ +struct uprobe_sleepable_stack *skel =NULL ; +cpu_set_t old_mask, mask; +bool affinity_set = false; +int cpu; + +#if !defined(__x86_64__) && !defined(__aarch64__) && !defined(__powerpc64__) +test__skip (); +return ; +#endif +if (!env.jit_enabled) { +test__skip (); +return ; + } + +if (!ASSERT_OK (sched_getaffinity (0 ,sizeof (old_mask), &old_mask), +"get_affinity" )) +return ; + +CPU_ZERO (&mask); +for (cpu =0 ; cpu <CPU_SETSIZE ; cpu++) { +if (CPU_ISSET (cpu, &old_mask)) { +CPU_SET (cpu, &mask); +break ; + } + } +if (!ASSERT_LT (cpu,CPU_SETSIZE ,"available_cpu" )) +return ; +/* Both triggers must run on the same CPU to test stack sharing */ +if (!ASSERT_OK (sched_setaffinity (0 ,sizeof (mask), &mask), +"set_affinity" )) +return ; + affinity_set = true; + + skel =uprobe_sleepable_stack__open_and_load (); +if (!ASSERT_OK_PTR (skel,"skel_open" )) + goto cleanup; +if (!skel->kconfig->CONFIG_PREEMPTION) { +test__skip (); + goto cleanup; + } + +if (test__start_subtest ("sleepable_classic_large" )) +run_test (skel, false, true); +if (test__start_subtest ("sleepable_classic_small" )) +run_test (skel, false, false); +if (test__start_subtest ("sleepable_multi_large" )) +run_test (skel, true, true); +if (test__start_subtest ("sleepable_multi_small" )) +run_test (skel, true, false); + +cleanup: +uprobe_sleepable_stack__destroy (skel); +if (affinity_set) +ASSERT_OK (sched_setaffinity (0 ,sizeof (old_mask), &old_mask), +"restore_affinity" ); +}
tools/testing/selftests/bpf/progs/uprobe_sleepable_stack.c new file +89 -0
89 lines not on this page · raw message
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH bpf v3 2/2] selftests/bpf: verify preemptible uprobes avoid private stack
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Spurious test failure (flakiness) due to insufficiently small timeout limit in bpf_loop. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Spurious test failure (flakiness) due to insufficiently small timeout limit in bpf_loop. --
--- Patch [2]: [PATCH bpf v3 2/2] selftests/bpf: verify preemptible uprobes avoid private stack --- commit c17909e48f5484c8e9bf3341a1fc87b98ad29988
Author: Christian Simon <[email protected]>
selftests/bpf: verify preemptible uprobes avoid private stack
Pin two threads to one CPU and overlap invocations of the same uprobe program. Verify that both invocations execute and that the first invocation's stack frame is not corrupted by the second.
Quoted 4 lines
tools/testing/selftests/bpf/progs/uprobe_sleepable_stack.c new file +0
-0
[ ... ]
Quoted 2 lines
tools/testing/selftests/bpf/progs/uprobe_sleepable_stack.c new file +2
-0+static __always_inline int run_stack_test(volatile __u64 *stack) +{
[ ... ]
tools/testing/selftests/bpf/progs/uprobe_sleepable_stack.c line 48
Quoted 4 lines
tools/testing/selftests/bpf/progs/uprobe_sleepable_stack.c new file +4
-0+ ready =1 ; +bpf_loop (1 <<23 , wait_for_release,NULL ,0 ); +if (!release) + loop_exhausted =1 ;
[Severity: Medium]
Could this iteration limit cause spurious test failures under load?
In a heavily loaded CI environment, the main thread's usleep(1000) can be delayed significantly (for example, if timer granularity is 10ms or due to hypervisor latency). On a fast CPU, 8.38M iterations of this trivial callback might complete in just a few milliseconds, exhausting the loop before the main thread wakes up and sets release.
If this happens, loop_exhausted is set to 1 and the selftest fails. Is it possible to safely extend this wait time by wrapping the bpf_loop call in an outer bounded loop?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[email protected]
AI reviewRe: [PATCH bpf v3 2/2] selftests/bpf: verify preemptible uprobes avoid private stack
In reply to Christian SimonA subsystem pattern flags this as potentially concerning: the BPF selftests guide asks whether a new test file for a narrow variant of behavior should instead be a new case in an existing more…
tools/testing/selftests/bpf/prog_tests/uprobe_sleepable_stack.c
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/uprobe_sleepable_stack.c new file +0
-0
A subsystem pattern flags this as potentially concerning: the BPF selftests guide asks whether a new test file for a narrow variant of behavior should instead be a new case in an existing more general test.
The behavior being tested - that the verifier disables private stack for sleepable uprobes - is a verifier-level decision made in bpf_enable_priv_stack() when prog->sleepable is set. The test suite already has tools/testing/selftests/bpf/progs/verifier_private_stack.c in the same directory, which covers private-stack selection deterministically by asserting on JIT output patterns with __jited() annotations.
This patch adds a 254-line, two-file runtime test that depends on thread timing and CPU affinity. Adding a SEC("uprobe.s") case to the verifier test with a 64-byte frame would pin the fix by asserting the absence of the private-frame-pointer setup instruction, without needing threads or timing dependencies.
The runtime test does exercise something the verifier test cannot - actual concurrent same-CPU execution - so this is raised as a question:
would extending verifier_private_stack.c have been a simpler way to pin the verifier change, or is the runtime concurrency test worth the added complexity?
[ ... ]
Quoted 1 line
tools/testing/selftests/bpf/prog_tests/uprobe_sleepable_stack.c new file +0
-0@@ -0,0 +1,165 @@
[ ... ]
Quoted 15 lines
tools/testing/selftests/bpf/prog_tests/uprobe_sleepable_stack.c new file +15
-0+void test_uprobe_sleepable_stack (void ) +{ +struct uprobe_sleepable_stack *skel =NULL ; +cpu_set_t old_mask, mask; +bool affinity_set = false; +int cpu; + +#if !defined(__x86_64__) && !defined(__aarch64__) && !defined(__powerpc64__) +test__skip (); +return ; +#endif +if (!env.jit_enabled) { +test__skip (); +return ; + }
[ ... ]
Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/uprobe_sleepable_stack.c new file +4
-0+if (!skel->kconfig->CONFIG_PREEMPTION) { +test__skip (); + goto cleanup; + }
[ ... ]
tools/testing/selftests/bpf/prog_tests/uprobe_sleepable_stack.c line 80
Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/uprobe_sleepable_stack.c new file +5
-0+ link_err =libbpf_get_error (link); +if (link_err == -EOPNOTSUPP ) { +test__skip (); +return ; + }
This isn't a bug, but could the skip paths include diagnostics? All four skip sites (unsupported architecture, JIT disabled, CONFIG_PREEMPTION unset, and attach returning -EOPNOTSUPP) call bare test__skip() with no message, so a skipped result gives the operator no way to tell which prerequisite was missing.
The test_progs framework's test__skip() takes no arguments, so this matches prevailing practice across the test suite. Is it worth noting because the missing prerequisite that matters most - runtime preemption actually being enabled - would be most helpful to communicate.
tools/testing/selftests/bpf/progs/uprobe_sleepable_stack.c
Quoted 1 line
tools/testing/selftests/bpf/progs/uprobe_sleepable_stack.c +0
-0
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32604492082