bpf: Keep progs alive until the trampoline image calling them is freed
PATCH 1/1 bpf: Keep progs alive until the trampoline image calling them is freed
bpf_tramp_image_put() makes sure a trampoline image is not freed while a task may still be running in it (call_rcu_tasks() + im->pcref), but nothing similar is done for the progs called by that image. Since commit e21aa341785c ("bpf: Fix fexit trampoline."), detach patches the return path so that a task still in the original function skips the fexit progs when it comes back, and counts on the prog's own RCU flavor to cover a task that is inside a prog. On that basis the last prog reference is dropped right away and the prog is freed after a single RCU / RCU tasks trace grace period.
That leaves out a task in the trampoline glue itself: between two progs, or already past the patched jump but not yet in the first fexit prog's enter helper. On !PREEMPT kernels this is a few instructions that cannot be preempted, so it did not matter. With CONFIG_PREEMPTION a task can sit there, in no RCU read section of any flavor and holding only im->pcref, for longer than it takes to free the prog it is about to call:
CPU 0 CPU 1
in image I, orig_call() returned
[preempted before lsm.s prog A]
bpf_tracing_link_release()
-> bpf_tramp_image_put(I)
bpf_link_dealloc()
bpf_prog_put(A), last ref
tasks trace GP, A's text freed
__bpf_prog_enter_sleepable(A)
call A->bpf_func
On x86 this is an int3 in poisoned bpf_prog_pack memory:
Oops: int3: 0000 [#1] SMP NOPTI CPU: 18 UID: 0 PID: 94573 Comm: x169 Not tainted 6.18.44 #1 PREEMPT(lazy) RIP: 0010:0xffffffffc0601d8d Call Trace: <TASK> ? bpf_trampoline_6442515411+0x1a4/0x21b bpf_lsm_bprm_committed_creds+0x5/0x10 security_bprm_committed_creds+0x5f/0x70 begin_new_exec+0x2d6/0x410 ...
We hit this in production on preemptible kernels when progs attached through trampolines got detached while their hooks were busy. Adding grace periods before the prog free would not help with sleepable progs:
neither RCU tasks nor RCU tasks trace waits for a task that slept in a prog and then got preempted in the gap after it.
Fix it by having the image take a reference on every prog it calls, in bpf_tramp_image_alloc(), and drop them in bpf_tramp_image_free(). A detached prog now stays loaded until the old image is gone, which reverts a deliberate choice of commit e21aa341785c ("bpf: Fix fexit trampoline."). Detached fexit progs still stop being called right away since the return path is patched.
---
This should also be queued for the stable trees, the same race exists everywhere since struct bpf_tramp_image was introduced (v5.12+).
Tested on x86_64 with PREEMPT_DYNAMIC/preempt=lazy by attaching and detaching a handful of sleepable and non-sleepable LSM progs in a loop next to an exec storm: unpatched 6.18.44 hits the int3 oops above within the hour, the patched kernel survived 10x that. The trampoline related test_progs selftests (fentry/fexit/modify_return/lsm/ trampoline_count/tracing_multi...) pass with KASAN and lockdep on both this commit and its parent.
include/linux/bpf.h | 2 ++ kernel/bpf/trampoline.c | 28 ++++++++++++++++++++++------ 2 files changed, 24 insertions(+), 6 deletions(-)
include/linux/bpf.h +2 -0
@@ -1368,6 +1368,7 @@ enum bpf_tramp_prog_type {struct bpf_tramp_image {void *image;int size; +int progs_cnt;struct bpf_ksym ksym;struct percpu_ref pcref;void *ip_after_call;
@@ -1376,6 +1377,7 @@ struct bpf_tramp_image {struct rcu_head rcu;struct work_struct work; }; + struct bpf_prog *progs[] __counted_by(progs_cnt); };struct bpf_trampoline {
kernel/bpf/trampoline.c +22 -6
@@ -531,10 +531,14 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_astatic void bpf_tramp_image_free (struct bpf_tramp_image *im) { +int i; +bpf_image_ksym_del (&im->ksym);arch_free_bpf_trampoline (im->image, im->size);bpf_jit_uncharge_modmem (im->size);percpu_ref_exit (&im->pcref); +for (i =0 ; i < im->progs_cnt; i++) +bpf_prog_put (im->progs[i]);kfree_rcu (im, rcu); }
@@ -588,12 +592,11 @@ static void bpf_tramp_image_put(struct bpf_tramp_image *im)* rcu tasks to protect trampoline asm not covered by percpu_ref * (which are few asm insns before __bpf_tramp_enter and * after __bpf_tramp_exit) +* im->progs refs to keep the progs alive as long as the image * * The trampoline is unreachable before bpf_tramp_image_put(). * * First, patch the trampoline to avoid calling into fexit progs. -* The progs will be freed even if the original function is still -* executing or sleeping. * In case of CONFIG_PREEMPT=y use call_rcu_tasks() to wait on * first few asm instructions to execute and call into * __bpf_tramp_enter->percpu_ref_get.
@@ -628,16 +631,20 @@ static void bpf_tramp_image_put(struct bpf_tramp_image *im)call_rcu_tasks_trace (&im->rcu, __bpf_tramp_image_put_rcu_tasks); } -static struct bpf_tramp_image *bpf_tramp_image_alloc (u64 key,int size) +static struct bpf_tramp_image *bpf_tramp_image_alloc (u64 key,int size, +struct bpf_tramp_nodes *tnodes, +int progs_cnt) {struct bpf_tramp_image *im;struct bpf_ksym *ksym; -void *image; +int kind, i, n =0 ;int err = -ENOMEM ; +void *image; - im =kzalloc_obj (*im); + im =kzalloc_flex (*im, progs, progs_cnt);if (!im) goto out; + im->progs_cnt = progs_cnt; err =bpf_jit_charge_modmem (size);if (err)
@@ -658,6 +665,15 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, int size)snprintf (ksym->name,KSYM_NAME_LEN ,"bpf_trampoline_%llu" , key);bpf_image_ksym_init (image, size, ksym);bpf_image_ksym_add (ksym); + +for (kind =0 ; kind <BPF_TRAMP_MAX ; kind++) { +for (i =0 ; i < tnodes[kind].nr_nodes; i++) { +struct bpf_prog *prog = tnodes[kind].nodes[i]->link->prog; + +bpf_prog_inc (prog); + im->progs[n++] = prog; + } + }return im; out_free_image:
@@ -734,7 +750,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut goto out; } - im =bpf_tramp_image_alloc (tr->key, size); + im =bpf_tramp_image_alloc (tr->key, size, tnodes, total);if (IS_ERR (im)) { err =PTR_ERR (im); goto out;
base-commit: a13307e97d5c54b65720bb71fa379960ded1e51a
Signature · 1 line
2.54.0
Quoted 10 lines · Florent Revest (Anthropic)
bpf_tramp_image_put()makes sure a trampoline image is not freed while a task may still be running in it (call_rcu_tasks()+im->pcref), but nothing similar is done for the progs called by that image. Since commite21aa341785c("bpf: Fix fexit trampoline."), detach patches the return path so that a task still in the original function skips the fexit progs when it comes back, and counts on the prog's own RCU flavor to cover a task that is inside a prog. On that basis the last prog reference is dropped right away and the prog is freed after a single RCU / RCU tasks trace grace period.
Quoted 8 lines
[...]
Fix it by having the image take a reference on every prog it calls, in
bpf_tramp_image_alloc(), and drop them inbpf_tramp_image_free(). A detached prog now stays loaded until the old image is gone, which reverts a deliberate choice of commite21aa341785c("bpf: Fix fexit trampoline."). Detached fexit progs still stop being called right away since the return path is patched.
This appears to be the same issue addressed by my earlier patch [1].
I think the flexible-array approach here is cleaner, so I'm fine with this version going forward. Could you please carry the original Reported-by tag?
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 32 lines · On Thu, Aug 20, 2026 at 04:06:30PM +0000, [email protected] wrote:
Quoted 30 lines · 20 août 2026 à 17:26 "Junseo Lim" <[email protected]…
Quoted 11 lines · Florent Revest (Anthropic)
bpf_tramp_image_put()makes sure a trampoline image is not freed while
a task may still be running in it (call_rcu_tasks()+im->pcref), but nothing similar is done for the progs called by that image. Since commite21aa341785c("bpf: Fix fexit trampoline."), detach patches the return path so that a task still in the original function skips the fexit progs when it comes back, and counts on the prog's own RCU flavor to cover a task that is inside a prog. On that basis the last prog reference is dropped right away and the prog is freed after a single RCU / RCU tasks trace grace period.Quoted 9 lines
[...]
Fix it by having the image take a reference on every prog it calls, in
bpf_tramp_image_alloc(), and drop them inbpf_tramp_image_free(). A detached prog now stays loaded until the old image is gone, which reverts a deliberate choice of commite21aa341785c("bpf: Fix fexit trampoline."). Detached fexit progs still stop being called right away since the return path is patched.This appears to be the same issue addressed by my earlier patch [1].
I think the flexible-array approach here is cleaner, so I'm fine with this version going forward. Could you please carry the original Reported-by tag?
Reported-by Sechang Lim <[email protected]>
Quoted 7 lines
Quoted 2 lines
Oh wow, sorry, I had no idea we raced on this!
I also like the idea of guarding this with
CONFIG_PREEMPTIONand of course I don't mind adding your Reported-by tag in a v2. I'll give a bit of time for others to chime in first.
Just in case it helps with the CONFIG_PREEMPTION angle: Sashiko pointed out on my patch that a CONFIG_PREEMPTION guard would leave the unlink/update failure path uncovered on CONFIG_PREEMPTION=n.
So your unconditional version looks right to me.
Cheers,
Junseo
Quoted 41 lines · Junseo Lim
Quoted 32 lines · On Thu, Aug 20, 2026 at 04:06:30PM +0000, [email protected] wrote:
Quoted 30 lines · 20 août 2026 à 17:26 "Junseo Lim" <[email protected]…
Quoted 11 lines · Florent Revest (Anthropic)
bpf_tramp_image_put()makes sure a trampoline image is not freed while
a task may still be running in it (call_rcu_tasks()+im->pcref), but nothing similar is done for the progs called by that image. Since commite21aa341785c("bpf: Fix fexit trampoline."), detach patches the return path so that a task still in the original function skips the fexit progs when it comes back, and counts on the prog's own RCU flavor to cover a task that is inside a prog. On that basis the last prog reference is dropped right away and the prog is freed after a single RCU / RCU tasks trace grace period.Quoted 9 lines
[...]
Fix it by having the image take a reference on every prog it calls, in
bpf_tramp_image_alloc(), and drop them inbpf_tramp_image_free(). A detached prog now stays loaded until the old image is gone, which reverts a deliberate choice of commite21aa341785c("bpf: Fix fexit trampoline."). Detached fexit progs still stop being called right away since the return path is patched.This appears to be the same issue addressed by my earlier patch [1].
I think the flexible-array approach here is cleaner, so I'm fine with this version going forward. Could you please carry the original Reported-by tag?
Reported-by Sechang Lim <[email protected]>Quoted 7 lines
Quoted 2 lines
Oh wow, sorry, I had no idea we raced on this!
I also like the idea of guarding this with
CONFIG_PREEMPTIONand of course I don't mind adding your Reported-by tag in a v2. I'll give a bit of time for others to chime in first.
Quoted 5 lines
Just in case it helps with the
CONFIG_PREEMPTIONangle: Sashiko pointed out on my patch that aCONFIG_PREEMPTIONguard would leave the unlink/update failure path uncovered onCONFIG_PREEMPTION=n.So your unconditional version looks right to me.
Actually, the #ifdef CONFIG_PREEMPT guard wouldn't have been enough anyway. :/ Preemption was one easy way to cause that bug but if a task sleeps in a sleepable prog and the prog that runs after it in the same image is detached and freed, the same problem would happen. That could also happen on PREEMPT_NONE.
But I'll keep your Reported-by in a v2 yep!
Quoted 55 lines · Florent Revest (Anthropic)
bpf_tramp_image_put()makes sure a trampoline image is not freed while a task may still be running in it (call_rcu_tasks()+im->pcref), but nothing similar is done for the progs called by that image. Since commite21aa341785c("bpf: Fix fexit trampoline."), detach patches the return path so that a task still in the original function skips the fexit progs when it comes back, and counts on the prog's own RCU flavor to cover a task that is inside a prog. On that basis the last prog reference is dropped right away and the prog is freed after a single RCU / RCU tasks trace grace period.That leaves out a task in the trampoline glue itself: between two progs, or already past the patched jump but not yet in the first fexit prog's enter helper. On !PREEMPT kernels this is a few instructions that cannot be preempted, so it did not matter. With
CONFIG_PREEMPTIONa task can sit there, in no RCU read section of any flavor and holding onlyim->pcref, for longer than it takes to free the prog it is about to call:CPU 0 CPU 1
in image I,orig_call()returned
[preempted before lsm.s prog A]
bpf_tracing_link_release()
->bpf_tramp_image_put(I)
bpf_link_dealloc()
bpf_prog_put(A), last ref
tasks trace GP, A's text freed
__bpf_prog_enter_sleepable(A)
callA->bpf_funcOn x86 this is an int3 in poisoned
bpf_prog_packmemory:Oops: int3: 0000 [#1] SMP NOPTI CPU: 18 UID: 0 PID: 94573 Comm: x169 Not tainted 6.18.44 #1 PREEMPT(lazy) RIP: 0010:0xffffffffc0601d8d Call Trace: <TASK> ? bpf_trampoline_6442515411+0x1a4/0x21b bpf_lsm_bprm_committed_creds+0x5/0x10 security_bprm_committed_creds+0x5f/0x70 begin_new_exec+0x2d6/0x410 ...We hit this in production on preemptible kernels when progs attached through trampolines got detached while their hooks were busy. Adding grace periods before the prog free would not help with sleepable progs:
neither RCU tasks nor RCU tasks trace waits for a task that slept in a prog and then got preempted in the gap after it.Fix it by having the image take a reference on every prog it calls, in
bpf_tramp_image_alloc(), and drop them inbpf_tramp_image_free(). A detached prog now stays loaded until the old image is gone, which reverts a deliberate choice of commite21aa341785c("bpf: Fix fexit trampoline."). Detached fexit progs still stop being called right away since the return path is patched.
Quoted 3 lines
Fixes e21aa341785c ("bpf: Fix fexit trampoline.")Assisted-by Claude:unspecifiedSigned-off-by Florent Revest (Anthropic) <[email protected]>
lgtm,
Quoted 1 line
[...]
Quoted 10 lines · Florent Revest (Anthropic)
bpf_tramp_image_put()makes sure a trampoline image is not freed while a task may still be running in it (call_rcu_tasks()+im->pcref), but nothing similar is done for the progs called by that image. Since commite21aa341785c("bpf: Fix fexit trampoline."), detach patches the return path so that a task still in the original function skips the fexit progs when it comes back, and counts on the prog's own RCU flavor to cover a task that is inside a prog. On that basis the last prog reference is dropped right away and the prog is freed after a single RCU / RCU tasks trace grace period.
Quoted 4 lines
That leaves out a task in the trampoline glue itself: between two progs, or already past the patched jump but not yet in the first fexit prog's enter helper. On !PREEMPT kernels this is a few instructions that cannot be preempted, so it did not matter. With
CONFIG_PREEMPTION
I guess it would make sense to highlight why it may not have mattered. I think the real reason was that on !PREEMPT kernels, the execution in the trampoline image counted as (implicit) RCU read section which caused program free path to wait for someone executing the trampoline image? For non-sleepable progs, they wait for RCU grace period already, for sleepable, RCU tasks trace has implicit RCU grace period wait as well, hence this never showed up on !PREEMPT.
Quoted 41 lines
a task can sit there, in no RCU read section of any flavor and holding only
im->pcref, for longer than it takes to free the prog it is about to call:CPU 0 CPU 1
in image I,orig_call()returned
[preempted before lsm.s prog A]
bpf_tracing_link_release()
->bpf_tramp_image_put(I)
bpf_link_dealloc()
bpf_prog_put(A), last ref
tasks trace GP, A's text freed
__bpf_prog_enter_sleepable(A)
callA->bpf_funcOn x86 this is an int3 in poisoned
bpf_prog_packmemory:Oops: int3: 0000 [#1] SMP NOPTI CPU: 18 UID: 0 PID: 94573 Comm: x169 Not tainted 6.18.44 #1 PREEMPT(lazy) RIP: 0010:0xffffffffc0601d8d Call Trace: <TASK> ? bpf_trampoline_6442515411+0x1a4/0x21b bpf_lsm_bprm_committed_creds+0x5/0x10 security_bprm_committed_creds+0x5f/0x70 begin_new_exec+0x2d6/0x410 ...We hit this in production on preemptible kernels when progs attached through trampolines got detached while their hooks were busy. Adding grace periods before the prog free would not help with sleepable progs:
neither RCU tasks nor RCU tasks trace waits for a task that slept in a prog and then got preempted in the gap after it.Fix it by having the image take a reference on every prog it calls, in
bpf_tramp_image_alloc(), and drop them inbpf_tramp_image_free(). A detached prog now stays loaded until the old image is gone, which reverts a deliberate choice of commite21aa341785c("bpf: Fix fexit trampoline."). Detached fexit progs still stop being called right away since the return path is patched.
Quoted 4 lines
Fixes e21aa341785c ("bpf: Fix fexit trampoline.")Assisted-by Claude:unspecifiedSigned-off-by Florent Revest (Anthropic) <[email protected]>---
Overall, looks good to me. Thanks for the fix!
Note for whoever applies this: please add Reported-by: tag for Sechang as well. Optionally, wordsmith the commit log with the suggestion above.
Quoted 1 line
[...]
Quoted 70 lines · Kumar Kartikeya Dwivedi
Quoted 10 lines · Florent Revest (Anthropic)
bpf_tramp_image_put()makes sure a trampoline image is not freed while a task may still be running in it (call_rcu_tasks()+im->pcref), but nothing similar is done for the progs called by that image. Since commite21aa341785c("bpf: Fix fexit trampoline."), detach patches the return path so that a task still in the original function skips the fexit progs when it comes back, and counts on the prog's own RCU flavor to cover a task that is inside a prog. On that basis the last prog reference is dropped right away and the prog is freed after a single RCU / RCU tasks trace grace period.Quoted 4 lines
That leaves out a task in the trampoline glue itself: between two progs, or already past the patched jump but not yet in the first fexit prog's enter helper. On !PREEMPT kernels this is a few instructions that cannot be preempted, so it did not matter. With
CONFIG_PREEMPTIONI guess it would make sense to highlight why it may not have mattered. I think the real reason was that on !PREEMPT kernels, the execution in the trampoline image counted as (implicit) RCU read section which caused program free path to wait for someone executing the trampoline image? For non-sleepable progs, they wait for RCU grace period already, for sleepable, RCU tasks trace has implicit RCU grace period wait as well, hence this never showed up on !PREEMPT.
Quoted 41 lines
a task can sit there, in no RCU read section of any flavor and holding only
im->pcref, for longer than it takes to free the prog it is about to call:CPU 0 CPU 1
in image I,orig_call()returned
[preempted before lsm.s prog A]
bpf_tracing_link_release()
->bpf_tramp_image_put(I)
bpf_link_dealloc()
bpf_prog_put(A), last ref
tasks trace GP, A's text freed
__bpf_prog_enter_sleepable(A)
callA->bpf_funcOn x86 this is an int3 in poisoned
bpf_prog_packmemory:Oops: int3: 0000 [#1] SMP NOPTI CPU: 18 UID: 0 PID: 94573 Comm: x169 Not tainted 6.18.44 #1 PREEMPT(lazy) RIP: 0010:0xffffffffc0601d8d Call Trace: <TASK> ? bpf_trampoline_6442515411+0x1a4/0x21b bpf_lsm_bprm_committed_creds+0x5/0x10 security_bprm_committed_creds+0x5f/0x70 begin_new_exec+0x2d6/0x410 ...We hit this in production on preemptible kernels when progs attached through trampolines got detached while their hooks were busy. Adding grace periods before the prog free would not help with sleepable progs:
neither RCU tasks nor RCU tasks trace waits for a task that slept in a prog and then got preempted in the gap after it.Fix it by having the image take a reference on every prog it calls, in
bpf_tramp_image_alloc(), and drop them inbpf_tramp_image_free(). A detached prog now stays loaded until the old image is gone, which reverts a deliberate choice of commite21aa341785c("bpf: Fix fexit trampoline."). Detached fexit progs still stop being called right away since the return path is patched.Quoted 4 lines
Fixes e21aa341785c ("bpf: Fix fexit trampoline.")Assisted-by Claude:unspecifiedSigned-off-by Florent Revest (Anthropic) <[email protected]>---
Quoted 6 lines
Overall, looks good to me. Thanks for the fix!
Note for whoever applies this: please add Reported-by: tag for Sechang as well. Optionally, wordsmith the commit log with the suggestion above.
Hold on. I don't think we can proceed with this fix.
It defeats the point of fexit jmp patching and keeps progs pinned until a sleepable kernel function that were attached to will return. Which means that the tracing prog attached to "unlucky" kernel function that sleeps for an hour will stay pinned for an hour. Let's think of a different way of fixing the race.
pw-bot: cr
Kumar Kartikeya Dwivedi
maintainerQuoted 79 lines · Alexei Starovoitov
Quoted 70 lines · Kumar Kartikeya Dwivedi
Quoted 10 lines · Florent Revest (Anthropic)
bpf_tramp_image_put()makes sure a trampoline image is not freed while a task may still be running in it (call_rcu_tasks()+im->pcref), but nothing similar is done for the progs called by that image. Since commite21aa341785c("bpf: Fix fexit trampoline."), detach patches the return path so that a task still in the original function skips the fexit progs when it comes back, and counts on the prog's own RCU flavor to cover a task that is inside a prog. On that basis the last prog reference is dropped right away and the prog is freed after a single RCU / RCU tasks trace grace period.Quoted 4 lines
That leaves out a task in the trampoline glue itself: between two progs, or already past the patched jump but not yet in the first fexit prog's enter helper. On !PREEMPT kernels this is a few instructions that cannot be preempted, so it did not matter. With
CONFIG_PREEMPTIONI guess it would make sense to highlight why it may not have mattered. I think the real reason was that on !PREEMPT kernels, the execution in the trampoline image counted as (implicit) RCU read section which caused program free path to wait for someone executing the trampoline image? For non-sleepable progs, they wait for RCU grace period already, for sleepable, RCU tasks trace has implicit RCU grace period wait as well, hence this never showed up on !PREEMPT.
Quoted 41 lines
a task can sit there, in no RCU read section of any flavor and holding only
im->pcref, for longer than it takes to free the prog it is about to call:CPU 0 CPU 1
in image I,orig_call()returned
[preempted before lsm.s prog A]
bpf_tracing_link_release()
->bpf_tramp_image_put(I)
bpf_link_dealloc()
bpf_prog_put(A), last ref
tasks trace GP, A's text freed
__bpf_prog_enter_sleepable(A)
callA->bpf_funcOn x86 this is an int3 in poisoned
bpf_prog_packmemory:Oops: int3: 0000 [#1] SMP NOPTI CPU: 18 UID: 0 PID: 94573 Comm: x169 Not tainted 6.18.44 #1 PREEMPT(lazy) RIP: 0010:0xffffffffc0601d8d Call Trace: <TASK> ? bpf_trampoline_6442515411+0x1a4/0x21b bpf_lsm_bprm_committed_creds+0x5/0x10 security_bprm_committed_creds+0x5f/0x70 begin_new_exec+0x2d6/0x410 ...We hit this in production on preemptible kernels when progs attached through trampolines got detached while their hooks were busy. Adding grace periods before the prog free would not help with sleepable progs:
neither RCU tasks nor RCU tasks trace waits for a task that slept in a prog and then got preempted in the gap after it.Fix it by having the image take a reference on every prog it calls, in
bpf_tramp_image_alloc(), and drop them inbpf_tramp_image_free(). A detached prog now stays loaded until the old image is gone, which reverts a deliberate choice of commite21aa341785c("bpf: Fix fexit trampoline."). Detached fexit progs still stop being called right away since the return path is patched.Quoted 4 lines
Fixes e21aa341785c ("bpf: Fix fexit trampoline.")Assisted-by Claude:unspecifiedSigned-off-by Florent Revest (Anthropic) <[email protected]>---
Quoted 6 lines
Overall, looks good to me. Thanks for the fix!
Note for whoever applies this: please add Reported-by: tag for Sechang as well. Optionally, wordsmith the commit log with the suggestion above.
Quoted 6 lines
Hold on. I don't think we can proceed with this fix.
It defeats the point of fexit jmp patching and keeps progs pinned until a sleepable kernel function that were attached to will return. Which means that the tracing prog attached to "unlucky" kernel function that sleeps for an hour will stay pinned for an hour. Let's think of a different way of fixing the race.
I don't have background on the original commit being fixed, but is that really realistic? Or worrisome even if it happens in practice, since worst case the program refcounts remains raised for that duration?
We have similar worst case for programs too (e.g. using bpf_copy_from_user on user controlled buffer in, say, LSM progs). At least here we won't be extending any RCU flavored GP.
That said I will think about alternative fixes in the meantime, if we accept the premise that we don't want to pin program references in the image and keep their lifetimes decoupled.
Quoted 2 lines
pw-bot: cr
Quoted 22 lines · Kumar Kartikeya Dwivedi
Quoted 10 lines · Alexei Starovoitov
Quoted 7 lines · Kumar Kartikeya Dwivedi
Overall, looks good to me. Thanks for the fix!
Note for whoever applies this: please add Reported-by: tag for Sechang as well. Optionally, wordsmith the commit log with the suggestion above.
Quoted 6 lines
Hold on. I don't think we can proceed with this fix.
It defeats the point of fexit jmp patching and keeps progs pinned until a sleepable kernel function that were attached to will return. Which means that the tracing prog attached to "unlucky" kernel function that sleeps for an hour will stay pinned for an hour. Let's think of a different way of fixing the race.I don't have background on the original commit being fixed, but is that really realistic? Or worrisome even if it happens in practice, since worst case the program refcounts remains raised for that duration?
Quoted 7 lines
We have similar worst case for programs too (e.g. using
bpf_copy_from_useron user controlled buffer in, say, LSM progs). At least here we won't be extending any RCU flavored GP.That said I will think about alternative fixes in the meantime, if we accept the premise that we don't want to pin program references in the image and keep their lifetimes decoupled.
How about having old trampolines skip freed programs instead of keeping them alive ?
For example, bpf_tramp_image could hold one RCU managed prog pointer per prog it calls and we could JIT trampolines that'd pass that slot to __bpf_prog_enter*() instead of the prog itself.
The enter helpers could rcu_dereference() the prog under their rcu_read_lock()/rcu_read_lock_trace() and return 0 when it is NULL. Would that work ?
Quoted 28 lines · Florent Revest
Quoted 26 lines · Kumar Kartikeya Dwivedi
Quoted 10 lines · Alexei Starovoitov
Quoted 7 lines · Kumar Kartikeya Dwivedi
Overall, looks good to me. Thanks for the fix!
Note for whoever applies this: please add Reported-by: tag for Sechang as well. Optionally, wordsmith the commit log with the suggestion above.
Quoted 6 lines
Hold on. I don't think we can proceed with this fix.
It defeats the point of fexit jmp patching and keeps progs pinned until a sleepable kernel function that were attached to will return. Which means that the tracing prog attached to "unlucky" kernel function that sleeps for an hour will stay pinned for an hour. Let's think of a different way of fixing the race.I don't have background on the original commit being fixed, but is that really realistic? Or worrisome even if it happens in practice, since worst case the program refcounts remains raised for that duration?
We have similar worst case for programs too (e.g. using
bpf_copy_from_useron user controlled buffer in, say, LSM progs). At least here we won't be extending any RCU flavored GP.
Quoted 6 lines
Quoted 3 lines
That said I will think about alternative fixes in the meantime, if we accept the premise that we don't want to pin program references in the image and keep their lifetimes decoupled.
How about having old trampolines skip freed programs instead of keeping them alive ?
Isn't it doing it now?
That was the whole point of patching nop to jmp in a trampoline.
Quoted 33 lines · Alexei Starovoitov
Quoted 32 lines · Florent Revest
Quoted 29 lines · Kumar Kartikeya Dwivedi
Quoted 10 lines · Alexei Starovoitov
Quoted 7 lines · Kumar Kartikeya Dwivedi
Overall, looks good to me. Thanks for the fix!
Note for whoever applies this: please add Reported-by: tag for Sechang as well. Optionally, wordsmith the commit log with the suggestion above.
Quoted 6 lines
Hold on. I don't think we can proceed with this fix.
It defeats the point of fexit jmp patching and keeps progs pinned until a sleepable kernel function that were attached to will return. Which means that the tracing prog attached to "unlucky" kernel function that sleeps for an hour will stay pinned for an hour. Let's think of a different way of fixing the race.I don't have background on the original commit being fixed, but is that really realistic? Or worrisome even if it happens in practice, since worst case the program refcounts remains raised for that duration?
We have similar worst case for programs too (e.g. using
bpf_copy_from_useron user controlled buffer in, say, LSM progs). At least here we won't be extending any RCU flavored GP.That said I will think about alternative fixes in the meantime, if we accept the premise that we don't want to pin program references in the image and keep their lifetimes decoupled.
Quoted 5 lines
Quoted 2 lines
How about having old trampolines skip freed programs instead of keeping them alive ?
Isn't it doing it now?
That was the whole point of patching nop to jmp in a trampoline.
Just to be sure we're talking about the same thing, you're referring to the "ip_after_call" patching done in bpf_tramp_image_put, right ?
My understanding is that this only helps in the case where a task is in the traced function while a prog is detached. It'd make it skip all fexit progs.
But if a task is sleeping in a fexit.s prog for example, it is already past ip_after_call, so this patching would not prevent it from calling a second, freed, fexit prog lined up after it in the trampoline, no ? The same situation should happen with a sleeping fentry.s prog followed by a fentry prog.
I have a reproducer for those scenarios that crashes bpf-next. I will make it a selftest in v2 like Jiri suggested.
Quoted 38 lines · Florent Revest
Quoted 36 lines · Alexei Starovoitov
Quoted 34 lines · Florent Revest
Quoted 22 lines · Kumar Kartikeya Dwivedi
Quoted 10 lines · Alexei Starovoitov
Quoted 7 lines · Kumar Kartikeya Dwivedi
Overall, looks good to me. Thanks for the fix!
Note for whoever applies this: please add Reported-by: tag for Sechang as well. Optionally, wordsmith the commit log with the suggestion above.
Quoted 6 lines
Hold on. I don't think we can proceed with this fix.
It defeats the point of fexit jmp patching and keeps progs pinned until a sleepable kernel function that were attached to will return. Which means that the tracing prog attached to "unlucky" kernel function that sleeps for an hour will stay pinned for an hour. Let's think of a different way of fixing the race.I don't have background on the original commit being fixed, but is that really realistic? Or worrisome even if it happens in practice, since worst case the program refcounts remains raised for that duration?
Quoted 7 lines
We have similar worst case for programs too (e.g. using
bpf_copy_from_useron user controlled buffer in, say, LSM progs). At least here we won't be extending any RCU flavored GP.That said I will think about alternative fixes in the meantime, if we accept the premise that we don't want to pin program references in the image and keep their lifetimes decoupled.
How about having old trampolines skip freed programs instead of keeping them alive ?
Quoted 8 lines
Quoted 2 lines
Isn't it doing it now?
That was the whole point of patching nop to jmp in a trampoline.Just to be sure we're talking about the same thing, you're referring to the "
ip_after_call" patching done inbpf_tramp_image_put, right ?My understanding is that this only helps in the case where a task is in the traced function while a prog is detached. It'd make it skip all fexit progs.
Right. That's the one I meant.
Quoted 4 lines
But if a task is sleeping in a fexit.s prog for example, it is already past
ip_after_call, so this patching would not prevent it from calling a second, freed, fexit prog lined up after it in the trampoline, no ? The same situation should happen with a sleeping fentry.s prog followed by a fentry prog.
Hmm. You mean like fexit prog A called a sleepable kfunc and another prog B is attached to the same trampoline.
Then the trampoline needs to be freed. ip_after_call patches jmp in, and trampoline proceeds to free progs A and B.
B is freed right away.
Eventually A's kfunc returns from sleep
and jmps back into a trampoline
which is not freed yet, since percpu_ref keeps it,
but prog B is gone, so it attempts to execute a freed prog B ?
Quoted 2 lines
I have a reproducer for those scenarios that crashes bpf-next. I will make it a selftest in v2 like Jiri suggested.
Please narrow down the reproducer (sounds like it's a random stress test at this point) to specific steps.
If my guess above is correct the reproducer will be short and deterministic.
Quoted 27 lines · Alexei Starovoitov
Quoted 7 lines · Florent Revest
Quoted 5 lines · Alexei Starovoitov
Quoted 3 lines · Florent Revest
How about having old trampolines skip freed programs instead of keeping them alive ?
Quoted 8 lines
Quoted 2 lines
Isn't it doing it now?
That was the whole point of patching nop to jmp in a trampoline.Just to be sure we're talking about the same thing, you're referring to the "
ip_after_call" patching done inbpf_tramp_image_put, right ?My understanding is that this only helps in the case where a task is in the traced function while a prog is detached. It'd make it skip all fexit progs.
Right. That's the one I meant.
Quoted 4 lines
But if a task is sleeping in a fexit.s prog for example, it is already past
ip_after_call, so this patching would not prevent it from calling a second, freed, fexit prog lined up after it in the trampoline, no ? The same situation should happen with a sleeping fentry.s prog followed by a fentry prog.Hmm. You mean like fexit prog A called a sleepable kfunc and another prog B is attached to the same trampoline.
Quoted 7 lines
Then the trampoline needs to be freed.
ip_after_callpatches jmp in, and trampoline proceeds to free progs A and B.
B is freed right away.
Eventually A's kfunc returns from sleep
and jmps back into a trampoline
which is not freed yet, sincepercpu_refkeeps it,
but prog B is gone, so it attempts to execute a freed prog B ?
Yep, exactly!
Quoted 8 lines
Quoted 2 lines
I have a reproducer for those scenarios that crashes bpf-next. I will make it a selftest in v2 like Jiri suggested.
Please narrow down the reproducer (sounds like it's a random stress test at this point) to specific steps.
If my guess above is correct the reproducer will be short and deterministic.
Indeed. When I originally sent this patch my repro was a stress test but when I found out about the above scenario I could make a short and deterministic repro
My only question then is, what kind of fix would you prefer in v2: keeping refs to progs while the trampoline is alive ? (like in this v1, but I think you nacked this approach already ?) or some other creative solution like having __bpf_prog_enter*() skip each freed prog individually ? Or maybe something else ?
Quoted 48 lines · Florent Revest
Quoted 27 lines · Alexei Starovoitov
Quoted 7 lines · Florent Revest
Quoted 5 lines · Alexei Starovoitov
Quoted 3 lines · Florent Revest
How about having old trampolines skip freed programs instead of keeping them alive ?
Quoted 8 lines
Quoted 2 lines
Isn't it doing it now?
That was the whole point of patching nop to jmp in a trampoline.Just to be sure we're talking about the same thing, you're referring to the "
ip_after_call" patching done inbpf_tramp_image_put, right ?My understanding is that this only helps in the case where a task is in the traced function while a prog is detached. It'd make it skip all fexit progs.
Right. That's the one I meant.
Quoted 4 lines
But if a task is sleeping in a fexit.s prog for example, it is already past
ip_after_call, so this patching would not prevent it from calling a second, freed, fexit prog lined up after it in the trampoline, no ? The same situation should happen with a sleeping fentry.s prog followed by a fentry prog.Hmm. You mean like fexit prog A called a sleepable kfunc and another prog B is attached to the same trampoline.
Quoted 7 lines
Then the trampoline needs to be freed.
ip_after_callpatches jmp in, and trampoline proceeds to free progs A and B.
B is freed right away.
Eventually A's kfunc returns from sleep
and jmps back into a trampoline
which is not freed yet, sincepercpu_refkeeps it,
but prog B is gone, so it attempts to execute a freed prog B ?Yep, exactly!
Quoted 8 lines
Quoted 2 lines
I have a reproducer for those scenarios that crashes bpf-next. I will make it a selftest in v2 like Jiri suggested.
Please narrow down the reproducer (sounds like it's a random stress test at this point) to specific steps.
If my guess above is correct the reproducer will be short and deterministic.
Quoted 6 lines
Indeed. When I originally sent this patch my repro was a stress test but when I found out about the above scenario I could make a short and deterministic repro
My only question then is, what kind of fix would you prefer in v2: keeping refs to progs while the trampoline is alive ? (like in this v1, but I think you nacked this approach already ?)
I'm worried about holding fexit progs potentially forever. There are kernel funcs that enter and never return. Attaching to them will pin fexit progs forever. While right now it's only the trampoline that gets pinned which is 4k.
Quoted 2 lines
or some other creative solution like having __bpf_prog_enter*() skip each freed prog individually ?
I don't want to add more overhead to __bpf_prog_enter*() wrappers either.
Quoted 2 lines
Or maybe something
else ?
Let's extend ip_after_call concept.
Instead of using a single NOP after the call to skip fexit progs let's add NOPs in front of every invoke_bpf() including fentry and fmod_ret.
Then on tramp destroy patch all nop-s to jmp epilogue.
llm-s helped me realize that fentry has the same issue. fentry prog A can sleep
while non-sleepable fentry prog B can detach and things will crash.
better ideas?
Quoted 10 lines · Alexei Starovoitov
Let's extend
ip_after_callconcept.
Instead of using a single NOP after the call to skip fexit progs let's add NOPs in front of everyinvoke_bpf()including fentry andfmod_ret.Then on tramp destroy patch all nop-s to jmp epilogue.
llm-s helped me realize that fentry has the same issue. fentry prog A can sleep
while non-sleepable fentry prog B can detach and things will crash.
Florent,
are you planning to respin with the above fix or should I ask my clanker ?
Quoted 9 lines · Alexei Starovoitov
Quoted 7 lines · Alexei Starovoitov
Let's extend
ip_after_callconcept.
Instead of using a single NOP after the call to skip fexit progs let's add NOPs in front of everyinvoke_bpf()including fentry andfmod_ret.Then on tramp destroy patch all nop-s to jmp epilogue.
Quoted 7 lines
Quoted 3 lines
llm-s helped me realize that fentry has the same issue. fentry prog A can sleep
while non-sleepable fentry prog B can detach and things will crash.Florent,
are you planning to respin with the above fix or should I ask my clanker ?
Sorry for the wait, this week was busier than I expected! I just sent v2! :)
Quoted 18 lines · Florent Revest (Anthropic)
bpf_tramp_image_put()makes sure a trampoline image is not freed while a task may still be running in it (call_rcu_tasks()+im->pcref), but nothing similar is done for the progs called by that image. Since commite21aa341785c("bpf: Fix fexit trampoline."), detach patches the return path so that a task still in the original function skips the fexit progs when it comes back, and counts on the prog's own RCU flavor to cover a task that is inside a prog. On that basis the last prog reference is dropped right away and the prog is freed after a single RCU / RCU tasks trace grace period.That leaves out a task in the trampoline glue itself: between two progs, or already past the patched jump but not yet in the first fexit prog's enter helper. On !PREEMPT kernels this is a few instructions that cannot be preempted, so it did not matter. With
CONFIG_PREEMPTIONa task can sit there, in no RCU read section of any flavor and holding onlyim->pcref, for longer than it takes to free the prog it is about to call:
Quoted 10 lines
CPU 0 CPU 1
in image I,orig_call()returned
[preempted before lsm.s prog A]
bpf_tracing_link_release()
->bpf_tramp_image_put(I)
bpf_link_dealloc()
bpf_prog_put(A), last ref
tasks trace GP, A's text freed
__bpf_prog_enter_sleepable(A)
callA->bpf_func
hi,
do you have a selftest or another reproducer for this?
thanks,
jirak
Quoted 154 lines
On x86 this is an int3 in poisoned
bpf_prog_packmemory:Oops: int3: 0000 [#1] SMP NOPTI CPU: 18 UID: 0 PID: 94573 Comm: x169 Not tainted 6.18.44 #1 PREEMPT(lazy) RIP: 0010:0xffffffffc0601d8d Call Trace: <TASK> ? bpf_trampoline_6442515411+0x1a4/0x21b bpf_lsm_bprm_committed_creds+0x5/0x10 security_bprm_committed_creds+0x5f/0x70 begin_new_exec+0x2d6/0x410 ...We hit this in production on preemptible kernels when progs attached through trampolines got detached while their hooks were busy. Adding grace periods before the prog free would not help with sleepable progs:
neither RCU tasks nor RCU tasks trace waits for a task that slept in a prog and then got preempted in the gap after it.Fix it by having the image take a reference on every prog it calls, in
bpf_tramp_image_alloc(), and drop them inbpf_tramp_image_free(). A detached prog now stays loaded until the old image is gone, which reverts a deliberate choice of commite21aa341785c("bpf: Fix fexit trampoline."). Detached fexit progs still stop being called right away since the return path is patched.Fixes e21aa341785c ("bpf: Fix fexit trampoline.")Assisted-by Claude:unspecifiedSigned-off-by Florent Revest (Anthropic) <[email protected]>---
This should also be queued for the stable trees, the same race exists everywhere since structbpf_tramp_imagewas introduced (v5.12+).Tested on
x86_64with PREEMPT_DYNAMIC/preempt=lazy by attaching and detaching a handful of sleepable and non-sleepable LSM progs in a loop next to an exec storm: unpatched 6.18.44 hits the int3 oops above within the hour, the patched kernel survived 10x that. The trampoline relatedtest_progsselftests (fentry/fexit/modify_return/lsm/ trampoline_count/tracing_multi...) pass with KASAN and lockdep on both this commit and its parent.include/linux/bpf.h | 2 ++ kernel/bpf/trampoline.c | 28 ++++++++++++++++++++++------ 2 files changed, 24 insertions(+), 6 deletions(-)include/linux/bpf.h +2
-0@@ -1368,6 +1368,7 @@ enum bpf_tramp_prog_type {struct bpf_tramp_image {void *image;int size; +int progs_cnt;struct bpf_ksym ksym;struct percpu_ref pcref;void *ip_after_call;@@ -1376,6 +1377,7 @@ struct bpf_tramp_image {struct rcu_head rcu;struct work_struct work; }; + struct bpf_prog *progs[] __counted_by(progs_cnt); };struct bpf_trampoline {kernel/bpf/trampoline.c +22
-6@@ -531,10 +531,14 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_astatic void bpf_tramp_image_free (struct bpf_tramp_image *im) { +int i; +bpf_image_ksym_del (&im->ksym);arch_free_bpf_trampoline (im->image, im->size);bpf_jit_uncharge_modmem (im->size);percpu_ref_exit (&im->pcref); +for (i =0 ; i < im->progs_cnt; i++) +bpf_prog_put (im->progs[i]);kfree_rcu (im, rcu); }@@ -588,12 +592,11 @@ static void bpf_tramp_image_put(struct bpf_tramp_image *im)* rcu tasks to protect trampoline asm not covered by percpu_ref * (which are few asm insns before __bpf_tramp_enter and * after __bpf_tramp_exit) +* im->progs refs to keep the progs alive as long as the image * * The trampoline is unreachable before bpf_tramp_image_put(). * * First, patch the trampoline to avoid calling into fexit progs. -* The progs will be freed even if the original function is still -* executing or sleeping. * In case of CONFIG_PREEMPT=y use call_rcu_tasks() to wait on * first few asm instructions to execute and call into * __bpf_tramp_enter->percpu_ref_get. @@ -628,16 +631,20 @@ static void bpf_tramp_image_put(struct bpf_tramp_image *im)call_rcu_tasks_trace (&im->rcu, __bpf_tramp_image_put_rcu_tasks); } -static struct bpf_tramp_image *bpf_tramp_image_alloc (u64 key,int size) +static struct bpf_tramp_image *bpf_tramp_image_alloc (u64 key,int size, +struct bpf_tramp_nodes *tnodes, +int progs_cnt) {struct bpf_tramp_image *im;struct bpf_ksym *ksym; -void *image; +int kind, i, n =0 ;int err = -ENOMEM ; +void *image; - im =kzalloc_obj (*im); + im =kzalloc_flex (*im, progs, progs_cnt);if (!im) goto out; + im->progs_cnt = progs_cnt; err =bpf_jit_charge_modmem (size);if (err)@@ -658,6 +665,15 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, int size)snprintf (ksym->name,KSYM_NAME_LEN ,"bpf_trampoline_%llu" , key);bpf_image_ksym_init (image, size, ksym);bpf_image_ksym_add (ksym); + +for (kind =0 ; kind <BPF_TRAMP_MAX ; kind++) { +for (i =0 ; i < tnodes[kind].nr_nodes; i++) { +struct bpf_prog *prog = tnodes[kind].nodes[i]->link->prog; + +bpf_prog_inc (prog); + im->progs[n++] = prog; + } + }return im; out_free_image:@@ -734,7 +750,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut goto out; } - im =bpf_tramp_image_alloc (tr->key, size); + im =bpf_tramp_image_alloc (tr->key, size, tnodes, total);if (IS_ERR (im)) { err =PTR_ERR (im); goto out;base-commit:
a13307e97d5c54b65720bb71fa379960ded1e51aSignature · 1 line
2.54.0
Quoted 30 lines · Jiri Olsa
Quoted 18 lines · Florent Revest (Anthropic)
bpf_tramp_image_put()makes sure a trampoline image is not freed while a task may still be running in it (call_rcu_tasks()+im->pcref), but nothing similar is done for the progs called by that image. Since commite21aa341785c("bpf: Fix fexit trampoline."), detach patches the return path so that a task still in the original function skips the fexit progs when it comes back, and counts on the prog's own RCU flavor to cover a task that is inside a prog. On that basis the last prog reference is dropped right away and the prog is freed after a single RCU / RCU tasks trace grace period.That leaves out a task in the trampoline glue itself: between two progs, or already past the patched jump but not yet in the first fexit prog's enter helper. On !PREEMPT kernels this is a few instructions that cannot be preempted, so it did not matter. With
CONFIG_PREEMPTIONa task can sit there, in no RCU read section of any flavor and holding onlyim->pcref, for longer than it takes to free the prog it is about to call:Quoted 10 lines
CPU 0 CPU 1
in image I,orig_call()returned
[preempted before lsm.s prog A]
bpf_tracing_link_release()
->bpf_tramp_image_put(I)
bpf_link_dealloc()
bpf_prog_put(A), last ref
tasks trace GP, A's text freed
__bpf_prog_enter_sleepable(A)
callA->bpf_func
Quoted 2 lines
hi,
do you have a selftest or another reproducer for this?
I originally only reproduced that with a stress test (an attach/detach loop during an exec storm).
But actually, preemption isn't needed! If a task sleeps in a sleepable prog and the prog that runs after it in the same image gets detached, it calls the freed prog when it wakes up. That case could be made deterministic with userfaultfd so I could write a selftest for it. I'll include it in v2. :)