bpf: keep trampoline progs alive until image release
PATCH v2 1/1 bpf: keep trampoline progs alive until image release
arch_prepare_bpf_trampoline() embeds program pointers in the generated image and passes them to __bpf_prog_enter_recur(). After bpf_trampoline_update() replaces the image, bpf_tramp_image_put() can keep the old image executable past a normal RCU grace period.
A detached non-sleepable prog can therefore be freed before a preempted task reaches rcu_read_lock_dont_migrate() in __bpf_prog_enter_recur(), leading to a use-after-free.
Keep image-local prog refs and drop them from bpf_tramp_image_free().
---
v1 -> v2:
- Factor prog ref get/put logic into helpers, per Leon Hwang's suggestion.
v1: https://lore.kernel.org/bpf/[email protected]/T/
include/linux/bpf.h | 5 +++++ kernel/bpf/trampoline.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+)
include/linux/bpf.h +5 -0
@@ -1376,6 +1376,11 @@ struct bpf_tramp_image {struct rcu_head rcu;struct work_struct work; }; +#ifdef CONFIG_PREEMPTION +/* Programs called from this image must outlive deferred image freeing. */ +struct bpf_prog *progs[BPF_MAX_TRAMP_LINKS ]; +int nr_progs; +#endif };struct bpf_trampoline {
kernel/bpf/trampoline.c +30 -0
@@ -529,8 +529,36 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_areturn tnodes; } +static void bpf_tramp_image_get_progs (struct bpf_tramp_image *im, +struct bpf_tramp_nodes *tnodes) +{ +#ifdef CONFIG_PREEMPTION +int i, kind; + +for (kind =0 ; kind <BPF_TRAMP_MAX ; kind++) +for (i =0 ; i < tnodes[kind].nr_nodes; i++) { +struct bpf_prog *prog = tnodes[kind].nodes[i]->link->prog; + +bpf_prog_inc (prog); + im->progs[im->nr_progs++] = prog; + } +#endif +} + +static void bpf_tramp_image_put_progs (struct bpf_tramp_image *im) +{ +#ifdef CONFIG_PREEMPTION +int i; + +for (i =0 ; i < im->nr_progs; i++) +bpf_prog_put (im->progs[i]); +#endif +} +static void bpf_tramp_image_free (struct bpf_tramp_image *im) { +bpf_tramp_image_put_progs (im); +bpf_image_ksym_del (&im->ksym);arch_free_bpf_trampoline (im->image, im->size);bpf_jit_uncharge_modmem (im->size);
@@ -740,6 +768,8 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut goto out; } +bpf_tramp_image_get_progs (im, tnodes); + err =arch_prepare_bpf_trampoline (im, im->image, im->image + size, &tr->func.model, tr->flags, tnodes, tr->func.addr);
Signature · 1 line
2.55.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] The patch restricts the bpf_tramp_image program reference counting fix to CONFIG_PREEMPTION=y, leaving CONFIG_PREEMPTION=n kernels vulnerable to a Use-After-Free if bpf_trampoline_update() fails during program unlinking. --
--- Patch [1]: [PATCH bpf v2] bpf: keep trampoline progs alive until image release --- commit e97f38fbbba7062b73e2785973928c70f36f1661
Author: Junseo Lim <[email protected]>
bpf: keep trampoline progs alive until image release
arch_prepare_bpf_trampoline() embeds program pointers in the generated image. After bpf_trampoline_update() replaces the image, bpf_tramp_image_put() can keep the old image executable past a normal RCU grace period, allowing a detached non-sleepable prog to be freed before a preempted task finishes, leading to a use-after-free. This patch keeps image-local prog refs and drops them from bpf_tramp_image_free().
Quoted 13 lines
include/linux/bpf.h +5
-0@@ -1376,6 +1376,11 @@ struct bpf_tramp_image {struct rcu_head rcu;struct work_struct work; }; +#ifdef CONFIG_PREEMPTION +/* Programs called from this image must outlive deferred image freeing. */ +struct bpf_prog *progs[BPF_MAX_TRAMP_LINKS ]; +int nr_progs; +#endif };
[ ... ]
kernel/bpf/trampoline.c bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a · line 535
Quoted 11 lines
kernel/bpf/trampoline.c +4
-0@@ -529,8 +529,36 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_areturn tnodes; } +static void bpf_tramp_image_get_progs (struct bpf_tramp_image *im, +struct bpf_tramp_nodes *tnodes) +{ +#ifdef CONFIG_PREEMPTION
^^^^
[Severity: High]
This is a pre-existing issue, but does restricting this reference counting to CONFIG_PREEMPTION=y leave CONFIG_PREEMPTION=n kernels vulnerable to a use-after-free if bpf_trampoline_update() fails during program unlinking?
If a user detaches a BPF tracing program, bpf_tracing_link_release() calls __bpf_trampoline_unlink_prog(), which removes the program from the trampoline list and calls bpf_trampoline_update().
If bpf_trampoline_update() fails (for example, due to -ENOMEM in bpf_tramp_image_alloc()), the error is absorbed by a WARN_ON_ONCE. The link's reference to the program is subsequently dropped, and the program is freed.
However, because the update failed, the old executable image remains active and continues to execute the embedded pointers to the freed program memory.
Could the CONFIG_PREEMPTION ifdef guards be removed so that the image always holds references to its programs, preventing the use-after-free for CONFIG_PREEMPTION=n kernels as well?
Quoted 11 lines
kernel/bpf/trampoline.c +11
-0+int i, kind; + +for (kind =0 ; kind <BPF_TRAMP_MAX ; kind++) +for (i =0 ; i < tnodes[kind].nr_nodes; i++) { +struct bpf_prog *prog = tnodes[kind].nodes[i]->link->prog; + +bpf_prog_inc (prog); + im->progs[im->nr_progs++] = prog; + } +#endif +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewThis isn't a bug, but would renaming bpf_tramp_image_get_progs() to something like bpf_tramp_image_hold_progs() help avoid confusion with the existing bpf_trampoline_get_progs() right above it? The…
Quoted 4 lines
include/linux/bpf.h +0
-0
[ ... ]
Quoted 13 lines
kernel/bpf/trampoline.c +5
-0@@ -529,8 +529,36 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_areturn tnodes; } +static void bpf_tramp_image_get_progs (struct bpf_tramp_image *im, +struct bpf_tramp_nodes *tnodes) +{ +#ifdef CONFIG_PREEMPTION +int i, kind;
kernel/bpf/trampoline.c bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a · line 546
Quoted 10 lines
kernel/bpf/trampoline.c +10
-0+ +for (kind =0 ; kind <BPF_TRAMP_MAX ; kind++) +for (i =0 ; i < tnodes[kind].nr_nodes; i++) { +struct bpf_prog *prog = tnodes[kind].nodes[i]->link->prog; + +bpf_prog_inc (prog); + im->progs[im->nr_progs++] = prog; + } +#endif +}
This isn't a bug, but would renaming bpf_tramp_image_get_progs() to something like bpf_tramp_image_hold_progs() help avoid confusion with the existing bpf_trampoline_get_progs() right above it? The two helpers do completely different things: bpf_trampoline_get_progs() collects nodes from hlists while bpf_tramp_image_get_progs() takes refcounts.
kernel/bpf/trampoline.c bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a · line 556
Quoted 10 lines
kernel/bpf/trampoline.c +10
-0+ +static void bpf_tramp_image_put_progs (struct bpf_tramp_image *im) +{ +#ifdef CONFIG_PREEMPTION +int i; + +for (i =0 ; i < im->nr_progs; i++) +bpf_prog_put (im->progs[i]); +#endif +}
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32326053450
Hi Junseo,
Quoted 5 lines · Junseo Lim
arch_prepare_bpf_trampoline()embeds program pointers in the generated image and passes them to__bpf_prog_enter_recur(). Afterbpf_trampoline_update()replaces the image,bpf_tramp_image_put()can keep the old image executable past a normal RCU grace period.
Quoted 7 lines
A detached non-sleepable prog can therefore be freed before a preempted task reaches
rcu_read_lock_dont_migrate()in__bpf_prog_enter_recur(), leading to a use-after-free.Keep image-local prog refs and drop them from
bpf_tramp_image_free().Fixes e21aa341785c ("bpf: Fix fexit trampoline.")
There was an existing fix [1]. Could you take a look at that fix?
[1]
https://lore.kernel.org/bpf/[email protected]/
Thanks,
Leon
Quoted 3 lines
Reported-by Sechang Lim <[email protected]>Signed-off-by Junseo Lim <[email protected]>--- [...]
Quoted 6 lines · Leon Hwang
There was an existing fix [1]. Could you take a look at that fix?
Thanks for the pointer. This appears to be the same issue as my v1. I took a look at Florent's patch, and I think its flexible-array approach is cleaner, so I'm fine with that version going forward.
Thanks,
Junseo