Skip to contentSkip to conversation

bpf: keep trampoline progs alive until image release

PATCH In review Junseo Lim 4 participants 5 messages Aug 20

Subsystems (2)
lore
Share & tools
lore

Junseo Lim

author
#0 lore raw

PATCH v2 1/1 bpf: keep trampoline progs alive until image release

arch_prepare_bpf_trampoline() embeds program pointers in the generated image and passes them to __bpf_prog_enter_recur(). After bpf_trampoline_update() replaces the image, bpf_tramp_image_put() can keep the old image executable past a normal RCU grace period.

A detached non-sleepable prog can therefore be freed before a preempted task reaches rcu_read_lock_dont_migrate() in __bpf_prog_enter_recur(), leading to a use-after-free.

Keep image-local prog refs and drop them from bpf_tramp_image_free().

Fixes e21aa341785c ("bpf: Fix fexit trampoline.")

---
v1 -> v2:
- Factor prog ref get/put logic into helpers, per Leon Hwang's suggestion.

v1: https://lore.kernel.org/bpf/[email protected]/T/

 include/linux/bpf.h     |  5 +++++
 kernel/bpf/trampoline.c | 30 ++++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+)
include/linux/bpf.h +5 -0
index 7719f6528445..bacb6bc2e27b 100644
@@ -1376,6 +1376,11 @@ struct bpf_tramp_image {
 		struct rcu_head rcu;
 		struct work_struct work;
 	};
+#ifdef CONFIG_PREEMPTION
+	/* Programs called from this image must outlive deferred image freeing. */
+	struct bpf_prog *progs[BPF_MAX_TRAMP_LINKS];
+	int nr_progs;
+#endif
 };
 
 struct bpf_trampoline {
kernel/bpf/trampoline.c +30 -0
index 1a721fc4bef5..33383d1dc1e0 100644
@@ -529,8 +529,36 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a
 	return tnodes;
 }
 
+static void bpf_tramp_image_get_progs(struct bpf_tramp_image *im,
+				      struct bpf_tramp_nodes *tnodes)
+{
+#ifdef CONFIG_PREEMPTION
+	int i, kind;
+
+	for (kind = 0; kind < BPF_TRAMP_MAX; kind++)
+		for (i = 0; i < tnodes[kind].nr_nodes; i++) {
+			struct bpf_prog *prog = tnodes[kind].nodes[i]->link->prog;
+
+			bpf_prog_inc(prog);
+			im->progs[im->nr_progs++] = prog;
+		}
+#endif
+}
+
+static void bpf_tramp_image_put_progs(struct bpf_tramp_image *im)
+{
+#ifdef CONFIG_PREEMPTION
+	int i;
+
+	for (i = 0; i < im->nr_progs; i++)
+		bpf_prog_put(im->progs[i]);
+#endif
+}
+
 static void bpf_tramp_image_free(struct bpf_tramp_image *im)
 {
+	bpf_tramp_image_put_progs(im);
+
 	bpf_image_ksym_del(&im->ksym);
 	arch_free_bpf_trampoline(im->image, im->size);
 	bpf_jit_uncharge_modmem(im->size);
@@ -740,6 +768,8 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
 		goto out;
 	}
 
+	bpf_tramp_image_get_progs(im, tnodes);
+
 	err = arch_prepare_bpf_trampoline(im, im->image, im->image + size,
 					  &tr->func.model, tr->flags, tnodes,
 					  tr->func.addr);
Signature · 1 line
2.55.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] The patch restricts the bpf_tramp_image program reference counting fix to CONFIG_PREEMPTION=y, leaving CONFIG_PREEMPTION=n kernels vulnerable to a Use-After-Free if bpf_trampoline_update() fails during program unlinking. --

--- Patch [1]: [PATCH bpf v2] bpf: keep trampoline progs alive until image release --- commit e97f38fbbba7062b73e2785973928c70f36f1661
Author: Junseo Lim <[email protected]>

bpf: keep trampoline progs alive until image release

arch_prepare_bpf_trampoline() embeds program pointers in the generated image. After bpf_trampoline_update() replaces the image, bpf_tramp_image_put() can keep the old image executable past a normal RCU grace period, allowing a detached non-sleepable prog to be freed before a preempted task finishes, leading to a use-after-free. This patch keeps image-local prog refs and drops them from bpf_tramp_image_free().

Quoted 13 lines
include/linux/bpf.h +5 -0
@@ -1376,6 +1376,11 @@ struct bpf_tramp_image {
 		struct rcu_head rcu;
 		struct work_struct work;
 	};
+#ifdef CONFIG_PREEMPTION
+	/* Programs called from this image must outlive deferred image freeing. */
+	struct bpf_prog *progs[BPF_MAX_TRAMP_LINKS];
+	int nr_progs;
+#endif
 };

[ ... ]

kernel/bpf/trampoline.c bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a · line 535

Quoted 11 lines
kernel/bpf/trampoline.c +4 -0
@@ -529,8 +529,36 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a
 	return tnodes;
 }
 
+static void bpf_tramp_image_get_progs(struct bpf_tramp_image *im,
+				      struct bpf_tramp_nodes *tnodes)
+{
+#ifdef CONFIG_PREEMPTION
        ^^^^

[Severity: High]
This is a pre-existing issue, but does restricting this reference counting to CONFIG_PREEMPTION=y leave CONFIG_PREEMPTION=n kernels vulnerable to a use-after-free if bpf_trampoline_update() fails during program unlinking?

If a user detaches a BPF tracing program, bpf_tracing_link_release() calls __bpf_trampoline_unlink_prog(), which removes the program from the trampoline list and calls bpf_trampoline_update().

If bpf_trampoline_update() fails (for example, due to -ENOMEM in bpf_tramp_image_alloc()), the error is absorbed by a WARN_ON_ONCE. The link's reference to the program is subsequently dropped, and the program is freed.

However, because the update failed, the old executable image remains active and continues to execute the embedded pointers to the freed program memory.

Could the CONFIG_PREEMPTION ifdef guards be removed so that the image always holds references to its programs, preventing the use-after-free for CONFIG_PREEMPTION=n kernels as well?

Quoted 11 lines
kernel/bpf/trampoline.c +11 -0
+	int i, kind;
+
+	for (kind = 0; kind < BPF_TRAMP_MAX; kind++)
+		for (i = 0; i < tnodes[kind].nr_nodes; i++) {
+			struct bpf_prog *prog = tnodes[kind].nodes[i]->link->prog;
+
+			bpf_prog_inc(prog);
+			im->progs[im->nr_progs++] = prog;
+		}
+#endif
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Junseo Lim
This isn't a bug, but would renaming bpf_tramp_image_get_progs() to something like bpf_tramp_image_hold_progs() help avoid confusion with the existing bpf_trampoline_get_progs() right above it? The…
Quoted 4 lines
include/linux/bpf.h +0 -0
index 7719f6528445..bacb6bc2e27b 100644

[ ... ]

Quoted 13 lines
kernel/bpf/trampoline.c +5 -0
index 1a721fc4bef5..33383d1dc1e0 100644
@@ -529,8 +529,36 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a
 	return tnodes;
 }
 
+static void bpf_tramp_image_get_progs(struct bpf_tramp_image *im,
+				      struct bpf_tramp_nodes *tnodes)
+{
+#ifdef CONFIG_PREEMPTION
+	int i, kind;

kernel/bpf/trampoline.c bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a · line 546

Quoted 10 lines
kernel/bpf/trampoline.c +10 -0
+
+	for (kind = 0; kind < BPF_TRAMP_MAX; kind++)
+		for (i = 0; i < tnodes[kind].nr_nodes; i++) {
+			struct bpf_prog *prog = tnodes[kind].nodes[i]->link->prog;
+
+			bpf_prog_inc(prog);
+			im->progs[im->nr_progs++] = prog;
+		}
+#endif
+}

This isn't a bug, but would renaming bpf_tramp_image_get_progs() to something like bpf_tramp_image_hold_progs() help avoid confusion with the existing bpf_trampoline_get_progs() right above it? The two helpers do completely different things: bpf_trampoline_get_progs() collects nodes from hlists while bpf_tramp_image_get_progs() takes refcounts.

kernel/bpf/trampoline.c bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a · line 556

Quoted 10 lines
kernel/bpf/trampoline.c +10 -0
+
+static void bpf_tramp_image_put_progs(struct bpf_tramp_image *im)
+{
+#ifdef CONFIG_PREEMPTION
+	int i;
+
+	for (i = 0; i < im->nr_progs; i++)
+		bpf_prog_put(im->progs[i]);
+#endif
+}

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32326053450

#3 lore raw
In reply to Junseo Lim

Hi Junseo,

Quoted 5 lines · Junseo Lim

arch_prepare_bpf_trampoline() embeds program pointers in the generated image and passes them to __bpf_prog_enter_recur(). After bpf_trampoline_update() replaces the image, bpf_tramp_image_put() can keep the old image executable past a normal RCU grace period.

Quoted 7 lines

A detached non-sleepable prog can therefore be freed before a preempted task reaches rcu_read_lock_dont_migrate() in __bpf_prog_enter_recur(), leading to a use-after-free.

Keep image-local prog refs and drop them from bpf_tramp_image_free().

Fixes e21aa341785c ("bpf: Fix fexit trampoline.")

There was an existing fix [1]. Could you take a look at that fix?

[1]
https://lore.kernel.org/bpf/[email protected]/

Thanks,
Leon

Quoted 3 lines

--- [...]

author
#4 lore raw
Quoted 6 lines · Leon Hwang

There was an existing fix [1]. Could you take a look at that fix?

[1]
https://lore.kernel.org/bpf/[email protected]/

Thanks for the pointer. This appears to be the same issue as my v1. I took a look at Florent's patch, and I think its flexible-array approach is cleaner, so I'm fine with that version going forward.

Thanks,
Junseo