Skip to contentSkip to conversation

bpf: Migrate bpf_task_work and file dynptr to kmalloc_nolock

PATCH Applied Mykyta Yatsenko 4 participants 9 messages Mar 30 – Apr 2 BPF

lore
Share & tools
lore
#0 lore raw

v2 cover bpf: Migrate bpf_task_work and file dynptr to kmalloc_nolock

Now that kmalloc can be used from NMI context via kmalloc_nolock(), migrate BPF internal allocations away from bpf_mem_alloc to use the standard slab allocator.

Use kfree_rcu() for deferred freeing, which waits for a regular RCU grace period before the memory is reclaimed. Sleepable BPF programs hold rcu_read_lock_trace but not regular rcu_read_lock, so patch 1 adds explicit rcu_read_lock/unlock around the pointer-to-refcount window to prevent kfree_rcu from freeing memory while a sleepable program is still between reading the pointer and acquiring a reference.

Patch 1 migrates bpf_task_work_ctx from bpf_mem_alloc/bpf_mem_free to kmalloc_nolock/kfree_rcu.

Patch 2 migrates bpf_dynptr_file_impl from bpf_mem_alloc/bpf_mem_free to kmalloc_nolock/kfree.

---
Changes in v2:
- Switch to scoped_guard in patch 1 (Kumar)
- Remove rcu gp wait in patch 2 (Kumar)
- Defer to irq_work when irqs disabled in patch 1
- use bpf_map_kmalloc_nolock() for bpf_task_work
- use kmalloc_nolock() for file dynptr
- Link to v1: https://lore.kernel.org/all/[email protected]/

---
Mykyta Yatsenko (2):
bpf: Migrate bpf_task_work to kmalloc_nolock
bpf: Migrate dynptr file to kmalloc_nolock

 kernel/bpf/helpers.c | 60 ++++++++++++++++++++++++++++++++++------------------
 1 file changed, 39 insertions(+), 21 deletions(-)

---
base-commit: 9f7d8fa6817e2709846fc7f5c9f60254e536d138
change-id: 20260223-kmalloc_special-933ec4c543d7

Best regards,

Signature · 1 line
Mykyta Yatsenko <[email protected]>
#1 lore raw

PATCH v2 1/2 bpf: Migrate bpf_task_work to kmalloc_nolock

From: Mykyta Yatsenko <[email protected]>

Replace bpf_mem_alloc/bpf_mem_free with
kmalloc_nolock/kfree_rcu for bpf_task_work_ctx.

Replace guard(rcu_tasks_trace)() with guard(rcu)() in bpf_task_work_irq(). The function only accesses ctx struct members (not map values), so tasks trace protection is not needed - regular RCU is sufficient since ctx is freed via kfree_rcu. The guard in bpf_task_work_callback() remains as tasks trace since it accesses map values from process context.

Sleepable BPF programs hold rcu_read_lock_trace but not regular rcu_read_lock. Since kfree_rcu
waits for a regular RCU grace period, the ctx memory can be freed while a sleepable program is still running. Add scoped_guard(rcu) around the pointer read and refcount tryget in bpf_task_work_acquire_ctx to close this race window.

Since kfree_rcu uses call_rcu internally which is not safe from NMI context, defer destruction via irq_work when irqs are disabled.

For the lost-cmpxchg path the ctx was never published, so kfree_nolock is safe.

---
 kernel/bpf/helpers.c | 56 ++++++++++++++++++++++++++++++++++------------------
 1 file changed, 37 insertions(+), 19 deletions(-)
kernel/bpf/helpers.c +37 -19
index cb6d242bd093..4c3011ef631f 100644
@@ -4165,17 +4165,25 @@ static bool bpf_task_work_ctx_tryget(struct bpf_task_work_ctx *ctx)
 	return refcount_inc_not_zero(&ctx->refcnt);
 }
 
+static void bpf_task_work_destroy(struct irq_work *irq_work)
+{
+	struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
+
+	bpf_task_work_ctx_reset(ctx);
+	kfree_rcu(ctx, rcu);
+}
+
 static void bpf_task_work_ctx_put(struct bpf_task_work_ctx *ctx)
 {
 	if (!refcount_dec_and_test(&ctx->refcnt))
 		return;
 
-	bpf_task_work_ctx_reset(ctx);
-
-	/* bpf_mem_free expects migration to be disabled */
-	migrate_disable();
-	bpf_mem_free(&bpf_global_ma, ctx);
-	migrate_enable();
+	if (irqs_disabled()) {
+		ctx->irq_work = IRQ_WORK_INIT(bpf_task_work_destroy);
+		irq_work_queue(&ctx->irq_work);
+	} else {
+		bpf_task_work_destroy(&ctx->irq_work);
+	}
 }
 
 static void bpf_task_work_cancel(struct bpf_task_work_ctx *ctx)
@@ -4229,7 +4237,7 @@ static void bpf_task_work_irq(struct irq_work *irq_work)
 	enum bpf_task_work_state state;
 	int err;
 
-	guard(rcu_tasks_trace)();
+	guard(rcu)();
 
 	if (cmpxchg(&ctx->state, BPF_TW_PENDING, BPF_TW_SCHEDULING) != BPF_TW_PENDING) {
 		bpf_task_work_ctx_put(ctx);
@@ -4251,9 +4259,9 @@ static void bpf_task_work_irq(struct irq_work *irq_work)
 	/*
 	 * It's technically possible for just scheduled task_work callback to
 	 * complete running by now, going SCHEDULING -> RUNNING and then
-	 * dropping its ctx refcount. Instead of capturing extra ref just to
-	 * protected below ctx->state access, we rely on RCU protection to
-	 * perform below SCHEDULING -> SCHEDULED attempt.
+	 * dropping its ctx refcount. Instead of capturing an extra ref just
+	 * to protect below ctx->state access, we rely on rcu_read_lock
+	 * above to prevent kfree_rcu from freeing ctx before we return.
 	 */
 	state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_SCHEDULED);
 	if (state == BPF_TW_FREED)
@@ -4270,7 +4278,7 @@ static struct bpf_task_work_ctx *bpf_task_work_fetch_ctx(struct bpf_task_work *t
 	if (ctx)
 		return ctx;
 
-	ctx = bpf_mem_alloc(&bpf_global_ma, sizeof(struct bpf_task_work_ctx));
+	ctx = bpf_map_kmalloc_nolock(map, sizeof(*ctx), 0, NUMA_NO_NODE);
 	if (!ctx)
 		return ERR_PTR(-ENOMEM);
 
@@ -4284,7 +4292,7 @@ static struct bpf_task_work_ctx *bpf_task_work_fetch_ctx(struct bpf_task_work *t
 		 * tw->ctx is set by concurrent BPF program, release allocated
 		 * memory and try to reuse already set context.
 		 */
-		bpf_mem_free(&bpf_global_ma, ctx);
+		kfree_nolock(ctx);
 		return old_ctx;
 	}
 
@@ -4296,13 +4304,23 @@ static struct bpf_task_work_ctx *bpf_task_work_acquire_ctx(struct bpf_task_work
 {
 	struct bpf_task_work_ctx *ctx;
 
-	ctx = bpf_task_work_fetch_ctx(tw, map);
-	if (IS_ERR(ctx))
-		return ctx;
-
-	/* try to get ref for task_work callback to hold */
-	if (!bpf_task_work_ctx_tryget(ctx))
-		return ERR_PTR(-EBUSY);
+	/*
+	 * Sleepable BPF programs hold rcu_read_lock_trace but not
+	 * regular rcu_read_lock. Since kfree_rcu waits for regular
+	 * RCU GP, the ctx can be freed while we're between reading
+	 * the pointer and incrementing the refcount. Take regular
+	 * rcu_read_lock to prevent kfree_rcu from freeing the ctx
+	 * before we can tryget it.
+	 */
+	scoped_guard(rcu) {
+		ctx = bpf_task_work_fetch_ctx(tw, map);
+		if (IS_ERR(ctx))
+			return ctx;
+
+		/* try to get ref for task_work callback to hold */
+		if (!bpf_task_work_ctx_tryget(ctx))
+			return ERR_PTR(-EBUSY);
+	}
 
 	if (cmpxchg(&ctx->state, BPF_TW_STANDBY, BPF_TW_PENDING) != BPF_TW_STANDBY) {
 		/* lost acquiring race or map_release_uref() stole it from us, put ref and bail */
Signature · 1 line
2.52.0
#2 lore raw

Re: [PATCH bpf-next v2 1/2] bpf: Migrate bpf_task_work to kmalloc_nolock

Quoted 6 lines · Mykyta Yatsenko

From: Mykyta Yatsenko <[email protected]>

Replace bpf_mem_alloc/bpf_mem_free with
kmalloc_nolock/kfree_rcu for bpf_task_work_ctx.

Quoted 6 lines

Replace guard(rcu_tasks_trace)() with guard(rcu)() in bpf_task_work_irq(). The function only accesses ctx struct members (not map values), so tasks trace protection is not needed - regular RCU is sufficient since ctx is freed via kfree_rcu. The guard in bpf_task_work_callback() remains as tasks trace since it accesses map values from process context.

I didn't quite get if this change was necessary for correctness or it's just an optimization?

Quoted 19 lines

Sleepable BPF programs hold rcu_read_lock_trace but not regular rcu_read_lock. Since kfree_rcu
waits for a regular RCU grace period, the ctx memory can be freed while a sleepable program is still running. Add scoped_guard(rcu) around the pointer read and refcount tryget in bpf_task_work_acquire_ctx to close this race window.

Since kfree_rcu uses call_rcu internally which is not safe from NMI context, defer destruction via irq_work when irqs are disabled.

For the lost-cmpxchg path the ctx was never published, so kfree_nolock is safe.

---
 kernel/bpf/helpers.c | 56 ++++++++++++++++++++++++++++++++++------------------
 1 file changed, 37 insertions(+), 19 deletions(-)

[...]

#3 lore raw

Re: [PATCH bpf-next v2 1/2] bpf: Migrate bpf_task_work to kmalloc_nolock

Quoted 15 lines · Andrii Nakryiko
Quoted 6 lines · Mykyta Yatsenko

From: Mykyta Yatsenko <[email protected]>

Replace bpf_mem_alloc/bpf_mem_free with
kmalloc_nolock/kfree_rcu for bpf_task_work_ctx.

Quoted 6 lines

Replace guard(rcu_tasks_trace)() with guard(rcu)() in bpf_task_work_irq(). The function only accesses ctx struct members (not map values), so tasks trace protection is not needed - regular RCU is sufficient since ctx is freed via kfree_rcu. The guard in bpf_task_work_callback() remains as tasks trace since it accesses map values from process context.

Quoted 3 lines

I didn't quite get if this change was necessary for correctness or it's just an optimization?

Correctness - ctx is freed via kfree_rcu(), so we need to hold rcu read lock when we pass refcnt to the task_work_add() callback. It worked before on tasks trace rcu because bpf_mem_alloc() used it (with normal rcu chaining) before freeing the ctx.

Quoted 21 lines
Quoted 19 lines

Sleepable BPF programs hold rcu_read_lock_trace but not regular rcu_read_lock. Since kfree_rcu
waits for a regular RCU grace period, the ctx memory can be freed while a sleepable program is still running. Add scoped_guard(rcu) around the pointer read and refcount tryget in bpf_task_work_acquire_ctx to close this race window.

Since kfree_rcu uses call_rcu internally which is not safe from NMI context, defer destruction via irq_work when irqs are disabled.

For the lost-cmpxchg path the ctx was never published, so kfree_nolock is safe.

---
  kernel/bpf/helpers.c | 56 ++++++++++++++++++++++++++++++++++------------------
  1 file changed, 37 insertions(+), 19 deletions(-)

[...]

maintainer Acked-by on v2 1/2
#4 lore raw

Re: [PATCH bpf-next v2 1/2] bpf: Migrate bpf_task_work to kmalloc_nolock

In reply to Mykyta Yatsenko
Quoted 20 lines · Mykyta Yatsenko

From: Mykyta Yatsenko <[email protected]>

Replace bpf_mem_alloc/bpf_mem_free with
kmalloc_nolock/kfree_rcu for bpf_task_work_ctx.

Replace guard(rcu_tasks_trace)() with guard(rcu)() in bpf_task_work_irq(). The function only accesses ctx struct members (not map values), so tasks trace protection is not needed - regular RCU is sufficient since ctx is freed via kfree_rcu. The guard in bpf_task_work_callback() remains as tasks trace since it accesses map values from process context.

Sleepable BPF programs hold rcu_read_lock_trace but not regular rcu_read_lock. Since kfree_rcu
waits for a regular RCU grace period, the ctx memory can be freed while a sleepable program is still running. Add scoped_guard(rcu) around the pointer read and refcount tryget in bpf_task_work_acquire_ctx to close this race window.

Quoted 8 lines

Since kfree_rcu uses call_rcu internally which is not safe from NMI context, defer destruction via irq_work when irqs are disabled.

For the lost-cmpxchg path the ctx was never published, so kfree_nolock is safe.

---

Quoted 1 line

[...]

#5 lore raw

PATCH v2 2/2 bpf: Migrate dynptr file to kmalloc_nolock

In reply to Mykyta Yatsenko

From: Mykyta Yatsenko <[email protected]>

Replace bpf_mem_alloc/bpf_mem_free with kmalloc_nolock/kfree_nolock for bpf_dynptr_file_impl, continuing the migration away from bpf_mem_alloc now that kmalloc can be used from NMI context.

freader_cleanup() runs before kfree_nolock() while the dynptr still holds exclusive access, so plain kfree_nolock() is safe — no concurrent readers can access the object.

---
 kernel/bpf/helpers.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/bpf/helpers.c +2 -2
index 4c3011ef631f..7bb8b1339e2f 100644
@@ -4435,7 +4435,7 @@ static int make_file_dynptr(struct file *file, u32 flags, bool may_sleep,
 		return -EINVAL;
 	}
 
-	state = bpf_mem_alloc(&bpf_global_ma, sizeof(struct bpf_dynptr_file_impl));
+	state = kmalloc_nolock(sizeof(*state), 0, NUMA_NO_NODE);
 	if (!state) {
 		bpf_dynptr_set_null(ptr);
 		return -ENOMEM;
@@ -4467,7 +4467,7 @@ __bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)
 		return 0;
 
 	freader_cleanup(&df->freader);
-	bpf_mem_free(&bpf_global_ma, df);
+	kfree_nolock(df);
 	bpf_dynptr_set_null(ptr);
 	return 0;
 }
Signature · 1 line
2.52.0
Acked-by on v2 2/2
#6 lore raw

Re: [PATCH bpf-next v2 2/2] bpf: Migrate dynptr file to kmalloc_nolock

Quoted 7 lines · Mykyta Yatsenko

From: Mykyta Yatsenko <[email protected]>

Replace bpf_mem_alloc/bpf_mem_free with kmalloc_nolock/kfree_nolock for bpf_dynptr_file_impl, continuing the migration away from bpf_mem_alloc now that kmalloc can be used from NMI context.

Quoted 9 lines

freader_cleanup() runs before kfree_nolock() while the dynptr still holds exclusive access, so plain kfree_nolock() is safe — no concurrent readers can access the object.

---
 kernel/bpf/helpers.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

LGTM

Quoted 26 lines
kernel/bpf/helpers.c +2 -3
index 4c3011ef631f..7bb8b1339e2f 100644
@@ -4435,7 +4435,7 @@ static int make_file_dynptr(struct file *file, u32 flags, bool may_sleep,
                return -EINVAL;
        }
 
-       state = bpf_mem_alloc(&bpf_global_ma, sizeof(struct bpf_dynptr_file_impl));
+       state = kmalloc_nolock(sizeof(*state), 0, NUMA_NO_NODE);
        if (!state) {
                bpf_dynptr_set_null(ptr);
                return -ENOMEM;
@@ -4467,7 +4467,7 @@ __bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)
                return 0;
 
        freader_cleanup(&df->freader);
-       bpf_mem_free(&bpf_global_ma, df);
+       kfree_nolock(df);
        bpf_dynptr_set_null(ptr);
        return 0;
 }
 
--

2.52.0

maintainer Acked-by on v2 2/2
#7 lore raw

Re: [PATCH bpf-next v2 2/2] bpf: Migrate dynptr file to kmalloc_nolock

In reply to Mykyta Yatsenko
Quoted 7 lines · Mykyta Yatsenko

From: Mykyta Yatsenko <[email protected]>

Replace bpf_mem_alloc/bpf_mem_free with kmalloc_nolock/kfree_nolock for bpf_dynptr_file_impl, continuing the migration away from bpf_mem_alloc now that kmalloc can be used from NMI context.

Quoted 6 lines

freader_cleanup() runs before kfree_nolock() while the dynptr still holds exclusive access, so plain kfree_nolock() is safe — no concurrent readers can access the object.

---

Quoted 1 line

[...]

#8 lore raw
In reply to Mykyta Yatsenko
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Quoted 12 lines · On Mon, 30 Mar 2026 15:27:55 -0700 you wrote:

Now that kmalloc can be used from NMI context via kmalloc_nolock(), migrate BPF internal allocations away from bpf_mem_alloc to use the standard slab allocator.

Use kfree_rcu() for deferred freeing, which waits for a regular RCU grace period before the memory is reclaimed. Sleepable BPF programs hold rcu_read_lock_trace but not regular rcu_read_lock, so patch 1 adds explicit rcu_read_lock/unlock around the pointer-to-refcount window to prevent kfree_rcu from freeing memory while a sleepable program is still between reading the pointer and acquiring a reference.

Quoted 1 line

[...]

Here is the summary with links:
- [bpf-next,v2,1/2] bpf: Migrate bpf_task_work to kmalloc_nolock

    https://git.kernel.org/bpf/bpf-next/c/90f51ebff242
  - [bpf-next,v2,2/2] bpf: Migrate dynptr file to kmalloc_nolock
    https://git.kernel.org/bpf/bpf-next/c/cc878b414450

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html