bpf: Avoid locks in bpf_timer and bpf_wq
Subsystems (4)
v5 cover bpf: Avoid locks in bpf_timer and bpf_wq
This series reworks implementation of BPF timer and workqueue APIs. The goal is to make both timers and wq non-blocking, enabling their use in NMI context.
Today this code relies on a bpf_spin_lock embedded in the map element to serialize:
* init of the async object,
* setting/changing the callback and bpf_prog
* starting/cancelling the timer/work
* tearing down when the map element is deleted or the map’s user ref is dropped
The basic design approach in this series:
* Use irq_work to offload all blocking work from NMI
* Introduce refcount to guarantee lifetime of the bpf_async_cb structs deferred to potentially multiple irq_work callbacks
* Keep objects under RCU protection to make sure they are not freed while kfuncs/helpers access them (We can't use refcnt for this, as refcnt itself is part of the bpf_async_cb struct)
---
Changes in v5:
- Extracted lock-free algorithm for updating cb->prog andcb->callback_fn into a function bpf_async_update_prog_callback(),
added a new commit and introduces this function and uses it in __bpf_async_set_callback(), bpf_timer_cancel() and bpf_async_cancel_and_free().
This allows to move the change into the separate commit without breaking correctness.
- Handle NULL prog in bpf_async_update_prog_callback().
- Link to v4: https://lore.kernel.org/r/[email protected]
Changes in v4:
- Handle irq_work_queue failures in both schedule and cancel_and_free
paths: introduced bpf_async_refcnt_dec_cleanup() that decrements refcnt and makes sure if last reference is put, there is at least one irq_work scheduled to execute final cleanup.
- Additional refcnt inc/dec in set_callback() + rcu lock to make sure cleanup is not running at the same time as set_callback().
- Added READ_ONCE where it was needed.
- Squash 'bpf: Refactor __bpf_async_set_callback()' commit into 'bpf:
Add lock-free cell for NMI-safe
async operations'
- Removed mpmc_cell, use seqcount_latch_t instead.
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- Major rework
- Introduce mpmc_cell, allowing concurrent writes and reads
- Implement irq_work deferring
- Adding selftests
- Introduces bpf_timer_cancel_async kfunc
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Move refcnt initialization and put (from cancel_and_free())
from patch 5 into the patch 4, so that patch 4 has more clear and full implementation and use of refcnt
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Mykyta Yatsenko (10):
bpf: Factor out timer deletion helper
bpf: Remove unnecessary arguments from bpf_async_set_callback()
bpf: Introduce lock-free bpf_async_update_prog_callback()
bpf: Simplify bpf_timer_cancel()
bpf: Enable bpf timer and workqueue use in NMI
bpf: Add verifier support for bpf_timer argument in kfuncs
bpf: Introduce bpf_timer_cancel_async() kfunc selftests/bpf: Refactor timer selftests selftests/bpf: Add stress test for timer async cancel selftests/bpf: Verify bpf_timer_cancel_async works
kernel/bpf/helpers.c | 455 ++++++++++++++++--------- kernel/bpf/verifier.c | 59 +++- tools/testing/selftests/bpf/prog_tests/timer.c | 92 ++++- tools/testing/selftests/bpf/progs/timer.c | 37 +- 4 files changed, 458 insertions(+), 185 deletions(-)
---
base-commit: 46c76760febfb14618d88f6a01fca2d93d003082
change-id: 20251028-timer_nolock-457f5b9daace
Best regards,
Signature · 1 line
Mykyta Yatsenko <[email protected]>
RFC PATCH v5 1/10 bpf: Factor out timer deletion helper
From: Mykyta Yatsenko <[email protected]>
Move the timer deletion logic into a dedicated bpf_timer_delete() helper so it can be reused by later patches.
--- kernel/bpf/helpers.c | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-)
kernel/bpf/helpers.c +18 -11
@@ -1558,18 +1558,10 @@ static struct bpf_async_cb *__bpf_async_cancel_and_free(struct bpf_async_kern *areturn cb; } -/* This function is called by map_delete/update_elem for individual element and -* by ops->map_release_uref when the user space reference to a map reaches zero. -*/ -void bpf_timer_cancel_and_free (void *val) +static void bpf_timer_delete (struct bpf_hrtimer *t) { -struct bpf_hrtimer *t; - - t = (struct bpf_hrtimer *)__bpf_async_cancel_and_free (val); - -if (!t) -return ; -/* We check that bpf_map_delete/update_elem() was called from timer +/* +* We check that bpf_map_delete/update_elem() was called from timer * callback_fn. In such case we don't call hrtimer_cancel() (since it * will deadlock) and don't call hrtimer_try_to_cancel() (since it will * just return -1). Though callback_fn is still running on this cpu it's
@@ -1618,6 +1610,21 @@ void bpf_timer_cancel_and_free(void *val) } } +/* +* This function is called by map_delete/update_elem for individual element and +* by ops->map_release_uref when the user space reference to a map reaches zero. +*/ +void bpf_timer_cancel_and_free (void *val) +{ +struct bpf_hrtimer *t; + + t = (struct bpf_hrtimer *)__bpf_async_cancel_and_free (val); +if (!t) +return ; + +bpf_timer_delete (t); +} +/* This function is called by map_delete/update_elem for individual element and * by ops->map_release_uref when the user space reference to a map reaches zero. */
Signature · 1 line
2.52.0
RFC PATCH v5 2/10 bpf: Remove unnecessary arguments from bpf_async_set_callback()
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Remove unused arguments from __bpf_async_set_callback().
--- kernel/bpf/helpers.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
kernel/bpf/helpers.c +4 -5
@@ -1355,10 +1355,9 @@ static const struct bpf_func_proto bpf_timer_init_proto = { };static int __bpf_async_set_callback (struct bpf_async_kern *async,void *callback_fn, -struct bpf_prog_aux *aux,unsigned int flags, -enum bpf_async_type type) +struct bpf_prog *prog) { -struct bpf_prog *prev, *prog = aux->prog; +struct bpf_prog *prev;struct bpf_async_cb *cb;int ret =0 ;
@@ -1403,7 +1402,7 @@ static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn, struct bpf_prog_aux *, aux) { -return __bpf_async_set_callback (timer, callback_fn, aux,0 ,BPF_ASYNC_TYPE_TIMER ); +return __bpf_async_set_callback (timer, callback_fn, aux->prog); }static const struct bpf_func_proto bpf_timer_set_callback_proto = {
@@ -3138,7 +3137,7 @@ __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,if (flags)return -EINVAL ; -return __bpf_async_set_callback (async, callback_fn, aux, flags,BPF_ASYNC_TYPE_WQ ); +return __bpf_async_set_callback (async, callback_fn, aux->prog); } __bpf_kfunc void bpf_preempt_disable(void)
Signature · 1 line
2.52.0
RFC PATCH v5 3/10 bpf: Introduce lock-free bpf_async_update_prog_callback()
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Introduce bpf_async_update_prog_callback(): lock-free update of cb->prog and cb->callback_fn. This function allows updating prog and callback_fn fields of the struct bpf_async_cb without holding lock.
For now use it under the lock from __bpf_async_set_callback(), in the next patches that lock will be removed.
Lock-free algorithm:
* Acquire a guard reference on prog to prevent it from being freed during the retry loop.
* Retry loop:
1. Each iteration acquires a new prog reference and stores it in cb->prog via xchg. The previous prog is released.
2. The loop condition checks if both cb->prog and cb->callback_fn match what we just wrote. If either differs, a concurrent writer overwrote our value, and we must retry.
3. When we retry, our previously-stored prog was already released by the concurrent writer or will be released by us after overwriting.
* Release guard reference.
--- kernel/bpf/helpers.c | 67 +++++++++++++++++++++++++++++----------------------- 1 file changed, 37 insertions(+), 30 deletions(-)
kernel/bpf/helpers.c +37 -30
@@ -1354,10 +1354,43 @@ static const struct bpf_func_proto bpf_timer_init_proto = { .arg3_type = ARG_ANYTHING, }; +static int bpf_async_update_prog_callback (struct bpf_async_cb *cb,void *callback_fn, +struct bpf_prog *prog) +{ +struct bpf_prog *prev; + +/* Acquire a guard reference on prog to prevent it from being freed during the loop */ +if (prog) { + prog =bpf_prog_inc_not_zero (prog); +if (IS_ERR (prog)) +return PTR_ERR (prog); + } + +do { +if (prog) + prog =bpf_prog_inc_not_zero (prog); + prev =xchg (&cb->prog, prog); +rcu_assign_pointer (cb->callback_fn, callback_fn); + +/* +* Release previous prog, make sure that if other CPU is contending, +* to set bpf_prog, references are not leaked as each iteration acquires and +* releases one reference. +*/ +if (prev) +bpf_prog_put (prev); + + }while (READ_ONCE (cb->prog) != prog ||READ_ONCE (cb->callback_fn) != callback_fn); + +if (prog) +bpf_prog_put (prog); + +return 0 ; +} +static int __bpf_async_set_callback (struct bpf_async_kern *async,void *callback_fn,struct bpf_prog *prog) { -struct bpf_prog *prev;struct bpf_async_cb *cb;int ret =0 ;
@@ -1378,22 +1411,7 @@ static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback ret = -EPERM ; goto out; } - prev = cb->prog; -if (prev != prog) { -/* Bump prog refcnt once. Every bpf_timer_set_callback() -* can pick different callback_fn-s within the same prog. -*/ - prog =bpf_prog_inc_not_zero (prog); -if (IS_ERR (prog)) { - ret =PTR_ERR (prog); - goto out; - } -if (prev) -/* Drop prev prog refcnt when swapping with new prog */ -bpf_prog_put (prev); - cb->prog = prog; - } -rcu_assign_pointer (cb->callback_fn, callback_fn); + ret =bpf_async_update_prog_callback (cb, callback_fn, prog); out:__bpf_spin_unlock_irqrestore (&async->lock);return ret;
@@ -1453,17 +1471,6 @@ static const struct bpf_func_proto bpf_timer_start_proto = { .arg3_type = ARG_ANYTHING, }; -static void drop_prog_refcnt (struct bpf_async_cb *async) -{ -struct bpf_prog *prog = async->prog; - -if (prog) { -bpf_prog_put (prog); - async->prog =NULL ; -rcu_assign_pointer (async->callback_fn,NULL ); - } -} - BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, timer) {struct bpf_hrtimer *t, *cur_t;
@@ -1514,7 +1521,7 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, timer) goto out; } drop: -drop_prog_refcnt (&t->cb); +bpf_async_update_prog_callback (&t->cb,NULL ,NULL ); out:__bpf_spin_unlock_irqrestore (&timer->lock);/* Cancel the timer and wait for associated callback to finish
@@ -1547,7 +1554,7 @@ static struct bpf_async_cb *__bpf_async_cancel_and_free(struct bpf_async_kern *a cb = async->cb;if (!cb) goto out; -drop_prog_refcnt (cb); +bpf_async_update_prog_callback (cb,NULL ,NULL );/* The subsequent bpf_timer_start/cancel() helpers won't be able to use * this timer, since it won't be initialized. */
Signature · 1 line
2.52.0
Re: [PATCH RFC v5 03/10] bpf: Introduce lock-free bpf_async_update_prog_callback()
Quoted 23 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
Introduce
bpf_async_update_prog_callback(): lock-free update ofcb->progandcb->callback_fn. This function allows updating prog andcallback_fnfields of the structbpf_async_cbwithout holding lock.
For now use it under the lock from__bpf_async_set_callback(), in the next patches that lock will be removed.Lock-free algorithm:
* Acquire a guard reference on prog to prevent it from being freed during the retry loop.
* Retry loop:
1. Each iteration acquires a new prog reference and stores it incb->progvia xchg. The previous prog is released.
2. The loop condition checks if bothcb->progandcb->callback_fnmatch what we just wrote. If either differs, a concurrent writer overwrote our value, and we must retry.
3. When we retry, our previously-stored prog was already released by the concurrent writer or will be released by us after overwriting.
* Release guard reference.
Quoted 5 lines
Signed-off-by Mykyta Yatsenko <[email protected]>--- kernel/bpf/helpers.c | 67 +++++++++++++++++++++++++++++----------------------- 1 file changed, 37 insertions(+), 30 deletions(-)
LGTM
[...]
RFC PATCH v5 4/10 bpf: Simplify bpf_timer_cancel()
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Remove lock from the bpf_timer_cancel() helper. The lock does not protect from concurrent modification of the bpf_async_cb data fields as those are modified in the callback without locking.
Use guard(rcu)() instead of pair of explicit lock()/unlock().
--- kernel/bpf/helpers.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-)
kernel/bpf/helpers.c +11 -16
@@ -1471,7 +1471,7 @@ static const struct bpf_func_proto bpf_timer_start_proto = { .arg3_type = ARG_ANYTHING, }; -BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, timer) +BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, async) {struct bpf_hrtimer *t, *cur_t;bool inc = false;
@@ -1479,13 +1479,12 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, timer)if (in_nmi ())return -EOPNOTSUPP ; -rcu_read_lock (); -__bpf_spin_lock_irqsave (&timer->lock); - t = timer->timer; -if (!t) { - ret = -EINVAL ; - goto out; - } + +guard (rcu)(); + + t =READ_ONCE (async->timer); +if (!t) +return -EINVAL ; cur_t =this_cpu_read (hrtimer_running);if (cur_t == t) {
@@ -1493,8 +1492,7 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, timer)* its own timer the hrtimer_cancel() will deadlock * since it waits for callback_fn to finish. */ - ret = -EDEADLK ; - goto out; +return -EDEADLK ; }/* Only account in-flight cancellations when invoked from a timer
@@ -1517,20 +1515,17 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, timer)* cancelling and waiting for it synchronously, since it might * do the same. Bail! */ - ret = -EDEADLK ; - goto out; +atomic_dec (&t->cancelling); +return -EDEADLK ; } drop:bpf_async_update_prog_callback (&t->cb,NULL ,NULL ); -out: -__bpf_spin_unlock_irqrestore (&timer->lock);/* Cancel the timer and wait for associated callback to finish * if it was running. */ - ret = ret ?:hrtimer_cancel (&t->timer); + ret =hrtimer_cancel (&t->timer);if (inc)atomic_dec (&t->cancelling); -rcu_read_unlock ();return ret; }
Signature · 1 line
2.52.0
Re: [PATCH RFC v5 04/10] bpf: Simplify bpf_timer_cancel()
Quoted 7 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
Remove lock from the
bpf_timer_cancel()helper. The lock does not protect from concurrent modification of thebpf_async_cbdata fields as those are modified in the callback without locking.
Quoted 8 lines
Use guard(rcu)() instead of pair of explicit lock()/unlock().
Signed-off-by Mykyta Yatsenko <[email protected]>--- kernel/bpf/helpers.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-)
LGTM
[...]
RFC PATCH v5 5/10 bpf: Enable bpf timer and workqueue use in NMI
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Refactor bpf timer and workqueue helpers to allow calling them from NMI context by making all operations lock-free and deferring NMI-unsafe work to irq_work.
Previously, bpf_timer_start(), and bpf_wq_start()
could not be called from NMI context because they acquired bpf_spin_lock and called hrtimer/schedule_work APIs directly. This patch removes these limitations.
Key changes:
* Remove bpf_spin_lock from struct bpf_async_kern. Replace locked operations with atomic cmpxchg() for initialization and xchg() for cancel and free.
* Add per-async irq_work to defer NMI-unsafe operations (hrtimer_start, hrtimer_try_to_cancel, schedule_work) from NMI to softirq context.
* Use the lock-free seqcount_latch_t to pass operation commands (start/cancel/free) along with their parameters (nsec, mode) from NMI-safe callers to the irq_work handler.
* Add reference counting to bpf_async_cb to ensure the object stays alive until all scheduled irq_work completes and the timer/work callback finishes.
* Move bpf_prog_put() to RCU callback to handle races between set_callback() and cancel_and_free().
* Refactor __bpf_async_set_callback() getting rid of locks. Each iteration acquires a new reference and stores it
in cb->prog via xchg. The previous value is retrieved and released. The loop condition checks if both cb->prog and cb->callback_fn match what we just wrote. If either differs, a concurrent writer overwrote our value, and we must retry.
When we retry, our previously-stored prog was already put() by the concurrent writer or we put() it after xchg().
This enables BPF programs attached to NMI-context hooks (perf events) to use timers and workqueues for deferred processing.
--- kernel/bpf/helpers.c | 335 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 225 insertions(+), 110 deletions(-)
kernel/bpf/helpers.c +225 -110
@@ -29,6 +29,7 @@#include <linux/task_work.h> #include <linux/irq_work.h> #include <linux/buildid.h> +#include <linux/seqlock.h> #include "../../lib/kstrtox.h"
@@ -1095,6 +1096,23 @@ static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)return (void *)value -round_up (map->key_size,8 ); } +enum bpf_async_type { +BPF_ASYNC_TYPE_TIMER =0 , +BPF_ASYNC_TYPE_WQ , +}; + +enum bpf_async_op { +BPF_ASYNC_START , +BPF_ASYNC_CANCEL , +BPF_ASYNC_CANCEL_AND_FREE , +}; + +struct bpf_async_cmd { +u64 nsec; +u32 mode; +u32 op; +}; +struct bpf_async_cb {struct bpf_map *map;struct bpf_prog *prog;
@@ -1105,6 +1123,13 @@ struct bpf_async_cb {struct work_struct delete_work; };u64 flags; +struct irq_work worker; +atomic_t writer; +seqcount_latch_t latch; +struct bpf_async_cmd cmd[2 ]; +atomic_t last_seq; +refcount_t refcnt; +enum bpf_async_type type; };/* BPF map elements can contain 'struct bpf_timer'.
@@ -1142,18 +1167,9 @@ struct bpf_async_kern {struct bpf_hrtimer *timer;struct bpf_work *work; }; -/* bpf_spin_lock is used here instead of spinlock_t to make -* sure that it always fits into space reserved by struct bpf_timer -* regardless of LOCKDEP and spinlock debug flags. -*/ -struct bpf_spin_lock lock; +u32 __opaque; } __attribute__((aligned(8))); -enum bpf_async_type { -BPF_ASYNC_TYPE_TIMER =0 , -BPF_ASYNC_TYPE_WQ , -}; - static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running);static enum hrtimer_restart bpf_timer_cb (struct hrtimer *hrtimer)
@@ -1219,6 +1235,13 @@ static void bpf_async_cb_rcu_free(struct rcu_head *rcu) { struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu); +/* +* Drop the last reference to prog only after RCU GP, as set_callback() +* may race with cancel_and_free() +*/ +if (cb->prog) +bpf_prog_put (cb->prog); +kfree_nolock (cb); }
@@ -1246,18 +1269,17 @@ static void bpf_timer_delete_work(struct work_struct *work)call_rcu (&t->cb.rcu, bpf_async_cb_rcu_free); } +static void __bpf_async_cancel_and_free (struct bpf_async_kern *async); +static void bpf_async_irq_worker (struct irq_work *work); +static int __bpf_async_init (struct bpf_async_kern *async,struct bpf_map *map,u64 flags,enum bpf_async_type type) { -struct bpf_async_cb *cb; +struct bpf_async_cb *cb, *old_cb;struct bpf_hrtimer *t;struct bpf_work *w;clockid_t clockid;size_t size; -int ret =0 ; - -if (in_nmi ()) -return -EOPNOTSUPP ;switch (type) {case BPF_ASYNC_TYPE_TIMER :
@@ -1270,18 +1292,13 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, ureturn -EINVAL ; } -__bpf_spin_lock_irqsave (&async->lock); - t = async->timer; -if (t) { - ret = -EBUSY ; - goto out; - } + old_cb =READ_ONCE (async->cb); +if (old_cb) +return -EBUSY ; cb =bpf_map_kmalloc_nolock (map, size,0 , map->numa_node); -if (!cb) { - ret = -ENOMEM ; - goto out; - } +if (!cb) +return -ENOMEM ;switch (type) {case BPF_ASYNC_TYPE_TIMER :
@@ -1304,9 +1321,20 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u cb->map = map; cb->prog =NULL ; cb->flags = flags; + cb->worker =IRQ_WORK_INIT (bpf_async_irq_worker); +seqcount_latch_init (&cb->latch); +atomic_set (&cb->writer,0 ); +refcount_set (&cb->refcnt,1 );/* map's reference */ +atomic_set (&cb->last_seq,0 ); + cb->type = type;rcu_assign_pointer (cb->callback_fn,NULL ); -WRITE_ONCE (async->cb, cb); + old_cb =cmpxchg (&async->cb,NULL , cb); +if (old_cb) { +/* Lost the race to initialize this bpf_async_kern, drop the allocated object */ +kfree_nolock (cb); +return -EBUSY ; + }/* Guarantee the order between async->cb and map->usercnt. So * when there are concurrent uref release and bpf timer init, either * bpf_timer_cancel_and_free() called by uref release reads a no-NULL
@@ -1317,13 +1345,11 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u/* maps with timers must be either held by user space * or pinned in bpffs. */ -WRITE_ONCE (async->cb,NULL ); -kfree_nolock (cb); - ret = -EPERM ; +__bpf_async_cancel_and_free (async); +return -EPERM ; } -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return ret; + +return 0 ; } BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map,
@@ -1388,33 +1414,95 @@ static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbacreturn 0 ; } +/* Decrements bpf_async_cb refcnt, if it becomes 0 schedule cleanup irq_work */ +static void bpf_async_refcnt_dec_cleanup (struct bpf_async_cb *cb) +{ +if (!refcount_dec_and_test (&cb->refcnt)) +return ; + +/* +* At this point we took the last reference +* Try to schedule cleanup, either: +* - Set ref to 1 and succeed irq_work_queue +* - See non-zero refcnt after decrement - other irq_work is going to cleanup +*/ +do { +refcount_set (&cb->refcnt,1 ); +if (irq_work_queue (&cb->worker)) +break ; + }while (refcount_dec_and_test (&cb->refcnt)); +} + +static int bpf_async_schedule_op (struct bpf_async_cb *cb,u32 op,u64 nsec,u32 timer_mode) +{ +/* Acquire active writer atomic*/ +if (atomic_cmpxchg_acquire (&cb->writer,0 ,1 )) +return -EBUSY ; + +write_seqcount_latch_begin (&cb->latch); + cb->cmd[0 ].nsec = nsec; + cb->cmd[0 ].mode = timer_mode; + cb->cmd[0 ].op = op; +write_seqcount_latch (&cb->latch); + cb->cmd[1 ].nsec = nsec; + cb->cmd[1 ].mode = timer_mode; + cb->cmd[1 ].op = op; +write_seqcount_latch_end (&cb->latch); +atomic_set (&cb->writer,0 ); + +if (!refcount_inc_not_zero (&cb->refcnt)) +return -EBUSY ; + +/* TODO: Run operation without irq_work if not in NMI */ +if (!irq_work_queue (&cb->worker)) +/* irq_work is already scheduled on this CPU */ +bpf_async_refcnt_dec_cleanup (cb); + +return 0 ; +} + +static int bpf_async_read_op (struct bpf_async_cb *cb,enum bpf_async_op *op, +u64 *nsec,u32 *flags) +{ +u32 seq, last_seq, idx; + +while (true) { + last_seq =atomic_read_acquire (&cb->last_seq); + + seq =raw_read_seqcount_latch (&cb->latch); + +/* Return -EBUSY if current seq is consumed by another reader */ +if (seq == last_seq) +return -EBUSY ; + + idx = seq &1 ; + *nsec = cb->cmd[idx].nsec; + *flags = cb->cmd[idx].mode; + *op = cb->cmd[idx].op; + +if (raw_read_seqcount_latch_retry (&cb->latch, seq)) +continue ; +/* Commit read sequence number, own snapshot exclusively */ +if (atomic_cmpxchg_release (&cb->last_seq, last_seq, seq) == last_seq) +break ; + } + +return 0 ; +} +static int __bpf_async_set_callback (struct bpf_async_kern *async,void *callback_fn,struct bpf_prog *prog) {struct bpf_async_cb *cb; -int ret =0 ; -if (in_nmi ()) -return -EOPNOTSUPP ; -__bpf_spin_lock_irqsave (&async->lock); - cb = async->cb; -if (!cb) { - ret = -EINVAL ; - goto out; - } -if (!atomic64_read (&cb->map->usercnt)) { -/* maps with timers must be either held by user space -* or pinned in bpffs. Otherwise timer might still be -* running even when bpf prog is detached and user space -* is gone, since map_release_uref won't ever be called. -*/ - ret = -EPERM ; - goto out; - } - ret =bpf_async_update_prog_callback (cb, callback_fn, prog); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return ret; +/* Make sure bpf_async_cb_rcu_free() is not called while here */ +guard (rcu)(); + + cb =READ_ONCE (async->cb); +if (!cb) +return -EINVAL ; + +return bpf_async_update_prog_callback (cb, callback_fn, prog); } BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn,
@@ -1431,22 +1519,19 @@ static const struct bpf_func_proto bpf_timer_set_callback_proto = { .arg2_type = ARG_PTR_TO_FUNC, }; -BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, timer, u64, nsecs, u64, flags) +BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, async, u64, nsecs, u64, flags) {struct bpf_hrtimer *t; -int ret =0 ; -enum hrtimer_mode mode; +u32 mode; -if (in_nmi ()) -return -EOPNOTSUPP ;if (flags & ~(BPF_F_TIMER_ABS |BPF_F_TIMER_CPU_PIN ))return -EINVAL ; -__bpf_spin_lock_irqsave (&timer->lock); - t = timer->timer; -if (!t || !t->cb.prog) { - ret = -EINVAL ; - goto out; - } + +guard (rcu)(); + + t =READ_ONCE (async->timer); +if (!t || !READ_ONCE (t->cb.prog)) +return -EINVAL ;if (flags &BPF_F_TIMER_ABS ) mode =HRTIMER_MODE_ABS_SOFT ;
@@ -1456,10 +1541,7 @@ BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, timer, u64, nsecs, u64, flaif (flags &BPF_F_TIMER_CPU_PIN ) mode |=HRTIMER_MODE_PINNED ; -hrtimer_start (&t->timer,ns_to_ktime (nsecs), mode); -out: -__bpf_spin_unlock_irqrestore (&timer->lock); -return ret; +return bpf_async_schedule_op (&t->cb,BPF_ASYNC_START , nsecs, mode); }static const struct bpf_func_proto bpf_timer_start_proto = {
@@ -1536,27 +1618,16 @@ static const struct bpf_func_proto bpf_timer_cancel_proto = { .arg1_type = ARG_PTR_TO_TIMER, }; -static struct bpf_async_cb *__bpf_async_cancel_and_free (struct bpf_async_kern *async) +static void __bpf_async_cancel_and_free (struct bpf_async_kern *async) {struct bpf_async_cb *cb; -/* Performance optimization: read async->cb without lock first. */ -if (!READ_ONCE (async->cb)) -return NULL ; - -__bpf_spin_lock_irqsave (&async->lock); -/* re-read it under lock */ - cb = async->cb; + cb =xchg (&async->cb,NULL );if (!cb) - goto out; -bpf_async_update_prog_callback (cb,NULL ,NULL ); -/* The subsequent bpf_timer_start/cancel() helpers won't be able to use -* this timer, since it won't be initialized. -*/ -WRITE_ONCE (async->cb,NULL ); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return cb; +return ; + +/* Consume map's own refcnt, schedule cleanup irq_work if this is the last ref */ +bpf_async_refcnt_dec_cleanup (cb); }static void bpf_timer_delete (struct bpf_hrtimer *t)
@@ -1611,19 +1682,76 @@ static void bpf_timer_delete(struct bpf_hrtimer *t) } } +static void bpf_async_process_op (struct bpf_async_cb *cb,u32 op, +u64 timer_nsec,u32 timer_mode) +{ +switch (cb->type) { +case BPF_ASYNC_TYPE_TIMER : { + struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb); + +switch (op) { +case BPF_ASYNC_START : +hrtimer_start (&t->timer,ns_to_ktime (timer_nsec), timer_mode); +break ; +case BPF_ASYNC_CANCEL : +hrtimer_try_to_cancel (&t->timer); +break ; +case BPF_ASYNC_CANCEL_AND_FREE : +bpf_timer_delete (t); +break ; +default : +break ; + } +break ; + } +case BPF_ASYNC_TYPE_WQ : { + struct bpf_work *w = container_of(cb, struct bpf_work, cb); + +switch (op) { +case BPF_ASYNC_START : +schedule_work (&w->work); +break ; +case BPF_ASYNC_CANCEL_AND_FREE : +/* +* Trigger cancel of the sleepable work, but *do not* wait for +* it to finish. +* kfree will be called once the work has finished. +*/ +schedule_work (&w->delete_work); +break ; +default : +break ; + } +break ; + } + } +} + +static void bpf_async_irq_worker (struct irq_work *work) +{ + struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker); +u32 op, timer_mode; +u64 nsec; +int err; + + err =bpf_async_read_op (cb, &op, &nsec, &timer_mode); +if (err) + goto out; + +bpf_async_process_op (cb, op, nsec, timer_mode); + +out: +if (refcount_dec_and_test (&cb->refcnt)) +bpf_async_process_op (cb,BPF_ASYNC_CANCEL_AND_FREE ,0 ,0 ); +} +/* * This function is called by map_delete/update_elem for individual element and * by ops->map_release_uref when the user space reference to a map reaches zero. */ void bpf_timer_cancel_and_free (void *val) { -struct bpf_hrtimer *t; - - t = (struct bpf_hrtimer *)__bpf_async_cancel_and_free (val); -if (!t) -return ; - -bpf_timer_delete (t); +__bpf_async_cancel_and_free (val); }/* This function is called by map_delete/update_elem for individual element and
@@ -1631,19 +1759,7 @@ void bpf_timer_cancel_and_free(void *val)*/ void bpf_wq_cancel_and_free (void *val) { -struct bpf_work *work; - -BTF_TYPE_EMIT (struct bpf_wq ); - - work = (struct bpf_work *)__bpf_async_cancel_and_free (val); -if (!work) -return ; -/* Trigger cancel of the sleepable work, but *do not* wait for -* it to finish if it was running as we might not be in a -* sleepable context. -* kfree will be called once the work has finished. -*/ -schedule_work (&work->delete_work); +__bpf_async_cancel_and_free (val); } BPF_CALL_2(bpf_kptr_xchg, void *, dst, void *, ptr)
@@ -3116,16 +3232,15 @@ __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)struct bpf_async_kern *async = (struct bpf_async_kern *)wq;struct bpf_work *w; -if (in_nmi ()) -return -EOPNOTSUPP ;if (flags)return -EINVAL ; + +guard (rcu)(); w =READ_ONCE (async->work);if (!w || !READ_ONCE (w->cb.prog))return -EINVAL ; -schedule_work (&w->work); -return 0 ; +return bpf_async_schedule_op (&w->cb,BPF_ASYNC_START ,0 ,0 ); } __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,
Signature · 1 line
2.52.0
Re: [PATCH RFC v5 05/10] bpf: Enable bpf timer and workqueue use in NMI
Quoted 26 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
Refactor bpf timer and workqueue helpers to allow calling them from NMI context by making all operations lock-free and deferring NMI-unsafe work to
irq_work.Previously,
bpf_timer_start(), andbpf_wq_start()
could not be called from NMI context because they acquiredbpf_spin_lockand called hrtimer/schedule_work APIs directly. This patch removes these limitations.Key changes:
* Removebpf_spin_lockfrom structbpf_async_kern. Replace locked operations with atomiccmpxchg()for initialization andxchg()for cancel and free.
* Add per-asyncirq_workto defer NMI-unsafe operations (hrtimer_start,hrtimer_try_to_cancel,schedule_work) from NMI to softirq context.
* Use the lock-freeseqcount_latch_tto pass operation commands (start/cancel/free) along with their parameters (nsec, mode) from NMI-safe callers to theirq_workhandler.
* Add reference counting tobpf_async_cbto ensure the object stays alive until all scheduledirq_workcompletes and the timer/work callback finishes.
* Movebpf_prog_put()to RCU callback to handle races betweenset_callback()andcancel_and_free().
Quoted 8 lines
* Refactor
__bpf_async_set_callback()getting rid of locks. Each iteration acquires a new reference and stores it
incb->progvia xchg. The previous value is retrieved and released. The loop condition checks if bothcb->progandcb->callback_fnmatch what we just wrote. If either differs, a concurrent writer overwrote our value, and we must retry.
When we retry, our previously-stored prog was alreadyput()by the concurrent writer or weput()it afterxchg().
you already described that in earlier patch, no need to repeat that here, IMO
Quoted 30 lines
This enables BPF programs attached to NMI-context hooks (perf events) to use timers and workqueues for deferred processing.
Signed-off-by Mykyta Yatsenko <[email protected]>--- kernel/bpf/helpers.c | 335 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 225 insertions(+), 110 deletions(-)kernel/bpf/helpers.c +6
-0@@ -29,6 +29,7 @@#include <linux/task_work.h> #include <linux/irq_work.h> #include <linux/buildid.h> +#include <linux/seqlock.h> #include "../../lib/kstrtox.h" @@ -1095,6 +1096,23 @@ static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)return (void *)value -round_up (map->key_size,8 ); } +enum bpf_async_type { +BPF_ASYNC_TYPE_TIMER =0 , +BPF_ASYNC_TYPE_WQ , +}; +
kernel/bpf/helpers.c static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx) · line 1113
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+enum bpf_async_op { +BPF_ASYNC_START , +BPF_ASYNC_CANCEL , +BPF_ASYNC_CANCEL_AND_FREE , +}; + +struct bpf_async_cmd { +u64 nsec; +u32 mode; +u32 op;
enum bpf_async_op here?
Quoted 20 lines
kernel/bpf/helpers.c +9
-0+}; +struct bpf_async_cb {struct bpf_map *map;struct bpf_prog *prog;@@ -1105,6 +1123,13 @@ struct bpf_async_cb {struct work_struct delete_work; };u64 flags; +struct irq_work worker; +atomic_t writer; +seqcount_latch_t latch; +struct bpf_async_cmd cmd[2 ]; +atomic_t last_seq; +refcount_t refcnt; +enum bpf_async_type type; };/* BPF map elements can contain 'struct bpf_timer'.
kernel/bpf/helpers.c struct bpf_async_kern { · line 1170
Quoted 9 lines
kernel/bpf/helpers.c +1
-5@@ -1142,18 +1167,9 @@ struct bpf_async_kern {struct bpf_hrtimer *timer;struct bpf_work *work; }; -/* bpf_spin_lock is used here instead of spinlock_t to make -* sure that it always fits into space reserved by struct bpf_timer -* regardless of LOCKDEP and spinlock debug flags. -*/ -struct bpf_spin_lock lock; +u32 __opaque;
nit: I'd call it __pad because it's not used functionally anymore (maybe also mention in a short comment on the side that we used to have bpf_spin_lock here)
Quoted 122 lines
kernel/bpf/helpers.c +39
-27} __attribute__((aligned(8))); -enum bpf_async_type { -BPF_ASYNC_TYPE_TIMER =0 , -BPF_ASYNC_TYPE_WQ , -}; - static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running);static enum hrtimer_restart bpf_timer_cb (struct hrtimer *hrtimer)@@ -1219,6 +1235,13 @@ static void bpf_async_cb_rcu_free(struct rcu_head *rcu) { struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu); +/* +* Drop the last reference to prog only after RCU GP, as set_callback() +* may race with cancel_and_free() +*/ +if (cb->prog) +bpf_prog_put (cb->prog); +kfree_nolock (cb); }@@ -1246,18 +1269,17 @@ static void bpf_timer_delete_work(struct work_struct *work)call_rcu (&t->cb.rcu, bpf_async_cb_rcu_free); } +static void __bpf_async_cancel_and_free (struct bpf_async_kern *async); +static void bpf_async_irq_worker (struct irq_work *work); +static int __bpf_async_init (struct bpf_async_kern *async,struct bpf_map *map,u64 flags,enum bpf_async_type type) { -struct bpf_async_cb *cb; +struct bpf_async_cb *cb, *old_cb;struct bpf_hrtimer *t;struct bpf_work *w;clockid_t clockid;size_t size; -int ret =0 ; - -if (in_nmi ()) -return -EOPNOTSUPP ;switch (type) {case BPF_ASYNC_TYPE_TIMER :@@ -1270,18 +1292,13 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, ureturn -EINVAL ; } -__bpf_spin_lock_irqsave (&async->lock); - t = async->timer; -if (t) { - ret = -EBUSY ; - goto out; - } + old_cb =READ_ONCE (async->cb); +if (old_cb) +return -EBUSY ; cb =bpf_map_kmalloc_nolock (map, size,0 , map->numa_node); -if (!cb) { - ret = -ENOMEM ; - goto out; - } +if (!cb) +return -ENOMEM ;switch (type) {case BPF_ASYNC_TYPE_TIMER :@@ -1304,9 +1321,20 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u cb->map = map; cb->prog =NULL ; cb->flags = flags; + cb->worker =IRQ_WORK_INIT (bpf_async_irq_worker); +seqcount_latch_init (&cb->latch); +atomic_set (&cb->writer,0 ); +refcount_set (&cb->refcnt,1 );/* map's reference */ +atomic_set (&cb->last_seq,0 ); + cb->type = type;rcu_assign_pointer (cb->callback_fn,NULL ); -WRITE_ONCE (async->cb, cb); + old_cb =cmpxchg (&async->cb,NULL , cb); +if (old_cb) { +/* Lost the race to initialize this bpf_async_kern, drop the allocated object */ +kfree_nolock (cb); +return -EBUSY ; + }/* Guarantee the order between async->cb and map->usercnt. So * when there are concurrent uref release and bpf timer init, either * bpf_timer_cancel_and_free() called by uref release reads a no-NULL @@ -1317,13 +1345,11 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u/* maps with timers must be either held by user space * or pinned in bpffs. */ -WRITE_ONCE (async->cb,NULL ); -kfree_nolock (cb); - ret = -EPERM ; +__bpf_async_cancel_and_free (async); +return -EPERM ; } -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return ret; + +return 0 ; } BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map,@@ -1388,33 +1414,95 @@ static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbacreturn 0 ; } +/* Decrements bpf_async_cb refcnt, if it becomes 0 schedule cleanup irq_work */ +static void bpf_async_refcnt_dec_cleanup (struct bpf_async_cb *cb) +{ +if (!refcount_dec_and_test (&cb->refcnt)) +return ; + +/*
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1433
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+* At this point we took the last reference +* Try to schedule cleanup, either: +* - Set ref to 1 and succeed irq_work_queue +* - See non-zero refcnt after decrement - other irq_work is going to cleanup +*/ +do { +refcount_set (&cb->refcnt,1 ); +if (irq_work_queue (&cb->worker)) +break ; + }while (refcount_dec_and_test (&cb->refcnt));
I still don't understand why you think this hack is better than just marking cb as "destined for destruction" through one-way flagging that cannot be reset.
What you have here is some unconventional partial resurrection scheme. Where cb's refcount dropped to zero, but then we try to temporarily revive it for that CANCEL_AND_FREE cleanup with *our* scheduled irq_work callback (which *maybe* will happen), but meanwhile any BPF program that still has reference to bpf_async_cb (there might be many on different CPUs) can now suddenly succeed with inc_not_zero(cb) again...
Why so complicated?.. Refcount is supposed to make things simpler, but you are managing to abuse it here.
Let's do this:
- make last_seq into u64 (so we can have special value designating "this async is DONE, no more operations"), that value should not overlap with valid latch counter values, use U32MAX + 1, for example.
- in __bpf_async_cancel_and_free:
1. cb = xchg(async->cb, BPF_PTR_POISON) (see my notes about poisoning below)
2. if (cb && cb != BPF_PTR_POISON) xchg(cb->last_seq,WE_ARE_DONE_GAME_OVER = U32_MAX + 1)
3. irq_work_queue() (doesn't matter if succeeds, someone will pick
up on WE_ARE_DONE_GAME_OVER)
- in bpf_async_read_op:
- read cb->last_seq with acquire semantics
- if (seq == WE_ARE_DONE_GAME_OVER)
- cmpxchg(seq, WE_ARE_DONE_GAME_OVER, WE_ARE_DONE_GAME_OVER + 1)
- if we lost, bail, someone else already handled this
- if we won, we are the only ones that will do clean up
- bpf_async_process_op(cb, BPF_ASYNC_CANCEL_AND_FREE, 0, 0);
(but see below, I think currently we do too much here)
- done, no more command processing
- if (seq == last_seq) -> out, someone got here first (just like
you have right now)
- proceed as before, eventually dec_and_test(cb->refcnt), and if
it dropped to zero -> schedule freeing after call_rcu_tasks_trace+call_rcu
Let's leave refcount to just preserve the lifetime of cb while being scheduled and processed by irq_work callbacks. And nothing more. And then see comments later on about async->cb poisoning.
But we still need to keep in mind that there might be active BPF programs that still have access to this memory. But they won't be able to schedule anything because a) async->cb is poisoned, b) if they happen to already have cb, refcount is at zero, so inc_not_zero fails,
c) even if they got refcount, we are at GAME_OVER stage (or beyond it), so whatever command we write will be ignored.
And I think we can and should honor usercnt==0 handling, just like with task_work.
Do I miss something in the above?
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1438
Quoted 5 lines
kernel/bpf/helpers.c +5
-0+} + +static int bpf_async_schedule_op (struct bpf_async_cb *cb,u32 op,u64 nsec,u32 timer_mode) +{ +/* Acquire active writer atomic*/
space missing before */
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1451
Quoted 13 lines
kernel/bpf/helpers.c +13
-0+if (atomic_cmpxchg_acquire (&cb->writer,0 ,1 )) +return -EBUSY ; + +write_seqcount_latch_begin (&cb->latch); + cb->cmd[0 ].nsec = nsec; + cb->cmd[0 ].mode = timer_mode; + cb->cmd[0 ].op = op; +write_seqcount_latch (&cb->latch); + cb->cmd[1 ].nsec = nsec; + cb->cmd[1 ].mode = timer_mode; + cb->cmd[1 ].op = op; +write_seqcount_latch_end (&cb->latch); +atomic_set (&cb->writer,0 );
Hm... this double write of the same data is interesting, I didn't realize seqcount_latch requires it. Good to know, but if we had more than 16 bytes to write, it would be a bit of concern for me, tbh.
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1458
Quoted 7 lines
kernel/bpf/helpers.c +7
-0+ +if (!refcount_inc_not_zero (&cb->refcnt)) +return -EBUSY ; + +/* TODO: Run operation without irq_work if not in NMI */ +if (!irq_work_queue (&cb->worker)) +/* irq_work is already scheduled on this CPU */
misleading comment, it doesn't matter if it's on this or another CPU, it's just irq_work is pending *somewhere*, no?
Quoted 59 lines
kernel/bpf/helpers.c +34
-20+bpf_async_refcnt_dec_cleanup (cb); + +return 0 ; +} + +static int bpf_async_read_op (struct bpf_async_cb *cb,enum bpf_async_op *op, +u64 *nsec,u32 *flags) +{ +u32 seq, last_seq, idx; + +while (true) { + last_seq =atomic_read_acquire (&cb->last_seq); + + seq =raw_read_seqcount_latch (&cb->latch); + +/* Return -EBUSY if current seq is consumed by another reader */ +if (seq == last_seq) +return -EBUSY ; + + idx = seq &1 ; + *nsec = cb->cmd[idx].nsec; + *flags = cb->cmd[idx].mode; + *op = cb->cmd[idx].op; + +if (raw_read_seqcount_latch_retry (&cb->latch, seq)) +continue ; +/* Commit read sequence number, own snapshot exclusively */ +if (atomic_cmpxchg_release (&cb->last_seq, last_seq, seq) == last_seq) +break ; + } + +return 0 ; +} +static int __bpf_async_set_callback (struct bpf_async_kern *async,void *callback_fn,struct bpf_prog *prog) {struct bpf_async_cb *cb; -int ret =0 ; -if (in_nmi ()) -return -EOPNOTSUPP ; -__bpf_spin_lock_irqsave (&async->lock); - cb = async->cb; -if (!cb) { - ret = -EINVAL ; - goto out; - } -if (!atomic64_read (&cb->map->usercnt)) { -/* maps with timers must be either held by user space -* or pinned in bpffs. Otherwise timer might still be -* running even when bpf prog is detached and user space -* is gone, since map_release_uref won't ever be called. -*/ - ret = -EPERM ; - goto out; - } - ret =bpf_async_update_prog_callback (cb, callback_fn, prog); -out:
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1505
Quoted 10 lines
kernel/bpf/helpers.c +8
-2-__bpf_spin_unlock_irqrestore (&async->lock); -return ret; +/* Make sure bpf_async_cb_rcu_free() is not called while here */ +guard (rcu)(); + + cb =READ_ONCE (async->cb); +if (!cb) +return -EINVAL ; + +return bpf_async_update_prog_callback (cb, callback_fn, prog);
shouldn't we do map->usercnt check after successful bpf_async_update_prog_callback(), and if usercnt dropped to zero, we just basically request cancle_and_free. We do something like that for task_work, no?
Quoted 53 lines
kernel/bpf/helpers.c +10
-16} BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn,@@ -1431,22 +1519,19 @@ static const struct bpf_func_proto bpf_timer_set_callback_proto = { .arg2_type = ARG_PTR_TO_FUNC, }; -BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, timer, u64, nsecs, u64, flags) +BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, async, u64, nsecs, u64, flags) {struct bpf_hrtimer *t; -int ret =0 ; -enum hrtimer_mode mode; +u32 mode; -if (in_nmi ()) -return -EOPNOTSUPP ;if (flags & ~(BPF_F_TIMER_ABS |BPF_F_TIMER_CPU_PIN ))return -EINVAL ; -__bpf_spin_lock_irqsave (&timer->lock); - t = timer->timer; -if (!t || !t->cb.prog) { - ret = -EINVAL ; - goto out; - } + +guard (rcu)(); + + t =READ_ONCE (async->timer); +if (!t || !READ_ONCE (t->cb.prog)) +return -EINVAL ;if (flags &BPF_F_TIMER_ABS ) mode =HRTIMER_MODE_ABS_SOFT ;@@ -1456,10 +1541,7 @@ BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, timer, u64, nsecs, u64, flaif (flags &BPF_F_TIMER_CPU_PIN ) mode |=HRTIMER_MODE_PINNED ; -hrtimer_start (&t->timer,ns_to_ktime (nsecs), mode); -out: -__bpf_spin_unlock_irqrestore (&timer->lock); -return ret; +return bpf_async_schedule_op (&t->cb,BPF_ASYNC_START , nsecs, mode); }static const struct bpf_func_proto bpf_timer_start_proto = {@@ -1536,27 +1618,16 @@ static const struct bpf_func_proto bpf_timer_cancel_proto = { .arg1_type = ARG_PTR_TO_TIMER, }; -static struct bpf_async_cb *__bpf_async_cancel_and_free (struct bpf_async_kern *async) +static void __bpf_async_cancel_and_free (struct bpf_async_kern *async) {
kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1625
Quoted 10 lines
kernel/bpf/helpers.c +1
-7struct bpf_async_cb *cb; -/* Performance optimization: read async->cb without lock first. */ -if (!READ_ONCE (async->cb)) -return NULL ; - -__bpf_spin_lock_irqsave (&async->lock); -/* re-read it under lock */ - cb = async->cb; + cb =xchg (&async->cb,NULL );
What prevents bpf_timer_init() from recreating async->cb? Should we "poison" this pointer here (i.e., xchg() to BPF_PTR_POISON and handle that in timer_init properly)?
Quoted 27 lines
kernel/bpf/helpers.c +10
-9if (!cb) - goto out; -bpf_async_update_prog_callback (cb,NULL ,NULL ); -/* The subsequent bpf_timer_start/cancel() helpers won't be able to use -* this timer, since it won't be initialized. -*/ -WRITE_ONCE (async->cb,NULL ); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return cb; +return ; + +/* Consume map's own refcnt, schedule cleanup irq_work if this is the last ref */ +bpf_async_refcnt_dec_cleanup (cb); }static void bpf_timer_delete (struct bpf_hrtimer *t)@@ -1611,19 +1682,76 @@ static void bpf_timer_delete(struct bpf_hrtimer *t) } } +static void bpf_async_process_op (struct bpf_async_cb *cb,u32 op, +u64 timer_nsec,u32 timer_mode) +{ +switch (cb->type) { +case BPF_ASYNC_TYPE_TIMER : { + struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
kernel/bpf/helpers.c static void bpf_timer_delete(struct bpf_hrtimer *t) · line 1700
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+ +switch (op) { +case BPF_ASYNC_START : +hrtimer_start (&t->timer,ns_to_ktime (timer_nsec), timer_mode); +break ; +case BPF_ASYNC_CANCEL : +hrtimer_try_to_cancel (&t->timer); +break ; +case BPF_ASYNC_CANCEL_AND_FREE : +bpf_timer_delete (t);
make sure bpf_timer_delete() doesn't do anything stupid and unnecessary. It shouldn't free any memory, just cancel timer. Let's analyze whether we need all that queue_work stuff we have there right now. We will be calling this from a well-defined irq_work context, so maybe it's good enough to just perform hrtimer_cancel() directly? In any case we should not do call_rcu(&w->cb.rcu, bpf_async_cb_rcu_free);, that will be done when cb->refcnt drops to zero afterwards.
Similar points for wq's CANCEL_AND_FREE, it should only ensure workqueue is canceled. There is no point in doing cancel_work_sync(), no one is waiting for irq_work to be done, so just cancel_work()?
Quoted 6 lines
kernel/bpf/helpers.c +6
-0+break ; +default : +break ; + } +break ; + }
[...]
Re: [PATCH RFC v5 05/10] bpf: Enable bpf timer and workqueue use in NMI
Quoted 297 lines · Andrii Nakryiko
Quoted 25 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
Refactor bpf timer and workqueue helpers to allow calling them from NMI context by making all operations lock-free and deferring NMI-unsafe work to
irq_work.Previously,
bpf_timer_start(), andbpf_wq_start()
could not be called from NMI context because they acquiredbpf_spin_lockand called hrtimer/schedule_work APIs directly. This patch removes these limitations.Key changes:
* Removebpf_spin_lockfrom structbpf_async_kern. Replace locked operations with atomiccmpxchg()for initialization andxchg()for cancel and free.
* Add per-asyncirq_workto defer NMI-unsafe operations (hrtimer_start,hrtimer_try_to_cancel,schedule_work) from NMI to softirq context.
* Use the lock-freeseqcount_latch_tto pass operation commands (start/cancel/free) along with their parameters (nsec, mode) from NMI-safe callers to theirq_workhandler.
* Add reference counting tobpf_async_cbto ensure the object stays alive until all scheduledirq_workcompletes and the timer/work callback finishes.
* Movebpf_prog_put()to RCU callback to handle races betweenset_callback()andcancel_and_free().Quoted 8 lines
* Refactor
__bpf_async_set_callback()getting rid of locks. Each iteration acquires a new reference and stores it
incb->progvia xchg. The previous value is retrieved and released. The loop condition checks if bothcb->progandcb->callback_fnmatch what we just wrote. If either differs, a concurrent writer overwrote our value, and we must retry.
When we retry, our previously-stored prog was alreadyput()by the concurrent writer or weput()it afterxchg().you already described that in earlier patch, no need to repeat that here, IMO
Quoted 29 lines
This enables BPF programs attached to NMI-context hooks (perf events) to use timers and workqueues for deferred processing.
Signed-off-by Mykyta Yatsenko <[email protected]>--- kernel/bpf/helpers.c | 335 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 225 insertions(+), 110 deletions(-)kernel/bpf/helpers.c +6
-0@@ -29,6 +29,7 @@#include <linux/task_work.h> #include <linux/irq_work.h> #include <linux/buildid.h> +#include <linux/seqlock.h> #include "../../lib/kstrtox.h" @@ -1095,6 +1096,23 @@ static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)return (void *)value -round_up (map->key_size,8 ); } +enum bpf_async_type { +BPF_ASYNC_TYPE_TIMER =0 , +BPF_ASYNC_TYPE_WQ , +}; +kernel/bpf/helpers.c static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx) · line 1113
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+enum bpf_async_op { +BPF_ASYNC_START , +BPF_ASYNC_CANCEL , +BPF_ASYNC_CANCEL_AND_FREE , +}; + +struct bpf_async_cmd { +u64 nsec; +u32 mode; +u32 op;enum
bpf_async_ophere?Quoted 20 lines
kernel/bpf/helpers.c +9
-0+}; +struct bpf_async_cb {struct bpf_map *map;struct bpf_prog *prog;@@ -1105,6 +1123,13 @@ struct bpf_async_cb {struct work_struct delete_work; };u64 flags; +struct irq_work worker; +atomic_t writer; +seqcount_latch_t latch; +struct bpf_async_cmd cmd[2 ]; +atomic_t last_seq; +refcount_t refcnt; +enum bpf_async_type type; };/* BPF map elements can contain 'struct bpf_timer'. kernel/bpf/helpers.c struct bpf_async_kern { · line 1170
Quoted 9 lines
kernel/bpf/helpers.c +1
-5@@ -1142,18 +1167,9 @@ struct bpf_async_kern {struct bpf_hrtimer *timer;struct bpf_work *work; }; -/* bpf_spin_lock is used here instead of spinlock_t to make -* sure that it always fits into space reserved by struct bpf_timer -* regardless of LOCKDEP and spinlock debug flags. -*/ -struct bpf_spin_lock lock; +u32 __opaque;nit: I'd call it
__padbecause it's not used functionally anymore (maybe also mention in a short comment on the side that we used to havebpf_spin_lockhere)Quoted 122 lines
kernel/bpf/helpers.c +39
-27} __attribute__((aligned(8))); -enum bpf_async_type { -BPF_ASYNC_TYPE_TIMER =0 , -BPF_ASYNC_TYPE_WQ , -}; - static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running);static enum hrtimer_restart bpf_timer_cb (struct hrtimer *hrtimer)@@ -1219,6 +1235,13 @@ static void bpf_async_cb_rcu_free(struct rcu_head *rcu) { struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu); +/* +* Drop the last reference to prog only after RCU GP, as set_callback() +* may race with cancel_and_free() +*/ +if (cb->prog) +bpf_prog_put (cb->prog); +kfree_nolock (cb); }@@ -1246,18 +1269,17 @@ static void bpf_timer_delete_work(struct work_struct *work)call_rcu (&t->cb.rcu, bpf_async_cb_rcu_free); } +static void __bpf_async_cancel_and_free (struct bpf_async_kern *async); +static void bpf_async_irq_worker (struct irq_work *work); +static int __bpf_async_init (struct bpf_async_kern *async,struct bpf_map *map,u64 flags,enum bpf_async_type type) { -struct bpf_async_cb *cb; +struct bpf_async_cb *cb, *old_cb;struct bpf_hrtimer *t;struct bpf_work *w;clockid_t clockid;size_t size; -int ret =0 ; - -if (in_nmi ()) -return -EOPNOTSUPP ;switch (type) {case BPF_ASYNC_TYPE_TIMER :@@ -1270,18 +1292,13 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, ureturn -EINVAL ; } -__bpf_spin_lock_irqsave (&async->lock); - t = async->timer; -if (t) { - ret = -EBUSY ; - goto out; - } + old_cb =READ_ONCE (async->cb); +if (old_cb) +return -EBUSY ; cb =bpf_map_kmalloc_nolock (map, size,0 , map->numa_node); -if (!cb) { - ret = -ENOMEM ; - goto out; - } +if (!cb) +return -ENOMEM ;switch (type) {case BPF_ASYNC_TYPE_TIMER :@@ -1304,9 +1321,20 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u cb->map = map; cb->prog =NULL ; cb->flags = flags; + cb->worker =IRQ_WORK_INIT (bpf_async_irq_worker); +seqcount_latch_init (&cb->latch); +atomic_set (&cb->writer,0 ); +refcount_set (&cb->refcnt,1 );/* map's reference */ +atomic_set (&cb->last_seq,0 ); + cb->type = type;rcu_assign_pointer (cb->callback_fn,NULL ); -WRITE_ONCE (async->cb, cb); + old_cb =cmpxchg (&async->cb,NULL , cb); +if (old_cb) { +/* Lost the race to initialize this bpf_async_kern, drop the allocated object */ +kfree_nolock (cb); +return -EBUSY ; + }/* Guarantee the order between async->cb and map->usercnt. So * when there are concurrent uref release and bpf timer init, either * bpf_timer_cancel_and_free() called by uref release reads a no-NULL @@ -1317,13 +1345,11 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u/* maps with timers must be either held by user space * or pinned in bpffs. */ -WRITE_ONCE (async->cb,NULL ); -kfree_nolock (cb); - ret = -EPERM ; +__bpf_async_cancel_and_free (async); +return -EPERM ; } -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return ret; + +return 0 ; } BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map,@@ -1388,33 +1414,95 @@ static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbacreturn 0 ; } +/* Decrements bpf_async_cb refcnt, if it becomes 0 schedule cleanup irq_work */ +static void bpf_async_refcnt_dec_cleanup (struct bpf_async_cb *cb) +{ +if (!refcount_dec_and_test (&cb->refcnt)) +return ; + +/* kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1433
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+* At this point we took the last reference +* Try to schedule cleanup, either: +* - Set ref to 1 and succeed irq_work_queue +* - See non-zero refcnt after decrement - other irq_work is going to cleanup +*/ +do { +refcount_set (&cb->refcnt,1 ); +if (irq_work_queue (&cb->worker)) +break ; + }while (refcount_dec_and_test (&cb->refcnt));I still don't understand why you think this hack is better than just marking cb as "destined for destruction" through one-way flagging that cannot be reset.
What you have here is some unconventional partial resurrection scheme. Where cb's refcount dropped to zero, but then we try to temporarily revive it for that
CANCEL_AND_FREEcleanup with *our* scheduledirq_workcallback (which *maybe* will happen), but meanwhile any BPF program that still has reference tobpf_async_cb(there might be many on different CPUs) can now suddenly succeed withinc_not_zero(cb)again...Why so complicated?.. Refcount is supposed to make things simpler, but you are managing to abuse it here.
Let's do this:
- make
last_seqinto u64 (so we can have special value designating "this async is DONE, no more operations"), that value should not overlap with valid latch counter values, use U32MAX + 1, for example.- in
__bpf_async_cancel_and_free:
1. cb =xchg(async->cb, BPF_PTR_POISON)(see my notes about poisoning below)
2. if (cb && cb !=BPF_PTR_POISON) xchg(cb->last_seq,WE_ARE_DONE_GAME_OVER=U32_MAX+ 1)
3.irq_work_queue()(doesn't matter if succeeds, someone will pick
up onWE_ARE_DONE_GAME_OVER)- in
bpf_async_read_op:
- readcb->last_seqwith acquire semantics
- if (seq ==WE_ARE_DONE_GAME_OVER)
-cmpxchg(seq, WE_ARE_DONE_GAME_OVER, WE_ARE_DONE_GAME_OVER + 1)
- if we lost, bail, someone else already handled this
- if we won, we are the only ones that will do clean up
-bpf_async_process_op(cb, BPF_ASYNC_CANCEL_AND_FREE, 0, 0);
(but see below, I think currently we do too much here)
- done, no more command processing
- if (seq ==last_seq) -> out, someone got here first (just like
you have right now)
- proceed as before, eventuallydec_and_test(cb->refcnt), and if
it dropped to zero -> schedule freeing after call_rcu_tasks_trace+call_rcuLet's leave refcount to just preserve the lifetime of cb while being scheduled and processed by
irq_workcallbacks. And nothing more. And then see comments later on aboutasync->cbpoisoning.But we still need to keep in mind that there might be active BPF programs that still have access to this memory. But they won't be able to schedule anything because a)
async->cbis poisoned, b) if they happen to already have cb, refcount is at zero, soinc_not_zerofails,
c) even if they got refcount, we are atGAME_OVERstage (or beyond it), so whatever command we write will be ignored.
Quoted 4 lines
And I think we can and should honor usercnt==0 handling, just like with
task_work.Do I miss something in the above?
Poisoning thing won't work, because for array maps we can do multiple cancel_and_free()/init() cycles for the same value, so right after cancel_and_free() we should be able to take over the cb pointer with init().
I think there is a small confusion, let me clarify few things:
- bpf_async_cancel_and_free() is called by the map synchronously, in case of array map
we have to make map value available for the reuse immediately
(that's why detach (xchg(NULL)) and schedule free for the detached cb).
- bpf_async_process_op(cb, BPF_ASYNC_CANCEL_AND_FREE, 0, 0) cancels timer and schedules cb freeing,
after calling it we can't access cb.
So in the above approach, the action you do on winning "cmpxchg(seq, WE_ARE_DONE_GAME_OVER, WE_ARE_DONE_GAME_OVER + 1)" and
"dec_and_test(cb->refcnt) dropped to zero"
is the same - cancel and free the cb.
Quoted 25 lines
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1438
Quoted 5 lines
kernel/bpf/helpers.c +5
-0+} + +static int bpf_async_schedule_op (struct bpf_async_cb *cb,u32 op,u64 nsec,u32 timer_mode) +{ +/* Acquire active writer atomic*/ space missing before */
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1451
Quoted 13 lines
kernel/bpf/helpers.c +13
-0+if (atomic_cmpxchg_acquire (&cb->writer,0 ,1 )) +return -EBUSY ; + +write_seqcount_latch_begin (&cb->latch); + cb->cmd[0 ].nsec = nsec; + cb->cmd[0 ].mode = timer_mode; + cb->cmd[0 ].op = op; +write_seqcount_latch (&cb->latch); + cb->cmd[1 ].nsec = nsec; + cb->cmd[1 ].mode = timer_mode; + cb->cmd[1 ].op = op; +write_seqcount_latch_end (&cb->latch); +atomic_set (&cb->writer,0 );Hm... this double write of the same data is interesting, I didn't realize
seqcount_latchrequires it. Good to know, but if we had more than 16 bytes to write, it would be a bit of concern for me, tbh.
Quoted 9 lines
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1458
Quoted 7 lines
kernel/bpf/helpers.c +7
-0+ +if (!refcount_inc_not_zero (&cb->refcnt)) +return -EBUSY ; + +/* TODO: Run operation without irq_work if not in NMI */ +if (!irq_work_queue (&cb->worker)) +/* irq_work is already scheduled on this CPU */ misleading comment, it doesn't matter if it's on this or another CPU, it's just
irq_workis pending *somewhere*, no?
Right, thanks for the correction.
Quoted 66 lines
Quoted 66 lines
kernel/bpf/helpers.c +38
-22+bpf_async_refcnt_dec_cleanup (cb); + +return 0 ; +} + +static int bpf_async_read_op (struct bpf_async_cb *cb,enum bpf_async_op *op, +u64 *nsec,u32 *flags) +{ +u32 seq, last_seq, idx; + +while (true) { + last_seq =atomic_read_acquire (&cb->last_seq); + + seq =raw_read_seqcount_latch (&cb->latch); + +/* Return -EBUSY if current seq is consumed by another reader */ +if (seq == last_seq) +return -EBUSY ; + + idx = seq &1 ; + *nsec = cb->cmd[idx].nsec; + *flags = cb->cmd[idx].mode; + *op = cb->cmd[idx].op; + +if (raw_read_seqcount_latch_retry (&cb->latch, seq)) +continue ; +/* Commit read sequence number, own snapshot exclusively */ +if (atomic_cmpxchg_release (&cb->last_seq, last_seq, seq) == last_seq) +break ; + } + +return 0 ; +} +static int __bpf_async_set_callback (struct bpf_async_kern *async,void *callback_fn,struct bpf_prog *prog) {struct bpf_async_cb *cb; -int ret =0 ; -if (in_nmi ()) -return -EOPNOTSUPP ; -__bpf_spin_lock_irqsave (&async->lock); - cb = async->cb; -if (!cb) { - ret = -EINVAL ; - goto out; - } -if (!atomic64_read (&cb->map->usercnt)) { -/* maps with timers must be either held by user space -* or pinned in bpffs. Otherwise timer might still be -* running even when bpf prog is detached and user space -* is gone, since map_release_uref won't ever be called. -*/ - ret = -EPERM ; - goto out; - } - ret =bpf_async_update_prog_callback (cb, callback_fn, prog); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return ret; +/* Make sure bpf_async_cb_rcu_free() is not called while here */ +guard (rcu)(); + + cb =READ_ONCE (async->cb);
Quoted 8 lines
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1505
Quoted 4 lines
kernel/bpf/helpers.c +4
-0+if (!cb) +return -EINVAL ; + +return bpf_async_update_prog_callback (cb, callback_fn, prog);shouldn't we do
map->usercntcheck after successfulbpf_async_update_prog_callback(), and if usercnt dropped to zero, we just basically requestcancle_and_free. We do something like that fortask_work, no?
I don't think map->usercnt check is needed here.
The risks here are:
* bpf_prog refcnt leak
* UAF for cb
Both are mitigated by the rcu lock:
* bpf_prog refcnt: last refcnt is put() in the rcu callback.
* cb is freed in rcu callback.
So even if cancel_and_free() is called concurrently and this function operate on the detached cb, it should be safe, as far as I understand.
Quoted 56 lines
Quoted 56 lines
kernel/bpf/helpers.c +10
-16} BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn,@@ -1431,22 +1519,19 @@ static const struct bpf_func_proto bpf_timer_set_callback_proto = { .arg2_type = ARG_PTR_TO_FUNC, }; -BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, timer, u64, nsecs, u64, flags) +BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, async, u64, nsecs, u64, flags) {struct bpf_hrtimer *t; -int ret =0 ; -enum hrtimer_mode mode; +u32 mode; -if (in_nmi ()) -return -EOPNOTSUPP ;if (flags & ~(BPF_F_TIMER_ABS |BPF_F_TIMER_CPU_PIN ))return -EINVAL ; -__bpf_spin_lock_irqsave (&timer->lock); - t = timer->timer; -if (!t || !t->cb.prog) { - ret = -EINVAL ; - goto out; - } + +guard (rcu)(); + + t =READ_ONCE (async->timer); +if (!t || !READ_ONCE (t->cb.prog)) +return -EINVAL ;if (flags &BPF_F_TIMER_ABS ) mode =HRTIMER_MODE_ABS_SOFT ;@@ -1456,10 +1541,7 @@ BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, timer, u64, nsecs, u64, flaif (flags &BPF_F_TIMER_CPU_PIN ) mode |=HRTIMER_MODE_PINNED ; -hrtimer_start (&t->timer,ns_to_ktime (nsecs), mode); -out: -__bpf_spin_unlock_irqrestore (&timer->lock); -return ret; +return bpf_async_schedule_op (&t->cb,BPF_ASYNC_START , nsecs, mode); }static const struct bpf_func_proto bpf_timer_start_proto = {@@ -1536,27 +1618,16 @@ static const struct bpf_func_proto bpf_timer_cancel_proto = { .arg1_type = ARG_PTR_TO_TIMER, }; -static struct bpf_async_cb *__bpf_async_cancel_and_free (struct bpf_async_kern *async) +static void __bpf_async_cancel_and_free (struct bpf_async_kern *async) {struct bpf_async_cb *cb;
Quoted 11 lines
kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1625
Quoted 8 lines
kernel/bpf/helpers.c +1
-7-/* Performance optimization: read async->cb without lock first. */ -if (!READ_ONCE (async->cb)) -return NULL ; - -__bpf_spin_lock_irqsave (&async->lock); -/* re-read it under lock */ - cb = async->cb; + cb =xchg (&async->cb,NULL );What prevents
bpf_timer_init()from recreatingasync->cb? Should we "poison" this pointer here (i.e.,xchg()toBPF_PTR_POISONand handle that intimer_initproperly)?
This is by design, on update element, array map does cancel_and_free() and the value should be ready for reuse immediately.
Quoted 38 lines
Quoted 28 lines
kernel/bpf/helpers.c +10
-9if (!cb) - goto out; -bpf_async_update_prog_callback (cb,NULL ,NULL ); -/* The subsequent bpf_timer_start/cancel() helpers won't be able to use -* this timer, since it won't be initialized. -*/ -WRITE_ONCE (async->cb,NULL ); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return cb; +return ; + +/* Consume map's own refcnt, schedule cleanup irq_work if this is the last ref */ +bpf_async_refcnt_dec_cleanup (cb); }static void bpf_timer_delete (struct bpf_hrtimer *t)@@ -1611,19 +1682,76 @@ static void bpf_timer_delete(struct bpf_hrtimer *t) } } +static void bpf_async_process_op (struct bpf_async_cb *cb,u32 op, +u64 timer_nsec,u32 timer_mode) +{ +switch (cb->type) { +case BPF_ASYNC_TYPE_TIMER : { + struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);kernel/bpf/helpers.c static void bpf_timer_delete(struct bpf_hrtimer *t) · line 1700
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+ +switch (op) { +case BPF_ASYNC_START : +hrtimer_start (&t->timer,ns_to_ktime (timer_nsec), timer_mode); +break ; +case BPF_ASYNC_CANCEL : +hrtimer_try_to_cancel (&t->timer); +break ; +case BPF_ASYNC_CANCEL_AND_FREE : +bpf_timer_delete (t);
Quoted 8 lines
make sure
bpf_timer_delete()doesn't do anything stupid and unnecessary. It shouldn't free any memory, just cancel timer. Let's analyze whether we need all thatqueue_workstuff we have there right now. We will be calling this from a well-definedirq_workcontext, so maybe it's good enough to just performhrtimer_cancel()directly? In any case we should not docall_rcu(&w->cb.rcu, bpf_async_cb_rcu_free);, that will be done whencb->refcntdrops to zero afterwards.
bpf_timer_delete() does freeing, that's the point of it, pure cancel is done on the BPF_ASYNC_CANCEL above.
Quoted 12 lines
Similar points for wq's
CANCEL_AND_FREE, it should only ensure workqueue is canceled. There is no point in doingcancel_work_sync(), no one is waiting forirq_workto be done, so justcancel_work()?Quoted 6 lines
kernel/bpf/helpers.c +6
-0+break ; +default : +break ; + } +break ; + }[...]
Re: [PATCH RFC v5 05/10] bpf: Enable bpf timer and workqueue use in NMI
Quoted 47 lines · Mykyta Yatsenko
Quoted 45 lines · Andrii Nakryiko
Quoted 25 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
Refactor bpf timer and workqueue helpers to allow calling them from NMI context by making all operations lock-free and deferring NMI-unsafe work to
irq_work.Previously,
bpf_timer_start(), andbpf_wq_start()
could not be called from NMI context because they acquiredbpf_spin_lockand called hrtimer/schedule_work APIs directly. This patch removes these limitations.Key changes:
* Removebpf_spin_lockfrom structbpf_async_kern. Replace locked operations with atomiccmpxchg()for initialization andxchg()for cancel and free.
* Add per-asyncirq_workto defer NMI-unsafe operations (hrtimer_start,hrtimer_try_to_cancel,schedule_work) from NMI to softirq context.
* Use the lock-freeseqcount_latch_tto pass operation commands (start/cancel/free) along with their parameters (nsec, mode) from NMI-safe callers to theirq_workhandler.
* Add reference counting tobpf_async_cbto ensure the object stays alive until all scheduledirq_workcompletes and the timer/work callback finishes.
* Movebpf_prog_put()to RCU callback to handle races betweenset_callback()andcancel_and_free().Quoted 8 lines
* Refactor
__bpf_async_set_callback()getting rid of locks. Each iteration acquires a new reference and stores it
incb->progvia xchg. The previous value is retrieved and released. The loop condition checks if bothcb->progandcb->callback_fnmatch what we just wrote. If either differs, a concurrent writer overwrote our value, and we must retry.
When we retry, our previously-stored prog was alreadyput()by the concurrent writer or weput()it afterxchg().you already described that in earlier patch, no need to repeat that here, IMO
Quoted 8 lines
This enables BPF programs attached to NMI-context hooks (perf events) to use timers and workqueues for deferred processing.
Signed-off-by Mykyta Yatsenko <[email protected]>--- kernel/bpf/helpers.c | 335 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 225 insertions(+), 110 deletions(-)
[...]
Quoted 72 lines
Quoted 72 lines
Quoted 8 lines
kernel/bpf/helpers.c +7
-0+/* Decrements bpf_async_cb refcnt, if it becomes 0 schedule cleanup irq_work */ +static void bpf_async_refcnt_dec_cleanup (struct bpf_async_cb *cb) +{ +if (!refcount_dec_and_test (&cb->refcnt)) +return ; + +/* kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1433
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+* At this point we took the last reference +* Try to schedule cleanup, either: +* - Set ref to 1 and succeed irq_work_queue +* - See non-zero refcnt after decrement - other irq_work is going to cleanup +*/ +do { +refcount_set (&cb->refcnt,1 ); +if (irq_work_queue (&cb->worker)) +break ; + }while (refcount_dec_and_test (&cb->refcnt));I still don't understand why you think this hack is better than just marking cb as "destined for destruction" through one-way flagging that cannot be reset.
What you have here is some unconventional partial resurrection scheme. Where cb's refcount dropped to zero, but then we try to temporarily revive it for that
CANCEL_AND_FREEcleanup with *our* scheduledirq_workcallback (which *maybe* will happen), but meanwhile any BPF program that still has reference tobpf_async_cb(there might be many on different CPUs) can now suddenly succeed withinc_not_zero(cb)again...Why so complicated?.. Refcount is supposed to make things simpler, but you are managing to abuse it here.
Let's do this:
- make
last_seqinto u64 (so we can have special value designating "this async is DONE, no more operations"), that value should not overlap with valid latch counter values, use U32MAX + 1, for example.- in
__bpf_async_cancel_and_free:
1. cb =xchg(async->cb, BPF_PTR_POISON)(see my notes about poisoning below)
2. if (cb && cb !=BPF_PTR_POISON) xchg(cb->last_seq,WE_ARE_DONE_GAME_OVER=U32_MAX+ 1)
3.irq_work_queue()(doesn't matter if succeeds, someone will pick
up onWE_ARE_DONE_GAME_OVER)- in
bpf_async_read_op:
- readcb->last_seqwith acquire semantics
- if (seq ==WE_ARE_DONE_GAME_OVER)
-cmpxchg(seq, WE_ARE_DONE_GAME_OVER, WE_ARE_DONE_GAME_OVER + 1)
- if we lost, bail, someone else already handled this
- if we won, we are the only ones that will do clean up
-bpf_async_process_op(cb, BPF_ASYNC_CANCEL_AND_FREE, 0, 0);
(but see below, I think currently we do too much here)
- done, no more command processing
- if (seq ==last_seq) -> out, someone got here first (just like
you have right now)
- proceed as before, eventuallydec_and_test(cb->refcnt), and if
it dropped to zero -> schedule freeing after call_rcu_tasks_trace+call_rcuLet's leave refcount to just preserve the lifetime of cb while being scheduled and processed by
irq_workcallbacks. And nothing more. And then see comments later on aboutasync->cbpoisoning.But we still need to keep in mind that there might be active BPF programs that still have access to this memory. But they won't be able to schedule anything because a)
async->cbis poisoned, b) if they happen to already have cb, refcount is at zero, soinc_not_zerofails,
c) even if they got refcount, we are atGAME_OVERstage (or beyond it), so whatever command we write will be ignored.
Quoted 8 lines
Quoted 4 lines
And I think we can and should honor usercnt==0 handling, just like with
task_work.Do I miss something in the above?
Poisoning thing won't work, because for array maps we can do multiple cancel_and_free()/init() cycles for the same value, so right after
cancel_and_free()
we should be able to take over the cb pointer withinit().
Yeah, and it's not just array maps, hashmaps can reuse same memory for a different key/value while some BPF program still keeps the pointer assuming that elevent is it's key's value. So yeah, you are right, we can't poison, it's just inherent in how element deletion and memory reuse is done in BPF.
Quoted 8 lines
I think there is a small confusion, let me clarify few things:
-
bpf_async_cancel_and_free()is called by the map synchronously, in case of array map
we have to make map value available for the reuse immediately
(that's why detach (xchg(NULL)) and schedule free for the detached cb).
-bpf_async_process_op(cb, BPF_ASYNC_CANCEL_AND_FREE, 0, 0)cancels timer and schedules cb freeing,
after calling it we can't access cb.
Just for others that follow. We chatted w/ Mykyta offline about all this. One point of confusion was that bpf_async_cancel_and_free() has to schedule memory freeing (though call_rcu). That's how it works right now, but that's not how it should work with refcnt and this whole irq_work workflow. Mykyta will rework this logic such that bpf_async_cancel_and_free() only schedules cancellation and dropping of that initial refcnt ("map's refcnt"). And orthogonally to that, when cb's refcnt drops to zero (no matter who does that last and in what context), that will trigger *only* RCU-delayed memory freeing. By that time all the timer cancellation should have happened one way or another.
Quoted 7 lines
So in the above approach, the action you do on winning "
cmpxchg(seq, WE_ARE_DONE_GAME_OVER, WE_ARE_DONE_GAME_OVER + 1)" and
"dec_and_test(cb->refcnt)dropped to zero"
is the same - cancel and free the cb.
[...]
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
kernel/bpf/helpers.c +4
-0+/* Make sure bpf_async_cb_rcu_free() is not called while here */ +guard (rcu)(); + + cb =READ_ONCE (async->cb);Quoted 8 lines
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1505
Quoted 4 lines
kernel/bpf/helpers.c +4
-0+if (!cb) +return -EINVAL ; + +return bpf_async_update_prog_callback (cb, callback_fn, prog);shouldn't we do
map->usercntcheck after successfulbpf_async_update_prog_callback(), and if usercnt dropped to zero, we just basically requestcancle_and_free. We do something like that fortask_work, no?
Quoted 1 line
I don't think
map->usercntcheck is needed here.
We chatted about this as well. Indeed, this is set_callback's usercnt check, which actually seems redundant either with current implementation or with the new nmi-compatible one. One way or another cancel_and_free will put whatever program reference we might still have set. usercnt check is only necessary in bpf_timer_init(), because we should not be able to successfully initialize timer after map->usercnt drops to zero, and with no timer initialized, we can't set_callback.
Quoted 12 lines
The risks here are:
*bpf_progrefcnt leak
* UAF for cb
Both are mitigated by the rcu lock:
*bpf_progrefcnt: last refcnt isput()in the rcu callback.
* cb is freed in rcu callback.
So even ifcancel_and_free()is called concurrently and this function operate on
the detached cb, it should be safe, as far as I understand.Quoted 3 lines
Quoted 3 lines
}
[...]
Quoted 16 lines
Quoted 5 lines
Quoted 5 lines
kernel/bpf/helpers.c +1
-1-static struct bpf_async_cb *__bpf_async_cancel_and_free (struct bpf_async_kern *async) +static void __bpf_async_cancel_and_free (struct bpf_async_kern *async) {struct bpf_async_cb *cb;Quoted 11 lines
kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1625
Quoted 8 lines
kernel/bpf/helpers.c +1
-7-/* Performance optimization: read async->cb without lock first. */ -if (!READ_ONCE (async->cb)) -return NULL ; - -__bpf_spin_lock_irqsave (&async->lock); -/* re-read it under lock */ - cb = async->cb; + cb =xchg (&async->cb,NULL );What prevents
bpf_timer_init()from recreatingasync->cb? Should we "poison" this pointer here (i.e.,xchg()toBPF_PTR_POISONand handle that intimer_initproperly)?
Quoted 2 lines
This is by design, on update element, array map does
cancel_and_free()and the value should be ready for reuse immediately.
yep, you are right, we shouldn't poison
[...]
Quoted 11 lines
Quoted 10 lines
kernel/bpf/helpers.c static void bpf_timer_delete(struct bpf_hrtimer *t) · line 1700
Quoted 2 lines
kernel/bpf/helpers.c +2
-0+case BPF_ASYNC_CANCEL_AND_FREE : +bpf_timer_delete (t);make sure
bpf_timer_delete()doesn't do anything stupid and unnecessary. It shouldn't free any memory, just cancel timer. Let's analyze whether we need all thatqueue_workstuff we have there right now. We will be calling this from a well-definedirq_workcontext, so maybe it's good enough to just performhrtimer_cancel()directly? In any case we should not docall_rcu(&w->cb.rcu, bpf_async_cb_rcu_free);, that will be done whencb->refcntdrops to zero afterwards.
Quoted 2 lines
bpf_timer_delete()does freeing, that's the point of it, pure cancel is done on theBPF_ASYNC_CANCELabove.
that was the biggest confusion we clarified, bpf_timer_delete() will basically be replaced by BPF_ASYNC_CANCEL and that GAME_OVER one-way transition, plus dropping cb's "map's refcnt", as described above. We think that entire bpf_timer_delete() complication with work queues can be just removed because we now have irq_work doing all the work, so there shouldn't be any waiting (and thus deadlocking) involved.
Quoted 14 lines
Quoted 13 lines
Similar points for wq's
CANCEL_AND_FREE, it should only ensure workqueue is canceled. There is no point in doingcancel_work_sync(), no one is waiting forirq_workto be done, so justcancel_work()?Quoted 6 lines
kernel/bpf/helpers.c +6
-0+break ; +default : +break ; + } +break ; + }[...]
RFC PATCH v5 6/10 bpf: Add verifier support for bpf_timer argument in kfuncs
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Extend the verifier to recognize struct bpf_timer as a valid kfunc argument type. Previously, bpf_timer was only supported in BPF helpers.
This prepares for adding timer-related kfuncs in subsequent patches.
--- kernel/bpf/verifier.c | 59 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 53 insertions(+), 6 deletions(-)
kernel/bpf/verifier.c +53 -6
@@ -8566,17 +8566,15 @@ static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno, }static int process_timer_func (struct bpf_verifier_env *env,int regno, -struct bpf_call_arg_meta *meta) +struct bpf_map *map) { -struct bpf_reg_state *reg =reg_state (env, regno); -struct bpf_map *map = reg->map_ptr;int err; err =check_map_field_pointer (env, regno,BPF_TIMER );if (err)return err; -if (meta->map_ptr) { +if (map) {verifier_bug (env,"Two map pointers in a timer helper" );return -EFAULT ; }
@@ -8584,8 +8582,36 @@ static int process_timer_func(struct bpf_verifier_env *env, int regno,verbose (env,"bpf_timer cannot be used for PREEMPT_RT.\n" );return -EOPNOTSUPP ; } +return 0 ; +} + +static int process_timer_helper (struct bpf_verifier_env *env,int regno, +struct bpf_call_arg_meta *meta) +{ +struct bpf_reg_state *reg =reg_state (env, regno); +int err; + + err =process_timer_func (env, regno, meta->map_ptr); +if (err) +return err; + meta->map_uid = reg->map_uid; - meta->map_ptr = map; + meta->map_ptr = reg->map_ptr; +return 0 ; +} + +static int process_timer_kfunc (struct bpf_verifier_env *env,int regno, +struct bpf_kfunc_call_arg_meta *meta) +{ +struct bpf_reg_state *reg =reg_state (env, regno); +int err; + + err =process_timer_func (env, regno, meta->map.ptr); +if (err) +return err; + + meta->map.uid = reg->map_uid; + meta->map.ptr = reg->map_ptr;return 0 ; }
@@ -9908,7 +9934,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, }break ;case ARG_PTR_TO_TIMER : - err =process_timer_func (env, regno, meta); + err =process_timer_helper (env, regno, meta);if (err)return err;break ;
@@ -12161,6 +12187,7 @@ enum {KF_ARG_WORKQUEUE_ID ,KF_ARG_RES_SPIN_LOCK_ID ,KF_ARG_TASK_WORK_ID , +KF_ARG_TIMER_ID , };BTF_ID_LIST (kf_arg_btf_ids)
@@ -12172,6 +12199,7 @@ BTF_ID(struct, bpf_rb_node) BTF_ID(struct, bpf_wq) BTF_ID(struct, bpf_res_spin_lock) BTF_ID(struct, bpf_task_work) +BTF_ID(struct, bpf_timer)static bool __is_kfunc_ptr_arg_type (const struct btf *btf,const struct btf_param *arg,int type)
@@ -12215,6 +12243,11 @@ static bool is_kfunc_arg_rbtree_node(const struct btf *btf, const struct btf_parreturn __is_kfunc_ptr_arg_type (btf, arg,KF_ARG_RB_NODE_ID ); } +static bool is_kfunc_arg_timer (const struct btf *btf,const struct btf_param *arg) +{ +return __is_kfunc_ptr_arg_type (btf, arg,KF_ARG_TIMER_ID ); +} +static bool is_kfunc_arg_wq (const struct btf *btf,const struct btf_param *arg) {return __is_kfunc_ptr_arg_type (btf, arg,KF_ARG_WORKQUEUE_ID );
@@ -12309,6 +12342,7 @@ enum kfunc_ptr_arg_type {KF_ARG_PTR_TO_NULL ,KF_ARG_PTR_TO_CONST_STR ,KF_ARG_PTR_TO_MAP , +KF_ARG_PTR_TO_TIMER ,KF_ARG_PTR_TO_WORKQUEUE ,KF_ARG_PTR_TO_IRQ_FLAG , KF_ARG_PTR_TO_RES_SPIN_LOCK,
@@ -12554,6 +12588,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,if (is_kfunc_arg_wq (meta->btf, &args[argno]))return KF_ARG_PTR_TO_WORKQUEUE ; +if (is_kfunc_arg_timer (meta->btf, &args[argno])) +return KF_ARG_PTR_TO_TIMER ; +if (is_kfunc_arg_task_work (meta->btf, &args[argno]))return KF_ARG_PTR_TO_TASK_WORK ;
@@ -13340,6 +13377,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_case KF_ARG_PTR_TO_REFCOUNTED_KPTR :case KF_ARG_PTR_TO_CONST_STR :case KF_ARG_PTR_TO_WORKQUEUE : +case KF_ARG_PTR_TO_TIMER :case KF_ARG_PTR_TO_TASK_WORK :case KF_ARG_PTR_TO_IRQ_FLAG :case KF_ARG_PTR_TO_RES_SPIN_LOCK :
@@ -13639,6 +13677,15 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_if (ret <0 )return ret;break ; +case KF_ARG_PTR_TO_TIMER : +if (reg->type !=PTR_TO_MAP_VALUE ) { +verbose (env,"arg#%d doesn't point to a map value\n" , i); +return -EINVAL ; + } + ret =process_timer_kfunc (env, regno, meta); +if (ret <0 ) +return ret; +break ;case KF_ARG_PTR_TO_TASK_WORK :if (reg->type !=PTR_TO_MAP_VALUE ) {verbose (env,"arg#%d doesn't point to a map value\n" , i);
Signature · 1 line
2.52.0
RFC PATCH v5 7/10 bpf: Introduce bpf_timer_cancel_async() kfunc
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
introducing bpf timer cancel kfunc that attempts canceling timer asynchronously, hence, supports working in NMI context.
--- kernel/bpf/helpers.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
kernel/bpf/helpers.c +14 -0
@@ -4483,6 +4483,19 @@ __bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)return 0 ; } +__bpf_kfunc int bpf_timer_cancel_async(struct bpf_timer *timer) +{ +struct bpf_async_cb *cb; +struct bpf_async_kern *async = (void *)timer; + +guard (rcu)(); + cb =READ_ONCE (async->cb); +if (!cb) +return -EINVAL ; + +return bpf_async_schedule_op (cb,BPF_ASYNC_CANCEL ,0 ,0 ); +} +__bpf_kfunc_end_defs ();static void bpf_task_work_cancel_scheduled (struct irq_work *irq_work)
@@ -4664,6 +4677,7 @@ BTF_ID_FLAGS(func, bpf_task_work_schedule_signal_impl) BTF_ID_FLAGS(func, bpf_task_work_schedule_resume_impl) BTF_ID_FLAGS(func, bpf_dynptr_from_file) BTF_ID_FLAGS(func, bpf_dynptr_file_discard) +BTF_ID_FLAGS(func, bpf_timer_cancel_async)BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {
Signature · 1 line
2.52.0
RFC PATCH v5 8/10 selftests/bpf: Refactor timer selftests
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Refactor timer selftests, extracting stress test into a separate test. This makes it easier to debug test failures and allows to extend.
--- tools/testing/selftests/bpf/prog_tests/timer.c | 55 +++++++++++++++++--------- 1 file changed, 36 insertions(+), 19 deletions(-)
tools/testing/selftests/bpf/prog_tests/timer.c +36 -19
@@ -22,13 +22,35 @@ static void *spin_lock_thread(void *arg)pthread_exit (arg); } -static int timer (struct timer *timer_skel) + +static int timer_stress (struct timer *timer_skel) { -int i, err, prog_fd; +int i, err =1 , prog_fd;LIBBPF_OPTS (bpf_test_run_opts, topts);pthread_t thread_id[NUM_THR ];void *ret; + prog_fd =bpf_program__fd (timer_skel->progs.race); +for (i =0 ; i <NUM_THR ; i++) { + err =pthread_create (&thread_id[i],NULL , + &spin_lock_thread, &prog_fd); +if (!ASSERT_OK (err,"pthread_create" )) +break ; + } + +while (i) { + err =pthread_join (thread_id[--i], &ret); +if (ASSERT_OK (err,"pthread_join" )) +ASSERT_EQ (ret, (void *)&prog_fd,"pthread_join" ); + } +return err; +} + +static int timer (struct timer *timer_skel) +{ +int err, prog_fd; +LIBBPF_OPTS (bpf_test_run_opts, topts); + err =timer__attach (timer_skel);if (!ASSERT_OK (err,"timer_attach" ))return err;
@@ -63,25 +85,10 @@ static int timer(struct timer *timer_skel)/* check that code paths completed */ ASSERT_EQ (timer_skel->bss->ok,1 |2 |4 ,"ok" ); - prog_fd =bpf_program__fd (timer_skel->progs.race); -for (i =0 ; i <NUM_THR ; i++) { - err =pthread_create (&thread_id[i],NULL , - &spin_lock_thread, &prog_fd); -if (!ASSERT_OK (err,"pthread_create" )) -break ; - } - -while (i) { - err =pthread_join (thread_id[--i], &ret); -if (ASSERT_OK (err,"pthread_join" )) -ASSERT_EQ (ret, (void *)&prog_fd,"pthread_join" ); - } -return 0 ; } -/* TODO: use pid filtering */ -void serial_test_timer (void ) +static void test_timer (int (*timer_test_fn)(struct timer *timer_skel)) {struct timer *timer_skel =NULL ;int err;
@@ -94,13 +101,23 @@ void serial_test_timer(void)if (!ASSERT_OK_PTR (timer_skel,"timer_skel_load" ))return ; - err =timer (timer_skel); + err =timer_test_fn (timer_skel);ASSERT_OK (err,"timer" );timer__destroy (timer_skel); +} + +void serial_test_timer (void ) +{ +test_timer (timer);RUN_TESTS (timer_failure); } +void serial_test_timer_stress (void ) +{ +test_timer (timer_stress); +} +void test_timer_interrupt (void ) {struct timer_interrupt *skel =NULL ;
Signature · 1 line
2.52.0
RFC PATCH v5 9/10 selftests/bpf: Add stress test for timer async cancel
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Extend BPF timer selftest to run stress test for async cancel.
--- tools/testing/selftests/bpf/prog_tests/timer.c | 18 +++++++++++++++++- tools/testing/selftests/bpf/progs/timer.c | 14 +++++++++++--- 2 files changed, 28 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/prog_tests/timer.c +17 -1
@@ -23,13 +23,14 @@ static void *spin_lock_thread(void *arg) } -static int timer_stress (struct timer *timer_skel) +static int timer_stress_runner (struct timer *timer_skel,bool async_cancel) {int i, err =1 , prog_fd;LIBBPF_OPTS (bpf_test_run_opts, topts);pthread_t thread_id[NUM_THR ];void *ret; + timer_skel->bss->async_cancel = async_cancel; prog_fd =bpf_program__fd (timer_skel->progs.race);for (i =0 ; i <NUM_THR ; i++) { err = pthread_create(&thread_id[i], NULL,
@@ -46,6 +47,16 @@ static int timer_stress(struct timer *timer_skel)return err; } +static int timer_stress (struct timer *timer_skel) +{ +return timer_stress_runner (timer_skel, false); +} + +static int timer_stress_async_cancel (struct timer *timer_skel) +{ +return timer_stress_runner (timer_skel, true); +} +static int timer (struct timer *timer_skel) {int err, prog_fd;
@@ -118,6 +129,11 @@ void serial_test_timer_stress(void)test_timer (timer_stress); } +void serial_test_timer_stress_async_cancel (void ) +{ +test_timer (timer_stress_async_cancel); +} +void test_timer_interrupt (void ) {struct timer_interrupt *skel =NULL ;
tools/testing/selftests/bpf/progs/timer.c +11 -3
@@ -1,13 +1,17 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2021 Facebook */ -#include <linux/bpf.h> -#include <time.h> + +#include <vmlinux.h> #include <stdbool.h> #include <errno.h> #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> +#define CLOCK_MONOTONIC 1 +#define CLOCK_BOOTTIME 7 + char _license[] SEC("license") = "GPL"; +struct hmap_elem {int counter;struct bpf_timer timer;
@@ -63,6 +67,7 @@ __u64 callback_check = 52;__u64 callback2_check =52 ;__u64 pinned_callback_check;__s32 pinned_cpu; +bool async_cancel =0 ;#define ARRAY 1#define HTAB 2
@@ -419,7 +424,10 @@ int race(void *ctx)bpf_timer_set_callback (timer, race_timer_callback);bpf_timer_start (timer,0 ,0 ); -bpf_timer_cancel (timer); +if (async_cancel) +bpf_timer_cancel_async (timer); +else +bpf_timer_cancel (timer);return 0 ; }
Signature · 1 line
2.52.0
RFC PATCH v5 10/10 selftests/bpf: Verify bpf_timer_cancel_async works
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Add test that verifies that bpf_timer_cancel_async works: can cancel callback successfully.
--- tools/testing/selftests/bpf/prog_tests/timer.c | 25 +++++++++++++++++++++++++ tools/testing/selftests/bpf/progs/timer.c | 23 +++++++++++++++++++++++ 2 files changed, 48 insertions(+)
tools/testing/selftests/bpf/prog_tests/timer.c +25 -0
@@ -99,6 +99,26 @@ static int timer(struct timer *timer_skel)return 0 ; } +static int timer_cancel_async (struct timer *timer_skel) +{ +int err, prog_fd; +LIBBPF_OPTS (bpf_test_run_opts, topts); + + prog_fd =bpf_program__fd (timer_skel->progs.test_async_cancel_succeed); + err =bpf_prog_test_run_opts (prog_fd, &topts); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (topts.retval,0 ,"test_run" ); + +usleep (500 ); +/* check that there were no errors in timer execution */ +ASSERT_EQ (timer_skel->bss->err,0 ,"err" ); + +/* check that code paths completed */ +ASSERT_EQ (timer_skel->bss->ok,1 |2 |4 ,"ok" ); + +return 0 ; +} +static void test_timer (int (*timer_test_fn)(struct timer *timer_skel)) {struct timer *timer_skel =NULL ;
@@ -134,6 +154,11 @@ void serial_test_timer_stress_async_cancel(void)test_timer (timer_stress_async_cancel); } +void serial_test_timer_async_cancel (void ) +{ +test_timer (timer_cancel_async); +} +void test_timer_interrupt (void ) {struct timer_interrupt *skel =NULL ;
tools/testing/selftests/bpf/progs/timer.c +23 -0
@@ -169,6 +169,29 @@ int BPF_PROG2(test1, int, a)return 0 ; } +static int timer_error (void *map,int *key,struct bpf_timer *timer) +{ + err =42 ; +return 0 ; +} + +SEC("syscall") +int test_async_cancel_succeed (void *ctx) +{ +struct bpf_timer *arr_timer; +int array_key =ARRAY ; + + arr_timer =bpf_map_lookup_elem (&array, &array_key); +if (!arr_timer) +return 0 ; +bpf_timer_init (arr_timer, &array,CLOCK_MONOTONIC ); +bpf_timer_set_callback (arr_timer, timer_error); +bpf_timer_start (arr_timer,100000 /* 100us */ ,0 ); +bpf_timer_cancel_async (arr_timer); + ok =7 ; +return 0 ; +} +/* callback for prealloc and non-prealloca hashtab timers */ static int timer_cb2 (void *map,int *key,struct hmap_elem *val) {
Signature · 1 line
2.52.0