bpf: Avoid locks in bpf_timer and bpf_wq
Subsystems (4)
v8 cover bpf: Avoid locks in bpf_timer and bpf_wq
This series reworks implementation of BPF timer and workqueue APIs. The goal is to make both timers and wq non-blocking, enabling their use in NMI context.
Today this code relies on a bpf_spin_lock embedded in the map element to serialize:
* init of the async object,
* setting/changing the callback and bpf_prog
* starting/cancelling the timer/work
* tearing down when the map element is deleted or the map’s user ref is dropped
The basic design approach in this series:
* Use irq_work to offload all blocking work from NMI
* Introduce refcount to guarantee lifetime of the bpf_async_cb structs deferred to potentially multiple irq_work callbacks
* Keep objects under RCU protection to make sure they are not freed while kfuncs/helpers access them (We can't use refcnt for this, as refcnt itself is part of the bpf_async_cb struct)
---
Changes in v8:
- Return -EBUSY in bpf_async_read_op() if last_seq is failed to be set
- In bpf_async_cancel_and_free() drop bpf_async_cb ref after calling bpf_async_process()
- Link to v7: https://lore.kernel.org/r/[email protected]
Changes in v7:
- Addressed Andrii's review points from the previous version - nothing very significang.
- Added NMI stress tests for bpf_timer - hit few verifier failing checks and removed them.
- Address sparse warning in the bpf_async_update_prog_callback()
- Link to v6: https://lore.kernel.org/r/[email protected]
Changes in v6:
- Reworked destruction and refcnt use:
- On cancel_and_free() set last_seq to BPF_ASYNC_DESTROY value, drop map's reference
- In irq work callback, atomically switch DESTROY to DESTROYED, cancel timer/wq
- Free bpf_async_cb on refcnt going to 0.
- Link to v5: https://lore.kernel.org/r/[email protected]
Changes in v5:
- Extracted lock-free algorithm for updating cb->prog andcb->callback_fn into a function bpf_async_update_prog_callback(),
added a new commit and introduces this function and uses it in __bpf_async_set_callback(), bpf_timer_cancel() and bpf_async_cancel_and_free().
This allows to move the change into the separate commit without breaking correctness.
- Handle NULL prog in bpf_async_update_prog_callback().
- Link to v4: https://lore.kernel.org/r/[email protected]
Changes in v4:
- Handle irq_work_queue failures in both schedule and cancel_and_free
paths: introduced bpf_async_refcnt_dec_cleanup() that decrements refcnt and makes sure if last reference is put, there is at least one irq_work scheduled to execute final cleanup.
- Additional refcnt inc/dec in set_callback() + rcu lock to make sure cleanup is not running at the same time as set_callback().
- Added READ_ONCE where it was needed.
- Squash 'bpf: Refactor __bpf_async_set_callback()' commit into 'bpf:
Add lock-free cell for NMI-safe
async operations'
- Removed mpmc_cell, use seqcount_latch_t instead.
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- Major rework
- Introduce mpmc_cell, allowing concurrent writes and reads
- Implement irq_work deferring
- Adding selftests
- Introduces bpf_timer_cancel_async kfunc
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Move refcnt initialization and put (from cancel_and_free())
from patch 5 into the patch 4, so that patch 4 has more clear and full implementation and use of refcnt
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Mykyta Yatsenko (8):
bpf: Enable bpf timer and workqueue use in NMI
bpf: Add verifier support for bpf_timer argument in kfuncs
bpf: Introduce bpf_timer_cancel_async() kfunc selftests/bpf: Refactor timer selftests selftests/bpf: Add stress test for timer async cancel selftests/bpf: Verify bpf_timer_cancel_async works selftests/bpf: Add timer stress test in NMI context selftests/bpf: Removed obsolete tests
kernel/bpf/helpers.c | 449 ++++++++++++--------- kernel/bpf/verifier.c | 75 +++- tools/testing/selftests/bpf/prog_tests/timer.c | 250 +++++++++++- tools/testing/selftests/bpf/progs/timer.c | 118 +++++- .../bpf/progs/verifier_helper_restricted.c | 111 ----- 5 files changed, 659 insertions(+), 344 deletions(-)
---
base-commit: ba335bf3a5b8c5f47d56e9be3f96e0989dad5346
change-id: 20251028-timer_nolock-457f5b9daace
Best regards,
Signature · 1 line
Mykyta Yatsenko <[email protected]>
PATCH v8 1/8 bpf: Enable bpf timer and workqueue use in NMI
From: Mykyta Yatsenko <[email protected]>
Refactor bpf timer and workqueue helpers to allow calling them from NMI context by making all operations lock-free and deferring NMI-unsafe work to irq_work.
Previously, bpf_timer_start(), and bpf_wq_start()
could not be called from NMI context because they acquired bpf_spin_lock and called hrtimer/schedule_work APIs directly. This patch removes these limitations.
Key changes:
* Remove bpf_spin_lock from struct bpf_async_kern.
* Initialize/Destroy via setting/unsetting bpf_async_cb pointer atomically.
* Add per-bpf_async_cb irq_work to defer NMI-unsafe operations (hrtimer_start, hrtimer_try_to_cancel, schedule_work) from NMI to softirq context.
* Use the lock-free seqcount_latch_t to pass operation commands (start/cancel/free) and parameters
from NMI-safe callers to the irq_work handler.
* Add reference counting to bpf_async_cb to ensure the object stays alive until all scheduled irq_work completes.
* Move bpf_prog_put() to RCU callback to handle races between set_callback() and cancel_and_free().
* Modify cancel_and_free() path:
* Detach bpf_async_cb.
* Signal destruction to irq_work side via setting last_seq to BPF_ASYNC_DESTROY.
* On receiving BPF_ASYNC_DESTROY, cancel timer/wq.
* Free bpf_async_cb on refcnt reaching 0, wait for both rcu and rcu task trace grace periods before freeing the bpf_async_cb. Removed unnecessary rcu locks, as kfunc/helper allways assumes rcu or rcu task trace lock.
This enables BPF programs attached to NMI-context hooks (perf events) to use timers and workqueues for deferred processing.
--- kernel/bpf/helpers.c | 436 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 254 insertions(+), 182 deletions(-)
kernel/bpf/helpers.c +254 -182
@@ -29,6 +29,7 @@#include <linux/task_work.h> #include <linux/irq_work.h> #include <linux/buildid.h> +#include <linux/seqlock.h> #include "../../lib/kstrtox.h"
@@ -1095,16 +1096,41 @@ static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)return (void *)value -round_up (map->key_size,8 ); } +enum bpf_async_type { +BPF_ASYNC_TYPE_TIMER =0 , +BPF_ASYNC_TYPE_WQ , +}; + +enum bpf_async_op { +BPF_ASYNC_START , +BPF_ASYNC_CANCEL +}; + +enum bpf_async_seq_state { +BPF_ASYNC_DESTROY = (u64 )U32_MAX +1 , +BPF_ASYNC_DESTROYED = (u64 )U32_MAX +2 , +}; + +struct bpf_async_cmd { +u64 nsec; +u32 mode; +enum bpf_async_op op; +}; +struct bpf_async_cb {struct bpf_map *map;struct bpf_prog *prog;void __rcu *callback_fn;void *value; -union { -struct rcu_head rcu; -struct work_struct delete_work; - }; +struct rcu_head rcu;u64 flags; +struct irq_work worker; +atomic_t writer; +seqcount_latch_t latch; +struct bpf_async_cmd cmd[2 ]; +atomic64_t last_seq; +refcount_t refcnt; +enum bpf_async_type type; };/* BPF map elements can contain 'struct bpf_timer'.
@@ -1132,7 +1158,6 @@ struct bpf_hrtimer {struct bpf_work {struct bpf_async_cb cb;struct work_struct work; -struct work_struct delete_work; };/* the actual struct hidden inside uapi struct bpf_timer and bpf_wq */
@@ -1142,18 +1167,9 @@ struct bpf_async_kern {struct bpf_hrtimer *timer;struct bpf_work *work; }; -/* bpf_spin_lock is used here instead of spinlock_t to make -* sure that it always fits into space reserved by struct bpf_timer -* regardless of LOCKDEP and spinlock debug flags. -*/ -struct bpf_spin_lock lock; +u32 __pad;/* Left for binary compatibility, previously stored spinlock */ } __attribute__((aligned(8))); -enum bpf_async_type { -BPF_ASYNC_TYPE_TIMER =0 , -BPF_ASYNC_TYPE_WQ , -}; - static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running);static enum hrtimer_restart bpf_timer_cb (struct hrtimer *hrtimer)
@@ -1219,45 +1235,54 @@ static void bpf_async_cb_rcu_free(struct rcu_head *rcu) { struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu); +/* +* Drop the last reference to prog only after RCU GP, as set_callback() +* may race with cancel_and_free() +*/ +if (cb->prog) +bpf_prog_put (cb->prog); +kfree_nolock (cb); } -static void bpf_wq_delete_work (struct work_struct *work) +/* Callback from call_rcu_tasks_trace, chains to call_rcu for final free */ +static void bpf_async_cb_rcu_tasks_trace_free (struct rcu_head *rcu) { - struct bpf_work *w = container_of(work, struct bpf_work, delete_work); - -cancel_work_sync (&w->work); - -call_rcu (&w->cb.rcu, bpf_async_cb_rcu_free); +/* +* If RCU Tasks Trace grace period implies RCU grace period, +* there is no need to invoke call_rcu(). +*/ +if (rcu_trace_implies_rcu_gp ()) +bpf_async_cb_rcu_free (rcu); +else +call_rcu (rcu, bpf_async_cb_rcu_free); } -static void bpf_timer_delete_work (struct work_struct *work) +/* +* Decrement refcount and if it reaches zero, schedule deferred cleanup +* through call_rcu_tasks_trace() -> call_rcu() -> bpf_prog_put()/kfree() +*/ +static void bpf_async_refcount_put (struct bpf_async_cb *cb) { - struct bpf_hrtimer *t = container_of(work, struct bpf_hrtimer, cb.delete_work); +if (!refcount_dec_and_test (&cb->refcnt)) +return ; -/* Cancel the timer and wait for callback to complete if it was running. -* If hrtimer_cancel() can be safely called it's safe to call -* call_rcu() right after for both preallocated and non-preallocated -* maps. The async->cb = NULL was already done and no code path can see -* address 't' anymore. Timer if armed for existing bpf_hrtimer before -* bpf_timer_cancel_and_free will have been cancelled. -*/ -hrtimer_cancel (&t->timer); -call_rcu (&t->cb.rcu, bpf_async_cb_rcu_free); +/* Both timer and wq callbacks run under RCU lock, UAF should not be possible there */ +call_rcu_tasks_trace (&cb->rcu, bpf_async_cb_rcu_tasks_trace_free); } +static void __bpf_async_cancel_and_free (struct bpf_async_kern *async); +static void bpf_async_irq_worker (struct irq_work *work); +static int bpf_async_process (struct bpf_async_cb *cb); +static int __bpf_async_init (struct bpf_async_kern *async,struct bpf_map *map,u64 flags,enum bpf_async_type type) { -struct bpf_async_cb *cb; +struct bpf_async_cb *cb, *old_cb;struct bpf_hrtimer *t;struct bpf_work *w;clockid_t clockid;size_t size; -int ret =0 ; - -if (in_nmi ()) -return -EOPNOTSUPP ;switch (type) {case BPF_ASYNC_TYPE_TIMER :
@@ -1270,18 +1295,13 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, ureturn -EINVAL ; } -__bpf_spin_lock_irqsave (&async->lock); - t = async->timer; -if (t) { - ret = -EBUSY ; - goto out; - } + old_cb =READ_ONCE (async->cb); +if (old_cb) +return -EBUSY ; cb =bpf_map_kmalloc_nolock (map, size,0 , map->numa_node); -if (!cb) { - ret = -ENOMEM ; - goto out; - } +if (!cb) +return -ENOMEM ;switch (type) {case BPF_ASYNC_TYPE_TIMER :
@@ -1289,7 +1309,6 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u t = (struct bpf_hrtimer *)cb;atomic_set (&t->cancelling,0 ); -INIT_WORK (&t->cb.delete_work, bpf_timer_delete_work);hrtimer_setup (&t->timer, bpf_timer_cb, clockid,HRTIMER_MODE_REL_SOFT ); cb->value = (void *)async - map->record->timer_off;break ;
@@ -1297,16 +1316,26 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u w = (struct bpf_work *)cb;INIT_WORK (&w->work, bpf_wq_work); -INIT_WORK (&w->delete_work, bpf_wq_delete_work); cb->value = (void *)async - map->record->wq_off;break ; } cb->map = map; cb->prog =NULL ; cb->flags = flags; + cb->worker =IRQ_WORK_INIT (bpf_async_irq_worker); +seqcount_latch_init (&cb->latch); +atomic_set (&cb->writer,0 ); +refcount_set (&cb->refcnt,1 );/* map's reference */ +atomic64_set (&cb->last_seq,0 ); + cb->type = type;rcu_assign_pointer (cb->callback_fn,NULL ); -WRITE_ONCE (async->cb, cb); + old_cb =cmpxchg (&async->cb,NULL , cb); +if (old_cb) { +/* Lost the race to initialize this bpf_async_kern, drop the allocated object */ +kfree_nolock (cb); +return -EBUSY ; + }/* Guarantee the order between async->cb and map->usercnt. So * when there are concurrent uref release and bpf timer init, either * bpf_timer_cancel_and_free() called by uref release reads a no-NULL
@@ -1317,13 +1346,11 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u/* maps with timers must be either held by user space * or pinned in bpffs. */ -WRITE_ONCE (async->cb,NULL ); -kfree_nolock (cb); - ret = -EPERM ; +__bpf_async_cancel_and_free (async); +return -EPERM ; } -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return ret; + +return 0 ; } BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map,
@@ -1354,8 +1381,9 @@ static const struct bpf_func_proto bpf_timer_init_proto = { .arg3_type = ARG_ANYTHING, }; -static int bpf_async_update_prog_callback (struct bpf_async_cb *cb,void *callback_fn, -struct bpf_prog *prog) +static int bpf_async_update_prog_callback (struct bpf_async_cb *cb, +struct bpf_prog *prog, +void *callback_fn) {struct bpf_prog *prev;
@@ -1380,7 +1408,8 @@ static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbacif (prev)bpf_prog_put (prev); - }while (READ_ONCE (cb->prog) != prog ||READ_ONCE (cb->callback_fn) != callback_fn); + }while (READ_ONCE (cb->prog) != prog || + (void __force *)READ_ONCE (cb->callback_fn) != callback_fn);if (prog)bpf_prog_put (prog);
@@ -1388,33 +1417,95 @@ static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbacreturn 0 ; } +static int bpf_async_schedule_op (struct bpf_async_cb *cb,enum bpf_async_op op, +u64 nsec,u32 timer_mode) +{ +/* Acquire active writer */ +if (atomic_cmpxchg_acquire (&cb->writer,0 ,1 )) +return -EBUSY ; + +write_seqcount_latch_begin (&cb->latch); + cb->cmd[0 ].nsec = nsec; + cb->cmd[0 ].mode = timer_mode; + cb->cmd[0 ].op = op; +write_seqcount_latch (&cb->latch); + cb->cmd[1 ].nsec = nsec; + cb->cmd[1 ].mode = timer_mode; + cb->cmd[1 ].op = op; +write_seqcount_latch_end (&cb->latch); + +atomic_set_release (&cb->writer,0 ); + +if (!in_nmi ()) +return bpf_async_process (cb); + +/* NMI path: defer to irq_work with refcount protection */ +if (!refcount_inc_not_zero (&cb->refcnt)) +return -EBUSY ; + +if (!irq_work_queue (&cb->worker)) +bpf_async_refcount_put (cb); + +return 0 ; +} + +static int bpf_async_read_op (struct bpf_async_cb *cb,enum bpf_async_op *op, +u64 *nsec,u32 *flags) +{ +u32 seq, idx; +s64 last_seq; + +while (true) { + last_seq =atomic64_read_acquire (&cb->last_seq); +if (unlikely (last_seq >U32_MAX )) {/* Check if terminal seq num has been set */ +s64 expected =BPF_ASYNC_DESTROY ; + +if (last_seq !=BPF_ASYNC_DESTROY ) +return -EBUSY ; + +if (!atomic64_try_cmpxchg_release (&cb->last_seq, + &expected, +BPF_ASYNC_DESTROYED )) +return -EBUSY ;/* Someone else set it to DESTROYED, bail */ + + *op =BPF_ASYNC_CANCEL ; +return 0 ; + } + + seq =raw_read_seqcount_latch (&cb->latch); + +/* Return -EBUSY if current seq is consumed by another reader */ +if (seq == last_seq) +return -EBUSY ; + + idx = seq &1 ; + *nsec = cb->cmd[idx].nsec; + *flags = cb->cmd[idx].mode; + *op = cb->cmd[idx].op; + +if (raw_read_seqcount_latch_retry (&cb->latch, seq)) +continue ; + +/* +* Try to commit read sequence number to own snapshot exclusively +* if other reader won, return -EBUSY +*/ +return atomic64_try_cmpxchg_release (&cb->last_seq, &last_seq, seq) ?0 : -EBUSY ; + } + +return 0 ; +} +static int __bpf_async_set_callback (struct bpf_async_kern *async,void *callback_fn,struct bpf_prog *prog) {struct bpf_async_cb *cb; -int ret =0 ; -if (in_nmi ()) -return -EOPNOTSUPP ; -__bpf_spin_lock_irqsave (&async->lock); - cb = async->cb; -if (!cb) { - ret = -EINVAL ; - goto out; - } -if (!atomic64_read (&cb->map->usercnt)) { -/* maps with timers must be either held by user space -* or pinned in bpffs. Otherwise timer might still be -* running even when bpf prog is detached and user space -* is gone, since map_release_uref won't ever be called. -*/ - ret = -EPERM ; - goto out; - } - ret =bpf_async_update_prog_callback (cb, callback_fn, prog); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return ret; + cb =READ_ONCE (async->cb); +if (!cb) +return -EINVAL ; + +return bpf_async_update_prog_callback (cb, prog, callback_fn); } BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn,
@@ -1431,22 +1522,17 @@ static const struct bpf_func_proto bpf_timer_set_callback_proto = { .arg2_type = ARG_PTR_TO_FUNC, }; -BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, timer, u64, nsecs, u64, flags) +BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, async, u64, nsecs, u64, flags) {struct bpf_hrtimer *t; -int ret =0 ; -enum hrtimer_mode mode; +u32 mode; -if (in_nmi ()) -return -EOPNOTSUPP ;if (flags & ~(BPF_F_TIMER_ABS |BPF_F_TIMER_CPU_PIN ))return -EINVAL ; -__bpf_spin_lock_irqsave (&timer->lock); - t = timer->timer; -if (!t || !t->cb.prog) { - ret = -EINVAL ; - goto out; - } + + t =READ_ONCE (async->timer); +if (!t || !READ_ONCE (t->cb.prog)) +return -EINVAL ;if (flags &BPF_F_TIMER_ABS ) mode =HRTIMER_MODE_ABS_SOFT ;
@@ -1456,10 +1542,7 @@ BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, timer, u64, nsecs, u64, flaif (flags &BPF_F_TIMER_CPU_PIN ) mode |=HRTIMER_MODE_PINNED ; -hrtimer_start (&t->timer,ns_to_ktime (nsecs), mode); -out: -__bpf_spin_unlock_irqrestore (&timer->lock); -return ret; +return bpf_async_schedule_op (&t->cb,BPF_ASYNC_START , nsecs, mode); }static const struct bpf_func_proto bpf_timer_start_proto = {
@@ -1480,8 +1563,6 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, async)if (in_nmi ())return -EOPNOTSUPP ; -guard (rcu)(); - t =READ_ONCE (async->timer);if (!t)return -EINVAL ;
@@ -1536,94 +1617,99 @@ static const struct bpf_func_proto bpf_timer_cancel_proto = { .arg1_type = ARG_PTR_TO_TIMER, }; -static struct bpf_async_cb *__bpf_async_cancel_and_free (struct bpf_async_kern *async) +static void __bpf_async_cancel_and_free (struct bpf_async_kern *async) {struct bpf_async_cb *cb;/* Performance optimization: read async->cb without lock first. */ if (!READ_ONCE (async->cb)) -return NULL ; +return ; -__bpf_spin_lock_irqsave (&async->lock); -/* re-read it under lock */ - cb = async->cb; + cb =xchg (&async->cb,NULL );if (!cb) - goto out; -bpf_async_update_prog_callback (cb,NULL ,NULL ); -/* The subsequent bpf_timer_start/cancel() helpers won't be able to use -* this timer, since it won't be initialized. +return ; + +atomic64_set (&cb->last_seq,BPF_ASYNC_DESTROY ); +if (!in_nmi ()) { +bpf_async_process (cb); +bpf_async_refcount_put (cb); +return ; + } +/* +* Attempt to pass our initial map's reference to +* irq_work callback, if we fail to schedule it (meaning it is +* already scheduled by someone else having their own ref), we have to +* put that ref here. */ -WRITE_ONCE (async->cb,NULL ); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return cb; +if (!irq_work_queue (&cb->worker)) +bpf_async_refcount_put (cb); } -static void bpf_timer_delete (struct bpf_hrtimer *t) +static void bpf_async_process_op (struct bpf_async_cb *cb,u32 op, +u64 timer_nsec,u32 timer_mode) { -/* -* We check that bpf_map_delete/update_elem() was called from timer -* callback_fn. In such case we don't call hrtimer_cancel() (since it -* will deadlock) and don't call hrtimer_try_to_cancel() (since it will -* just return -1). Though callback_fn is still running on this cpu it's -* safe to do kfree(t) because bpf_timer_cb() read everything it needed -* from 't'. The bpf subprog callback_fn won't be able to access 't', -* since async->cb = NULL was already done. The timer will be -* effectively cancelled because bpf_timer_cb() will return -* HRTIMER_NORESTART. -* -* However, it is possible the timer callback_fn calling us armed the -* timer _before_ calling us, such that failing to cancel it here will -* cause it to possibly use struct hrtimer after freeing bpf_hrtimer. -* Therefore, we _need_ to cancel any outstanding timers before we do -* call_rcu, even though no more timers can be armed. -* -* Moreover, we need to schedule work even if timer does not belong to -* the calling callback_fn, as on two different CPUs, we can end up in a -* situation where both sides run in parallel, try to cancel one -* another, and we end up waiting on both sides in hrtimer_cancel -* without making forward progress, since timer1 depends on time2 -* callback to finish, and vice versa. -* -* CPU 1 (timer1_cb) CPU 2 (timer2_cb) -* bpf_timer_cancel_and_free(timer2) bpf_timer_cancel_and_free(timer1) -* -* To avoid these issues, punt to workqueue context when we are in a -* timer callback. -*/ -if (this_cpu_read (hrtimer_running)) { -queue_work (system_dfl_wq, &t->cb.delete_work); -return ; - } +switch (cb->type) { +case BPF_ASYNC_TYPE_TIMER : { + struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb); -if (IS_ENABLED (CONFIG_PREEMPT_RT )) { -/* If the timer is running on other CPU, also use a kworker to -* wait for the completion of the timer instead of trying to -* acquire a sleepable lock in hrtimer_cancel() to wait for its -* completion. -*/ -if (hrtimer_try_to_cancel (&t->timer) >=0 ) -call_rcu (&t->cb.rcu, bpf_async_cb_rcu_free); -else -queue_work (system_dfl_wq, &t->cb.delete_work); - }else { -bpf_timer_delete_work (&t->cb.delete_work); +switch (op) { +case BPF_ASYNC_START : +hrtimer_start (&t->timer,ns_to_ktime (timer_nsec), timer_mode); +break ; +case BPF_ASYNC_CANCEL : +hrtimer_try_to_cancel (&t->timer); +break ; + } +break ; + } +case BPF_ASYNC_TYPE_WQ : { + struct bpf_work *w = container_of(cb, struct bpf_work, cb); + +switch (op) { +case BPF_ASYNC_START : +schedule_work (&w->work); +break ; +case BPF_ASYNC_CANCEL : +/* Use non-blocking cancel, safe in irq_work context. +* RCU grace period ensures callback completes before free. +*/ +cancel_work (&w->work); +break ; + } +break ; + } } } +static int bpf_async_process (struct bpf_async_cb *cb) +{ +u32 op, timer_mode; +u64 nsec; +int err; + + err =bpf_async_read_op (cb, &op, &nsec, &timer_mode); +if (err) +return err; + +bpf_async_process_op (cb, op, nsec, timer_mode); +return 0 ; +} + +static void bpf_async_irq_worker (struct irq_work *work) +{ + struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker); + +bpf_async_process (cb); +bpf_async_refcount_put (cb); +} +/* * This function is called by map_delete/update_elem for individual element and * by ops->map_release_uref when the user space reference to a map reaches zero. */ void bpf_timer_cancel_and_free (void *val) { -struct bpf_hrtimer *t; - - t = (struct bpf_hrtimer *)__bpf_async_cancel_and_free (val); -if (!t) -return ; - -bpf_timer_delete (t); +__bpf_async_cancel_and_free (val); }/* This function is called by map_delete/update_elem for individual element and
@@ -1631,19 +1717,7 @@ void bpf_timer_cancel_and_free(void *val)*/ void bpf_wq_cancel_and_free (void *val) { -struct bpf_work *work; - -BTF_TYPE_EMIT (struct bpf_wq ); - - work = (struct bpf_work *)__bpf_async_cancel_and_free (val); -if (!work) -return ; -/* Trigger cancel of the sleepable work, but *do not* wait for -* it to finish if it was running as we might not be in a -* sleepable context. -* kfree will be called once the work has finished. -*/ -schedule_work (&work->delete_work); +__bpf_async_cancel_and_free (val); } BPF_CALL_2(bpf_kptr_xchg, void *, dst, void *, ptr)
@@ -3116,16 +3190,14 @@ __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)struct bpf_async_kern *async = (struct bpf_async_kern *)wq;struct bpf_work *w; -if (in_nmi ()) -return -EOPNOTSUPP ;if (flags)return -EINVAL ; + w =READ_ONCE (async->work);if (!w || !READ_ONCE (w->cb.prog))return -EINVAL ; -schedule_work (&w->work); -return 0 ; +return bpf_async_schedule_op (&w->cb,BPF_ASYNC_START ,0 ,0 ); } __bpf_kfunc int bpf_wq_set_callback(struct bpf_wq *wq,
Signature · 1 line
2.52.0
Re: [PATCH bpf-next v8 1/8] bpf: Enable bpf timer and workqueue use in NMI
Quoted 23 lines · Mykyta Yatsenko
kernel/bpf/helpers.c +21
-0+static int bpf_async_schedule_op (struct bpf_async_cb *cb,enum bpf_async_op op, +u64 nsec,u32 timer_mode) +{ +/* Acquire active writer */ +if (atomic_cmpxchg_acquire (&cb->writer,0 ,1 )) +return -EBUSY ; + +write_seqcount_latch_begin (&cb->latch); + cb->cmd[0 ].nsec = nsec; + cb->cmd[0 ].mode = timer_mode; + cb->cmd[0 ].op = op; +write_seqcount_latch (&cb->latch); + cb->cmd[1 ].nsec = nsec; + cb->cmd[1 ].mode = timer_mode; + cb->cmd[1 ].op = op; +write_seqcount_latch_end (&cb->latch); + +atomic_set_release (&cb->writer,0 ); + +if (!in_nmi ()) +return bpf_async_process (cb);
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1450
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+ +/* NMI path: defer to irq_work with refcount protection */ +if (!refcount_inc_not_zero (&cb->refcnt)) +return -EBUSY ; + +if (!irq_work_queue (&cb->worker)) +bpf_async_refcount_put (cb); + +return 0 ; +}
We've been discussing it in private. Here is the race that prevented this patchset from landing:
CPU 1 (bpf_timer_start) CPU 2 (map_delete_elem)
───────────────────────── ─────────────────────────
bpf_async_schedule_op()
writes cmd[] = START
!in_nmi() -> bpf_async_process()
bpf_async_read_op() reads START
bpf_async_process_op()
// about to call hrtimer_start()
__bpf_async_cancel_and_free()
xchg(&async->cb, NULL)
atomic64_set(last_seq, DESTROY)
bpf_async_process()
bpf_async_read_op() -> CANCEL
hrtimer_try_to_cancel() // NOP!
bpf_async_refcount_put()
refcnt -> 0
call_rcu_tasks_trace()
hrtimer_start() // TIMER STARTED!
return ... RCU tasks trace grace period ...
memory freed
TIMER FIRES -> UAF!
PATCH v8 2/8 bpf: Add verifier support for bpf_timer argument in kfuncs
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Extend the verifier to recognize struct bpf_timer as a valid kfunc argument type. Previously, bpf_timer was only supported in BPF helpers.
This prepares for adding timer-related kfuncs in subsequent patches.
--- kernel/bpf/verifier.c | 75 ++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 54 insertions(+), 21 deletions(-)
kernel/bpf/verifier.c +54 -21
@@ -8656,17 +8656,15 @@ static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno, }static int process_timer_func (struct bpf_verifier_env *env,int regno, -struct bpf_call_arg_meta *meta) +struct bpf_map *map) { -struct bpf_reg_state *reg =reg_state (env, regno); -struct bpf_map *map = reg->map_ptr;int err; err =check_map_field_pointer (env, regno,BPF_TIMER );if (err)return err; -if (meta->map_ptr) { +if (map) {verifier_bug (env,"Two map pointers in a timer helper" );return -EFAULT ; }
@@ -8674,8 +8672,36 @@ static int process_timer_func(struct bpf_verifier_env *env, int regno,verbose (env,"bpf_timer cannot be used for PREEMPT_RT.\n" );return -EOPNOTSUPP ; } +return 0 ; +} + +static int process_timer_helper (struct bpf_verifier_env *env,int regno, +struct bpf_call_arg_meta *meta) +{ +struct bpf_reg_state *reg =reg_state (env, regno); +int err; + + err =process_timer_func (env, regno, meta->map_ptr); +if (err) +return err; + meta->map_uid = reg->map_uid; - meta->map_ptr = map; + meta->map_ptr = reg->map_ptr; +return 0 ; +} + +static int process_timer_kfunc (struct bpf_verifier_env *env,int regno, +struct bpf_kfunc_call_arg_meta *meta) +{ +struct bpf_reg_state *reg =reg_state (env, regno); +int err; + + err =process_timer_func (env, regno, meta->map.ptr); +if (err) +return err; + + meta->map.uid = reg->map_uid; + meta->map.ptr = reg->map_ptr;return 0 ; }
@@ -10003,7 +10029,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, }break ;case ARG_PTR_TO_TIMER : - err =process_timer_func (env, regno, meta); + err =process_timer_helper (env, regno, meta);if (err)return err;break ;
@@ -12268,7 +12294,8 @@ enum {KF_ARG_WORKQUEUE_ID ,KF_ARG_RES_SPIN_LOCK_ID ,KF_ARG_TASK_WORK_ID , - KF_ARG_PROG_AUX_ID +KF_ARG_PROG_AUX_ID , + KF_ARG_TIMER_ID };BTF_ID_LIST (kf_arg_btf_ids)
@@ -12281,6 +12308,7 @@ BTF_ID(struct, bpf_wq) BTF_ID(struct, bpf_res_spin_lock) BTF_ID(struct, bpf_task_work) BTF_ID(struct, bpf_prog_aux) +BTF_ID(struct, bpf_timer)static bool __is_kfunc_ptr_arg_type (const struct btf *btf,const struct btf_param *arg,int type)
@@ -12324,6 +12352,11 @@ static bool is_kfunc_arg_rbtree_node(const struct btf *btf, const struct btf_parreturn __is_kfunc_ptr_arg_type (btf, arg,KF_ARG_RB_NODE_ID ); } +static bool is_kfunc_arg_timer (const struct btf *btf,const struct btf_param *arg) +{ +return __is_kfunc_ptr_arg_type (btf, arg,KF_ARG_TIMER_ID ); +} +static bool is_kfunc_arg_wq (const struct btf *btf,const struct btf_param *arg) {return __is_kfunc_ptr_arg_type (btf, arg,KF_ARG_WORKQUEUE_ID );
@@ -12423,6 +12456,7 @@ enum kfunc_ptr_arg_type {KF_ARG_PTR_TO_NULL ,KF_ARG_PTR_TO_CONST_STR ,KF_ARG_PTR_TO_MAP , +KF_ARG_PTR_TO_TIMER ,KF_ARG_PTR_TO_WORKQUEUE ,KF_ARG_PTR_TO_IRQ_FLAG , KF_ARG_PTR_TO_RES_SPIN_LOCK,
@@ -12668,6 +12702,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,if (is_kfunc_arg_wq (meta->btf, &args[argno]))return KF_ARG_PTR_TO_WORKQUEUE ; +if (is_kfunc_arg_timer (meta->btf, &args[argno])) +return KF_ARG_PTR_TO_TIMER ; +if (is_kfunc_arg_task_work (meta->btf, &args[argno]))return KF_ARG_PTR_TO_TASK_WORK ;
@@ -13454,6 +13491,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_case KF_ARG_PTR_TO_REFCOUNTED_KPTR :case KF_ARG_PTR_TO_CONST_STR :case KF_ARG_PTR_TO_WORKQUEUE : +case KF_ARG_PTR_TO_TIMER :case KF_ARG_PTR_TO_TASK_WORK :case KF_ARG_PTR_TO_IRQ_FLAG :case KF_ARG_PTR_TO_RES_SPIN_LOCK :
@@ -13753,6 +13791,15 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_if (ret <0 )return ret;break ; +case KF_ARG_PTR_TO_TIMER : +if (reg->type !=PTR_TO_MAP_VALUE ) { +verbose (env,"arg#%d doesn't point to a map value\n" , i); +return -EINVAL ; + } + ret =process_timer_kfunc (env, regno, meta); +if (ret <0 ) +return ret; +break ;case KF_ARG_PTR_TO_TASK_WORK :if (reg->type !=PTR_TO_MAP_VALUE ) {verbose (env,"arg#%d doesn't point to a map value\n" , i);
@@ -21331,20 +21378,6 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, } } -if (btf_record_has_field (map->record,BPF_TIMER )) { -if (is_tracing_prog_type (prog_type)) { -verbose (env,"tracing progs cannot use bpf_timer yet\n" ); -return -EINVAL ; - } - } - -if (btf_record_has_field (map->record,BPF_WORKQUEUE )) { -if (is_tracing_prog_type (prog_type)) { -verbose (env,"tracing progs cannot use bpf_wq yet\n" ); -return -EINVAL ; - } - } -if ((bpf_prog_is_offloaded (prog->aux) ||bpf_map_is_offloaded (map)) && !bpf_offload_prog_map_match (prog, map)) {verbose (env,"offload device mismatch between prog and map\n" );
Signature · 1 line
2.52.0
PATCH v8 3/8 bpf: Introduce bpf_timer_cancel_async() kfunc
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
introducing bpf timer cancel kfunc that attempts canceling timer asynchronously, hence, supports working in NMI context.
--- kernel/bpf/helpers.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
kernel/bpf/helpers.c +13 -0
@@ -4437,6 +4437,18 @@ __bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)return 0 ; } +__bpf_kfunc int bpf_timer_cancel_async(struct bpf_timer *timer) +{ +struct bpf_async_cb *cb; +struct bpf_async_kern *async = (void *)timer; + + cb =READ_ONCE (async->cb); +if (!cb) +return -EINVAL ; + +return bpf_async_schedule_op (cb,BPF_ASYNC_CANCEL ,0 ,0 ); +} +__bpf_kfunc_end_defs ();static void bpf_task_work_cancel_scheduled (struct irq_work *irq_work)
@@ -4618,6 +4630,7 @@ BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_dynptr_from_file) BTF_ID_FLAGS(func, bpf_dynptr_file_discard) +BTF_ID_FLAGS(func, bpf_timer_cancel_async)BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {
Signature · 1 line
2.52.0
PATCH v8 4/8 selftests/bpf: Refactor timer selftests
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Refactor timer selftests, extracting stress test into a separate test. This makes it easier to debug test failures and allows to extend.
--- tools/testing/selftests/bpf/prog_tests/timer.c | 55 +++++++++++++++++--------- 1 file changed, 36 insertions(+), 19 deletions(-)
tools/testing/selftests/bpf/prog_tests/timer.c +36 -19
@@ -22,13 +22,35 @@ static void *spin_lock_thread(void *arg)pthread_exit (arg); } -static int timer (struct timer *timer_skel) + +static int timer_stress (struct timer *timer_skel) { -int i, err, prog_fd; +int i, err =1 , prog_fd;LIBBPF_OPTS (bpf_test_run_opts, topts);pthread_t thread_id[NUM_THR ];void *ret; + prog_fd =bpf_program__fd (timer_skel->progs.race); +for (i =0 ; i <NUM_THR ; i++) { + err =pthread_create (&thread_id[i],NULL , + &spin_lock_thread, &prog_fd); +if (!ASSERT_OK (err,"pthread_create" )) +break ; + } + +while (i) { + err =pthread_join (thread_id[--i], &ret); +if (ASSERT_OK (err,"pthread_join" )) +ASSERT_EQ (ret, (void *)&prog_fd,"pthread_join" ); + } +return err; +} + +static int timer (struct timer *timer_skel) +{ +int err, prog_fd; +LIBBPF_OPTS (bpf_test_run_opts, topts); + err =timer__attach (timer_skel);if (!ASSERT_OK (err,"timer_attach" ))return err;
@@ -63,25 +85,10 @@ static int timer(struct timer *timer_skel)/* check that code paths completed */ ASSERT_EQ (timer_skel->bss->ok,1 |2 |4 ,"ok" ); - prog_fd =bpf_program__fd (timer_skel->progs.race); -for (i =0 ; i <NUM_THR ; i++) { - err =pthread_create (&thread_id[i],NULL , - &spin_lock_thread, &prog_fd); -if (!ASSERT_OK (err,"pthread_create" )) -break ; - } - -while (i) { - err =pthread_join (thread_id[--i], &ret); -if (ASSERT_OK (err,"pthread_join" )) -ASSERT_EQ (ret, (void *)&prog_fd,"pthread_join" ); - } -return 0 ; } -/* TODO: use pid filtering */ -void serial_test_timer (void ) +static void test_timer (int (*timer_test_fn)(struct timer *timer_skel)) {struct timer *timer_skel =NULL ;int err;
@@ -94,13 +101,23 @@ void serial_test_timer(void)if (!ASSERT_OK_PTR (timer_skel,"timer_skel_load" ))return ; - err =timer (timer_skel); + err =timer_test_fn (timer_skel);ASSERT_OK (err,"timer" );timer__destroy (timer_skel); +} + +void serial_test_timer (void ) +{ +test_timer (timer);RUN_TESTS (timer_failure); } +void serial_test_timer_stress (void ) +{ +test_timer (timer_stress); +} +void test_timer_interrupt (void ) {struct timer_interrupt *skel =NULL ;
Signature · 1 line
2.52.0
PATCH v8 5/8 selftests/bpf: Add stress test for timer async cancel
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Extend BPF timer selftest to run stress test for async cancel.
--- tools/testing/selftests/bpf/prog_tests/timer.c | 18 +++++++++++++++++- tools/testing/selftests/bpf/progs/timer.c | 14 +++++++++++--- 2 files changed, 28 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/prog_tests/timer.c +17 -1
@@ -23,13 +23,14 @@ static void *spin_lock_thread(void *arg) } -static int timer_stress (struct timer *timer_skel) +static int timer_stress_runner (struct timer *timer_skel,bool async_cancel) {int i, err =1 , prog_fd;LIBBPF_OPTS (bpf_test_run_opts, topts);pthread_t thread_id[NUM_THR ];void *ret; + timer_skel->bss->async_cancel = async_cancel; prog_fd =bpf_program__fd (timer_skel->progs.race);for (i =0 ; i <NUM_THR ; i++) { err = pthread_create(&thread_id[i], NULL,
@@ -46,6 +47,16 @@ static int timer_stress(struct timer *timer_skel)return err; } +static int timer_stress (struct timer *timer_skel) +{ +return timer_stress_runner (timer_skel, false); +} + +static int timer_stress_async_cancel (struct timer *timer_skel) +{ +return timer_stress_runner (timer_skel, true); +} +static int timer (struct timer *timer_skel) {int err, prog_fd;
@@ -118,6 +129,11 @@ void serial_test_timer_stress(void)test_timer (timer_stress); } +void serial_test_timer_stress_async_cancel (void ) +{ +test_timer (timer_stress_async_cancel); +} +void test_timer_interrupt (void ) {struct timer_interrupt *skel =NULL ;
tools/testing/selftests/bpf/progs/timer.c +11 -3
@@ -1,13 +1,17 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2021 Facebook */ -#include <linux/bpf.h> -#include <time.h> + +#include <vmlinux.h> #include <stdbool.h> #include <errno.h> #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> +#define CLOCK_MONOTONIC 1 +#define CLOCK_BOOTTIME 7 + char _license[] SEC("license") = "GPL"; +struct hmap_elem {int counter;struct bpf_timer timer;
@@ -63,6 +67,7 @@ __u64 callback_check = 52;__u64 callback2_check =52 ;__u64 pinned_callback_check;__s32 pinned_cpu; +bool async_cancel =0 ;#define ARRAY 1#define HTAB 2
@@ -419,7 +424,10 @@ int race(void *ctx)bpf_timer_set_callback (timer, race_timer_callback);bpf_timer_start (timer,0 ,0 ); -bpf_timer_cancel (timer); +if (async_cancel) +bpf_timer_cancel_async (timer); +else +bpf_timer_cancel (timer);return 0 ; }
Signature · 1 line
2.52.0
PATCH v8 6/8 selftests/bpf: Verify bpf_timer_cancel_async works
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Add test that verifies that bpf_timer_cancel_async works: can cancel callback successfully.
--- tools/testing/selftests/bpf/prog_tests/timer.c | 25 +++++++++++++++++++++++++ tools/testing/selftests/bpf/progs/timer.c | 23 +++++++++++++++++++++++ 2 files changed, 48 insertions(+)
tools/testing/selftests/bpf/prog_tests/timer.c +25 -0
@@ -99,6 +99,26 @@ static int timer(struct timer *timer_skel)return 0 ; } +static int timer_cancel_async (struct timer *timer_skel) +{ +int err, prog_fd; +LIBBPF_OPTS (bpf_test_run_opts, topts); + + prog_fd =bpf_program__fd (timer_skel->progs.test_async_cancel_succeed); + err =bpf_prog_test_run_opts (prog_fd, &topts); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (topts.retval,0 ,"test_run" ); + +usleep (500 ); +/* check that there were no errors in timer execution */ +ASSERT_EQ (timer_skel->bss->err,0 ,"err" ); + +/* check that code paths completed */ +ASSERT_EQ (timer_skel->bss->ok,1 |2 |4 ,"ok" ); + +return 0 ; +} +static void test_timer (int (*timer_test_fn)(struct timer *timer_skel)) {struct timer *timer_skel =NULL ;
@@ -134,6 +154,11 @@ void serial_test_timer_stress_async_cancel(void)test_timer (timer_stress_async_cancel); } +void serial_test_timer_async_cancel (void ) +{ +test_timer (timer_cancel_async); +} +void test_timer_interrupt (void ) {struct timer_interrupt *skel =NULL ;
tools/testing/selftests/bpf/progs/timer.c +23 -0
@@ -169,6 +169,29 @@ int BPF_PROG2(test1, int, a)return 0 ; } +static int timer_error (void *map,int *key,struct bpf_timer *timer) +{ + err =42 ; +return 0 ; +} + +SEC("syscall") +int test_async_cancel_succeed (void *ctx) +{ +struct bpf_timer *arr_timer; +int array_key =ARRAY ; + + arr_timer =bpf_map_lookup_elem (&array, &array_key); +if (!arr_timer) +return 0 ; +bpf_timer_init (arr_timer, &array,CLOCK_MONOTONIC ); +bpf_timer_set_callback (arr_timer, timer_error); +bpf_timer_start (arr_timer,100000 /* 100us */ ,0 ); +bpf_timer_cancel_async (arr_timer); + ok =7 ; +return 0 ; +} +/* callback for prealloc and non-prealloca hashtab timers */ static int timer_cb2 (void *map,int *key,struct hmap_elem *val) {
Signature · 1 line
2.52.0
PATCH v8 7/8 selftests/bpf: Add timer stress test in NMI context
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Add stress tests for BPF timers that run in NMI context using perf_event programs attached to PERF_COUNT_HW_CPU_CYCLES.
The tests cover three scenarios:
- nmi_race: Tests concurrent timer start and async cancel operations
- nmi_update: Tests updating a map element (effectively deleting and inserting new for array map) from within a timer callback
- nmi_cancel: Tests timer self-cancellation attempt.
A common test_common() helper is used to share timer setup logic across all test modes.
The tests spawn multiple threads in a child process to generate perf events, which trigger the BPF programs in NMI context. Hit counters verify that the NMI code paths were actually exercised.
--- tools/testing/selftests/bpf/prog_tests/timer.c | 158 +++++++++++++++++++++++++ tools/testing/selftests/bpf/progs/timer.c | 85 +++++++++++-- 2 files changed, 231 insertions(+), 12 deletions(-)
tools/testing/selftests/bpf/prog_tests/timer.c +158 -0
@@ -1,12 +1,27 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2021 Facebook */ +#include <sched.h> #include <test_progs.h> +#include <linux/perf_event.h> +#include <sys/syscall.h> #include "timer.skel.h" #include "timer_failure.skel.h" #include "timer_interrupt.skel.h" #define NUM_THR 8 +static int perf_event_open (__u32 type,__u64 config,int pid,int cpu) +{ +struct perf_event_attr attr = { + .type = type, + .config = config, + .size =sizeof (struct perf_event_attr ), + .sample_period =10000 , + }; + +return syscall (__NR_perf_event_open, &attr, pid, cpu,-1 ,0 ); +} +static void *spin_lock_thread (void *arg) {int i, err, prog_fd = *(int *)arg;
@@ -57,6 +72,134 @@ static int timer_stress_async_cancel(struct timer *timer_skel)return timer_stress_runner (timer_skel, true); } +static void *nmi_cpu_worker (void *arg) +{ +volatile __u64 num =1 ; +int i; + +for (i =0 ; i <500000000 ; ++i) + num *= (i %7 ) +1 ; + (void )num; + +return NULL ; +} + +static int run_nmi_test (struct timer *timer_skel,struct bpf_program *prog) +{ +struct bpf_link *link =NULL ; +int pe_fd =-1 , pipefd[2 ] = {-1 ,-1 }, pid =0 , status; +char buf =0 ; +int ret =-1 ; + +if (!ASSERT_OK (pipe (pipefd),"pipe" )) + goto cleanup; + + pid =fork (); +if (pid ==0 ) { +/* Child: spawn multiple threads to consume multiple CPUs */ +pthread_t threads[NUM_THR ]; +int i; + +close (pipefd[1 ]); +read (pipefd[0 ], &buf,1 ); +close (pipefd[0 ]); + +for (i =0 ; i <NUM_THR ; i++) +pthread_create (&threads[i],NULL , nmi_cpu_worker,NULL ); +for (i =0 ; i <NUM_THR ; i++) +pthread_join (threads[i],NULL ); +exit (0 ); + } + +if (!ASSERT_GE (pid,0 ,"fork" )) + goto cleanup; + +/* Open perf event for child process across all CPUs */ + pe_fd =perf_event_open (PERF_TYPE_HARDWARE , +PERF_COUNT_HW_CPU_CYCLES , + pid,/* measure child process */ +-1 );/* on any CPU */ +if (pe_fd <0 ) { +if (errno ==ENOENT || errno ==EOPNOTSUPP ) { +printf ("SKIP:no PERF_COUNT_HW_CPU_CYCLES\n" ); +test__skip (); + ret =EOPNOTSUPP ; + goto cleanup; + } +ASSERT_GE (pe_fd,0 ,"perf_event_open" ); + goto cleanup; + } + + link =bpf_program__attach_perf_event (prog, pe_fd); +if (!ASSERT_OK_PTR (link,"attach_perf_event" )) + goto cleanup; + pe_fd =-1 ;/* Ownership transferred to link */ + +/* Signal child to start CPU work */ +close (pipefd[0 ]); + pipefd[0 ] =-1 ; +write (pipefd[1 ], &buf,1 ); +close (pipefd[1 ]); + pipefd[1 ] =-1 ; + +waitpid (pid, &status,0 ); + pid =0 ; + +/* Verify NMI context was hit */ +ASSERT_GT (timer_skel->bss->test_hits,0 ,"test_hits" ); + ret =0 ; + +cleanup: +bpf_link__destroy (link); +if (pe_fd >=0 ) +close (pe_fd); +if (pid >0 ) { +write (pipefd[1 ], &buf,1 ); +waitpid (pid, &status,0 ); + } +if (pipefd[0 ] >=0 ) +close (pipefd[0 ]); +if (pipefd[1 ] >=0 ) +close (pipefd[1 ]); +return ret; +} + +static int timer_stress_nmi_race (struct timer *timer_skel) +{ +int err; + + err =run_nmi_test (timer_skel, timer_skel->progs.nmi_race); +if (err ==EOPNOTSUPP ) +return 0 ; +return err; +} + +static int timer_stress_nmi_update (struct timer *timer_skel) +{ +int err; + + err =run_nmi_test (timer_skel, timer_skel->progs.nmi_update); +if (err ==EOPNOTSUPP ) +return 0 ; +if (err) +return err; +ASSERT_GT (timer_skel->bss->update_hits,0 ,"update_hits" ); +return 0 ; +} + +static int timer_stress_nmi_cancel (struct timer *timer_skel) +{ +int err; + + err =run_nmi_test (timer_skel, timer_skel->progs.nmi_cancel); +if (err ==EOPNOTSUPP ) +return 0 ; +if (err) +return err; +ASSERT_GT (timer_skel->bss->cancel_hits,0 ,"cancel_hits" ); +return 0 ; +} +static int timer (struct timer *timer_skel) {int err, prog_fd;
@@ -159,6 +302,21 @@ void serial_test_timer_async_cancel(void)test_timer (timer_cancel_async); } +void serial_test_timer_stress_nmi_race (void ) +{ +test_timer (timer_stress_nmi_race); +} + +void serial_test_timer_stress_nmi_update (void ) +{ +test_timer (timer_stress_nmi_update); +} + +void serial_test_timer_stress_nmi_cancel (void ) +{ +test_timer (timer_stress_nmi_cancel); +} +void test_timer_interrupt (void ) {struct timer_interrupt *skel =NULL ;
tools/testing/selftests/bpf/progs/timer.c +73 -12
109 lines not on this page · raw message
Signature · 1 line
2.52.0
PATCH v8 8/8 selftests/bpf: Removed obsolete tests
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Now bpf_timer can be used in tracepoints, so these tests are no longer relevant.
--- .../bpf/progs/verifier_helper_restricted.c | 111 --------------------- 1 file changed, 111 deletions(-)
tools/testing/selftests/bpf/progs/verifier_helper_restricted.c +0 -111
@@ -17,17 +17,6 @@ struct { __type(value, struct val); } map_spin_lock SEC(".maps"); -struct timer { -struct bpf_timer t; -}; - -struct { - __uint(type, BPF_MAP_TYPE_ARRAY); - __uint(max_entries, 1); - __type(key, int); - __type(value, struct timer); -} map_timer SEC(".maps"); - SEC("kprobe") __description("bpf_ktime_get_coarse_ns is forbidden in BPF_PROG_TYPE_KPROBE") __failure __msg("program of this type cannot use helper bpf_ktime_get_coarse_ns")
@@ -84,106 +73,6 @@ __naked void bpf_prog_type_raw_tracepoint_1(void) : __clobber_all); } -SEC("kprobe") -__description("bpf_timer_init isn restricted in BPF_PROG_TYPE_KPROBE") -__failure __msg("tracing progs cannot use bpf_timer yet") -__naked void in_bpf_prog_type_kprobe_2(void) -{ - asmvolatile (" \ -r2 = r10; \ -r2 += -8; \ -r1 = 0; \ -*(u64*)(r2 + 0) = r1; \ -r1 = %[map_timer] ll; \ -call %[bpf_map_lookup_elem]; \ -if r0 == 0 goto l0_%=; \ -r1 = r0; \ -r2 = %[map_timer] ll; \ -r3 = 1; \ -l0_%=: call %[bpf_timer_init]; \ -exit; \ -" : - : __imm(bpf_map_lookup_elem), -__imm (bpf_timer_init), -__imm_addr (map_timer) - : __clobber_all); -} - -SEC("perf_event") -__description("bpf_timer_init is forbidden in BPF_PROG_TYPE_PERF_EVENT") -__failure __msg("tracing progs cannot use bpf_timer yet") -__naked void bpf_prog_type_perf_event_2(void) -{ - asmvolatile (" \ -r2 = r10; \ -r2 += -8; \ -r1 = 0; \ -*(u64*)(r2 + 0) = r1; \ -r1 = %[map_timer] ll; \ -call %[bpf_map_lookup_elem]; \ -if r0 == 0 goto l0_%=; \ -r1 = r0; \ -r2 = %[map_timer] ll; \ -r3 = 1; \ -l0_%=: call %[bpf_timer_init]; \ -exit; \ -" : - : __imm(bpf_map_lookup_elem), -__imm (bpf_timer_init), -__imm_addr (map_timer) - : __clobber_all); -} - -SEC("tracepoint") -__description("bpf_timer_init is forbidden in BPF_PROG_TYPE_TRACEPOINT") -__failure __msg("tracing progs cannot use bpf_timer yet") -__naked void in_bpf_prog_type_tracepoint_2(void) -{ - asmvolatile (" \ -r2 = r10; \ -r2 += -8; \ -r1 = 0; \ -*(u64*)(r2 + 0) = r1; \ -r1 = %[map_timer] ll; \ -call %[bpf_map_lookup_elem]; \ -if r0 == 0 goto l0_%=; \ -r1 = r0; \ -r2 = %[map_timer] ll; \ -r3 = 1; \ -l0_%=: call %[bpf_timer_init]; \ -exit; \ -" : - : __imm(bpf_map_lookup_elem), -__imm (bpf_timer_init), -__imm_addr (map_timer) - : __clobber_all); -} - -SEC("raw_tracepoint") -__description("bpf_timer_init is forbidden in BPF_PROG_TYPE_RAW_TRACEPOINT") -__failure __msg("tracing progs cannot use bpf_timer yet") -__naked void bpf_prog_type_raw_tracepoint_2(void) -{ - asmvolatile (" \ -r2 = r10; \ -r2 += -8; \ -r1 = 0; \ -*(u64*)(r2 + 0) = r1; \ -r1 = %[map_timer] ll; \ -call %[bpf_map_lookup_elem]; \ -if r0 == 0 goto l0_%=; \ -r1 = r0; \ -r2 = %[map_timer] ll; \ -r3 = 1; \ -l0_%=: call %[bpf_timer_init]; \ -exit; \ -" : - : __imm(bpf_map_lookup_elem), -__imm (bpf_timer_init), -__imm_addr (map_timer) - : __clobber_all); -} - SEC("kprobe") __description("bpf_spin_lock is forbidden in BPF_PROG_TYPE_KPROBE") __failure __msg("tracing progs cannot use bpf_spin_lock yet")
Signature · 1 line
2.52.0