bpf: Avoid locks in bpf_timer and bpf_wq
Subsystems (4)
v6 cover bpf: Avoid locks in bpf_timer and bpf_wq
This series reworks implementation of BPF timer and workqueue APIs. The goal is to make both timers and wq non-blocking, enabling their use in NMI context.
Today this code relies on a bpf_spin_lock embedded in the map element to serialize:
* init of the async object,
* setting/changing the callback and bpf_prog
* starting/cancelling the timer/work
* tearing down when the map element is deleted or the map’s user ref is dropped
The basic design approach in this series:
* Use irq_work to offload all blocking work from NMI
* Introduce refcount to guarantee lifetime of the bpf_async_cb structs deferred to potentially multiple irq_work callbacks
* Keep objects under RCU protection to make sure they are not freed while kfuncs/helpers access them (We can't use refcnt for this, as refcnt itself is part of the bpf_async_cb struct)
---
Changes in v6:
- Reworked destruction and refcnt use:
- On cancel_and_free() set last_seq to BPF_ASYNC_DESTROY value, drop map's reference
- In irq work callback, atomically switch DESTROY to DESTROYED, cancel timer/wq
- Free bpf_async_cb on refcnt going to 0.
- Link to v5: https://lore.kernel.org/r/[email protected]
Changes in v5:
- Extracted lock-free algorithm for updating cb->prog andcb->callback_fn into a function bpf_async_update_prog_callback(),
added a new commit and introduces this function and uses it in __bpf_async_set_callback(), bpf_timer_cancel() and bpf_async_cancel_and_free().
This allows to move the change into the separate commit without breaking correctness.
- Handle NULL prog in bpf_async_update_prog_callback().
- Link to v4: https://lore.kernel.org/r/[email protected]
Changes in v4:
- Handle irq_work_queue failures in both schedule and cancel_and_free
paths: introduced bpf_async_refcnt_dec_cleanup() that decrements refcnt and makes sure if last reference is put, there is at least one irq_work scheduled to execute final cleanup.
- Additional refcnt inc/dec in set_callback() + rcu lock to make sure cleanup is not running at the same time as set_callback().
- Added READ_ONCE where it was needed.
- Squash 'bpf: Refactor __bpf_async_set_callback()' commit into 'bpf:
Add lock-free cell for NMI-safe
async operations'
- Removed mpmc_cell, use seqcount_latch_t instead.
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- Major rework
- Introduce mpmc_cell, allowing concurrent writes and reads
- Implement irq_work deferring
- Adding selftests
- Introduces bpf_timer_cancel_async kfunc
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Move refcnt initialization and put (from cancel_and_free())
from patch 5 into the patch 4, so that patch 4 has more clear and full implementation and use of refcnt
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Mykyta Yatsenko (10):
bpf: Factor out timer deletion helper
bpf: Remove unnecessary arguments from bpf_async_set_callback()
bpf: Introduce lock-free bpf_async_update_prog_callback()
bpf: Simplify bpf_timer_cancel()
bpf: Enable bpf timer and workqueue use in NMI
bpf: Add verifier support for bpf_timer argument in kfuncs
bpf: Introduce bpf_timer_cancel_async() kfunc selftests/bpf: Refactor timer selftests selftests/bpf: Add stress test for timer async cancel selftests/bpf: Verify bpf_timer_cancel_async works
kernel/bpf/helpers.c | 526 ++++++++++++++----------- kernel/bpf/verifier.c | 59 ++- tools/testing/selftests/bpf/prog_tests/timer.c | 92 ++++- tools/testing/selftests/bpf/progs/timer.c | 37 +- 4 files changed, 464 insertions(+), 250 deletions(-)
---
base-commit: efad162f5a840ae178e7761c176c49f433c7bb68
change-id: 20251028-timer_nolock-457f5b9daace
Best regards,
Signature · 1 line
Mykyta Yatsenko <[email protected]>
PATCH v6 1/10 bpf: Factor out timer deletion helper
From: Mykyta Yatsenko <[email protected]>
Move the timer deletion logic into a dedicated bpf_timer_delete() helper so it can be reused by later patches.
--- kernel/bpf/helpers.c | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-)
kernel/bpf/helpers.c +18 -11
@@ -1558,18 +1558,10 @@ static struct bpf_async_cb *__bpf_async_cancel_and_free(struct bpf_async_kern *areturn cb; } -/* This function is called by map_delete/update_elem for individual element and -* by ops->map_release_uref when the user space reference to a map reaches zero. -*/ -void bpf_timer_cancel_and_free (void *val) +static void bpf_timer_delete (struct bpf_hrtimer *t) { -struct bpf_hrtimer *t; - - t = (struct bpf_hrtimer *)__bpf_async_cancel_and_free (val); - -if (!t) -return ; -/* We check that bpf_map_delete/update_elem() was called from timer +/* +* We check that bpf_map_delete/update_elem() was called from timer * callback_fn. In such case we don't call hrtimer_cancel() (since it * will deadlock) and don't call hrtimer_try_to_cancel() (since it will * just return -1). Though callback_fn is still running on this cpu it's
@@ -1618,6 +1610,21 @@ void bpf_timer_cancel_and_free(void *val) } } +/* +* This function is called by map_delete/update_elem for individual element and +* by ops->map_release_uref when the user space reference to a map reaches zero. +*/ +void bpf_timer_cancel_and_free (void *val) +{ +struct bpf_hrtimer *t; + + t = (struct bpf_hrtimer *)__bpf_async_cancel_and_free (val); +if (!t) +return ; + +bpf_timer_delete (t); +} +/* This function is called by map_delete/update_elem for individual element and * by ops->map_release_uref when the user space reference to a map reaches zero. */
Signature · 1 line
2.52.0
PATCH v6 2/10 bpf: Remove unnecessary arguments from bpf_async_set_callback()
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Remove unused arguments from __bpf_async_set_callback().
--- kernel/bpf/helpers.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
kernel/bpf/helpers.c +4 -5
@@ -1355,10 +1355,9 @@ static const struct bpf_func_proto bpf_timer_init_proto = { };static int __bpf_async_set_callback (struct bpf_async_kern *async,void *callback_fn, -struct bpf_prog_aux *aux,unsigned int flags, -enum bpf_async_type type) +struct bpf_prog *prog) { -struct bpf_prog *prev, *prog = aux->prog; +struct bpf_prog *prev;struct bpf_async_cb *cb;int ret =0 ;
@@ -1403,7 +1402,7 @@ static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn, struct bpf_prog_aux *, aux) { -return __bpf_async_set_callback (timer, callback_fn, aux,0 ,BPF_ASYNC_TYPE_TIMER ); +return __bpf_async_set_callback (timer, callback_fn, aux->prog); }static const struct bpf_func_proto bpf_timer_set_callback_proto = {
@@ -3138,7 +3137,7 @@ __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,if (flags)return -EINVAL ; -return __bpf_async_set_callback (async, callback_fn, aux, flags,BPF_ASYNC_TYPE_WQ ); +return __bpf_async_set_callback (async, callback_fn, aux->prog); } __bpf_kfunc void bpf_preempt_disable(void)
Signature · 1 line
2.52.0
PATCH v6 3/10 bpf: Introduce lock-free bpf_async_update_prog_callback()
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Introduce bpf_async_update_prog_callback(): lock-free update of cb->prog and cb->callback_fn. This function allows updating prog and callback_fn fields of the struct bpf_async_cb without holding lock.
For now use it under the lock from __bpf_async_set_callback(), in the next patches that lock will be removed.
Lock-free algorithm:
* Acquire a guard reference on prog to prevent it from being freed during the retry loop.
* Retry loop:
1. Each iteration acquires a new prog reference and stores it in cb->prog via xchg. The previous prog is released.
2. The loop condition checks if both cb->prog and cb->callback_fn match what we just wrote. If either differs, a concurrent writer overwrote our value, and we must retry.
3. When we retry, our previously-stored prog was already released by the concurrent writer or will be released by us after overwriting.
* Release guard reference.
--- kernel/bpf/helpers.c | 67 +++++++++++++++++++++++++++++----------------------- 1 file changed, 37 insertions(+), 30 deletions(-)
kernel/bpf/helpers.c +37 -30
@@ -1354,10 +1354,43 @@ static const struct bpf_func_proto bpf_timer_init_proto = { .arg3_type = ARG_ANYTHING, }; +static int bpf_async_update_prog_callback (struct bpf_async_cb *cb,void *callback_fn, +struct bpf_prog *prog) +{ +struct bpf_prog *prev; + +/* Acquire a guard reference on prog to prevent it from being freed during the loop */ +if (prog) { + prog =bpf_prog_inc_not_zero (prog); +if (IS_ERR (prog)) +return PTR_ERR (prog); + } + +do { +if (prog) + prog =bpf_prog_inc_not_zero (prog); + prev =xchg (&cb->prog, prog); +rcu_assign_pointer (cb->callback_fn, callback_fn); + +/* +* Release previous prog, make sure that if other CPU is contending, +* to set bpf_prog, references are not leaked as each iteration acquires and +* releases one reference. +*/ +if (prev) +bpf_prog_put (prev); + + }while (READ_ONCE (cb->prog) != prog ||READ_ONCE (cb->callback_fn) != callback_fn); + +if (prog) +bpf_prog_put (prog); + +return 0 ; +} +static int __bpf_async_set_callback (struct bpf_async_kern *async,void *callback_fn,struct bpf_prog *prog) { -struct bpf_prog *prev;struct bpf_async_cb *cb;int ret =0 ;
@@ -1378,22 +1411,7 @@ static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback ret = -EPERM ; goto out; } - prev = cb->prog; -if (prev != prog) { -/* Bump prog refcnt once. Every bpf_timer_set_callback() -* can pick different callback_fn-s within the same prog. -*/ - prog =bpf_prog_inc_not_zero (prog); -if (IS_ERR (prog)) { - ret =PTR_ERR (prog); - goto out; - } -if (prev) -/* Drop prev prog refcnt when swapping with new prog */ -bpf_prog_put (prev); - cb->prog = prog; - } -rcu_assign_pointer (cb->callback_fn, callback_fn); + ret =bpf_async_update_prog_callback (cb, callback_fn, prog); out:__bpf_spin_unlock_irqrestore (&async->lock);return ret;
@@ -1453,17 +1471,6 @@ static const struct bpf_func_proto bpf_timer_start_proto = { .arg3_type = ARG_ANYTHING, }; -static void drop_prog_refcnt (struct bpf_async_cb *async) -{ -struct bpf_prog *prog = async->prog; - -if (prog) { -bpf_prog_put (prog); - async->prog =NULL ; -rcu_assign_pointer (async->callback_fn,NULL ); - } -} - BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, timer) {struct bpf_hrtimer *t, *cur_t;
@@ -1514,7 +1521,7 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, timer) goto out; } drop: -drop_prog_refcnt (&t->cb); +bpf_async_update_prog_callback (&t->cb,NULL ,NULL ); out:__bpf_spin_unlock_irqrestore (&timer->lock);/* Cancel the timer and wait for associated callback to finish
@@ -1547,7 +1554,7 @@ static struct bpf_async_cb *__bpf_async_cancel_and_free(struct bpf_async_kern *a cb = async->cb;if (!cb) goto out; -drop_prog_refcnt (cb); +bpf_async_update_prog_callback (cb,NULL ,NULL );/* The subsequent bpf_timer_start/cancel() helpers won't be able to use * this timer, since it won't be initialized. */
Signature · 1 line
2.52.0
kernel test robot
build botRe: [PATCH bpf-next v6 03/10] bpf: Introduce lock-free bpf_async_update_prog_callback()
kernel test robot noticed the following build warnings:
Hi Mykyta,
kernel test robot noticed the following build warnings:
[auto build test WARNING on efad162f5a840ae178e7761c176c49f433c7bb68]
url: https://github.com/intel-lab-lkp/linux/commits/Mykyta-Yatsenko/bpf-Factor-out-timer-deletion-helper/20260121-031948 base: efad162f5a840ae178e7761c176c49f433c7bb68 patch link: https://lore.kernel.org/r/20260120-timer_nolock-v6-3-670ffdd787b4%40meta.com patch subject: [PATCH bpf-next v6 03/10] bpf: Introduce lock-free bpf_async_update_prog_callback() config: arc-randconfig-r112-20260121 (https://download.01.org/0day-ci/archive/20260121/[email protected]/config) compiler: arc-linux-gcc (GCC) 13.4.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260121/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
kernel/bpf/helpers.c:1203:21: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected unsigned long long ( *[usertype] callback_fn )( ... ) @@ got void [noderef] __rcu * @@
kernel/bpf/helpers.c:1203:21: sparse: expected unsigned long long ( *[usertype] callback_fn )( ... )
kernel/bpf/helpers.c:1203:21: sparse: got void [noderef] __rcu *
>> kernel/bpf/helpers.c:1383:76: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/bpf/helpers.c:1383:76: sparse: void [noderef] __rcu *
kernel/bpf/helpers.c:1383:76: sparse: void *
kernel/bpf/helpers.c:2546:18: sparse: sparse: symbol 'bpf_task_release_dtor' was not declared. Should it be static? kernel/bpf/helpers.c:2576:18: sparse: sparse: symbol 'bpf_cgroup_release_dtor' was not declared. Should it be static? kernel/bpf/helpers.c:3045:18: sparse: sparse: context imbalance in 'bpf_rcu_read_lock' - wrong count at exit kernel/bpf/helpers.c:3050:18: sparse: sparse: context imbalance in 'bpf_rcu_read_unlock' - unexpected unlock
vim +1383 kernel/bpf/helpers.c
1190
1191 static void bpf_wq_work(struct work_struct *work)
1192 {
1193 struct bpf_work *w = container_of(work, struct bpf_work, work);
1194 struct bpf_async_cb *cb = &w->cb;
1195 struct bpf_map *map = cb->map;
1196 bpf_callback_t callback_fn;
1197 void *value = cb->value;
1198 void *key;
1199 u32 idx;
1200
1201 BTF_TYPE_EMIT(struct bpf_wq);
1202
> 1203 callback_fn = READ_ONCE(cb->callback_fn);
1204 if (!callback_fn)
1205 return;
1206
1207 key = map_key_from_value(map, value, &idx);
1208
1209 rcu_read_lock_trace();
1210 migrate_disable();
1211
1212 callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
1213
1214 migrate_enable();
1215 rcu_read_unlock_trace();
1216 }
1217
1218 static void bpf_async_cb_rcu_free(struct rcu_head *rcu)
1219 {
1220 struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
1221
1222 kfree_nolock(cb);
1223 }
1224
1225 static void bpf_wq_delete_work(struct work_struct *work)
1226 {
1227 struct bpf_work *w = container_of(work, struct bpf_work, delete_work);
1228
1229 cancel_work_sync(&w->work);
1230
1231 call_rcu(&w->cb.rcu, bpf_async_cb_rcu_free);
1232 }
1233
1234 static void bpf_timer_delete_work(struct work_struct *work)
1235 {
1236 struct bpf_hrtimer *t = container_of(work, struct bpf_hrtimer, cb.delete_work);
1237
1238 /* Cancel the timer and wait for callback to complete if it was running.
1239 * If hrtimer_cancel() can be safely called it's safe to call
1240 * call_rcu() right after for both preallocated and non-preallocated
1241 * maps. The async->cb = NULL was already done and no code path can see
1242 * address 't' anymore. Timer if armed for existing bpf_hrtimer before
1243 * bpf_timer_cancel_and_free will have been cancelled.
1244 */
1245 hrtimer_cancel(&t->timer);
1246 call_rcu(&t->cb.rcu, bpf_async_cb_rcu_free);
1247 }
1248
1249 static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u64 flags,
1250 enum bpf_async_type type)
1251 {
1252 struct bpf_async_cb *cb;
1253 struct bpf_hrtimer *t;
1254 struct bpf_work *w;
1255 clockid_t clockid;
1256 size_t size;
1257 int ret = 0;
1258
1259 if (in_nmi())
1260 return -EOPNOTSUPP;
1261
1262 switch (type) {
1263 case BPF_ASYNC_TYPE_TIMER:
1264 size = sizeof(struct bpf_hrtimer);
1265 break;
1266 case BPF_ASYNC_TYPE_WQ:
1267 size = sizeof(struct bpf_work);
1268 break;
1269 default:
1270 return -EINVAL;
1271 }
1272
1273 __bpf_spin_lock_irqsave(&async->lock);
1274 t = async->timer;
1275 if (t) {
1276 ret = -EBUSY;
1277 goto out;
1278 }
1279
1280 cb = bpf_map_kmalloc_nolock(map, size, 0, map->numa_node);
1281 if (!cb) {
1282 ret = -ENOMEM;
1283 goto out;
1284 }
1285
1286 switch (type) {
1287 case BPF_ASYNC_TYPE_TIMER:
1288 clockid = flags & (MAX_CLOCKS - 1);
1289 t = (struct bpf_hrtimer *)cb;
1290
1291 atomic_set(&t->cancelling, 0);
1292 INIT_WORK(&t->cb.delete_work, bpf_timer_delete_work);
1293 hrtimer_setup(&t->timer, bpf_timer_cb, clockid, HRTIMER_MODE_REL_SOFT);
1294 cb->value = (void *)async - map->record->timer_off;
1295 break;
1296 case BPF_ASYNC_TYPE_WQ:
1297 w = (struct bpf_work *)cb;
1298
1299 INIT_WORK(&w->work, bpf_wq_work);
1300 INIT_WORK(&w->delete_work, bpf_wq_delete_work);
1301 cb->value = (void *)async - map->record->wq_off;
1302 break;
1303 }
1304 cb->map = map;
1305 cb->prog = NULL;
1306 cb->flags = flags;
1307 rcu_assign_pointer(cb->callback_fn, NULL);
1308
1309 WRITE_ONCE(async->cb, cb);
1310 /* Guarantee the order between async->cb and map->usercnt. So
1311 * when there are concurrent uref release and bpf timer init, either
1312 * bpf_timer_cancel_and_free() called by uref release reads a no-NULL
1313 * timer or atomic64_read() below returns a zero usercnt.
1314 */
1315 smp_mb();
1316 if (!atomic64_read(&map->usercnt)) {
1317 /* maps with timers must be either held by user space
1318 * or pinned in bpffs.
1319 */
1320 WRITE_ONCE(async->cb, NULL);
1321 kfree_nolock(cb);
1322 ret = -EPERM;
1323 }
1324 out:
1325 __bpf_spin_unlock_irqrestore(&async->lock);
1326 return ret;
1327 }
1328
1329 BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map,
1330 u64, flags)
1331 {
1332 clock_t clockid = flags & (MAX_CLOCKS - 1);
1333
1334 BUILD_BUG_ON(MAX_CLOCKS != 16);
1335 BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_timer));
1336 BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_timer));
1337
1338 if (flags >= MAX_CLOCKS ||
1339 /* similar to timerfd except _ALARM variants are not supported */
1340 (clockid != CLOCK_MONOTONIC &&
1341 clockid != CLOCK_REALTIME &&
1342 clockid != CLOCK_BOOTTIME))
1343 return -EINVAL;
1344
1345 return __bpf_async_init(timer, map, flags, BPF_ASYNC_TYPE_TIMER);
1346 }
1347
1348 static const struct bpf_func_proto bpf_timer_init_proto = {
1349 .func = bpf_timer_init,
1350 .gpl_only = true,
1351 .ret_type = RET_INTEGER,
1352 .arg1_type = ARG_PTR_TO_TIMER,
1353 .arg2_type = ARG_CONST_MAP_PTR,
1354 .arg3_type = ARG_ANYTHING,
1355 };
1356
1357 static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callback_fn,
1358 struct bpf_prog *prog)
1359 {
1360 struct bpf_prog *prev;
1361
1362 /* Acquire a guard reference on prog to prevent it from being freed during the loop */
1363 if (prog) {
1364 prog = bpf_prog_inc_not_zero(prog);
1365 if (IS_ERR(prog))
1366 return PTR_ERR(prog);
1367 }
1368
1369 do {
1370 if (prog)
1371 prog = bpf_prog_inc_not_zero(prog);
1372 prev = xchg(&cb->prog, prog);
1373 rcu_assign_pointer(cb->callback_fn, callback_fn);
1374
1375 /*
1376 * Release previous prog, make sure that if other CPU is contending,
1377 * to set bpf_prog, references are not leaked as each iteration acquires and
1378 * releases one reference.
1379 */
1380 if (prev)
1381 bpf_prog_put(prev);
1382
> 1383 } while (READ_ONCE(cb->prog) != prog || READ_ONCE(cb->callback_fn) != callback_fn); 1384
1385 if (prog)
1386 bpf_prog_put(prog);
1387
1388 return 0;
1389 }
1390
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH v6 4/10 bpf: Simplify bpf_timer_cancel()
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Remove lock from the bpf_timer_cancel() helper. The lock does not protect from concurrent modification of the bpf_async_cb data fields as those are modified in the callback without locking.
Use guard(rcu)() instead of pair of explicit lock()/unlock().
--- kernel/bpf/helpers.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-)
kernel/bpf/helpers.c +11 -16
@@ -1471,7 +1471,7 @@ static const struct bpf_func_proto bpf_timer_start_proto = { .arg3_type = ARG_ANYTHING, }; -BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, timer) +BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, async) {struct bpf_hrtimer *t, *cur_t;bool inc = false;
@@ -1479,13 +1479,12 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, timer)if (in_nmi ())return -EOPNOTSUPP ; -rcu_read_lock (); -__bpf_spin_lock_irqsave (&timer->lock); - t = timer->timer; -if (!t) { - ret = -EINVAL ; - goto out; - } + +guard (rcu)(); + + t =READ_ONCE (async->timer); +if (!t) +return -EINVAL ; cur_t =this_cpu_read (hrtimer_running);if (cur_t == t) {
@@ -1493,8 +1492,7 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, timer)* its own timer the hrtimer_cancel() will deadlock * since it waits for callback_fn to finish. */ - ret = -EDEADLK ; - goto out; +return -EDEADLK ; }/* Only account in-flight cancellations when invoked from a timer
@@ -1517,20 +1515,17 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, timer)* cancelling and waiting for it synchronously, since it might * do the same. Bail! */ - ret = -EDEADLK ; - goto out; +atomic_dec (&t->cancelling); +return -EDEADLK ; } drop:bpf_async_update_prog_callback (&t->cb,NULL ,NULL ); -out: -__bpf_spin_unlock_irqrestore (&timer->lock);/* Cancel the timer and wait for associated callback to finish * if it was running. */ - ret = ret ?:hrtimer_cancel (&t->timer); + ret =hrtimer_cancel (&t->timer);if (inc)atomic_dec (&t->cancelling); -rcu_read_unlock ();return ret; }
Signature · 1 line
2.52.0
PATCH v6 5/10 bpf: Enable bpf timer and workqueue use in NMI
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Refactor bpf timer and workqueue helpers to allow calling them from NMI context by making all operations lock-free and deferring NMI-unsafe work to irq_work.
Previously, bpf_timer_start(), and bpf_wq_start()
could not be called from NMI context because they acquired bpf_spin_lock and called hrtimer/schedule_work APIs directly. This patch removes these limitations.
Key changes:
* Remove bpf_spin_lock from struct bpf_async_kern.
* Initialize/Destroy via setting/unsetting bpf_async_cb pointer atomically.
* Add per-bpf_async_cb irq_work to defer NMI-unsafe operations (hrtimer_start, hrtimer_try_to_cancel, schedule_work) from NMI to softirq context.
* Use the lock-free seqcount_latch_t to pass operation commands (start/cancel/free) and parameters
from NMI-safe callers to the irq_work handler.
* Add reference counting to bpf_async_cb to ensure the object stays alive until all scheduled irq_work completes.
* Move bpf_prog_put() to RCU callback to handle races between set_callback() and cancel_and_free().
* Modify cancel_and_free() path:
* Detach bpf_async_cb.
* Signal destruction to irq_work side via setting last_seq to BPF_ASYNC_DESTROY.
* On receiving BPF_ASYNC_DESTROY, cancel timer/wq.
* Free bpf_async_cb on refcnt reaching 0, wait for both rcu and rcu task trace grace periods before freeing the bpf_async_cb. Removed unnecessary rcu locks, as kfunc/helper allways assumes rcu or rcu task trace lock.
This enables BPF programs attached to NMI-context hooks (perf events) to use timers and workqueues for deferred processing.
--- kernel/bpf/helpers.c | 423 +++++++++++++++++++++++++++++---------------------- 1 file changed, 240 insertions(+), 183 deletions(-)
kernel/bpf/helpers.c +240 -183
@@ -29,6 +29,7 @@#include <linux/task_work.h> #include <linux/irq_work.h> #include <linux/buildid.h> +#include <linux/seqlock.h> #include "../../lib/kstrtox.h"
@@ -1095,16 +1096,42 @@ static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)return (void *)value -round_up (map->key_size,8 ); } +enum bpf_async_type { +BPF_ASYNC_TYPE_TIMER =0 , +BPF_ASYNC_TYPE_WQ , +}; + +enum bpf_async_op { +BPF_ASYNC_START , +BPF_ASYNC_CANCEL , +BPF_ASYNC_CANCEL_AND_FREE , +}; + +enum bpf_async_seq_state { +BPF_ASYNC_DESTROY = (u64 )U32_MAX +1 , +BPF_ASYNC_DESTROYED = (u64 )U32_MAX +2 , +}; + +struct bpf_async_cmd { +u64 nsec; +u32 mode; +enum bpf_async_op op; +}; +struct bpf_async_cb {struct bpf_map *map;struct bpf_prog *prog;void __rcu *callback_fn;void *value; -union { -struct rcu_head rcu; -struct work_struct delete_work; - }; +struct rcu_head rcu;u64 flags; +struct irq_work worker; +atomic_t writer; +seqcount_latch_t latch; +struct bpf_async_cmd cmd[2 ]; +atomic64_t last_seq; +refcount_t refcnt; +enum bpf_async_type type; };/* BPF map elements can contain 'struct bpf_timer'.
@@ -1132,7 +1159,6 @@ struct bpf_hrtimer {struct bpf_work {struct bpf_async_cb cb;struct work_struct work; -struct work_struct delete_work; };/* the actual struct hidden inside uapi struct bpf_timer and bpf_wq */
@@ -1142,18 +1168,9 @@ struct bpf_async_kern {struct bpf_hrtimer *timer;struct bpf_work *work; }; -/* bpf_spin_lock is used here instead of spinlock_t to make -* sure that it always fits into space reserved by struct bpf_timer -* regardless of LOCKDEP and spinlock debug flags. -*/ -struct bpf_spin_lock lock; +u32 __pad;/* Left for binary compatibility, previously stored spinlock */ } __attribute__((aligned(8))); -enum bpf_async_type { -BPF_ASYNC_TYPE_TIMER =0 , -BPF_ASYNC_TYPE_WQ , -}; - static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running);static enum hrtimer_restart bpf_timer_cb (struct hrtimer *hrtimer)
@@ -1219,45 +1236,53 @@ static void bpf_async_cb_rcu_free(struct rcu_head *rcu) { struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu); +/* +* Drop the last reference to prog only after RCU GP, as set_callback() +* may race with cancel_and_free() +*/ +if (cb->prog) +bpf_prog_put (cb->prog); +kfree_nolock (cb); } -static void bpf_wq_delete_work (struct work_struct *work) +/* Callback from call_rcu_tasks_trace, chains to call_rcu for final free */ +static void bpf_async_cb_rcu_tasks_trace_free (struct rcu_head *rcu) { - struct bpf_work *w = container_of(work, struct bpf_work, delete_work); - -cancel_work_sync (&w->work); - -call_rcu (&w->cb.rcu, bpf_async_cb_rcu_free); +/* +* If RCU Tasks Trace grace period implies RCU grace period, +* there is no need to invoke call_rcu(). +*/ +if (rcu_trace_implies_rcu_gp ()) +bpf_async_cb_rcu_free (rcu); +else +call_rcu (rcu, bpf_async_cb_rcu_free); } -static void bpf_timer_delete_work (struct work_struct *work) +/* +* Decrement refcount and if it reaches zero, schedule deferred cleanup +* through call_rcu_tasks_trace() -> call_rcu() -> bpf_prog_put()/kfree() +*/ +static void bpf_async_refcount_put (struct bpf_async_cb *cb) { - struct bpf_hrtimer *t = container_of(work, struct bpf_hrtimer, cb.delete_work); +if (!refcount_dec_and_test (&cb->refcnt)) +return ; -/* Cancel the timer and wait for callback to complete if it was running. -* If hrtimer_cancel() can be safely called it's safe to call -* call_rcu() right after for both preallocated and non-preallocated -* maps. The async->cb = NULL was already done and no code path can see -* address 't' anymore. Timer if armed for existing bpf_hrtimer before -* bpf_timer_cancel_and_free will have been cancelled. -*/ -hrtimer_cancel (&t->timer); -call_rcu (&t->cb.rcu, bpf_async_cb_rcu_free); +/* Both timer and wq callbacks run under RCU lock, UAF should not be possible there */ +call_rcu_tasks_trace (&cb->rcu, bpf_async_cb_rcu_tasks_trace_free); } +static void __bpf_async_cancel_and_free (struct bpf_async_kern *async); +static void bpf_async_irq_worker (struct irq_work *work); +static int __bpf_async_init (struct bpf_async_kern *async,struct bpf_map *map,u64 flags,enum bpf_async_type type) { -struct bpf_async_cb *cb; +struct bpf_async_cb *cb, *old_cb;struct bpf_hrtimer *t;struct bpf_work *w;clockid_t clockid;size_t size; -int ret =0 ; - -if (in_nmi ()) -return -EOPNOTSUPP ;switch (type) {case BPF_ASYNC_TYPE_TIMER :
@@ -1270,18 +1295,13 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, ureturn -EINVAL ; } -__bpf_spin_lock_irqsave (&async->lock); - t = async->timer; -if (t) { - ret = -EBUSY ; - goto out; - } + old_cb =READ_ONCE (async->cb); +if (old_cb) +return -EBUSY ; cb =bpf_map_kmalloc_nolock (map, size,0 , map->numa_node); -if (!cb) { - ret = -ENOMEM ; - goto out; - } +if (!cb) +return -ENOMEM ;switch (type) {case BPF_ASYNC_TYPE_TIMER :
@@ -1289,7 +1309,6 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u t = (struct bpf_hrtimer *)cb;atomic_set (&t->cancelling,0 ); -INIT_WORK (&t->cb.delete_work, bpf_timer_delete_work);hrtimer_setup (&t->timer, bpf_timer_cb, clockid,HRTIMER_MODE_REL_SOFT ); cb->value = (void *)async - map->record->timer_off;break ;
@@ -1297,16 +1316,26 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u w = (struct bpf_work *)cb;INIT_WORK (&w->work, bpf_wq_work); -INIT_WORK (&w->delete_work, bpf_wq_delete_work); cb->value = (void *)async - map->record->wq_off;break ; } cb->map = map; cb->prog =NULL ; cb->flags = flags; + cb->worker =IRQ_WORK_INIT (bpf_async_irq_worker); +seqcount_latch_init (&cb->latch); +atomic_set (&cb->writer,0 ); +refcount_set (&cb->refcnt,1 );/* map's reference */ +atomic64_set (&cb->last_seq,0 ); + cb->type = type;rcu_assign_pointer (cb->callback_fn,NULL ); -WRITE_ONCE (async->cb, cb); + old_cb =cmpxchg (&async->cb,NULL , cb); +if (old_cb) { +/* Lost the race to initialize this bpf_async_kern, drop the allocated object */ +kfree_nolock (cb); +return -EBUSY ; + }/* Guarantee the order between async->cb and map->usercnt. So * when there are concurrent uref release and bpf timer init, either * bpf_timer_cancel_and_free() called by uref release reads a no-NULL
@@ -1317,13 +1346,11 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u/* maps with timers must be either held by user space * or pinned in bpffs. */ -WRITE_ONCE (async->cb,NULL ); -kfree_nolock (cb); - ret = -EPERM ; +__bpf_async_cancel_and_free (async); +return -EPERM ; } -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return ret; + +return 0 ; } BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map,
@@ -1388,33 +1415,97 @@ static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbacreturn 0 ; } +static int bpf_async_schedule_op (struct bpf_async_cb *cb,enum bpf_async_op op, +u64 nsec,u32 timer_mode) +{ +/* Acquire active writer */ +if (atomic_cmpxchg_acquire (&cb->writer,0 ,1 )) +return -EBUSY ; + +write_seqcount_latch_begin (&cb->latch); + cb->cmd[0 ].nsec = nsec; + cb->cmd[0 ].mode = timer_mode; + cb->cmd[0 ].op = op; +write_seqcount_latch (&cb->latch); + cb->cmd[1 ].nsec = nsec; + cb->cmd[1 ].mode = timer_mode; + cb->cmd[1 ].op = op; +write_seqcount_latch_end (&cb->latch); + +atomic_set_release (&cb->writer,0 ); + +if (!refcount_inc_not_zero (&cb->refcnt)) +return -EBUSY ; + +if (!in_nmi ()) { +bpf_async_irq_worker (&cb->worker); +return 0 ; + } + +if (!irq_work_queue (&cb->worker)) +bpf_async_refcount_put (cb); + +return 0 ; +} + +static int bpf_async_handle_terminal_seq (struct bpf_async_cb *cb,s64 last_seq, +enum bpf_async_op *op) +{ +s64 expected =BPF_ASYNC_DESTROY ; + +if (last_seq !=BPF_ASYNC_DESTROY ) +return -EBUSY ; + +if (!atomic64_try_cmpxchg_release (&cb->last_seq, &expected,BPF_ASYNC_DESTROYED )) +return -EBUSY ;/* Someone else set it to DESTROYED, bail */ + + *op =BPF_ASYNC_CANCEL ; +return 0 ; +} + +static int bpf_async_read_op (struct bpf_async_cb *cb,enum bpf_async_op *op, +u64 *nsec,u32 *flags) +{ +u32 seq, idx; +s64 last_seq; + +while (true) { + last_seq =atomic64_read_acquire (&cb->last_seq); +if (last_seq >U32_MAX )/* Check if terminal seq num has been set */ +return bpf_async_handle_terminal_seq (cb, last_seq, op); + + seq =raw_read_seqcount_latch (&cb->latch); + +/* Return -EBUSY if current seq is consumed by another reader */ +if (seq == last_seq) +return -EBUSY ; + + idx = seq &1 ; + *nsec = cb->cmd[idx].nsec; + *flags = cb->cmd[idx].mode; + *op = cb->cmd[idx].op; + +if (raw_read_seqcount_latch_retry (&cb->latch, seq)) +continue ; + +/* Commit read sequence number, own snapshot exclusively */ +if (atomic64_try_cmpxchg_release (&cb->last_seq, &last_seq, seq)) +break ; + } + +return 0 ; +} +static int __bpf_async_set_callback (struct bpf_async_kern *async,void *callback_fn,struct bpf_prog *prog) {struct bpf_async_cb *cb; -int ret =0 ; -if (in_nmi ()) -return -EOPNOTSUPP ; -__bpf_spin_lock_irqsave (&async->lock); - cb = async->cb; -if (!cb) { - ret = -EINVAL ; - goto out; - } -if (!atomic64_read (&cb->map->usercnt)) { -/* maps with timers must be either held by user space -* or pinned in bpffs. Otherwise timer might still be -* running even when bpf prog is detached and user space -* is gone, since map_release_uref won't ever be called. -*/ - ret = -EPERM ; - goto out; - } - ret =bpf_async_update_prog_callback (cb, callback_fn, prog); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return ret; + cb =READ_ONCE (async->cb); +if (!cb) +return -EINVAL ; + +return bpf_async_update_prog_callback (cb, callback_fn, prog); } BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn,
@@ -1431,22 +1522,17 @@ static const struct bpf_func_proto bpf_timer_set_callback_proto = { .arg2_type = ARG_PTR_TO_FUNC, }; -BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, timer, u64, nsecs, u64, flags) +BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, async, u64, nsecs, u64, flags) {struct bpf_hrtimer *t; -int ret =0 ; -enum hrtimer_mode mode; +u32 mode; -if (in_nmi ()) -return -EOPNOTSUPP ;if (flags & ~(BPF_F_TIMER_ABS |BPF_F_TIMER_CPU_PIN ))return -EINVAL ; -__bpf_spin_lock_irqsave (&timer->lock); - t = timer->timer; -if (!t || !t->cb.prog) { - ret = -EINVAL ; - goto out; - } + + t =READ_ONCE (async->timer); +if (!t || !READ_ONCE (t->cb.prog)) +return -EINVAL ;if (flags &BPF_F_TIMER_ABS ) mode =HRTIMER_MODE_ABS_SOFT ;
@@ -1456,10 +1542,7 @@ BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, timer, u64, nsecs, u64, flaif (flags &BPF_F_TIMER_CPU_PIN ) mode |=HRTIMER_MODE_PINNED ; -hrtimer_start (&t->timer,ns_to_ktime (nsecs), mode); -out: -__bpf_spin_unlock_irqrestore (&timer->lock); -return ret; +return bpf_async_schedule_op (&t->cb,BPF_ASYNC_START , nsecs, mode); }static const struct bpf_func_proto bpf_timer_start_proto = {
@@ -1480,8 +1563,6 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, async)if (in_nmi ())return -EOPNOTSUPP ; -guard (rcu)(); - t =READ_ONCE (async->timer);if (!t)return -EINVAL ;
@@ -1536,79 +1617,75 @@ static const struct bpf_func_proto bpf_timer_cancel_proto = { .arg1_type = ARG_PTR_TO_TIMER, }; -static struct bpf_async_cb *__bpf_async_cancel_and_free (struct bpf_async_kern *async) +static void __bpf_async_cancel_and_free (struct bpf_async_kern *async) {struct bpf_async_cb *cb; -/* Performance optimization: read async->cb without lock first. */ -if (!READ_ONCE (async->cb)) -return NULL ; - -__bpf_spin_lock_irqsave (&async->lock); -/* re-read it under lock */ - cb = async->cb; + cb =xchg (&async->cb,NULL );if (!cb) - goto out; -bpf_async_update_prog_callback (cb,NULL ,NULL ); -/* The subsequent bpf_timer_start/cancel() helpers won't be able to use -* this timer, since it won't be initialized. -*/ -WRITE_ONCE (async->cb,NULL ); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return cb; +return ; + +atomic64_set (&cb->last_seq,BPF_ASYNC_DESTROY ); +/* Pass map's reference to irq_work callback */ +if (!irq_work_queue (&cb->worker)) +bpf_async_refcount_put (cb); } -static void bpf_timer_delete (struct bpf_hrtimer *t) +static void bpf_async_process_op (struct bpf_async_cb *cb,u32 op, +u64 timer_nsec,u32 timer_mode) { -/* -* We check that bpf_map_delete/update_elem() was called from timer -* callback_fn. In such case we don't call hrtimer_cancel() (since it -* will deadlock) and don't call hrtimer_try_to_cancel() (since it will -* just return -1). Though callback_fn is still running on this cpu it's -* safe to do kfree(t) because bpf_timer_cb() read everything it needed -* from 't'. The bpf subprog callback_fn won't be able to access 't', -* since async->cb = NULL was already done. The timer will be -* effectively cancelled because bpf_timer_cb() will return -* HRTIMER_NORESTART. -* -* However, it is possible the timer callback_fn calling us armed the -* timer _before_ calling us, such that failing to cancel it here will -* cause it to possibly use struct hrtimer after freeing bpf_hrtimer. -* Therefore, we _need_ to cancel any outstanding timers before we do -* call_rcu, even though no more timers can be armed. -* -* Moreover, we need to schedule work even if timer does not belong to -* the calling callback_fn, as on two different CPUs, we can end up in a -* situation where both sides run in parallel, try to cancel one -* another, and we end up waiting on both sides in hrtimer_cancel -* without making forward progress, since timer1 depends on time2 -* callback to finish, and vice versa. -* -* CPU 1 (timer1_cb) CPU 2 (timer2_cb) -* bpf_timer_cancel_and_free(timer2) bpf_timer_cancel_and_free(timer1) -* -* To avoid these issues, punt to workqueue context when we are in a -* timer callback. -*/ -if (this_cpu_read (hrtimer_running)) { -queue_work (system_dfl_wq, &t->cb.delete_work); -return ; - } +switch (cb->type) { +case BPF_ASYNC_TYPE_TIMER : { + struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb); -if (IS_ENABLED (CONFIG_PREEMPT_RT )) { -/* If the timer is running on other CPU, also use a kworker to -* wait for the completion of the timer instead of trying to -* acquire a sleepable lock in hrtimer_cancel() to wait for its -* completion. -*/ -if (hrtimer_try_to_cancel (&t->timer) >=0 ) -call_rcu (&t->cb.rcu, bpf_async_cb_rcu_free); -else -queue_work (system_dfl_wq, &t->cb.delete_work); - }else { -bpf_timer_delete_work (&t->cb.delete_work); +switch (op) { +case BPF_ASYNC_START : +hrtimer_start (&t->timer,ns_to_ktime (timer_nsec), timer_mode); +break ; +case BPF_ASYNC_CANCEL : +hrtimer_try_to_cancel (&t->timer); +break ; +default : +break ; + } +break ; + } +case BPF_ASYNC_TYPE_WQ : { + struct bpf_work *w = container_of(cb, struct bpf_work, cb); + +switch (op) { +case BPF_ASYNC_START : +schedule_work (&w->work); +break ; +case BPF_ASYNC_CANCEL : +/* Use non-blocking cancel, safe in irq_work context. +* RCU grace period ensures callback completes before free. +*/ +cancel_work (&w->work); +break ; +default : +break ; + } +break ; } + } +} + +static void bpf_async_irq_worker (struct irq_work *work) +{ + struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker); +u32 op, timer_mode; +u64 nsec; +int err; + + err =bpf_async_read_op (cb, &op, &nsec, &timer_mode); +if (err) + goto out; + +bpf_async_process_op (cb, op, nsec, timer_mode); + +out: +bpf_async_refcount_put (cb); }/*
@@ -1617,13 +1694,7 @@ static void bpf_timer_delete(struct bpf_hrtimer *t)*/ void bpf_timer_cancel_and_free (void *val) { -struct bpf_hrtimer *t; - - t = (struct bpf_hrtimer *)__bpf_async_cancel_and_free (val); -if (!t) -return ; - -bpf_timer_delete (t); +__bpf_async_cancel_and_free (val); }/* This function is called by map_delete/update_elem for individual element and
@@ -1631,19 +1702,7 @@ void bpf_timer_cancel_and_free(void *val)*/ void bpf_wq_cancel_and_free (void *val) { -struct bpf_work *work; - -BTF_TYPE_EMIT (struct bpf_wq ); - - work = (struct bpf_work *)__bpf_async_cancel_and_free (val); -if (!work) -return ; -/* Trigger cancel of the sleepable work, but *do not* wait for -* it to finish if it was running as we might not be in a -* sleepable context. -* kfree will be called once the work has finished. -*/ -schedule_work (&work->delete_work); +__bpf_async_cancel_and_free (val); } BPF_CALL_2(bpf_kptr_xchg, void *, dst, void *, ptr)
@@ -3116,16 +3175,14 @@ __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)struct bpf_async_kern *async = (struct bpf_async_kern *)wq;struct bpf_work *w; -if (in_nmi ()) -return -EOPNOTSUPP ;if (flags)return -EINVAL ; + w =READ_ONCE (async->work);if (!w || !READ_ONCE (w->cb.prog))return -EINVAL ; -schedule_work (&w->work); -return 0 ; +return bpf_async_schedule_op (&w->cb,BPF_ASYNC_START ,0 ,0 ); } __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,
Signature · 1 line
2.52.0
Re: [PATCH bpf-next v6 05/10] bpf: Enable bpf timer and workqueue use in NMI
Quoted 36 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
Refactor bpf timer and workqueue helpers to allow calling them from NMI context by making all operations lock-free and deferring NMI-unsafe work to
irq_work.Previously,
bpf_timer_start(), andbpf_wq_start()
could not be called from NMI context because they acquiredbpf_spin_lockand called hrtimer/schedule_work APIs directly. This patch removes these limitations.Key changes:
* Removebpf_spin_lockfrom structbpf_async_kern.
* Initialize/Destroy via setting/unsettingbpf_async_cbpointer atomically.
* Add per-bpf_async_cbirq_workto defer NMI-unsafe operations (hrtimer_start,hrtimer_try_to_cancel,schedule_work) from NMI to softirq context.
* Use the lock-freeseqcount_latch_tto pass operation commands (start/cancel/free) and parameters
from NMI-safe callers to theirq_workhandler.
* Add reference counting tobpf_async_cbto ensure the object stays alive until all scheduledirq_workcompletes.
* Movebpf_prog_put()to RCU callback to handle races betweenset_callback()andcancel_and_free().
* Modifycancel_and_free()path:
* Detachbpf_async_cb.
* Signal destruction toirq_workside via settinglast_seqtoBPF_ASYNC_DESTROY.
* On receivingBPF_ASYNC_DESTROY, cancel timer/wq.
* Freebpf_async_cbon refcnt reaching 0, wait for both rcu and rcu task trace grace periods before freeing thebpf_async_cb. Removed unnecessary rcu locks, as kfunc/helper allways assumes rcu or rcu task trace lock.
Quoted 7 lines
This enables BPF programs attached to NMI-context hooks (perf events) to use timers and workqueues for deferred processing.
Signed-off-by Mykyta Yatsenko <[email protected]>--- kernel/bpf/helpers.c | 423 +++++++++++++++++++++++++++++---------------------- 1 file changed, 240 insertions(+), 183 deletions(-)
I think we'll need another revision, but all the things I pointed out below do not change the logic in any big way, so it would be great for people that were following along but waited for the overall implementation to stabilize to actually check the implementation now. I think it's basically ready, hopefully the next revision will land, so let's do another round of thorough reviews.
[...]
Quoted 11 lines
kernel/bpf/helpers.c +11
-0+static int bpf_async_schedule_op (struct bpf_async_cb *cb,enum bpf_async_op op, +u64 nsec,u32 timer_mode) +{ +/* Acquire active writer */ +if (atomic_cmpxchg_acquire (&cb->writer,0 ,1 )) +return -EBUSY ; + +write_seqcount_latch_begin (&cb->latch); + cb->cmd[0 ].nsec = nsec; + cb->cmd[0 ].mode = timer_mode; + cb->cmd[0 ].op = op;
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1438
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+write_seqcount_latch (&cb->latch); + cb->cmd[1 ].nsec = nsec; + cb->cmd[1 ].mode = timer_mode; + cb->cmd[1 ].op = op; +write_seqcount_latch_end (&cb->latch); + +atomic_set_release (&cb->writer,0 ); + +if (!refcount_inc_not_zero (&cb->refcnt)) +return -EBUSY ;
let's not do refcount bump unless we are going to schedule irq_work, so move this after !in_nmi() check, and instead of bpf_async_irq_worker() call only part of it that does the work, but doesn't put the refcount. This will speed up common non-NMI case.
[...]
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1474
Quoted 9 lines
kernel/bpf/helpers.c +9
-0+static int bpf_async_read_op (struct bpf_async_cb *cb,enum bpf_async_op *op, +u64 *nsec,u32 *flags) +{ +u32 seq, idx; +s64 last_seq; + +while (true) { + last_seq =atomic64_read_acquire (&cb->last_seq); +if (last_seq >U32_MAX )/* Check if terminal seq num has been set */
unlikely() seems to be warranted here to optimize (a little bit) a common case
Quoted 1 line
+ return
bpf_async_handle_terminal_seq(cb, last_seq, op);
why this helper function, it's called once. It's actually a distraction, IMO, that this logic is split out. Can you please inline it here?
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1481
Quoted 6 lines
kernel/bpf/helpers.c +6
-0+ + seq =raw_read_seqcount_latch (&cb->latch); + +/* Return -EBUSY if current seq is consumed by another reader */ +if (seq == last_seq) +return -EBUSY ;
I'd drop this check, below atomic64_try_cmpxchg() should handle this unlikely situation anyways, no?
Quoted 3 lines
kernel/bpf/helpers.c +3
-0+ + idx = seq &1 ; + *nsec = cb->cmd[idx].nsec;
[...]
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1508
Quoted 9 lines
kernel/bpf/helpers.c +5
-4- ret =bpf_async_update_prog_callback (cb, callback_fn, prog); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return ret; + cb =READ_ONCE (async->cb); +if (!cb) +return -EINVAL ; + +return bpf_async_update_prog_callback (cb, callback_fn, prog);
nit: if you end up with another revision, consider swapping callback and prog arguments order, it doesn't make sense for callback (that belongs to that prog) to be specified before the prog. Super minor, but reads very backwards.
Quoted 3 lines
}
BPF_CALL_3(bpf_timer_set_callback, struct
bpf_async_kern*, timer, void *,callback_fn,
[...]
kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1546
Quoted 13 lines
kernel/bpf/helpers.c +1
-5@@ -1536,79 +1617,75 @@ static const struct bpf_func_proto bpf_timer_cancel_proto = { .arg1_type = ARG_PTR_TO_TIMER, }; -static struct bpf_async_cb *__bpf_async_cancel_and_free (struct bpf_async_kern *async) +static void __bpf_async_cancel_and_free (struct bpf_async_kern *async) {struct bpf_async_cb *cb; -/* Performance optimization: read async->cb without lock first. */ -if (!READ_ONCE (async->cb)) -return NULL ; -
I don't feel strongly about this, but I guess we could leave this performance optimization in place without changing anything else, so why not?
Quoted 8 lines
kernel/bpf/helpers.c +1
-6-__bpf_spin_lock_irqsave (&async->lock); -/* re-read it under lock */ - cb = async->cb; + cb =xchg (&async->cb,NULL );if (!cb) - goto out; -bpf_async_update_prog_callback (cb,NULL ,NULL ); -/* The subsequent bpf_timer_start/cancel() helpers won't be able to use
kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1629
Quoted 10 lines
kernel/bpf/helpers.c +4
-6-* this timer, since it won't be initialized. -*/ -WRITE_ONCE (async->cb,NULL ); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return cb; +return ; + +atomic64_set (&cb->last_seq,BPF_ASYNC_DESTROY ); +/* Pass map's reference to irq_work callback */
I'd expand here, because it's subtle but important. Just to make it very clear: we *attempt* to pass our initial map's reference to irq_work callback, but if we fail to schedule it (meaning it is already scheduled by someone else having their own ref), we have to put that ref here.
It's important to have a noticeable comment here because refcounting here looks asymmetrical (and thus broken at first sight), but it's actually not.
But also, can't we do the same !in_nmi() optimization here as with schedule_op above?
Quoted 4 lines
kernel/bpf/helpers.c +2
-0+if (!irq_work_queue (&cb->worker)) +bpf_async_refcount_put (cb); }
[...]
Quoted 8 lines
kernel/bpf/helpers.c +0
-8-if (IS_ENABLED (CONFIG_PREEMPT_RT )) { -/* If the timer is running on other CPU, also use a kworker to -* wait for the completion of the timer instead of trying to -* acquire a sleepable lock in hrtimer_cancel() to wait for its -* completion. -*/ -if (hrtimer_try_to_cancel (&t->timer) >=0 ) -call_rcu (&t->cb.rcu, bpf_async_cb_rcu_free);
kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1646
Quoted 10 lines
kernel/bpf/helpers.c +6
-4-else -queue_work (system_dfl_wq, &t->cb.delete_work); - }else { -bpf_timer_delete_work (&t->cb.delete_work); +switch (op) { +case BPF_ASYNC_START : +hrtimer_start (&t->timer,ns_to_ktime (timer_nsec), timer_mode); +break ; +case BPF_ASYNC_CANCEL : +hrtimer_try_to_cancel (&t->timer);
I think it's important to point out (in patch description) that we now have asynchronous timer cancellation on cancel_and_free, and thus we don't need all that hrtimer_running logic that you removed above (and a huge comment accompanying it).
Review
Quoted 2 lines
(quoted diff) +2
-0+ break; + default:
Drop BPF_ASYNC_CANCEL_AND_FREE which we don't use anymore, and then you won't need default (and compiler will complain if we add new operation that isn't handled explicitly).
If that doesn't work for some reason, at least WARN_ON_ONCE() here. This shouldn't ever happen, but if it does during development, we better know about that.
Quoted 8 lines
kernel/bpf/helpers.c +8
-0+break ; + } +break ; + } +case BPF_ASYNC_TYPE_WQ : { + struct bpf_work *w = container_of(cb, struct bpf_work, cb); + +switch (op) {
kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1666
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+case BPF_ASYNC_START : +schedule_work (&w->work); +break ; +case BPF_ASYNC_CANCEL : +/* Use non-blocking cancel, safe in irq_work context. +* RCU grace period ensures callback completes before free. +*/ +cancel_work (&w->work); +break ; +default :
same as above
Quoted 9 lines
kernel/bpf/helpers.c +8
-0+break ; + } +break ; } + } +} + +static void bpf_async_irq_worker (struct irq_work *work) +{
kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1685
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+ struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker); +u32 op, timer_mode; +u64 nsec; +int err; + + err =bpf_async_read_op (cb, &op, &nsec, &timer_mode); +if (err) + goto out; + +bpf_async_process_op (cb, op, nsec, timer_mode);
if you split read_op+process_op combo into a separate function, then you won't need refcount manipulations outside of in_nmi context. Let's do it.
Quoted 6 lines
kernel/bpf/helpers.c +3
-0+ +out: +bpf_async_refcount_put (cb); }/*
[...]
Re: [PATCH bpf-next v6 05/10] bpf: Enable bpf timer and workqueue use in NMI
Quoted 71 lines · Andrii Nakryiko
Quoted 35 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
Refactor bpf timer and workqueue helpers to allow calling them from NMI context by making all operations lock-free and deferring NMI-unsafe work to
irq_work.Previously,
bpf_timer_start(), andbpf_wq_start()
could not be called from NMI context because they acquiredbpf_spin_lockand called hrtimer/schedule_work APIs directly. This patch removes these limitations.Key changes:
* Removebpf_spin_lockfrom structbpf_async_kern.
* Initialize/Destroy via setting/unsettingbpf_async_cbpointer atomically.
* Add per-bpf_async_cbirq_workto defer NMI-unsafe operations (hrtimer_start,hrtimer_try_to_cancel,schedule_work) from NMI to softirq context.
* Use the lock-freeseqcount_latch_tto pass operation commands (start/cancel/free) and parameters
from NMI-safe callers to theirq_workhandler.
* Add reference counting tobpf_async_cbto ensure the object stays alive until all scheduledirq_workcompletes.
* Movebpf_prog_put()to RCU callback to handle races betweenset_callback()andcancel_and_free().
* Modifycancel_and_free()path:
* Detachbpf_async_cb.
* Signal destruction toirq_workside via settinglast_seqtoBPF_ASYNC_DESTROY.
* On receivingBPF_ASYNC_DESTROY, cancel timer/wq.
* Freebpf_async_cbon refcnt reaching 0, wait for both rcu and rcu task trace grace periods before freeing thebpf_async_cb. Removed unnecessary rcu locks, as kfunc/helper allways assumes rcu or rcu task trace lock.Quoted 7 lines
This enables BPF programs attached to NMI-context hooks (perf events) to use timers and workqueues for deferred processing.
Signed-off-by Mykyta Yatsenko <[email protected]>--- kernel/bpf/helpers.c | 423 +++++++++++++++++++++++++++++---------------------- 1 file changed, 240 insertions(+), 183 deletions(-)I think we'll need another revision, but all the things I pointed out below do not change the logic in any big way, so it would be great for people that were following along but waited for the overall implementation to stabilize to actually check the implementation now. I think it's basically ready, hopefully the next revision will land, so let's do another round of thorough reviews.
[...]
Quoted 17 lines
kernel/bpf/helpers.c +17
-0+static int bpf_async_schedule_op (struct bpf_async_cb *cb,enum bpf_async_op op, +u64 nsec,u32 timer_mode) +{ +/* Acquire active writer */ +if (atomic_cmpxchg_acquire (&cb->writer,0 ,1 )) +return -EBUSY ; + +write_seqcount_latch_begin (&cb->latch); + cb->cmd[0 ].nsec = nsec; + cb->cmd[0 ].mode = timer_mode; + cb->cmd[0 ].op = op; +write_seqcount_latch (&cb->latch); + cb->cmd[1 ].nsec = nsec; + cb->cmd[1 ].mode = timer_mode; + cb->cmd[1 ].op = op; +write_seqcount_latch_end (&cb->latch); +
Quoted 8 lines
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1438
Quoted 4 lines
kernel/bpf/helpers.c +4
-0+atomic_set_release (&cb->writer,0 ); + +if (!refcount_inc_not_zero (&cb->refcnt)) +return -EBUSY ;let's not do refcount bump unless we are going to schedule
irq_work, so move this after !in_nmi() check, and instead ofbpf_async_irq_worker()call only part of it that does the work, but doesn't put the refcount. This will speed up common non-NMI case.
Thanks for reminding about this one.
Quoted 14 lines
[...]
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1474
Quoted 9 lines
kernel/bpf/helpers.c +9
-0+static int bpf_async_read_op (struct bpf_async_cb *cb,enum bpf_async_op *op, +u64 *nsec,u32 *flags) +{ +u32 seq, idx; +s64 last_seq; + +while (true) { + last_seq =atomic64_read_acquire (&cb->last_seq); +if (last_seq >U32_MAX )/* Check if terminal seq num has been set */
unlikely()seems to be warranted here to optimize (a little bit) a common case
Quoted 4 lines
Quoted 1 line
+ return
bpf_async_handle_terminal_seq(cb, last_seq, op);why this helper function, it's called once. It's actually a distraction, IMO, that this logic is split out. Can you please inline it here?
It looked ugly inlined, I saw value in attaching word "terminal" to the functionality. I'll send inlined in the next version.
Quoted 9 lines
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1481
Quoted 7 lines
kernel/bpf/helpers.c +6
-0+ + seq =raw_read_seqcount_latch (&cb->latch); + +/* Return -EBUSY if current seq is consumed by another reader */ +if (seq == last_seq) +return -EBUSY ;I'd drop this check, below
atomic64_try_cmpxchg()should handle this unlikely situation anyways, no?
I don't think cmpxchg handles this, if seq == last_seq, cmpxchg still succeeds, it only fails if cb->last_seq != last_seq.
Quoted 146 lines
Quoted 4 lines
kernel/bpf/helpers.c +3
-0+ + idx = seq &1 ; + *nsec = cb->cmd[idx].nsec;[...]
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1508
Quoted 9 lines
kernel/bpf/helpers.c +5
-4- ret =bpf_async_update_prog_callback (cb, callback_fn, prog); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return ret; + cb =READ_ONCE (async->cb); +if (!cb) +return -EINVAL ; + +return bpf_async_update_prog_callback (cb, callback_fn, prog);nit: if you end up with another revision, consider swapping callback and prog arguments order, it doesn't make sense for callback (that belongs to that prog) to be specified before the prog. Super minor, but reads very backwards.
Quoted 3 lines
}BPF_CALL_3(bpf_timer_set_callback, struct
bpf_async_kern*, timer, void *,callback_fn,[...]
kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1546
Quoted 13 lines
kernel/bpf/helpers.c +1
-5@@ -1536,79 +1617,75 @@ static const struct bpf_func_proto bpf_timer_cancel_proto = { .arg1_type = ARG_PTR_TO_TIMER, }; -static struct bpf_async_cb *__bpf_async_cancel_and_free (struct bpf_async_kern *async) +static void __bpf_async_cancel_and_free (struct bpf_async_kern *async) {struct bpf_async_cb *cb; -/* Performance optimization: read async->cb without lock first. */ -if (!READ_ONCE (async->cb)) -return NULL ; -I don't feel strongly about this, but I guess we could leave this performance optimization in place without changing anything else, so why not?
Quoted 8 lines
kernel/bpf/helpers.c +1
-6-__bpf_spin_lock_irqsave (&async->lock); -/* re-read it under lock */ - cb = async->cb; + cb =xchg (&async->cb,NULL );if (!cb) - goto out; -bpf_async_update_prog_callback (cb,NULL ,NULL ); -/* The subsequent bpf_timer_start/cancel() helpers won't be able to use kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1629
Quoted 10 lines
kernel/bpf/helpers.c +4
-6-* this timer, since it won't be initialized. -*/ -WRITE_ONCE (async->cb,NULL ); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return cb; +return ; + +atomic64_set (&cb->last_seq,BPF_ASYNC_DESTROY ); +/* Pass map's reference to irq_work callback */ I'd expand here, because it's subtle but important. Just to make it very clear: we *attempt* to pass our initial map's reference to
irq_workcallback, but if we fail to schedule it (meaning it is already scheduled by someone else having their own ref), we have to put that ref here.It's important to have a noticeable comment here because refcounting here looks asymmetrical (and thus broken at first sight), but it's actually not.
But also, can't we do the same !in_nmi() optimization here as with
schedule_opabove?Quoted 4 lines
kernel/bpf/helpers.c +2
-0+if (!irq_work_queue (&cb->worker)) +bpf_async_refcount_put (cb); }[...]
Quoted 8 lines
kernel/bpf/helpers.c +0
-8-if (IS_ENABLED (CONFIG_PREEMPT_RT )) { -/* If the timer is running on other CPU, also use a kworker to -* wait for the completion of the timer instead of trying to -* acquire a sleepable lock in hrtimer_cancel() to wait for its -* completion. -*/ -if (hrtimer_try_to_cancel (&t->timer) >=0 ) -call_rcu (&t->cb.rcu, bpf_async_cb_rcu_free);kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1646
Quoted 10 lines
kernel/bpf/helpers.c +6
-4-else -queue_work (system_dfl_wq, &t->cb.delete_work); - }else { -bpf_timer_delete_work (&t->cb.delete_work); +switch (op) { +case BPF_ASYNC_START : +hrtimer_start (&t->timer,ns_to_ktime (timer_nsec), timer_mode); +break ; +case BPF_ASYNC_CANCEL : +hrtimer_try_to_cancel (&t->timer);I think it's important to point out (in patch description) that we now have asynchronous timer cancellation on
cancel_and_free, and thus we don't need all thathrtimer_runninglogic that you removed above (and a huge comment accompanying it).Review
Quoted 2 lines
(quoted diff) +2
-0+ break; + default:Drop
BPF_ASYNC_CANCEL_AND_FREEwhich we don't use anymore, and then you won't need default (and compiler will complain if we add new operation that isn't handled explicitly).If that doesn't work for some reason, at least
WARN_ON_ONCE()here. This shouldn't ever happen, but if it does during development, we better know about that.Quoted 8 lines
kernel/bpf/helpers.c +8
-0+break ; + } +break ; + } +case BPF_ASYNC_TYPE_WQ : { + struct bpf_work *w = container_of(cb, struct bpf_work, cb); + +switch (op) {kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1666
Quoted 10 lines
kernel/bpf/helpers.c +10
-0+case BPF_ASYNC_START : +schedule_work (&w->work); +break ; +case BPF_ASYNC_CANCEL : +/* Use non-blocking cancel, safe in irq_work context. +* RCU grace period ensures callback completes before free. +*/ +cancel_work (&w->work); +break ; +default :same as above
Quoted 14 lines
kernel/bpf/helpers.c +13
-0+break ; + } +break ; } + } +} + +static void bpf_async_irq_worker (struct irq_work *work) +{ + struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker); +u32 op, timer_mode; +u64 nsec; +int err; +
Quoted 8 lines
kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1685
Quoted 5 lines
kernel/bpf/helpers.c +5
-0+ err =bpf_async_read_op (cb, &op, &nsec, &timer_mode); +if (err) + goto out; + +bpf_async_process_op (cb, op, nsec, timer_mode);if you split read_op+process_op combo into a separate function, then you won't need refcount manipulations outside of
in_nmicontext. Let's do it.
Thanks!
Quoted 7 lines
Quoted 6 lines
kernel/bpf/helpers.c +3
-0+ +out: +bpf_async_refcount_put (cb); }/* [...]
Re: [PATCH bpf-next v6 05/10] bpf: Enable bpf timer and workqueue use in NMI
Quoted 54 lines · Mykyta Yatsenko
Quoted 52 lines · Andrii Nakryiko
Quoted 35 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
Refactor bpf timer and workqueue helpers to allow calling them from NMI context by making all operations lock-free and deferring NMI-unsafe work to
irq_work.Previously,
bpf_timer_start(), andbpf_wq_start()
could not be called from NMI context because they acquiredbpf_spin_lockand called hrtimer/schedule_work APIs directly. This patch removes these limitations.Key changes:
* Removebpf_spin_lockfrom structbpf_async_kern.
* Initialize/Destroy via setting/unsettingbpf_async_cbpointer atomically.
* Add per-bpf_async_cbirq_workto defer NMI-unsafe operations (hrtimer_start,hrtimer_try_to_cancel,schedule_work) from NMI to softirq context.
* Use the lock-freeseqcount_latch_tto pass operation commands (start/cancel/free) and parameters
from NMI-safe callers to theirq_workhandler.
* Add reference counting tobpf_async_cbto ensure the object stays alive until all scheduledirq_workcompletes.
* Movebpf_prog_put()to RCU callback to handle races betweenset_callback()andcancel_and_free().
* Modifycancel_and_free()path:
* Detachbpf_async_cb.
* Signal destruction toirq_workside via settinglast_seqtoBPF_ASYNC_DESTROY.
* On receivingBPF_ASYNC_DESTROY, cancel timer/wq.
* Freebpf_async_cbon refcnt reaching 0, wait for both rcu and rcu task trace grace periods before freeing thebpf_async_cb. Removed unnecessary rcu locks, as kfunc/helper allways assumes rcu or rcu task trace lock.Quoted 7 lines
This enables BPF programs attached to NMI-context hooks (perf events) to use timers and workqueues for deferred processing.
Signed-off-by Mykyta Yatsenko <[email protected]>--- kernel/bpf/helpers.c | 423 +++++++++++++++++++++++++++++---------------------- 1 file changed, 240 insertions(+), 183 deletions(-)I think we'll need another revision, but all the things I pointed out below do not change the logic in any big way, so it would be great for people that were following along but waited for the overall implementation to stabilize to actually check the implementation now. I think it's basically ready, hopefully the next revision will land, so let's do another round of thorough reviews.
[...]
Quoted 13 lines
Quoted 13 lines
[...]
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1474
Quoted 9 lines
kernel/bpf/helpers.c +9
-0+static int bpf_async_read_op (struct bpf_async_cb *cb,enum bpf_async_op *op, +u64 *nsec,u32 *flags) +{ +u32 seq, idx; +s64 last_seq; + +while (true) { + last_seq =atomic64_read_acquire (&cb->last_seq); +if (last_seq >U32_MAX )/* Check if terminal seq num has been set */
unlikely()seems to be warranted here to optimize (a little bit) a common case
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
+ return
bpf_async_handle_terminal_seq(cb, last_seq, op);why this helper function, it's called once. It's actually a distraction, IMO, that this logic is split out. Can you please inline it here?
It looked ugly inlined, I saw value in attaching word "terminal" to the functionality.
I'll send inlined in the next version.
I don't know, BPF_ASYNC_DESTROY -> BPF_ASYNC_DESTROYED has the same terminal viber :)
As for ugly, I don't know, let's see...
if (unlikely(last_seq >= BPF_ASYNC_DESTROY)) {
u64 expected = BPF_ASYNC_DESTROY;
if (!try_cmpxchg_release(&cb->last_seq, &expected, BPF_ASYNC_DESTROYED))
return -EBUSY;
*op = BPF_ASYNC_CANCEL;
return 0;
}Looks fine to me, tbh.
And note, you don't need to check last_seq ahead of time, this is rare event, no need to optimize with this speculative read check (unlike the cancel_and_free one I mentioned in last reply, that one across millions of map elements might add up to something)
Quoted 9 lines
Quoted 9 lines
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1481
Quoted 7 lines
kernel/bpf/helpers.c +6
-0+ + seq =raw_read_seqcount_latch (&cb->latch); + +/* Return -EBUSY if current seq is consumed by another reader */ +if (seq == last_seq) +return -EBUSY ;I'd drop this check, below
atomic64_try_cmpxchg()should handle this unlikely situation anyways, no?
Quoted 3 lines
I don't think cmpxchg handles this, if seq ==
last_seq, cmpxchg still succeeds,
it only fails ifcb->last_seq!=last_seq.
doh, of course, what I've been thinking, sorry, stupid suggestion on my part
Quoted 34 lines
Quoted 34 lines
Quoted 4 lines
kernel/bpf/helpers.c +3
-0+ + idx = seq &1 ; + *nsec = cb->cmd[idx].nsec;[...]
kernel/bpf/helpers.c static int bpf_async_update_prog_callback(struct bpf_async_cb *cb, void *callbac · line 1508
Quoted 9 lines
kernel/bpf/helpers.c +5
-4- ret =bpf_async_update_prog_callback (cb, callback_fn, prog); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return ret; + cb =READ_ONCE (async->cb); +if (!cb) +return -EINVAL ; + +return bpf_async_update_prog_callback (cb, callback_fn, prog);nit: if you end up with another revision, consider swapping callback and prog arguments order, it doesn't make sense for callback (that belongs to that prog) to be specified before the prog. Super minor, but reads very backwards.
Quoted 3 lines
}BPF_CALL_3(bpf_timer_set_callback, struct
bpf_async_kern*, timer, void *,callback_fn,[...]
Quoted 9 lines
kernel/bpf/helpers.c +1
-1@@ -1536,79 +1617,75 @@ static const struct bpf_func_proto bpf_timer_cancel_proto = { .arg1_type = ARG_PTR_TO_TIMER, }; -static struct bpf_async_cb *__bpf_async_cancel_and_free (struct bpf_async_kern *async) +static void __bpf_async_cancel_and_free (struct bpf_async_kern *async) {struct bpf_async_cb *cb;
Quoted 7 lines
Quoted 7 lines
kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1546
Quoted 4 lines
kernel/bpf/helpers.c +0
-4-/* Performance optimization: read async->cb without lock first. */ -if (!READ_ONCE (async->cb)) -return NULL ; -I don't feel strongly about this, but I guess we could leave this performance optimization in place without changing anything else, so why not?
It's useful for reviewers and readers to see acknowledgement of requests like this, as it's not always clear if this was missed or you are just going to apply the suggestion in the next revision. So please consider replying even if you agree.
Quoted 25 lines
Quoted 25 lines
Quoted 9 lines
kernel/bpf/helpers.c +1
-6-__bpf_spin_lock_irqsave (&async->lock); -/* re-read it under lock */ - cb = async->cb; + cb =xchg (&async->cb,NULL );if (!cb) - goto out; -bpf_async_update_prog_callback (cb,NULL ,NULL ); -/* The subsequent bpf_timer_start/cancel() helpers won't be able to use kernel/bpf/helpers.c static const struct bpf_func_proto bpf_timer_cancel_proto = { · line 1629
Quoted 10 lines
kernel/bpf/helpers.c +4
-6-* this timer, since it won't be initialized. -*/ -WRITE_ONCE (async->cb,NULL ); -out: -__bpf_spin_unlock_irqrestore (&async->lock); -return cb; +return ; + +atomic64_set (&cb->last_seq,BPF_ASYNC_DESTROY ); +/* Pass map's reference to irq_work callback */ I'd expand here, because it's subtle but important. Just to make it very clear: we *attempt* to pass our initial map's reference to
irq_workcallback, but if we fail to schedule it (meaning it is already scheduled by someone else having their own ref), we have to put that ref here.
Quoted 7 lines
Quoted 7 lines
It's important to have a noticeable comment here because refcounting here looks asymmetrical (and thus broken at first sight), but it's actually not.
But also, can't we do the same !in_nmi() optimization here as with
schedule_opabove?
same point as above (and few more below), I assume you agree, but it would be nice to ack this to clarify (this is a general point for email-based reviews)
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
kernel/bpf/helpers.c +2
-0+if (!irq_work_queue (&cb->worker)) +bpf_async_refcount_put (cb); }
[...]
PATCH v6 6/10 bpf: Add verifier support for bpf_timer argument in kfuncs
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Extend the verifier to recognize struct bpf_timer as a valid kfunc argument type. Previously, bpf_timer was only supported in BPF helpers.
This prepares for adding timer-related kfuncs in subsequent patches.
--- kernel/bpf/verifier.c | 59 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 53 insertions(+), 6 deletions(-)
kernel/bpf/verifier.c +53 -6
@@ -8566,17 +8566,15 @@ static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno, }static int process_timer_func (struct bpf_verifier_env *env,int regno, -struct bpf_call_arg_meta *meta) +struct bpf_map *map) { -struct bpf_reg_state *reg =reg_state (env, regno); -struct bpf_map *map = reg->map_ptr;int err; err =check_map_field_pointer (env, regno,BPF_TIMER );if (err)return err; -if (meta->map_ptr) { +if (map) {verifier_bug (env,"Two map pointers in a timer helper" );return -EFAULT ; }
@@ -8584,8 +8582,36 @@ static int process_timer_func(struct bpf_verifier_env *env, int regno,verbose (env,"bpf_timer cannot be used for PREEMPT_RT.\n" );return -EOPNOTSUPP ; } +return 0 ; +} + +static int process_timer_helper (struct bpf_verifier_env *env,int regno, +struct bpf_call_arg_meta *meta) +{ +struct bpf_reg_state *reg =reg_state (env, regno); +int err; + + err =process_timer_func (env, regno, meta->map_ptr); +if (err) +return err; + meta->map_uid = reg->map_uid; - meta->map_ptr = map; + meta->map_ptr = reg->map_ptr; +return 0 ; +} + +static int process_timer_kfunc (struct bpf_verifier_env *env,int regno, +struct bpf_kfunc_call_arg_meta *meta) +{ +struct bpf_reg_state *reg =reg_state (env, regno); +int err; + + err =process_timer_func (env, regno, meta->map.ptr); +if (err) +return err; + + meta->map.uid = reg->map_uid; + meta->map.ptr = reg->map_ptr;return 0 ; }
@@ -9913,7 +9939,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, }break ;case ARG_PTR_TO_TIMER : - err =process_timer_func (env, regno, meta); + err =process_timer_helper (env, regno, meta);if (err)return err;break ;
@@ -12166,6 +12192,7 @@ enum {KF_ARG_WORKQUEUE_ID ,KF_ARG_RES_SPIN_LOCK_ID ,KF_ARG_TASK_WORK_ID , +KF_ARG_TIMER_ID , };BTF_ID_LIST (kf_arg_btf_ids)
@@ -12177,6 +12204,7 @@ BTF_ID(struct, bpf_rb_node) BTF_ID(struct, bpf_wq) BTF_ID(struct, bpf_res_spin_lock) BTF_ID(struct, bpf_task_work) +BTF_ID(struct, bpf_timer)static bool __is_kfunc_ptr_arg_type (const struct btf *btf,const struct btf_param *arg,int type)
@@ -12220,6 +12248,11 @@ static bool is_kfunc_arg_rbtree_node(const struct btf *btf, const struct btf_parreturn __is_kfunc_ptr_arg_type (btf, arg,KF_ARG_RB_NODE_ID ); } +static bool is_kfunc_arg_timer (const struct btf *btf,const struct btf_param *arg) +{ +return __is_kfunc_ptr_arg_type (btf, arg,KF_ARG_TIMER_ID ); +} +static bool is_kfunc_arg_wq (const struct btf *btf,const struct btf_param *arg) {return __is_kfunc_ptr_arg_type (btf, arg,KF_ARG_WORKQUEUE_ID );
@@ -12314,6 +12347,7 @@ enum kfunc_ptr_arg_type {KF_ARG_PTR_TO_NULL ,KF_ARG_PTR_TO_CONST_STR ,KF_ARG_PTR_TO_MAP , +KF_ARG_PTR_TO_TIMER ,KF_ARG_PTR_TO_WORKQUEUE ,KF_ARG_PTR_TO_IRQ_FLAG , KF_ARG_PTR_TO_RES_SPIN_LOCK,
@@ -12559,6 +12593,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,if (is_kfunc_arg_wq (meta->btf, &args[argno]))return KF_ARG_PTR_TO_WORKQUEUE ; +if (is_kfunc_arg_timer (meta->btf, &args[argno])) +return KF_ARG_PTR_TO_TIMER ; +if (is_kfunc_arg_task_work (meta->btf, &args[argno]))return KF_ARG_PTR_TO_TASK_WORK ;
@@ -13345,6 +13382,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_case KF_ARG_PTR_TO_REFCOUNTED_KPTR :case KF_ARG_PTR_TO_CONST_STR :case KF_ARG_PTR_TO_WORKQUEUE : +case KF_ARG_PTR_TO_TIMER :case KF_ARG_PTR_TO_TASK_WORK :case KF_ARG_PTR_TO_IRQ_FLAG :case KF_ARG_PTR_TO_RES_SPIN_LOCK :
@@ -13644,6 +13682,15 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_if (ret <0 )return ret;break ; +case KF_ARG_PTR_TO_TIMER : +if (reg->type !=PTR_TO_MAP_VALUE ) { +verbose (env,"arg#%d doesn't point to a map value\n" , i); +return -EINVAL ; + } + ret =process_timer_kfunc (env, regno, meta); +if (ret <0 ) +return ret; +break ;case KF_ARG_PTR_TO_TASK_WORK :if (reg->type !=PTR_TO_MAP_VALUE ) {verbose (env,"arg#%d doesn't point to a map value\n" , i);
Signature · 1 line
2.52.0
PATCH v6 7/10 bpf: Introduce bpf_timer_cancel_async() kfunc
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
introducing bpf timer cancel kfunc that attempts canceling timer asynchronously, hence, supports working in NMI context.
--- kernel/bpf/helpers.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
kernel/bpf/helpers.c +13 -0
@@ -4425,6 +4425,18 @@ __bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)return 0 ; } +__bpf_kfunc int bpf_timer_cancel_async(struct bpf_timer *timer) +{ +struct bpf_async_cb *cb; +struct bpf_async_kern *async = (void *)timer; + + cb =READ_ONCE (async->cb); +if (!cb) +return -EINVAL ; + +return bpf_async_schedule_op (cb,BPF_ASYNC_CANCEL ,0 ,0 ); +} +__bpf_kfunc_end_defs ();static void bpf_task_work_cancel_scheduled (struct irq_work *irq_work)
@@ -4606,6 +4618,7 @@ BTF_ID_FLAGS(func, bpf_task_work_schedule_signal_impl) BTF_ID_FLAGS(func, bpf_task_work_schedule_resume_impl) BTF_ID_FLAGS(func, bpf_dynptr_from_file) BTF_ID_FLAGS(func, bpf_dynptr_file_discard) +BTF_ID_FLAGS(func, bpf_timer_cancel_async)BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {
Signature · 1 line
2.52.0
PATCH v6 8/10 selftests/bpf: Refactor timer selftests
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Refactor timer selftests, extracting stress test into a separate test. This makes it easier to debug test failures and allows to extend.
--- tools/testing/selftests/bpf/prog_tests/timer.c | 55 +++++++++++++++++--------- 1 file changed, 36 insertions(+), 19 deletions(-)
tools/testing/selftests/bpf/prog_tests/timer.c +36 -19
@@ -22,13 +22,35 @@ static void *spin_lock_thread(void *arg)pthread_exit (arg); } -static int timer (struct timer *timer_skel) + +static int timer_stress (struct timer *timer_skel) { -int i, err, prog_fd; +int i, err =1 , prog_fd;LIBBPF_OPTS (bpf_test_run_opts, topts);pthread_t thread_id[NUM_THR ];void *ret; + prog_fd =bpf_program__fd (timer_skel->progs.race); +for (i =0 ; i <NUM_THR ; i++) { + err =pthread_create (&thread_id[i],NULL , + &spin_lock_thread, &prog_fd); +if (!ASSERT_OK (err,"pthread_create" )) +break ; + } + +while (i) { + err =pthread_join (thread_id[--i], &ret); +if (ASSERT_OK (err,"pthread_join" )) +ASSERT_EQ (ret, (void *)&prog_fd,"pthread_join" ); + } +return err; +} + +static int timer (struct timer *timer_skel) +{ +int err, prog_fd; +LIBBPF_OPTS (bpf_test_run_opts, topts); + err =timer__attach (timer_skel);if (!ASSERT_OK (err,"timer_attach" ))return err;
@@ -63,25 +85,10 @@ static int timer(struct timer *timer_skel)/* check that code paths completed */ ASSERT_EQ (timer_skel->bss->ok,1 |2 |4 ,"ok" ); - prog_fd =bpf_program__fd (timer_skel->progs.race); -for (i =0 ; i <NUM_THR ; i++) { - err =pthread_create (&thread_id[i],NULL , - &spin_lock_thread, &prog_fd); -if (!ASSERT_OK (err,"pthread_create" )) -break ; - } - -while (i) { - err =pthread_join (thread_id[--i], &ret); -if (ASSERT_OK (err,"pthread_join" )) -ASSERT_EQ (ret, (void *)&prog_fd,"pthread_join" ); - } -return 0 ; } -/* TODO: use pid filtering */ -void serial_test_timer (void ) +static void test_timer (int (*timer_test_fn)(struct timer *timer_skel)) {struct timer *timer_skel =NULL ;int err;
@@ -94,13 +101,23 @@ void serial_test_timer(void)if (!ASSERT_OK_PTR (timer_skel,"timer_skel_load" ))return ; - err =timer (timer_skel); + err =timer_test_fn (timer_skel);ASSERT_OK (err,"timer" );timer__destroy (timer_skel); +} + +void serial_test_timer (void ) +{ +test_timer (timer);RUN_TESTS (timer_failure); } +void serial_test_timer_stress (void ) +{ +test_timer (timer_stress); +} +void test_timer_interrupt (void ) {struct timer_interrupt *skel =NULL ;
Signature · 1 line
2.52.0
PATCH v6 9/10 selftests/bpf: Add stress test for timer async cancel
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Extend BPF timer selftest to run stress test for async cancel.
--- tools/testing/selftests/bpf/prog_tests/timer.c | 18 +++++++++++++++++- tools/testing/selftests/bpf/progs/timer.c | 14 +++++++++++--- 2 files changed, 28 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/prog_tests/timer.c +17 -1
@@ -23,13 +23,14 @@ static void *spin_lock_thread(void *arg) } -static int timer_stress (struct timer *timer_skel) +static int timer_stress_runner (struct timer *timer_skel,bool async_cancel) {int i, err =1 , prog_fd;LIBBPF_OPTS (bpf_test_run_opts, topts);pthread_t thread_id[NUM_THR ];void *ret; + timer_skel->bss->async_cancel = async_cancel; prog_fd =bpf_program__fd (timer_skel->progs.race);for (i =0 ; i <NUM_THR ; i++) { err = pthread_create(&thread_id[i], NULL,
@@ -46,6 +47,16 @@ static int timer_stress(struct timer *timer_skel)return err; } +static int timer_stress (struct timer *timer_skel) +{ +return timer_stress_runner (timer_skel, false); +} + +static int timer_stress_async_cancel (struct timer *timer_skel) +{ +return timer_stress_runner (timer_skel, true); +} +static int timer (struct timer *timer_skel) {int err, prog_fd;
@@ -118,6 +129,11 @@ void serial_test_timer_stress(void)test_timer (timer_stress); } +void serial_test_timer_stress_async_cancel (void ) +{ +test_timer (timer_stress_async_cancel); +} +void test_timer_interrupt (void ) {struct timer_interrupt *skel =NULL ;
tools/testing/selftests/bpf/progs/timer.c +11 -3
@@ -1,13 +1,17 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2021 Facebook */ -#include <linux/bpf.h> -#include <time.h> + +#include <vmlinux.h> #include <stdbool.h> #include <errno.h> #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> +#define CLOCK_MONOTONIC 1 +#define CLOCK_BOOTTIME 7 + char _license[] SEC("license") = "GPL"; +struct hmap_elem {int counter;struct bpf_timer timer;
@@ -63,6 +67,7 @@ __u64 callback_check = 52;__u64 callback2_check =52 ;__u64 pinned_callback_check;__s32 pinned_cpu; +bool async_cancel =0 ;#define ARRAY 1#define HTAB 2
@@ -419,7 +424,10 @@ int race(void *ctx)bpf_timer_set_callback (timer, race_timer_callback);bpf_timer_start (timer,0 ,0 ); -bpf_timer_cancel (timer); +if (async_cancel) +bpf_timer_cancel_async (timer); +else +bpf_timer_cancel (timer);return 0 ; }
Signature · 1 line
2.52.0
PATCH v6 10/10 selftests/bpf: Verify bpf_timer_cancel_async works
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Add test that verifies that bpf_timer_cancel_async works: can cancel callback successfully.
--- tools/testing/selftests/bpf/prog_tests/timer.c | 25 +++++++++++++++++++++++++ tools/testing/selftests/bpf/progs/timer.c | 23 +++++++++++++++++++++++ 2 files changed, 48 insertions(+)
tools/testing/selftests/bpf/prog_tests/timer.c +25 -0
@@ -99,6 +99,26 @@ static int timer(struct timer *timer_skel)return 0 ; } +static int timer_cancel_async (struct timer *timer_skel) +{ +int err, prog_fd; +LIBBPF_OPTS (bpf_test_run_opts, topts); + + prog_fd =bpf_program__fd (timer_skel->progs.test_async_cancel_succeed); + err =bpf_prog_test_run_opts (prog_fd, &topts); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (topts.retval,0 ,"test_run" ); + +usleep (500 ); +/* check that there were no errors in timer execution */ +ASSERT_EQ (timer_skel->bss->err,0 ,"err" ); + +/* check that code paths completed */ +ASSERT_EQ (timer_skel->bss->ok,1 |2 |4 ,"ok" ); + +return 0 ; +} +static void test_timer (int (*timer_test_fn)(struct timer *timer_skel)) {struct timer *timer_skel =NULL ;
@@ -134,6 +154,11 @@ void serial_test_timer_stress_async_cancel(void)test_timer (timer_stress_async_cancel); } +void serial_test_timer_async_cancel (void ) +{ +test_timer (timer_cancel_async); +} +void test_timer_interrupt (void ) {struct timer_interrupt *skel =NULL ;
tools/testing/selftests/bpf/progs/timer.c +23 -0
@@ -169,6 +169,29 @@ int BPF_PROG2(test1, int, a)return 0 ; } +static int timer_error (void *map,int *key,struct bpf_timer *timer) +{ + err =42 ; +return 0 ; +} + +SEC("syscall") +int test_async_cancel_succeed (void *ctx) +{ +struct bpf_timer *arr_timer; +int array_key =ARRAY ; + + arr_timer =bpf_map_lookup_elem (&array, &array_key); +if (!arr_timer) +return 0 ; +bpf_timer_init (arr_timer, &array,CLOCK_MONOTONIC ); +bpf_timer_set_callback (arr_timer, timer_error); +bpf_timer_start (arr_timer,100000 /* 100us */ ,0 ); +bpf_timer_cancel_async (arr_timer); + ok =7 ; +return 0 ; +} +/* callback for prealloc and non-prealloca hashtab timers */ static int timer_cb2 (void *map,int *key,struct hmap_elem *val) {
Signature · 1 line
2.52.0
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 11 lines · On Tue, 20 Jan 2026 15:59:09 +0000 you wrote:
This series reworks implementation of BPF timer and workqueue APIs. The goal is to make both timers and wq non-blocking, enabling their use in NMI context.
Today this code relies on abpf_spin_lockembedded in the map element to serialize:
* init of the async object,
* setting/changing the callback andbpf_prog
* starting/cancelling the timer/work
* tearing down when the map element is deleted or the map’s user ref is dropped
Quoted 1 line
[...]
Here is the summary with links:
- [bpf-next,v6,01/10] bpf: Factor out timer deletion helper
https://git.kernel.org/bpf/bpf-next/c/c1f2c449de27 - [bpf-next,v6,02/10] bpf: Remove unnecessary arguments from bpf_async_set_callback() https://git.kernel.org/bpf/bpf-next/c/57d31e72dbdd - [bpf-next,v6,03/10] bpf: Introduce lock-free bpf_async_update_prog_callback() https://git.kernel.org/bpf/bpf-next/c/8bb1e32b3fac - [bpf-next,v6,04/10] bpf: Simplify bpf_timer_cancel() https://git.kernel.org/bpf/bpf-next/c/83c9030cdc45
- [bpf-next,v6,05/10] bpf: Enable bpf timer and workqueue use in NMI (no matching commit)
- [bpf-next,v6,06/10] bpf: Add verifier support for bpf_timer argument in kfuncs (no matching commit)
- [bpf-next,v6,07/10] bpf: Introduce bpf_timer_cancel_async() kfunc (no matching commit)
- [bpf-next,v6,08/10] selftests/bpf: Refactor timer selftests
(no matching commit)
- [bpf-next,v6,09/10] selftests/bpf: Add stress test for timer async cancel (no matching commit)
- [bpf-next,v6,10/10] selftests/bpf: Verify bpf_timer_cancel_async works (no matching commit)
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html