io_uring: futex wakeups that a preempted lower-priority task can't delay
Subsystems (2)
cover io_uring: futex wakeups that a preempted lower-priority task can't delay
Thanks for the prompt reviews, Jens!
I run robot control systems on kernels built with CONFIG_PREEMPT_RT. Each process runs an io_uring event loop, and processes publish to each other through shared-memory queues, with the publisher waking each subscriber's loop. Real-time and lower-priority processes publish and subscribe on the same queues. The real-time ones run on CPUs kept free of interrupts, and the interrupt threads and everything else run on the rest.
io_uring fits this well, for the same reasons it is fast: most of its state is per ring. The submission and completion rings are shared with userspace without a lock, and the kernel-side ring locks belong to the ring. Requests and single futex wait data come from per-ring caches once warm, not the slab. With DEFER_TASKRUN, completions run in the ring's own task when it waits, so no other task posts to its completion ring, and another task hands it work with a push onto a lock-free list and a wake up, taking no lock that sleeps on PREEMPT_RT. Each of those removes a place where a lower-priority task could be holding something the real-time task needs. And io_uring already lets a ring register files and buffers once, so each request skips looking them up again.
The wakeup between event loops is where that breaks down. What I need is for a real-time publisher to wake a real-time subscriber on time, whatever the lower-priority processes on the same queue are doing and however busy their CPUs are with interrupts. Which mechanism carries the wakeup doesn't matter to me. Running the real-time tasks above the interrupt threads isn't a great answer: a lower-priority task boosted to their priority would then outrank the interrupt threads on the CPUs they were moved to, which undoes the isolation.
Each mechanism I've looked at lets a preempted lower-priority task hold that wakeup back:
- A futex wake takes the futex hash bucket lock, and so does every one-shot IORING_OP_FUTEX_WAIT arm, which an event loop does after every wakeup. On PREEMPT_RT that is a sleeping lock, and priority inheritance lifts the lower-priority holder only to the waker's priority, which loses to threaded interrupt handlers above it.
- IORING_OP_MSG_RING allocates an io_kiocb per send from a shared slab cache, whose slow path takes the slab's list_lock, a sleeping lock on PREEMPT_RT that any task on the system can be holding.
- eventfd and signals take a spinlock_t, the eventfd's wait queue lock or the target's siglock, which on PREEMPT_RT is a sleeping lock that lower-priority writers and readers of the same eventfd or process hold too.
I measured each on a Jetson Orin Nano with PREEMPT_RT. A SCHED_FIFO 40 sender on CPU 4 sends a message every 500us to a SCHED_FIFO 40 receiver's event loop on CPU 5. On CPUs 0 and 1, lower-priority processes send and receive on the same queue every 200us, starting and stopping, beside SCHED_FIFO 50 threads running 20us to 1ms bursts and interrupt and system load. Ten minutes and 1.2 million messages a run. Counted here is the real-time sender's time in the call that wakes the receiver, which is where it waits on the lock:
runs over 100us longest
futex, one-shot io_uring wait (7.4) 1 10,285 1.0ms MSG_RING (7.4) 1 55 1.0ms signal, signalfd (6.18) 2 33,080-36,122 1.7ms
Traced on 6.18, the futex sender waits on the bucket lock, the MSG_RING sender on the slab's list_lock, and the signal sender on the receiver's siglock, each held by a lower-priority task boosted to 40 and preempted by the SCHED_FIFO 50 threads. The receiver's one-shot futex arms wait on the same bucket lock, so for futexes 10,348 messages reached the receiver over 100us after they were sent, worst 3.5ms.
The options I see:
1. Persistent futex waits and registered wakes: look the futex up once and cache it. The waker registers the futex word with its ring once, and the waiter arms a wait on it once and keeps it armed. Both meet in one node for that futex, so a wake goes straight to the cached node and walks its waiters, with no key lookup, no futex hash bucket, no lock that sleeps on PREEMPT_RT and no allocation. Patch 2; prototyped and measured.
2. An allocation-free MSG_RING: IORING_MSG_DOORBELL. A ring's task registers a doorbell, and a sender ORs bits into a word in the target ctx and wakes its task; the target posts one CQE per drain. Real-time kernels solve this problem the same way: seL4 notifications and FreeRTOS task notifications OR bits into a word the waiter owns. Prototyped on 6.18, not posted: at SCHED_FIFO 40, no message over 100us late, worst 43us.
3. Keep MSG_RING's data messages, but take the io_kiocb from a cache on the target ring instead of the slab. Not prototyped.
Patch 2 is option 1. A persistent wait posts one CQE per wakeup and is re-armed by its slot, without looking the futex up again. Futexes also simplify discovery: a publisher wakes whoever waits on the word without knowing who they are, where a doorbell or MSG_RING needs every subscriber's ring fd. In the same test, with persistent waits no send took over 100us, the longest 25us, and no message reached the receiver over 100us after it was sent, the worst 30.5us.
Patch 2 is here to start the discussion, not as a polished solution. It is probably more useful to discuss the design before digging too far into the details of that patch.
Patch 1 is a fix to the DEFER_TASKRUN wake up that patch 2 depends on. It stands on its own; its commit message has the details.
I'd love any feedback you've got, or suggestions on other options that I've missed.
Austin Schuh (2):
io_uring: don't let a preempted task_work add hold back the wake up io_uring/futex: add persistent waits and registered wakes
include/linux/io_uring_types.h | 6 + include/uapi/linux/io_uring.h | 52 ++- io_uring/futex.c | 788 ++++++++++++++++++++++++++++++++- io_uring/futex.h | 28 ++ io_uring/io_uring.c | 1 + io_uring/register.c | 13 + io_uring/tw.c | 124 ++++-- io_uring/tw.h | 4 +- 8 files changed, 983 insertions(+), 33 deletions(-)
base-commit: 5b01d5955ac206e38097f777eff6ab2a8b948c41
Signature · 1 line
2.47.3
RFC PATCH 1/2 io_uring: don't let a preempted task_work add hold back the wake up
Only one local task_work add per wait cycle wakes the waiter: the one that takes ->cq_wait_nr's credit. Every later add sees the credit gone and returns. The credit is taken after the entry is queued and the wake up comes after that, with nothing held in between, so an adder preempted in that window leaves the waiter asleep with work queued. A lazy add preempted before it counts itself down does the same: the add that would have reached zero stops one short.
The waiter stays asleep for as long as anything of higher priority runs on the adder's CPU, whatever the waiter's own priority. Nothing boosts the adder, since it holds no lock. This bites IORING_OP_MSG_RING into a DEFER_TASKRUN ring in particular, where the adds come from other tasks of any priority: a real-time sender's message waits for a lower-priority sender that took the credit and was preempted. On PREEMPT_RT, where threaded interrupt handlers preempt everything below them, that is routine.
Traced on a Jetson Orin Nano (6.18.39-rt), with a SCHED_FIFO 60 sender posting to a SCHED_FIFO 60 receiver every 500us while lower-priority processes post to the same ring from CPUs loaded with SCHED_FIFO 50
bursts: of the 17 wakeups over 100us late in 60s, 15 were delivered by a lower-priority sender whose MSG_RING had been in flight since before the real-time sender's returned, for 12 to 479us.
On for-7.4/io_uring with PREEMPT_RT, before and after this patch, the real-time sender on CPU 4 posts a MSG_RING to the real-time receiver's DEFER_TASKRUN ring on CPU 5 every 500us. On CPUs 0 and 1, lower-priority processes post to the same ring every 200us, alongside SCHED_FIFO 50 threads running 20us to 1ms bursts and normal-priority interrupt and system load. Ten minutes a run, 1.2 million messages each, timed from the send to the receiver handling it:
late by late by
runs over 100us over 1ms longest
FIFO 40, before 3 889-1139 341-417 8.6-11.8ms FIFO 40, after 3 15-21 0 0.69-0.94ms FIFO 60, before 1 1058 438 8.3ms FIFO 60, after 1 0 0 71us
The three runs at FIFO 40 are that setup as is, plus a thread in each real-time process mapping and unmapping memory on CPU 0 or 1, and with no lower-priority processes starting or stopping. What remains at FIFO 40 is the sender's own MSG_RING being slow, 44 to 60 sends over 100us a run, before and after. Traced on 6.18, its request allocation waits on the slab node's list_lock, held by a lower-priority process that inheritance only lifts to 40, under the FIFO 50 bursts. At 60, nothing is left.
Keep preemption disabled from queueing the entry to the wake up, as mpscq_push() already does for the push itself. That covers a few atomics and try_to_wake_up(). eventfd_signal() takes a spinlock_t, which sleeps on PREEMPT_RT, so signal the eventfd and wake ring pollers after that instead of before.
--- io_uring/tw.c | 80 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 52 insertions(+), 28 deletions(-)
io_uring/tw.c +52 -28
@@ -179,37 +179,14 @@ static void io_local_work_wake_poll(struct io_ring_ctx *ctx, unsigned int flags)call_rcu_hurry (&ctx->poll_wq_rcu, io_poll_wq_wake_rcu); } -void io_req_local_work_add (struct io_kiocb *req,unsigned flags) +/* +* Issue the wake up of the waiter's cycle if this add is the one that takes +* ->cq_wait_nr's credit, see io_cq_wait_arm(). +*/ +static void io_local_work_wake (struct io_ring_ctx *ctx,unsigned int flags) { -struct io_ring_ctx *ctx = req->ctx;int nr_wait; -/* pairs with synchronize_rcu() in io_ring_exit_work() */ -guard (rcu)(); - -/* -* We don't know how many requests there are in the link and whether -* they can even be queued lazily, fall back to non-lazy. -*/ -if (req->flags &IO_REQ_LINK_FLAGS ) - flags &= ~IOU_F_TWQ_LAZY_WAKE ; - -/* -* The xchg() in mpscq_push() implies a full barrier, which pairs with -* the barrier in io_cq_wait_arm() on the io_cqring_wait() side. This -* ensures that either we see the updated ->cq_wait_nr, or waiters going -* to sleep will observe the work added to the list, which is similar to -* the wait/wake task state sync. -*/ -if (mpscq_push (&ctx->work_list, &req->io_task_work.node)) { -io_ctx_mark_taskrun (ctx); -if (data_race (ctx->int_flags) &IO_RING_F_HAS_EVFD ) -io_eventfd_signal (ctx, false, flags &IOU_F_TWQ_IN_WAKE ); -/* whoever sets the flag wakes ->poll_wq afterwards */ -if (data_race (ctx->int_flags) &IO_RING_F_POLL_ACTIVATED ) -io_local_work_wake_poll (ctx, flags); - } -/* * No one is waiting (IO_CQ_WAKE_INIT), or this cycle's wake up has * already been issued (zero or negative, see below).
@@ -239,6 +216,53 @@ void io_req_local_work_add(struct io_kiocb *req, unsigned flags)wake_up_state (ctx->submitter_task,TASK_INTERRUPTIBLE ); } +void io_req_local_work_add (struct io_kiocb *req,unsigned flags) +{ +struct io_ring_ctx *ctx = req->ctx; +bool first; + +/* pairs with synchronize_rcu() in io_ring_exit_work() */ +guard (rcu)(); + +/* +* We don't know how many requests there are in the link and whether +* they can even be queued lazily, fall back to non-lazy. +*/ +if (req->flags &IO_REQ_LINK_FLAGS ) + flags &= ~IOU_F_TWQ_LAZY_WAKE ; + +/* +* Only the add that takes the credit wakes the waiter, later ones +* return early, and a lazy add that hasn't counted itself down yet +* holds back the one that would. Neither may be preempted between +* queueing its entry and the wake up: the waiter, whatever its +* priority, sleeps with the entry queued for as long as anything of +* higher priority runs on the adder's CPU, and nothing boosts the +* adder, as it holds no lock. +* +* The xchg() in mpscq_push() implies a full barrier, which pairs with +* the barrier in io_cq_wait_arm() on the io_cqring_wait() side. This +* ensures that either we see the updated ->cq_wait_nr, or waiters going +* to sleep will observe the work added to the list, which is similar to +* the wait/wake task state sync. +*/ +preempt_disable (); + first =mpscq_push (&ctx->work_list, &req->io_task_work.node); +if (first) +io_ctx_mark_taskrun (ctx); +io_local_work_wake (ctx, flags); +preempt_enable (); + +/* eventfd_signal() takes a spinlock_t, not allowed above on PREEMPT_RT */ +if (first) { +if (data_race (ctx->int_flags) &IO_RING_F_HAS_EVFD ) +io_eventfd_signal (ctx, false, flags &IOU_F_TWQ_IN_WAKE ); +/* whoever sets the flag wakes ->poll_wq afterwards */ +if (data_race (ctx->int_flags) &IO_RING_F_POLL_ACTIVATED ) +io_local_work_wake_poll (ctx, flags); + } +} +void io_req_normal_work_add (struct io_kiocb *req) {struct io_uring_task *tctx = req->tctx;
Signature · 1 line
2.47.3
RFC PATCH 2/2 io_uring/futex: add persistent waits and registered wakes
In reply to Austin SchuhEvery futex wait and wake takes the futex hash bucket's lock, and an io_uring event loop arms a new one-shot wait after every wakeup. On PREEMPT_RT the lock sleeps, so a lower-priority task waiting or waking on the same word, or on any word in the same bucket, can be preempted holding it. Priority inheritance lifts it only to the real-time waker's priority, which loses to the interrupt threads above that on the holder's CPU, and the real-time waker waits.
Look the futex up once and keep what the lookup found, the way io_uring already keeps files and buffers:
- IORING_REGISTER_FUTEX registers a table of futex words with the waker's ring, as IORING_REGISTER_FILES2 does files, sparse if asked; IORING_REGISTER_FUTEX_UPDATE replaces slots of it, as IORING_REGISTER_FILES_UPDATE2 does, and IORING_UNREGISTER_FUTEX drops it.
- IORING_OP_FUTEX_WAIT with IORING_FUTEX_PERSISTENT arms a wait that stays armed until cancelled, in a slot of the ring's that sqe->file_index names as it does a direct descriptor's: index + 1, or IORING_FILE_INDEX_ALLOC for a free one, returned in res.
- IORING_OP_FUTEX_WAKE with IORING_FUTEX_REGISTERED wakes the persistent waits on registered slot sqe->buf_index's word whose bitset matches sqe->addr3. Each posts one CQE with IORING_CQE_F_MORE, and posts no more until IORING_OP_FUTEX_WAIT with IORING_FUTEX_REARM names its slot, as a close does; wakes in between are kept as one.
Wakers and waiters can be in different processes, so registering and arming look the word up by its futex key, as futex(2) does, and meet in one node per key. A node outlives any one wait, unlike a futex_q, so the nodes get a hash of their own, sized and hashed like the futex hash, which also leaves the futex core untouched. Only registering, arming and teardown take its locks; registering and arming need write access to the word, as a waker would. A registered wake reads the node from the ring's table, walks its waiters under RCU and moves each with a cmpxchg, then sets the wait's bit in its ring's pending bitmap and wakes the ring's task. It does no key lookup, takes no futex hash bucket and no lock that sleeps on PREEMPT_RT, and allocates nothing. The waker's store to the word and the waiter's link onto the node are each followed by a full barrier, so either the wake finds the waiter or the waiter sees the new value.
futex(2) is untouched: a persistent wait is not a futex_q, and a word is used one way or the other.
The ring must be IORING_SETUP_DEFER_TASKRUN, and a persistent wait can't be linked or skip its successes. A ring has as many wait slots as its CQ has entries, since each wait needs room for a CQE. Nodes, waits and tables are freed after an RCU grace period.
This depends on "io_uring: don't let a preempted task_work add hold back the wake up": the wake skips waking the ring's task when a task_work add has already claimed that wake up.
Measured on a Jetson Orin Nano running for-7.4/io_uring with PREEMPT_RT and the patch above, so both rows have its fixed wake up into the ring, and the one-shot delays are the bucket lock's. A real-time sender on CPU 4 wakes a real-time receiver's event loop on CPU 5 every 500us. On CPUs 0 and 1, lower-priority processes wake and wait on the same word, beside SCHED_FIFO 50 threads running 20us to 1ms bursts and interrupt load. Ten minutes and 1.2 million messages a run:
sends over late by late by
100us over 100us over 1ms longest
one-shot, FIFO 40 10285 10348 76 3.5ms
persistent, FIFO 40 0 0 0 30.5us
A 30s stress test, a SCHED_FIFO 40 waker waking a SCHED_FIFO 40 waiter while two more threads arm and cancel persistent waits on the same word as fast as they can, lost none of about 259,000 wakes, worst 25.5us.
---
This is to discuss the design, not to merge as is. It works and passes its tests on PREEMPT_RT, but it isn't polished. The cover letter has the background and the other options I looked at.
Open design questions:
- Should futex(2) and persistent waits work on the same word? As posted, a word is used one way or the other: FUTEX_WAKE doesn't see persistent waits and a registered wake doesn't see futex(2) waiters, so mixing them silently loses wakeups. Interoperating is more code, but fewer gotchas.
- Should wakes be ordered by the waiters' priority? As posted, a wake walks the waiters in the order they armed, newest first. The futex core sorts its waiters by priority when they queue (__futex_queue()).
- Persistent waits need IORING_SETUP_DEFER_TASKRUN, since their wakes are drained with the ring's local work. That could be expanded to other rings if there's a use for it.
include/linux/io_uring_types.h | 6 + include/uapi/linux/io_uring.h | 52 ++- io_uring/futex.c | 788 ++++++++++++++++++++++++++++++++- io_uring/futex.h | 28 ++ io_uring/io_uring.c | 1 + io_uring/register.c | 13 + io_uring/tw.c | 44 ++ io_uring/tw.h | 4 +- 8 files changed, 931 insertions(+), 5 deletions(-)
include/linux/io_uring_types.h +6 -0
@@ -453,6 +453,8 @@ struct io_ring_ctx {unsigned long check_cq;atomic_t cq_wait_nr;atomic_t cq_timeouts; +/* persistent futex waits notified; see io_ring_futex_kick() */ +atomic_t futex_pending;struct wait_queue_head cq_wait; }____cacheline_aligned_in_smp ;
@@ -473,6 +475,10 @@ struct io_ring_ctx {#ifdef CONFIG_FUTEX struct hlist_head futex_list;struct io_alloc_cache futex_cache; +/* IORING_REGISTER_FUTEX words; changes under uring_lock */ +struct io_futex_table __rcu *futex_table; +/* persistent futex waits' slots, from the first one on */ +struct io_futex_ring *futex_ring;#endif const struct cred *sq_creds;/* cred used for __io_sq_thread() */
include/uapi/linux/io_uring.h +51 -1
70 lines not on this page · raw message
io_uring/futex.c +785 -3
867 lines not on this page · raw message
io_uring/futex.h +28 -0
@@ -16,6 +16,13 @@ bool io_futex_remove_all(struct io_ring_ctx *ctx, struct io_uring_task *tctx, bool cancel_all);bool io_futex_cache_init (struct io_ring_ctx *ctx);void io_futex_cache_free (struct io_ring_ctx *ctx); +int io_futex_register (struct io_ring_ctx *ctx,void __user *arg, +unsigned int size); +int io_futex_register_update (struct io_ring_ctx *ctx,void __user *arg, +unsigned int size); +int io_futex_unregister (struct io_ring_ctx *ctx); +void io_futex_unregister_all (struct io_ring_ctx *ctx); +int io_futex_run_pending (struct io_ring_ctx *ctx);#else static inline int io_futex_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd,
@@ -35,4 +42,25 @@ static inline bool io_futex_cache_init(struct io_ring_ctx *ctx)static inline void io_futex_cache_free (struct io_ring_ctx *ctx) { } +static inline int io_futex_register (struct io_ring_ctx *ctx,void __user *arg, +unsigned int size) +{ +return -EOPNOTSUPP ; +} +static inline int io_futex_register_update (struct io_ring_ctx *ctx, +void __user *arg,unsigned int size) +{ +return -EOPNOTSUPP ; +} +static inline int io_futex_unregister (struct io_ring_ctx *ctx) +{ +return -EOPNOTSUPP ; +} +static inline void io_futex_unregister_all (struct io_ring_ctx *ctx) +{ +} +static inline int io_futex_run_pending (struct io_ring_ctx *ctx) +{ +return 0 ; +}#endif
io_uring/io_uring.c +1 -0
@@ -2260,6 +2260,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx)io_unregister_zcrx (ctx);io_cqring_overflow_kill (ctx);io_eventfd_unregister (ctx); +io_futex_unregister_all (ctx);io_free_alloc_caches (ctx);io_destroy_buffers (ctx);io_free_region (ctx->user, &ctx->param_region);
io_uring/register.c +13 -0
@@ -28,6 +28,7 @@#include "cancel.h" #include "kbuf.h" #include "napi.h" +#include "futex.h" #include "eventfd.h" #include "msg_ring.h" #include "memmap.h"
@@ -1026,6 +1027,18 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,WRITE_ONCE (ctx->bpf_filters, ctx->restrictions.bpf_filters->filters);break ; +case IORING_REGISTER_FUTEX : + ret =io_futex_register (ctx, arg, nr_args); +break ; +case IORING_UNREGISTER_FUTEX : + ret = -EINVAL ; +if (arg || nr_args) +break ; + ret =io_futex_unregister (ctx); +break ; +case IORING_REGISTER_FUTEX_UPDATE : + ret =io_futex_register_update (ctx, arg, nr_args); +break ;default : ret = -EINVAL ;break ;
io_uring/tw.c +44 -0
62 lines not on this page · raw message
io_uring/tw.h +3 -1
@@ -25,6 +25,7 @@ static inline bool io_should_terminate_tw(struct io_ring_ctx *ctx) }void io_req_task_work_add_remote (struct io_kiocb *req,unsigned flags); +void io_ring_futex_kick (struct io_ring_ctx *ctx);void tctx_task_work (struct callback_head *cb);void io_tctx_fallback_work (struct work_struct *work);int io_run_local_work (struct io_ring_ctx *ctx,int min_events,int max_events);
@@ -90,7 +91,8 @@ static inline int io_run_task_work(void)static inline bool io_local_work_pending (struct io_ring_ctx *ctx) { -return !mpscq_empty (&ctx->work_list); +return !mpscq_empty (&ctx->work_list) || +atomic_read (&ctx->futex_pending); }static inline bool io_task_work_pending (struct io_ring_ctx *ctx)
Signature · 1 line
2.47.3
[email protected]
AI reviewRe: [RFC PATCH 2/2] io_uring/futex: add persistent waits and registered wakes
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] io_uring/futex: NULL deref in io_futex_node_get() on !CONFIG_MMU - [High] io_uring/futex…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] io_uring/futex: NULL deref in io_futex_node_get() on !CONFIG_MMU
- [High] io_uring/futex: busy-spin on teardown from uncleared futex_pending
- [Medium] io_uring/futex: 32-bit truncation of SQE fields in io_futex_prep() --
commit d39ae03333a1ccb677abc6626226b97d34d99fdf
Author: Austin Schuh <[email protected]>
io_uring/futex: add persistent waits and registered wakes
This commit adds persistent futex waits and registered futex wakes to io_uring to avoid futex hash bucket lock contention on PREEMPT_RT. It introduces registration opcodes for futex words and uses an RCU-walked node table and per-ring pending bitmaps to notify waiting rings without taking sleeping locks in the wake path.
Quoted 4 lines
io_uring/futex.c +0
-0
[ ... ]
Quoted 51 lines
io_uring/futex.c +51
-0+/* +* A key without FUT_OFF_INODE names its mm by pointer. The node keeps the +* mm_struct allocated so the pointer can't be reused for another mm while +* the node can still be found by it. +*/ +static bool io_futex_key_has_mm (union futex_key *key) +{ +return !(key->both.offset &FUT_OFF_INODE ); +} + +static struct io_futex_node *io_futex_node_get (u32 __user *uaddr, +unsigned int flags) +{ +union futex_key key =FUTEX_KEY_INIT ; +struct io_futex_node *node, *new =NULL ; +struct io_futex_bucket *b; +int ret; + +/* Write access, as a waker needs: reading the word is not enough. */ + ret =get_futex_key (uaddr, flags, &key,FUTEX_WRITE ); +if (ret) +return ERR_PTR (ret); + b =io_futex_bucket (&key); +again: +spin_lock (&b->lock); +hlist_for_each_entry (node, &b->nodes, hnode) { +if (futex_match (&node->key, &key)) { +/* A node at zero has already left the bucket. */ +refcount_inc (&node->ref); +spin_unlock (&b->lock); +kfree (new); +return node; + } + } +if (!new) { +/* Allocate only on a miss, and not under the lock. */ +spin_unlock (&b->lock); + new =kzalloc_obj (*new,GFP_KERNEL_ACCOUNT ); +if (!new) +return ERR_PTR (-ENOMEM ); + goto again; + } + new->key = key; + new->bucket = b; +INIT_HLIST_HEAD (&new->members); +refcount_set (&new->ref,1 ); +if (io_futex_key_has_mm (&key)) +mmgrab (key.private.mm); +hlist_add_head (&new->hnode, &b->nodes); +spin_unlock (&b->lock); +return new;
io_uring/futex.c static void io_futexv_complete(struct io_tw_req tw_req, io_tw_token_t tw) · line 291
Quoted 10 lines
io_uring/futex.c +10
-0+} + +static void io_futex_node_put (struct io_futex_node *node) +{ +if (!refcount_dec_and_lock (&node->ref, &node->bucket->lock)) +return ; +hlist_del (&node->hnode); +spin_unlock (&node->bucket->lock); +if (io_futex_key_has_mm (&node->key)) +mmdrop (node->key.private.mm);
[Severity: High]
Can this trigger a NULL pointer dereference in io_futex_node_get() and io_futex_node_put() on !CONFIG_MMU kernels?
On !CONFIG_MMU builds, FLAGS_SHARED is defined as 0x0000 in kernel/futex/futex.h, so get_futex_key() in kernel/futex/core.c always takes the !fshared path, leaves FUT_OFF_INODE unset in key->both.offset, and sets key->private.mm to NULL:
kernel/futex/core.c:get_futex_key() {
if (!fshared) {
if (IS_ENABLED(CONFIG_MMU))
key->private.mm = mm;
else
key->private.mm = NULL;
key->private.address = address;
return 0;
}
}Because io_futex_key_has_mm() only checks !(key->both.offset & FUT_OFF_INODE), it returns true when CONFIG_MMU is disabled, causing io_futex_node_get() to call mmgrab(NULL) and io_futex_node_put() to call mmdrop(NULL). Should io_futex_key_has_mm() also check key->private.mm != NULL (or IS_ENABLED(CONFIG_MMU))?
Quoted 3 lines
io_uring/futex.c +3
-0+/* A registered wake may still be walking it. */ +kfree_rcu (node, rcu); +}
[ ... ]
Quoted 39 lines
io_uring/futex.c +24
-3@@ -127,23 +847,69 @@ bool io_futex_remove_all(struct io_ring_ctx *ctx, struct io_uring_task *tctx,int io_futex_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) { struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); -u32 flags; +u32 flags, uflags; -if (unlikely (sqe->len || sqe->futex_flags || sqe->buf_index || - sqe->file_index)) +if (unlikely (sqe->len))return -EINVAL ; + uflags =READ_ONCE (sqe->futex_flags); + iof->iou_flags = uflags; + iof->member =NULL ; iof->uaddr =u64_to_user_ptr (READ_ONCE (sqe->addr)); iof->futex_val =READ_ONCE (sqe->addr2); iof->futex_mask =READ_ONCE (sqe->addr3); flags =READ_ONCE (sqe->fd); +if (req->opcode ==IORING_OP_FUTEX_WAIT ) { +if (uflags & ~(IORING_FUTEX_PERSISTENT |IORING_FUTEX_REARM ) || + uflags == (IORING_FUTEX_PERSISTENT |IORING_FUTEX_REARM ) || + sqe->buf_index) +return -EINVAL ; +/* +* A persistent wait's slot, as a direct descriptor's: index + 1, +* or IORING_FILE_INDEX_ALLOC. A re-arm names the slot the same +* way, as a close does. +*/ + iof->futex_nr =READ_ONCE (sqe->file_index); +if (!(uflags & (IORING_FUTEX_PERSISTENT |IORING_FUTEX_REARM )) != + !iof->futex_nr) +return -EINVAL ; +if (uflags &IORING_FUTEX_REARM ) { +if (iof->uaddr || iof->futex_val || iof->futex_mask || + flags || iof->futex_nr ==IORING_FILE_INDEX_ALLOC ) +return -EINVAL ; + iof->futex_nr--;
io_uring/futex.c bool io_futex_remove_all(struct io_ring_ctx *ctx, struct io_uring_task *tctx, · line 891
Quoted 10 lines
io_uring/futex.c +10
-0+return 0 ; + } + }else if (sqe->file_index) { +return -EINVAL ; + }else if (uflags) { +/* Woken by slot: no word, no count, only the bitset. */ +if (uflags !=IORING_FUTEX_REGISTERED || iof->uaddr || + iof->futex_val || flags || !iof->futex_mask || + iof->futex_mask >U32_MAX ) +return -EINVAL ;
[Severity: Medium]
On 32-bit architectures, struct io_futex defines uaddr as void __user * and futex_val and futex_mask as unsigned long:
struct io_futex {
struct file *file;
void __user *uaddr;
unsigned long futex_val;
unsigned long futex_mask;
...
};When io_futex_prep() reads the 64-bit sqe->addr, sqe->addr2, and sqe->addr3 fields into iof->uaddr, iof->futex_val, and iof->futex_mask, doesn't that truncate the upper 32 bits on 32-bit kernels before the validation checks run?
In addition, on 32-bit architectures where unsigned long is 32 bits wide, won't the check iof->futex_mask > U32_MAX in io_futex_prep() always be false and trigger a compiler tautological-compare warning while failing to reject non-zero upper 32 bits in sqe->addr3?
[ ... ]
Quoted 4 lines
io_uring/tw.h +0
-0
[ ... ]
io_uring/tw.h static inline int io_run_task_work(void) · line 96
Quoted 8 lines
io_uring/tw.h +2
-1@@ -90,7 +91,8 @@ static inline int io_run_task_work(void)static inline bool io_local_work_pending (struct io_ring_ctx *ctx) { -return !mpscq_empty (&ctx->work_list); +return !mpscq_empty (&ctx->work_list) || +atomic_read (&ctx->futex_pending); }
[Severity: High]
Could including ctx->futex_pending in io_local_work_pending() cause a busy-spin in io_ring_ctx_wait_inflight() during ring teardown?
ctx->futex_pending is only cleared in __io_run_local_work(), which requires ctx->submitter_task == current. During ring teardown, io_ring_ctx_cancel() calls io_cancel_local_task_work(), which drains ctx->work_list under uring_lock but does not clear ctx->futex_pending:
io_uring/tw.c:io_cancel_local_task_work() {
guard(mutex)(&ctx->uring_lock);
while (!mpscq_empty(&ctx->work_list)) {
...
}
io_submit_flush_completions(ctx);
}If the ring fd is closed from a task other than ctx->submitter_task (or a concurrent waker calls io_ring_futex_kick() during teardown) while an uncancelable request such as storage I/O is still in flight, io_ring_ctx_wait_inflight() sees io_local_work_pending(ctx) remain true on every iteration:
io_uring/io_uring.c:io_ring_ctx_wait_inflight() {
/* don't sleep on work that's already there and that we can run */
if (ran_own && ((tctx && !mpscq_empty(&tctx->task_list)) ||
io_local_work_pending(ctx)))
__set_current_state(TASK_RUNNING);
else
schedule_timeout(1);
}Wouldn't that skip schedule_timeout(1) and spin in a tight loop until all in-flight requests complete unless io_cancel_local_task_work() also clears ctx->futex_pending?
Quoted 2 lines
io_uring/tw.h +0
-0static inline bool io_task_work_pending (struct io_ring_ctx *ctx)
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2