Skip to contentSkip to conversation

io_uring/futex: return a wait's data to the cache when it fails to arm

PATCH Applied Austin Schuh 2 participants 5 messages Oct 19 – Oct 20 io_uring

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 io_uring/futex: return a wait's data to the cache when it fails to arm

io_futex_wait() takes its io_futex_data from ctx->futex_cache, and a completed wait gives it back in io_futex_complete(). A wait that fails to arm kfree()s it instead.

The usual failure is -EAGAIN from futex_wait_setup(): the futex word no longer holds the expected value. That is routine for a waiter that re-arms with the value it last read, since the word changes whenever there is something to wake for. Each such failure takes an object out of the cache for good, so the cache drains and from then on waits allocate.

The cache exists so that arming a wait doesn't have to go to the slab allocator, and that matters most to the waiters that re-arm the most:
event loops, often real-time ones. A kmalloc() or kfree() that misses the per-CPU slab takes the node's list_lock, shared with every other user of that kmalloc size on the system, and may go on to the page allocator.

On PREEMPT_RT list_lock is a sleeping lock. A real-time task that finds it held waits for the holder, which may be any task on any CPU, to get to run and release it. Priority inheritance boosts the holder only up to the waiter's priority, so the waiter still sits behind whatever outranks that on the holder's CPU, such as threaded interrupt handlers. Re-arming a futex wait should not depend on any of that.

Put the data back in the cache on that path too, while the submit lock still protects it.

In an event loop re-arming one IORING_OP_FUTEX_WAIT per message under load, 0.39 arms per message failed with -EAGAIN, and io_futex_wait() did 0.39 kmalloc()s and kfree()s per message. With this change, 100,000 waits that all fail with -EAGAIN do one kmalloc() in total.

Fixes 194bb58c6090 ("io_uring: add support for futex wake and wait")
Assisted-by LLM

---

Notes:
Based on for-7.4/io_uring (5b01d5955ac2). Built for arm64 with PREEMPT_RT, W=1, no new warnings. The numbers are from an equivalent change on 6.18.39-rt on a Jetson Orin Nano, counting kmem:kmalloc/kfree by call site with bpftrace while a test loops on IORING_OP_FUTEX_WAIT with a value the futex word doesn't hold. Happy to turn that into a liburing test if it's useful.

 io_uring/futex.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
io_uring/futex.c +5 -1
index 9203998654ff..9e4e63ee7bc1 100644
@@ -353,12 +353,16 @@ int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags)
 	}
 
 done_unlock:
+	if (ifd) {
+		/* As in io_futex_complete(), or failed arms would drain it. */
+		io_cache_free(&ctx->futex_cache, ifd);
+		io_req_async_data_clear(req, 0);
+	}
 	io_ring_submit_unlock(ctx, issue_flags);
 done:
 	if (ret < 0)
 		req_set_fail(req);
 	io_req_set_res(req, ret, 0);
-	io_req_async_data_free(req);
 	return IOU_COMPLETE;
 }
 

base-commit: 5b01d5955ac206e38097f777eff6ab2a8b948c41

Signature · 1 line
2.47.3
maintainer
#1 lore raw
Quoted 25 lines · Austin Schuh

io_futex_wait() takes its io_futex_data from ctx->futex_cache, and a completed wait gives it back in io_futex_complete(). A wait that fails to arm kfree()s it instead.

The usual failure is -EAGAIN from futex_wait_setup(): the futex word no longer holds the expected value. That is routine for a waiter that re-arms with the value it last read, since the word changes whenever there is something to wake for. Each such failure takes an object out of the cache for good, so the cache drains and from then on waits allocate.

The cache exists so that arming a wait doesn't have to go to the slab allocator, and that matters most to the waiters that re-arm the most:
event loops, often real-time ones. A kmalloc() or kfree() that misses the per-CPU slab takes the node's list_lock, shared with every other user of that kmalloc size on the system, and may go on to the page allocator.

On PREEMPT_RT list_lock is a sleeping lock. A real-time task that finds it held waits for the holder, which may be any task on any CPU, to get to run and release it. Priority inheritance boosts the holder only up to the waiter's priority, so the waiter still sits behind whatever outranks that on the holder's CPU, such as threaded interrupt handlers. Re-arming a futex wait should not depend on any of that.

Quoted 7 lines

Put the data back in the cache on that path too, while the submit lock still protects it.

In an event loop re-arming one IORING_OP_FUTEX_WAIT per message under load, 0.39 arms per message failed with -EAGAIN, and io_futex_wait() did 0.39 kmalloc()s and kfree()s per message. With this change, 100,000 waits that all fail with -EAGAIN do one kmalloc() in total.

Patches like this have been posted before, and my question is always "is this a common case?"? Reading this:

Quoted 7 lines

Notes:
Based on for-7.4/io_uring (5b01d5955ac2). Built for arm64 with PREEMPT_RT, W=1, no new warnings. The numbers are from an equivalent change on 6.18.39-rt on a Jetson Orin Nano, counting kmem:kmalloc/kfree by call site with bpftrace while a test loops on IORING_OP_FUTEX_WAIT with a value the futex word doesn't hold. Happy to turn that into a liburing test if it's useful.

makes me very much think "nope, not at all", as it's a deliberately failing test case. And yes if you keep issuing failing waits, then it'll drain the cache. The cache is for the hot FAST path, which is not an error path.

Hmm?

Signature · 1 line
Jens Axboe
#2 lore raw
Quoted 11 lines · Jens Axboe

Patches like this have been posted before, and my question is always "is this a common case?"? Reading this:

Quoted 7 lines

Notes:
Based on for-7.4/io_uring (5b01d5955ac2). Built for arm64 with PREEMPT_RT, W=1, no new warnings. The numbers are from an equivalent change on 6.18.39-rt on a Jetson Orin Nano, counting kmem:kmalloc/kfree by call site with bpftrace while a test loops on IORING_OP_FUTEX_WAIT with a value the futex word doesn't hold. Happy to turn that into a liburing test if it's useful.

Quoted 4 lines

makes me very much think "nope, not at all", as it's a deliberately failing test case. And yes if you keep issuing failing waits, then it'll drain the cache. The cache is for the hot FAST path, which is not an error path.

Fair on the test case, it only showed the mechanism. But for an event loop, -EAGAIN from a futex wait isn't an error path. Right before io_uring_enter() re-arms the wait that just fired, the event loop loads its futex word. If the word has already changed, it handles the new message and re-arms with the new value instead. A message published between that load and futex_wait_setup() checking the word makes the wait return -EAGAIN, and that is how the loop learns there is more to do. The window can be shrunk, but not removed; nearly all of what is left is inside io_uring_enter().

Here's how often that happens, measured on a Jetson Orin Nano running for-7.4/io_uring with PREEMPT_RT. The event loops are AOS (a robotics framework). Each one keeps one IORING_OP_FUTEX_WAIT armed per channel, on the channel's queue index, and re-arms it after every wake. One
channel: a SCHED_FIFO 40 publisher at 2 kHz with a SCHED_FIFO 40 subscriber, then low-priority processes publishing and subscribing on the same channel as well, then interrupt and system load on top. bpftrace on fexit:io_futex_wait counted, per process, the waits armed (IORING_OP_FUTEX_WAIT issued) and how many of them returned -EAGAIN instead of sleeping. Two minutes each:

real-time subscriber low-priority procs
messages/s arms/s -EAGAIN rate arms/s -EAGAIN rate
pair only 2,000 2,000 0 0% - - -

  + low-priority     12,200  11,020    1,292   12%   14,760    1,229    8%
  + load             12,200   9,830    2,797   28%    4,120      377    9%

(messages/s is everything published on the channel, arms/s and -EAGAIN are per second. A subscriber arms fewer times than messages arrive because one wake can cover several messages. Under load the low-priority processes arm less often because the load takes their CPUs.)

In all, 2,520 and 3,170 arms a second failed in the last two runs. bpftrace also read the futex word as each failed arm returned. Of the 683,303 failures across both runs, 88% found the word exactly one message past the value the wait was armed with. So the arm raced a single publish, not a backlog. Load makes it more likely because the stretch between that load and the kernel's check gets longer.

Timed from syscall entry to io_futex_wait() returning, 74% of the real-time subscriber's failures came within 6 us. For the low-priority processes, which can be preempted in there, a third took more than 40 us.

Without the patch, each of those failures is a kfree() and then a kmalloc() on the next arm: about 3,000 a second here, from a slab cache shared system-wide, and more the busier the system gets. On PREEMPT_RT the slab slow path's list_lock is a sleeping lock, which is where this started for us. The patch doesn't change the armed path at all; it only stops the cache from draining.

I can send a v2 with these numbers and this explanation in the commit message, in place of the synthetic test.

What I'm after is a futex wait an event loop can reliably re-arm without latency spikes. Is there a better way to avoid the allocation here? If you'd rather not have every failure refill the cache, I can limit it to -EAGAIN, the one failure an event loop hits in normal operation, and keep the kfree() for the rest.

Thanks for the prompt reply!
Austin

maintainer
#3 lore raw
Quoted 68 lines · Austin Schuh
Quoted 11 lines · On Mon, Oct 19, 2026 at 5:54?PM Jens Axboe <[email protected]> wrote:

Patches like this have been posted before, and my question is always "is this a common case?"? Reading this:

Quoted 7 lines

Notes:
Based on for-7.4/io_uring (5b01d5955ac2). Built for arm64 with PREEMPT_RT, W=1, no new warnings. The numbers are from an equivalent change on 6.18.39-rt on a Jetson Orin Nano, counting kmem:kmalloc/kfree by call site with bpftrace while a test loops on IORING_OP_FUTEX_WAIT with a value the futex word doesn't hold. Happy to turn that into a liburing test if it's useful.

Quoted 4 lines

makes me very much think "nope, not at all", as it's a deliberately failing test case. And yes if you keep issuing failing waits, then it'll drain the cache. The cache is for the hot FAST path, which is not an error path.

Fair on the test case, it only showed the mechanism. But for an event loop, -EAGAIN from a futex wait isn't an error path. Right before io_uring_enter() re-arms the wait that just fired, the event loop loads its futex word. If the word has already changed, it handles the new message and re-arms with the new value instead. A message published between that load and futex_wait_setup() checking the word makes the wait return -EAGAIN, and that is how the loop learns there is more to do. The window can be shrunk, but not removed; nearly all of what is left is inside io_uring_enter().

Here's how often that happens, measured on a Jetson Orin Nano running for-7.4/io_uring with PREEMPT_RT. The event loops are AOS (a robotics framework). Each one keeps one IORING_OP_FUTEX_WAIT armed per channel, on the channel's queue index, and re-arms it after every wake. One
channel: a SCHED_FIFO 40 publisher at 2 kHz with a SCHED_FIFO 40 subscriber, then low-priority processes publishing and subscribing on the same channel as well, then interrupt and system load on top. bpftrace on fexit:io_futex_wait counted, per process, the waits armed (IORING_OP_FUTEX_WAIT issued) and how many of them returned -EAGAIN instead of sleeping. Two minutes each:

real-time subscriber low-priority procs
messages/s arms/s -EAGAIN rate arms/s -EAGAIN rate
pair only 2,000 2,000 0 0% - - -

  + low-priority     12,200  11,020    1,292   12%   14,760    1,229    8%
  + load             12,200   9,830    2,797   28%    4,120      377    9%

(messages/s is everything published on the channel, arms/s and -EAGAIN are per second. A subscriber arms fewer times than messages arrive because one wake can cover several messages. Under load the low-priority processes arm less often because the load takes their CPUs.)

In all, 2,520 and 3,170 arms a second failed in the last two runs. bpftrace also read the futex word as each failed arm returned. Of the 683,303 failures across both runs, 88% found the word exactly one message past the value the wait was armed with. So the arm raced a single publish, not a backlog. Load makes it more likely because the stretch between that load and the kernel's check gets longer.

Timed from syscall entry to io_futex_wait() returning, 74% of the real-time subscriber's failures came within 6 us. For the low-priority processes, which can be preempted in there, a third took more than 40 us.

Without the patch, each of those failures is a kfree() and then a kmalloc() on the next arm: about 3,000 a second here, from a slab cache shared system-wide, and more the busier the system gets. On PREEMPT_RT the slab slow path's list_lock is a sleeping lock, which is where this started for us. The patch doesn't change the armed path at all; it only stops the cache from draining.

Quoted 8 lines

I can send a v2 with these numbers and this explanation in the commit message, in place of the synthetic test.

What I'm after is a futex wait an event loop can reliably re-arm without latency spikes. Is there a better way to avoid the allocation here? If you'd rather not have every failure refill the cache, I can limit it to -EAGAIN, the one failure an event loop hits in normal operation, and keep the kfree() for the rest.

That's great info, and I agree it differs from the usual forced error path. Futexes are inherently racy. Thanks, I'll get this queued up.

Signature · 1 line
Jens Axboe
maintainer Applied
#4 lore raw
In reply to Austin Schuh
Quoted 4 lines · Austin Schuh

io_futex_wait() takes its io_futex_data from ctx->futex_cache, and a completed wait gives it back in io_futex_complete(). A wait that fails to arm kfree()s it instead.

Quoted 8 lines

The usual failure is -EAGAIN from futex_wait_setup(): the futex word no longer holds the expected value. That is routine for a waiter that re-arms with the value it last read, since the word changes whenever there is something to wake for. Each such failure takes an object out of the cache for good, so the cache drains and from then on waits allocate.

[...]

Applied, thanks!

[1/1] io_uring/futex: return a wait's data to the cache when it fails to arm

      commit: 02caa5922184e99da04b37fd3336066625ea8e2e

Best regards,

Signature · 1 line
Jens Axboe