Skip to contentSkip to conversation

io_uring changes for 6.13-rc1

GIT PULL Merged Jens Axboe 3 participants 6 messages Nov 18, 2024 – Nov 19, 2024 io_uring

lore
Share & tools
lore

Jens Axboe

author maintainer
#0 lore raw

Hi Linus,

Here are the io_uring changes slated for the 6.13 kernel release. This pull request contains:

- Cleanups of the eventfd handling code, making it fully private.

- Support for sending a sync message to another ring, without having a ring available to send a normal async message.

- Get rid of the separate unlocked hash table, unify everything around the single locked one.

- Add support for ring resizing. It can be hard to appropriately size the CQ ring upfront, if the application doesn't know how busy it will be. This results in applications sizing rings for the most busy case, which can be wasteful. With ring resizing, they can start small and grow the ring, if needed.

- Add support for fixed wait regions, rather than needing to copy the same wait data tons of times for each wait operation.

- Rewrite the resource node handling, which before was serialized per ring. This caused issues with particularly fixed files, where one file waiting on IO could hold up putting and freeing of other unrelated files. Now each node is handled separately. New code is much simpler too, and was a net 250 line reduction in code.

- Add support for just doing partial buffer clones, rather than always cloning the entire buffer table.

- Series adding static NAPI support, where a specific NAPI instance is used rather than having a list of them available that need lookup.

- Add support for mapped regions, and also convert the fixed wait support mentioned above to that concept. This avoids doing special mappings for various planned features, and folds the existing registered wait into that too.

- Add support for hybrid IO polling, which is a variant of strict IOPOLL but with an initial sleep delay to avoid spinning too early and wasting resources on devices that aren't necessarily in the < 5 usec category wrt latencies.

- Various cleanups and little fixes.

Please pull!

The following changes since commit 42f7652d3eb527d03665b09edac47f85fb600924:

Linux 6.12-rc4 (2024-10-20 15:19:38 -0700)

are available in the Git repository at:

git://git.kernel.dk/linux.git tags/for-6.13/io_uring-20241118

for you to fetch changes up to a652958888fb1ada3e4f6b548576c2d2c1b60d66:

io_uring/region: fix error codes after failed vmap (2024-11-17 09:01:35 -0700)

----------------------------------------------------------------
for-6.13/io_uring-20241118

----------------------------------------------------------------
Al Viro (1):
switch io_msg_ring() to CLASS(fd)

Jens Axboe (50):
io_uring/eventfd: abstract out ev_fd put helper io_uring/eventfd: check for the need to async notifier earlier io_uring/eventfd: move actual signaling part into separate helper io_uring/eventfd: move trigger check into a helper io_uring/eventfd: abstract out ev_fd grab + release helpers io_uring/eventfd: move ctx->evfd_last_cq_tail into io_ev_fd io_uring/msg_ring: refactor a few helper functions io_uring/msg_ring: add support for sending a sync message io_uring/poll: remove 'ctx' argument from io_poll_req_delete() io_uring/poll: get rid of unlocked cancel hash io_uring/poll: get rid of io_poll_tw_hash_eject() io_uring/poll: get rid of per-hashtable bucket locks io_uring/cancel: get rid of init_hash_table() helper
io_uring: move cancel hash tables to kvmalloc/kvfree io_uring/rsrc: don't assign bvec twice in io_import_fixed() io_uring/uring_cmd: get rid of using req->imu
io_uring/rw: get rid of using req->imu
io_uring: remove 'issue_flags' argument for io_req_set_rsrc_node() io_uring/net: move send zc fixed buffer import to issue path
io_uring: kill 'imu' from struct io_kiocb
io_uring: move max entry definition and ring sizing into header
io_uring: abstract out a bit of the ring filling logic io_uring/memmap: explicitly return -EFAULT for mmap on NULL rings io_uring/register: add IORING_REGISTER_RESIZE_RINGS io_uring/sqpoll: wait on sqd->wait for thread parking
io_uring: switch struct ext_arg from __kernel_timespec to timespec64
io_uring: change io_get_ext_arg() to use uaccess begin + end
io_uring: add support for fixed wait regions
io_uring/nop: add support for testing registered files and buffers io_uring/rsrc: move struct io_fixed_file to rsrc.h header
io_uring: specify freeptr usage for SLAB_TYPESAFE_BY_RCU io_kiocb cache io_uring/splice: open code 2nd direct file assignment io_uring/rsrc: kill io_charge_rsrc_node()
io_uring/rsrc: get rid of per-ring io_rsrc_node list io_uring/rsrc: get rid of io_rsrc_node allocation cache io_uring/rsrc: add an empty io_rsrc_node for sparse buffer entries
io_uring: only initialize io_kiocb rsrc_nodes when needed io_uring/rsrc: unify file and buffer resource tables io_uring/rsrc: add io_rsrc_node_lookup() helper io_uring/filetable: remove io_file_from_index() helper io_uring/filetable: kill io_reset_alloc_hint() helper io_uring/rsrc: add io_reset_rsrc_node() helper io_uring/rsrc: get rid of the empty node and dummy_ubuf io_uring/rsrc: allow cloning at an offset
io_uring/rsrc: allow cloning with node replacements io_uring/rsrc: encode node type and ctx together io_uring/rsrc: split io_kiocb node type assignments
io_uring: move cancelations to be io_uring_task based
io_uring: remove task ref helpers
io_uring: move struct io_kiocb from task_struct to io_uring_task

Ming Lei (4):
io_uring/rsrc: pass 'struct io_ring_ctx' reference to rsrc helpers io_uring/rsrc: remove '->ctx_ptr' of 'struct io_rsrc_node' io_uring/rsrc: add & apply io_req_assign_buf_node() io_uring/uring_cmd: fix buffer index retrieval

Olivier Langlois (6):
io_uring/napi: protect concurrent io_napi_entry timeout accesses io_uring/napi: fix io_napi_entry RCU accesses io_uring/napi: improve __io_napi_add
io_uring/napi: Use lock guards
io_uring/napi: clean up __io_napi_do_busy_loop io_uring/napi: add static napi tracking strategy

Pavel Begunkov (17):
io_uring: kill io_llist_xchg
io_uring: static_key for !IORING_SETUP_NO_SQARRAY
io_uring: clean up cqe trace points io_uring/net: split send and sendmsg prep helpers io_uring/net: don't store send address ptr io_uring/net: don't alias send user pointer reads io_uring/net: clean up io_msg_copy_hdr
io_uring: prevent speculating sq_array indexing
io_uring: avoid normal tw intermediate fallback
io_uring: fix invalid hybrid polling ctx leaks
io_uring: fortify io_pin_pages with a warning
io_uring: disable ENTER_EXT_ARG_REG for IOPOLL
io_uring: temporarily disable registered waits
io_uring: introduce concept of memory regions
io_uring: add memory region registration
io_uring: restore back registered wait arguments io_uring/region: fix error codes after failed vmap

hexue (1):
io_uring: add support for hybrid IOPOLL

 include/linux/io_uring/cmd.h    |   2 +-
 include/linux/io_uring_types.h  |  88 ++++--
 include/trace/events/io_uring.h |  24 +-
 include/uapi/linux/io_uring.h   | 119 +++++++-
 io_uring/cancel.c               |  20 +-
 io_uring/cancel.h               |   1 -
 io_uring/eventfd.c              | 137 ++++++---
 io_uring/fdinfo.c               |  88 ++++--
 io_uring/filetable.c            |  71 ++---
 io_uring/filetable.h            |  35 +--
 io_uring/futex.c                |   4 +-
 io_uring/futex.h                |   4 +-
 io_uring/io_uring.c             | 433 ++++++++++++++------------
 io_uring/io_uring.h             |  30 +-
 io_uring/memmap.c               |  83 +++++
 io_uring/memmap.h               |  14 +
 io_uring/msg_ring.c             |  91 ++++--
 io_uring/msg_ring.h             |   1 +
 io_uring/napi.c                 | 184 +++++++----
 io_uring/napi.h                 |   8 +-
 io_uring/net.c                  | 112 ++++---
 io_uring/nop.c                  |  47 ++-
 io_uring/notif.c                |   7 +-
 io_uring/opdef.c                |   2 +
 io_uring/poll.c                 | 181 +++--------
 io_uring/poll.h                 |   2 +-
 io_uring/register.c             | 299 +++++++++++++++++-
 io_uring/rsrc.c                 | 657 +++++++++++++++-------------------------
 io_uring/rsrc.h                 |  97 +++---
 io_uring/rw.c                   | 105 +++++--
 io_uring/splice.c               |  42 ++-
 io_uring/splice.h               |   1 +
 io_uring/sqpoll.c               |   3 +-
 io_uring/tctx.c                 |   1 +
 io_uring/timeout.c              |  16 +-
 io_uring/timeout.h              |   2 +-
 io_uring/uring_cmd.c            |  27 +-
 io_uring/uring_cmd.h            |   2 +-
 io_uring/waitid.c               |   6 +-
 io_uring/waitid.h               |   2 +-
 40 files changed, 1825 insertions(+), 1223 deletions(-)
Signature · 1 line
Jens Axboe
#1 lore raw

Hi Jens, Thomas,

Quoted 2 lines · Jens Axboe

hexue (1):
io_uring: add support for hybrid IOPOLL

After merging of this pull request into linus-next, I've started seeing build errors:

/builds/linux/io_uring/rw.c: In function 'io_hybrid_iopoll_delay':
/builds/linux/io_uring/rw.c:1179:2: error: implicit declaration of function 'hrtimer_init_sleeper_on_stack'; did you mean 'hrtimer_setup_sleeper_on_stack'? [-Werror=implicit-function-declaration]
hrtimer_init_sleeper_on_stack(&timer, CLOCK_MONOTONIC, mode);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
hrtimer_setup_sleeper_on_stack

This is because 01ee194d1aba ("io_uring: add support for hybrid IOPOLL") adds a call to hrtimer_init_sleeper_on_stack() which was removed earlier today in Thomas's PR[1], specifically in commit f3bef7aaa6c8 ("hrtimers: Delete hrtimer_init_sleeper_on_stack()").

[1] https://lore.kernel.org/all/173195758632.1896928.11371209657780930206.tglx@xen13/

Signature · 2 lines
Thanks,
Sasha
author maintainer
#2 lore raw
Quoted 15 lines · Sasha Levin

Hi Jens, Thomas,

Quoted 2 lines · Jens Axboe

hexue (1):
io_uring: add support for hybrid IOPOLL

After merging of this pull request into linus-next, I've started seeing build errors:

/builds/linux/io_uring/rw.c: In function 'io_hybrid_iopoll_delay':
/builds/linux/io_uring/rw.c:1179:2: error: implicit declaration of function 'hrtimer_init_sleeper_on_stack'; did you mean 'hrtimer_setup_sleeper_on_stack'? [-Werror=implicit-function-declaration]
hrtimer_init_sleeper_on_stack(&timer, CLOCK_MONOTONIC, mode);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
hrtimer_setup_sleeper_on_stack

Quoted 4 lines

This is because 01ee194d1aba ("io_uring: add support for hybrid IOPOLL") adds a call to hrtimer_init_sleeper_on_stack() which was removed earlier today in Thomas's PR[1], specifically in commit f3bef7aaa6c8 ("hrtimers: Delete hrtimer_init_sleeper_on_stack()").

Right, forgot to mention that. linux-next has been carrying a fixup for that which I was going to link, but it's not on a public list for some reason. In any case, it's trivial to fixup, just change

hrtimer_init_sleeper_on_stack(&timer, CLOCK_MONOTONIC, mode);

to

hrtimer_setup_sleeper_on_stack(&timer, CLOCK_MONOTONIC, mode);

in io_uring/rw.c while merging either of the two trees that go in last.

FWIW, this kind of change should be done before -rc1 rather than be queued up earlier. That's usually the best way to do these things and avoid -next or merge window breakage. I'm guessing this tree isn't the only one going to get hit by that.

Signature · 1 line
Jens Axboe
author maintainer
#3 lore raw
Quoted 16 lines · Jens Axboe
Quoted 15 lines · Sasha Levin

Hi Jens, Thomas,

Quoted 2 lines · Jens Axboe

hexue (1):
io_uring: add support for hybrid IOPOLL

After merging of this pull request into linus-next, I've started seeing build errors:

/builds/linux/io_uring/rw.c: In function 'io_hybrid_iopoll_delay':
/builds/linux/io_uring/rw.c:1179:2: error: implicit declaration of function 'hrtimer_init_sleeper_on_stack'; did you mean 'hrtimer_setup_sleeper_on_stack'? [-Werror=implicit-function-declaration]
hrtimer_init_sleeper_on_stack(&timer, CLOCK_MONOTONIC, mode);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
hrtimer_setup_sleeper_on_stack

Quoted 8 lines
Quoted 4 lines

This is because 01ee194d1aba ("io_uring: add support for hybrid IOPOLL") adds a call to hrtimer_init_sleeper_on_stack() which was removed earlier today in Thomas's PR[1], specifically in commit f3bef7aaa6c8 ("hrtimers: Delete hrtimer_init_sleeper_on_stack()").

Right, forgot to mention that. linux-next has been carrying a fixup for that which I was going to link, but it's not on a public list for some reason.

My bad, it is on lkml:

https://lore.kernel.org/lkml/[email protected]/

Signature · 1 line
Jens Axboe
#4 lore raw
In reply to Sasha Levin
Quoted 15 lines · Sasha Levin

Hi Jens, Thomas,

Quoted 2 lines · Jens Axboe

hexue (1):
io_uring: add support for hybrid IOPOLL

After merging of this pull request into linus-next, I've started seeing build errors:

/builds/linux/io_uring/rw.c: In function 'io_hybrid_iopoll_delay':
/builds/linux/io_uring/rw.c:1179:2: error: implicit declaration of function 'hrtimer_init_sleeper_on_stack'; did you mean 'hrtimer_setup_sleeper_on_stack'? [-Werror=implicit-function-declaration]
hrtimer_init_sleeper_on_stack(&timer, CLOCK_MONOTONIC, mode);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
hrtimer_setup_sleeper_on_stack

Quoted 4 lines

This is because 01ee194d1aba ("io_uring: add support for hybrid IOPOLL") adds a call to hrtimer_init_sleeper_on_stack() which was removed earlier today in Thomas's PR[1], specifically in commit f3bef7aaa6c8 ("hrtimers: Delete hrtimer_init_sleeper_on_stack()").

Linus,

Looks like this is a simple s/hrtimer_init_sleeper_on_stack/hrtimer_setup_sleeper_on_stack , so this issue could be addressed by replacing the new call with hrtimer_setup_sleeper_on_stack() either in the io_uring or the timers/core merge, whichever you pull last.

Signature · 2 lines
Thanks,
Sasha
merge bot Merged
#5 lore raw
In reply to Jens Axboe
The pull request you sent on Mon, 18 Nov 2024 07:22:59 -0700:

The pull request you sent on Mon, 18 Nov 2024 07:22:59 -0700:

Quoted 1 line

git://git.kernel.dk/linux.git tags/for-6.13/io_uring-20241118

has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/8350142a4b4cedebfa76cd4cc6e5a7ba6a330629

Thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/prtracker.html