io_uring updates for 5.16-rc1
Jens Axboe
maintainerHi Linus,
Sitting on top of the core block branch due to the polling changes, here are the io_uring updates for the 5.16-rc1 merge window. Light on new features (basically just the hybrid mode support), outside of that it's just fixes, cleanups, and performance improvements. In detail:
- Add ring related information to the fdinfo output (Hao)
- Hybrid async mode (Hao)
- Support for batched issue on block (me)
- sqe error trace improvement (me)
- IOPOLL efficiency improvements (Pavel)
- submit state cleanups and improvements (Pavel)
- Completion side improvements (Pavel)
- Drain improvements (Pavel)
- Buffer selection cleanups (Pavel)
- Fixed file node improvements (Pavel)
- io-wq setup cancelation fix (Pavel)
- Various other performance improvements and cleanups (Pavel)
- Misc fixes (Arnd, Bixuan, Changcheng, Hao, me, Noah)
This will throw two merge conflicts, see below for how I resolved it. There are two spots, one is trivial, and the other needs io_queue_linked_timeout() moved into io_queue_sqe_arm_apoll().
Please pull!
The following changes since commit e0d78afeb8d190164a823d5ef5821b0b3802af33:
block: fix too broad elevator check in blk_mq_free_request() (2021-10-19 05:48:15 -0600)
are available in the Git repository at:
git://git.kernel.dk/linux-block.git tags/for-5.16/io_uring-2021-10-29
for you to fetch changes up to 1d5f5ea7cb7d15b9fb1cc82673ebb054f02cd7d2:
io-wq: remove worker to owner tw dependency (2021-10-29 09:49:33 -0600)
----------------------------------------------------------------
for-5.16/io_uring-2021-10-29
----------------------------------------------------------------
Arnd Bergmann (1):
io_uring: warning about unused-but-set parameter
Bixuan Cui (1):
io-wq: Remove duplicate code in io_workqueue_create()
Changcheng Deng (1):
io_uring: Use ERR_CAST() instead of ERR_PTR(PTR_ERR())
Hao Xu (4):
io_uring: add more uring info to fdinfo for debug
io_uring: return boolean value for io_alloc_async_data
io_uring: split logic of force_nonblock
io_uring: implement async hybrid mode for pollable requests
Jens Axboe (4):
io_uring: dump sqe contents if issue fails
io_uring: inform block layer of how many requests we are submitting
io_uring: don't assign write hint in the read path
io_uring: harder fdinfo sq/cq ring iterating
Noah Goldstein (1):
fs/io_uring: Prioritise checking faster conditions first in io_write
Pavel Begunkov (85):
io_uring: kill off ios_left
io_uring: inline io_dismantle_req
io_uring: inline linked part of io_req_find_next
io_uring: dedup CQE flushing non-empty checks
io_uring: kill extra wake_up_process in tw add
io_uring: remove ctx referencing from complete_post
io_uring: optimise io_req_init() sqe flags checks
io_uring: mark having different creds unlikely
io_uring: force_nonspin
io_uring: make io_do_iopoll return number of reqs
io_uring: use slist for completion batching
io_uring: remove allocation cache array
io-wq: add io_wq_work_node based stack
io_uring: replace list with stack for req caches
io_uring: split iopoll loop
io_uring: use single linked list for iopoll
io_uring: add a helper for batch free
io_uring: convert iopoll_completed to store_release
io_uring: optimise batch completion
io_uring: inline completion batching helpers
io_uring: don't pass tail into io_free_batch_list
io_uring: don't pass state to io_submit_state_end
io_uring: deduplicate io_queue_sqe() call sites
io_uring: remove drain_active check from hot path
io_uring: split slow path from io_queue_sqe
io_uring: inline hot path of __io_queue_sqe()
io_uring: reshuffle queue_sqe completion handling
io_uring: restructure submit sqes to_submit checks
io_uring: kill off ->inflight_entry field
io_uring: comment why inline complete calls io_clean_op()
io_uring: disable draining earlier
io_uring: extra a helper for drain init
io_uring: don't return from io_drain_req()
io_uring: init opcode in io_init_req()
io_uring: clean up buffer select
io_uring: add flag to not fail link after timeout
io_uring: optimise kiocb layout
io_uring: add more likely/unlikely() annotations
io_uring: delay req queueing into compl-batch list
io_uring: optimise request allocation
io_uring: optimise INIT_WQ_LIST
io_uring: don't wake sqpoll in io_cqring_ev_posted
io_uring: merge CQ and poll waitqueues
io_uring: optimise ctx referencing by requests
io_uring: mark cold functions
io_uring: optimise io_free_batch_list()
io_uring: control ->async_data with a REQ_F flag
io_uring: remove struct io_completion
io_uring: inline io_req_needs_clean()
io_uring: inline io_poll_complete
io_uring: correct fill events helpers types
io_uring: optimise plugging
io_uring: safer fallback_work free
io_uring: reshuffle io_submit_state bits
io_uring: optimise out req->opcode reloading
io_uring: remove extra io_ring_exit_work wake up
io_uring: fix io_free_batch_list races
io_uring: optimise io_req_set_rsrc_node()
io_uring: optimise rsrc referencing
io_uring: consistent typing for issue_flags
io_uring: prioritise read success path over fails
io_uring: optimise rw comletion handlers
io_uring: encapsulate rw state
io_uring: optimise read/write iov state storing
io_uring: optimise io_import_iovec nonblock passing
io_uring: clean up io_import_iovec
io_uring: rearrange io_read()/write()
io_uring: optimise req->ctx reloads
io_uring: kill io_wq_current_is_worker() in iopoll
io_uring: optimise io_import_iovec fixed path
io_uring: return iovec from __io_import_iovec
io_uring: optimise fixed rw rsrc node setting
io_uring: clean io_prep_rw()
io_uring: arm poll for non-nowait files
io_uring: combine REQ_F_NOWAIT_{READ,WRITE} flags
io_uring: simplify io_file_supports_nowait()
io-wq: use helper for worker refcounting
io_uring: clean io_wq_submit_work()'s main loop
io_uring: clean iowq submit work cancellation
io_uring: check if opcode needs poll first on arming
io_uring: don't try io-wq polling if not supported
io_uring: clean up timeout async_data allocation
io_uring: kill unused param from io_file_supports_nowait
io_uring: clusterise ki_flags access in rw_prep
io-wq: remove worker to owner tw dependency
fs/io-wq.c | 58 +- fs/io-wq.h | 59 +- fs/io_uring.c | 1714 ++++++++++++++++++++------------------- include/trace/events/io_uring.h | 61 ++ include/uapi/linux/io_uring.h | 1 + 5 files changed, 1045 insertions(+), 848 deletions(-)
fs/io_uring.c +0 -0
@@@ -6945,7 -6906,33 +6903,33 @@@ static void io_queue_linked_timeout(strio_put_req (req); } - static void __io_queue_sqe(struct io_kiocb *req) +static void io_queue_sqe_arm_apoll (struct io_kiocb *req) + __must_hold(&req->ctx->uring_lock) + { +struct io_kiocb *linked_timeout =io_prep_linked_timeout (req); + +switch (io_arm_poll_handler (req)) { +case IO_APOLL_READY : +if (linked_timeout) { -io_unprep_linked_timeout (req); ++io_queue_linked_timeout (linked_timeout); + linked_timeout =NULL ; + } +io_req_task_queue (req); +break ; +case IO_APOLL_ABORTED : +/* +* Queued up for async execution, worker will release +* submit reference when the iocb is actually submitted. +*/ +io_queue_async_work (req,NULL ); +break ; + } + +if (linked_timeout) +io_queue_linked_timeout (linked_timeout); + } + +static inline void __io_queue_sqe (struct io_kiocb *req) __must_hold(&req->ctx->uring_lock) {struct io_kiocb *linked_timeout;return io_wq_cpu_affinity (tctx->io_wq,NULL ); } -static int io_register_iowq_max_workers (struct io_ring_ctx *ctx, - void __user *arg) +static __cold int io_register_iowq_max_workers (struct io_ring_ctx *ctx, + void __user *arg) +__must_hold (&ctx->uring_lock) { + struct io_tctx_node *node;struct io_uring_task *tctx =NULL ;struct io_sq_data *sqd =NULL ;__u32 new_count[2 ];
Signature · 1 line
Jens Axboe
Quoted 4 lines · Jens Axboe
This will throw two merge conflicts, see below for how I resolved it. There are two spots, one is trivial, and the other needs
io_queue_linked_timeout()moved intoio_queue_sqe_arm_apoll().
So I ended up resolving it the same way you did, because that was the mindless direct thing.
But I don't much like it.
Basically, io_queue_sqe_arm_apoll() now ends up doing
case IO_APOLL_READY:
if (linked_timeout) {
io_queue_linked_timeout(linked_timeout);
linked_timeout = NULL;
}
io_req_task_queue(req);
break;
...
if (linked_timeout)
io_queue_linked_timeout(linked_timeout);and that really seems *completely* pointless. Notice how it does that
if (linked_timeout)
io_queue_linked_timeout()
basically twice, and sets linked_timeout to NULL just to avoid the second one...
Why isn't it just
case IO_APOLL_READY:
io_req_task_queue(req);
break;
...
if (linked_timeout)
io_queue_linked_timeout(linked_timeout);where the only difference would seem to be the order of operations between io_req_task_queue() and io_queue_linked_timeout()?
Does the order of operations really matter here? As far as I can tell, io_req_task_queue() really just queues up work for later, so it's not really very ordered wrt that io_queue_linked_timeout(), and in the _other_ case statement it's apparently fine to do that io_queue_async_work() before the io_queue_linked_timeout()..
Again - I ended up resolving this the same way you had done, because I don't know the exact rules here well enough to do anything else. But it _looks_ a bit messy.
Hmm?
Linus
Jens Axboe
maintainerQuoted 49 lines · Linus Torvalds
Quoted 4 lines · Jens Axboe
This will throw two merge conflicts, see below for how I resolved it. There are two spots, one is trivial, and the other needs
io_queue_linked_timeout()moved intoio_queue_sqe_arm_apoll().So I ended up resolving it the same way you did, because that was the mindless direct thing.
But I don't much like it.
Basically,
io_queue_sqe_arm_apoll()now ends up doingcase IO_APOLL_READY: if (linked_timeout) { io_queue_linked_timeout(linked_timeout); linked_timeout = NULL; } io_req_task_queue(req); break; ... if (linked_timeout) io_queue_linked_timeout(linked_timeout);and that really seems *completely* pointless. Notice how it does that
if (
linked_timeout)
io_queue_linked_timeout()basically twice, and sets
linked_timeoutto NULL just to avoid the second one...Why isn't it just
case IO_APOLL_READY: io_req_task_queue(req); break; ... if (linked_timeout) io_queue_linked_timeout(linked_timeout);where the only difference would seem to be the order of operations between
io_req_task_queue()andio_queue_linked_timeout()?Does the order of operations really matter here? As far as I can tell,
io_req_task_queue()really just queues up work for later, so it's not really very ordered wrt thatio_queue_linked_timeout(), and in the _other_ case statement it's apparently fine to do thatio_queue_async_work()before theio_queue_linked_timeout()..
Quoted 3 lines
Again - I ended up resolving this the same way you had done, because I don't know the exact rules here well enough to do anything else. But it _looks_ a bit messy.
Yes I agree, and it's mostly just to keep the resolution simpler as I don't think the current construct makes too much sense when both of them end up being queueing the linked timeout. I think the cleanup done here made more sense in the context before, not now.
We'll get a cleanup done for this shortly.
Signature · 1 line
Jens Axboe
The pull request you sent on Sun, 31 Oct 2021 13:41:47 -0600:
The pull request you sent on Sun, 31 Oct 2021 13:41:47 -0600:
Quoted 1 line
git://git.kernel.dk/linux-block.gittags/for-5.16/io_uring-2021-10-29
has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/8d1f01775f8ead7ee313403158be95bffdbb3638
Thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/prtracker.html