Skip to contentSkip to conversation

io_uring updates for 5.16-rc1

GIT PULL Merged Jens Axboe 3 participants 4 messages Oct 31, 2021 – Nov 1, 2021

Subsystems (2)
lore
Share & tools
lore

Jens Axboe

author maintainer
#0 lore raw

Hi Linus,

Sitting on top of the core block branch due to the polling changes, here are the io_uring updates for the 5.16-rc1 merge window. Light on new features (basically just the hybrid mode support), outside of that it's just fixes, cleanups, and performance improvements. In detail:

- Add ring related information to the fdinfo output (Hao)

- Hybrid async mode (Hao)

- Support for batched issue on block (me)

- sqe error trace improvement (me)

- IOPOLL efficiency improvements (Pavel)

- submit state cleanups and improvements (Pavel)

- Completion side improvements (Pavel)

- Drain improvements (Pavel)

- Buffer selection cleanups (Pavel)

- Fixed file node improvements (Pavel)

- io-wq setup cancelation fix (Pavel)

- Various other performance improvements and cleanups (Pavel)

- Misc fixes (Arnd, Bixuan, Changcheng, Hao, me, Noah)

This will throw two merge conflicts, see below for how I resolved it. There are two spots, one is trivial, and the other needs io_queue_linked_timeout() moved into io_queue_sqe_arm_apoll().

Please pull!

The following changes since commit e0d78afeb8d190164a823d5ef5821b0b3802af33:

block: fix too broad elevator check in blk_mq_free_request() (2021-10-19 05:48:15 -0600)

are available in the Git repository at:

git://git.kernel.dk/linux-block.git tags/for-5.16/io_uring-2021-10-29

for you to fetch changes up to 1d5f5ea7cb7d15b9fb1cc82673ebb054f02cd7d2:

io-wq: remove worker to owner tw dependency (2021-10-29 09:49:33 -0600)

----------------------------------------------------------------
for-5.16/io_uring-2021-10-29

----------------------------------------------------------------
Arnd Bergmann (1):
io_uring: warning about unused-but-set parameter

Bixuan Cui (1):
io-wq: Remove duplicate code in io_workqueue_create()

Changcheng Deng (1):
io_uring: Use ERR_CAST() instead of ERR_PTR(PTR_ERR())

Hao Xu (4):
io_uring: add more uring info to fdinfo for debug
io_uring: return boolean value for io_alloc_async_data
io_uring: split logic of force_nonblock
io_uring: implement async hybrid mode for pollable requests

Jens Axboe (4):
io_uring: dump sqe contents if issue fails
io_uring: inform block layer of how many requests we are submitting
io_uring: don't assign write hint in the read path
io_uring: harder fdinfo sq/cq ring iterating

Noah Goldstein (1):
fs/io_uring: Prioritise checking faster conditions first in io_write

Pavel Begunkov (85):
io_uring: kill off ios_left
io_uring: inline io_dismantle_req
io_uring: inline linked part of io_req_find_next
io_uring: dedup CQE flushing non-empty checks
io_uring: kill extra wake_up_process in tw add
io_uring: remove ctx referencing from complete_post
io_uring: optimise io_req_init() sqe flags checks
io_uring: mark having different creds unlikely
io_uring: force_nonspin
io_uring: make io_do_iopoll return number of reqs
io_uring: use slist for completion batching
io_uring: remove allocation cache array
io-wq: add io_wq_work_node based stack
io_uring: replace list with stack for req caches
io_uring: split iopoll loop
io_uring: use single linked list for iopoll
io_uring: add a helper for batch free
io_uring: convert iopoll_completed to store_release
io_uring: optimise batch completion
io_uring: inline completion batching helpers
io_uring: don't pass tail into io_free_batch_list
io_uring: don't pass state to io_submit_state_end
io_uring: deduplicate io_queue_sqe() call sites
io_uring: remove drain_active check from hot path
io_uring: split slow path from io_queue_sqe
io_uring: inline hot path of __io_queue_sqe()
io_uring: reshuffle queue_sqe completion handling
io_uring: restructure submit sqes to_submit checks
io_uring: kill off ->inflight_entry field
io_uring: comment why inline complete calls io_clean_op()
io_uring: disable draining earlier
io_uring: extra a helper for drain init
io_uring: don't return from io_drain_req()
io_uring: init opcode in io_init_req()
io_uring: clean up buffer select
io_uring: add flag to not fail link after timeout
io_uring: optimise kiocb layout
io_uring: add more likely/unlikely() annotations
io_uring: delay req queueing into compl-batch list
io_uring: optimise request allocation
io_uring: optimise INIT_WQ_LIST
io_uring: don't wake sqpoll in io_cqring_ev_posted
io_uring: merge CQ and poll waitqueues
io_uring: optimise ctx referencing by requests
io_uring: mark cold functions
io_uring: optimise io_free_batch_list()
io_uring: control ->async_data with a REQ_F flag
io_uring: remove struct io_completion
io_uring: inline io_req_needs_clean()
io_uring: inline io_poll_complete
io_uring: correct fill events helpers types
io_uring: optimise plugging
io_uring: safer fallback_work free
io_uring: reshuffle io_submit_state bits
io_uring: optimise out req->opcode reloading
io_uring: remove extra io_ring_exit_work wake up
io_uring: fix io_free_batch_list races
io_uring: optimise io_req_set_rsrc_node()
io_uring: optimise rsrc referencing
io_uring: consistent typing for issue_flags
io_uring: prioritise read success path over fails
io_uring: optimise rw comletion handlers
io_uring: encapsulate rw state
io_uring: optimise read/write iov state storing
io_uring: optimise io_import_iovec nonblock passing
io_uring: clean up io_import_iovec
io_uring: rearrange io_read()/write()
io_uring: optimise req->ctx reloads
io_uring: kill io_wq_current_is_worker() in iopoll
io_uring: optimise io_import_iovec fixed path
io_uring: return iovec from __io_import_iovec
io_uring: optimise fixed rw rsrc node setting
io_uring: clean io_prep_rw()
io_uring: arm poll for non-nowait files
io_uring: combine REQ_F_NOWAIT_{READ,WRITE} flags
io_uring: simplify io_file_supports_nowait()
io-wq: use helper for worker refcounting
io_uring: clean io_wq_submit_work()'s main loop
io_uring: clean iowq submit work cancellation
io_uring: check if opcode needs poll first on arming
io_uring: don't try io-wq polling if not supported
io_uring: clean up timeout async_data allocation
io_uring: kill unused param from io_file_supports_nowait
io_uring: clusterise ki_flags access in rw_prep
io-wq: remove worker to owner tw dependency

 fs/io-wq.c                      |   58 +-
 fs/io-wq.h                      |   59 +-
 fs/io_uring.c                   | 1714 ++++++++++++++++++++-------------------
 include/trace/events/io_uring.h |   61 ++
 include/uapi/linux/io_uring.h   |    1 +
 5 files changed, 1045 insertions(+), 848 deletions(-)
fs/io_uring.c +0 -0
diff --cc fs/io_uring.c
index 5c11b8442743,c887e4e19e9e..5d7234363b9f
@@@ -6945,7 -6906,33 +6903,33 @@@ static void io_queue_linked_timeout(str
  	io_put_req(req);
  }
  
- static void __io_queue_sqe(struct io_kiocb *req)
+ static void io_queue_sqe_arm_apoll(struct io_kiocb *req)
+ 	__must_hold(&req->ctx->uring_lock)
+ {
+ 	struct io_kiocb *linked_timeout = io_prep_linked_timeout(req);
+ 
+ 	switch (io_arm_poll_handler(req)) {
+ 	case IO_APOLL_READY:
+ 		if (linked_timeout) {
 -			io_unprep_linked_timeout(req);
++			io_queue_linked_timeout(linked_timeout);
+ 			linked_timeout = NULL;
+ 		}
+ 		io_req_task_queue(req);
+ 		break;
+ 	case IO_APOLL_ABORTED:
+ 		/*
+ 		 * Queued up for async execution, worker will release
+ 		 * submit reference when the iocb is actually submitted.
+ 		 */
+ 		io_queue_async_work(req, NULL);
+ 		break;
+ 	}
+ 
+ 	if (linked_timeout)
+ 		io_queue_linked_timeout(linked_timeout);
+ }
+ 
+ static inline void __io_queue_sqe(struct io_kiocb *req)
  	__must_hold(&req->ctx->uring_lock)
  {
  	struct io_kiocb *linked_timeout;
@@@ -10645,11 -10697,9 +10703,11 @@@ static __cold int io_unregister_iowq_af
  	return io_wq_cpu_affinity(tctx->io_wq, NULL);
  }
  
- static int io_register_iowq_max_workers(struct io_ring_ctx *ctx,
- 					void __user *arg)
+ static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx,
+ 					       void __user *arg)
 +	__must_hold(&ctx->uring_lock)
  {
 +	struct io_tctx_node *node;
  	struct io_uring_task *tctx = NULL;
  	struct io_sq_data *sqd = NULL;
  	__u32 new_count[2];
Signature · 1 line
Jens Axboe
#1 lore raw
Quoted 4 lines · Jens Axboe

This will throw two merge conflicts, see below for how I resolved it. There are two spots, one is trivial, and the other needs io_queue_linked_timeout() moved into io_queue_sqe_arm_apoll().

So I ended up resolving it the same way you did, because that was the mindless direct thing.

But I don't much like it.

Basically, io_queue_sqe_arm_apoll() now ends up doing

        case IO_APOLL_READY:
                if (linked_timeout) {
                        io_queue_linked_timeout(linked_timeout);
                        linked_timeout = NULL;
                }
                io_req_task_queue(req);
                break;
    ...
        if (linked_timeout)
                io_queue_linked_timeout(linked_timeout);

and that really seems *completely* pointless. Notice how it does that

if (linked_timeout)
io_queue_linked_timeout()

basically twice, and sets linked_timeout to NULL just to avoid the second one...

Why isn't it just

        case IO_APOLL_READY:
                io_req_task_queue(req);
                break;
  ...
        if (linked_timeout)
                io_queue_linked_timeout(linked_timeout);

where the only difference would seem to be the order of operations between io_req_task_queue() and io_queue_linked_timeout()?

Does the order of operations really matter here? As far as I can tell, io_req_task_queue() really just queues up work for later, so it's not really very ordered wrt that io_queue_linked_timeout(), and in the _other_ case statement it's apparently fine to do that io_queue_async_work() before the io_queue_linked_timeout()..

Again - I ended up resolving this the same way you had done, because I don't know the exact rules here well enough to do anything else. But it _looks_ a bit messy.

Hmm?

Linus

author maintainer
#2 lore raw
Quoted 49 lines · Linus Torvalds
Quoted 4 lines · Jens Axboe

This will throw two merge conflicts, see below for how I resolved it. There are two spots, one is trivial, and the other needs io_queue_linked_timeout() moved into io_queue_sqe_arm_apoll().

So I ended up resolving it the same way you did, because that was the mindless direct thing.

But I don't much like it.

Basically, io_queue_sqe_arm_apoll() now ends up doing

        case IO_APOLL_READY:
                if (linked_timeout) {
                        io_queue_linked_timeout(linked_timeout);
                        linked_timeout = NULL;
                }
                io_req_task_queue(req);
                break;
    ...
        if (linked_timeout)
                io_queue_linked_timeout(linked_timeout);

and that really seems *completely* pointless. Notice how it does that

if (linked_timeout)
io_queue_linked_timeout()

basically twice, and sets linked_timeout to NULL just to avoid the second one...

Why isn't it just

        case IO_APOLL_READY:
                io_req_task_queue(req);
                break;
  ...
        if (linked_timeout)
                io_queue_linked_timeout(linked_timeout);

where the only difference would seem to be the order of operations between io_req_task_queue() and io_queue_linked_timeout()?

Does the order of operations really matter here? As far as I can tell, io_req_task_queue() really just queues up work for later, so it's not really very ordered wrt that io_queue_linked_timeout(), and in the _other_ case statement it's apparently fine to do that io_queue_async_work() before the io_queue_linked_timeout()..

Quoted 3 lines

Again - I ended up resolving this the same way you had done, because I don't know the exact rules here well enough to do anything else. But it _looks_ a bit messy.

Yes I agree, and it's mostly just to keep the resolution simpler as I don't think the current construct makes too much sense when both of them end up being queueing the linked timeout. I think the cleanup done here made more sense in the context before, not now.

We'll get a cleanup done for this shortly.

Signature · 1 line
Jens Axboe
merge bot Merged
#3 lore raw
In reply to Jens Axboe
The pull request you sent on Sun, 31 Oct 2021 13:41:47 -0600:

The pull request you sent on Sun, 31 Oct 2021 13:41:47 -0600:

Quoted 1 line

git://git.kernel.dk/linux-block.git tags/for-5.16/io_uring-2021-10-29

has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/8d1f01775f8ead7ee313403158be95bffdbb3638

Thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/prtracker.html