io_uring/uring_cmd: allow non-iopoll cmds with IORING_SETUP_IOPOLL
v5 cover io_uring/uring_cmd: allow non-iopoll cmds with IORING_SETUP_IOPOLL
Currently, creating an io_uring with IORING_SETUP_IOPOLL requires all requests issued to it to support iopoll. This prevents, for example, using ublk zero-copy together with IORING_SETUP_IOPOLL, as ublk zero-copy buffer registrations are performed using a uring_cmd. There's no technical reason why these non-iopoll uring_cmds can't be supported. They will either complete synchronously or via an external mechanism that calls io_uring_cmd_done(), io_uring_cmd_post_mshot_cqe32(), or io_uring_mshot_cmd_post_cqe(), so they don't need to be polled.
Allow uring_cmd requests to be issued to IORING_SETUP_IOPOLL io_urings even if their files don't implement ->uring_cmd_iopoll().
Use a new REQ_F_IOPOLL flag to track whether a request is using iopoll. This makes the iopoll_queue opcode definition flag unnecessary.
The last commit removes an unnecessary IO_URING_F_IOPOLL check in nvme_dev_uring_cmd() as NVMe admin passthru commands can be issued to IORING_SETUP_IOPOLL io_urings now.
v5: perform one iteration of iopoll when min_events == 0 (Claude)
v4: check non-iopoll CQEs against min_events in io_iopoll_check() (Ming)
v3: fix REW -> REQ typo (Anuj)
v2:
- Add REQ_F_IOPOLL request flag, remove redundant iopoll_queue
- Split IORING_OP_URING_CMD128 fix to a separate commit
Caleb Sander Mateos (5):
io_uring: add REQ_F_IOPOLL
io_uring: remove iopoll_queue from struct io_issue_def
io_uring: count CQEs in io_iopoll_check() io_uring/uring_cmd: allow non-iopoll cmds with IORING_SETUP_IOPOLL
nvme: remove nvme_dev_uring_cmd() IO_URING_F_IOPOLL check
drivers/nvme/host/ioctl.c | 4 ---- include/linux/io_uring_types.h | 3 +++ io_uring/io_uring.c | 19 ++++++------------- io_uring/opdef.c | 10 ---------- io_uring/opdef.h | 2 -- io_uring/rw.c | 11 ++++++----- io_uring/uring_cmd.c | 9 ++++----- 7 files changed, 19 insertions(+), 39 deletions(-)
Signature · 1 line
2.45.2
PATCH v5 1/5 io_uring: add REQ_F_IOPOLL
A subsequent commit will allow uring_cmds to files that don't implement ->uring_cmd_iopoll() to be issued to IORING_SETUP_IOPOLL io_urings. This means the ctx's IORING_SETUP_IOPOLL flag isn't sufficient to determine whether a given request needs to be iopolled.
Introduce a request flag REQ_F_IOPOLL set in ->issue() if a request needs to be iopolled to completion. Set the flag in io_rw_init_file() and io_uring_cmd() for requests issued to IORING_SETUP_IOPOLL ctxs. Use the request flag instead of IORING_SETUP_IOPOLL in places dealing with a specific request.
A future possibility would be to add an option to enable/disable iopoll in the io_uring SQE instead of determining it from IORING_SETUP_IOPOLL.
--- include/linux/io_uring_types.h | 3 +++ io_uring/io_uring.c | 9 ++++----- io_uring/rw.c | 11 ++++++----- io_uring/uring_cmd.c | 5 +++-- 4 files changed, 16 insertions(+), 12 deletions(-)
include/linux/io_uring_types.h +3 -0
@@ -541,10 +541,11 @@ enum {REQ_F_BUFFERS_COMMIT_BIT ,REQ_F_BUF_NODE_BIT ,REQ_F_HAS_METADATA_BIT ,REQ_F_IMPORT_BUFFER_BIT ,REQ_F_SQE_COPIED_BIT , +REQ_F_IOPOLL_BIT ,/* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, };
@@ -632,10 +633,12 @@ enum {* For SEND_ZC, whether to import buffers (i.e. the first issue). */ REQ_F_IMPORT_BUFFER =IO_REQ_FLAG (REQ_F_IMPORT_BUFFER_BIT ),/* ->sqe_copy() has been called, if necessary */ REQ_F_SQE_COPIED =IO_REQ_FLAG (REQ_F_SQE_COPIED_BIT ), +/* request must be iopolled to completion (set in ->issue()) */ + REQ_F_IOPOLL = IO_REQ_FLAG(REQ_F_IOPOLL_BIT), };struct io_tw_req {struct io_kiocb *req; };
io_uring/io_uring.c +4 -5
@@ -354,11 +354,10 @@ static struct io_kiocb *__io_prep_linked_timeout(struct io_kiocb *req) }static void io_prep_async_work (struct io_kiocb *req) {const struct io_issue_def *def = &io_issue_defs[req->opcode]; -struct io_ring_ctx *ctx = req->ctx;if (!(req->flags &REQ_F_CREDS )) { req->flags |=REQ_F_CREDS ; req->creds =get_current_cred (); }
@@ -376,11 +375,11 @@ static void io_prep_async_work(struct io_kiocb *req)/* don't serialize this request if the fs doesn't need it */ if (should_hash && (req->file->f_flags &O_DIRECT ) && (req->file->f_op->fop_flags &FOP_DIO_PARALLEL_WRITE )) should_hash = false; -if (should_hash || (ctx->flags &IORING_SETUP_IOPOLL )) +if (should_hash || (req->flags &REQ_F_IOPOLL ))io_wq_hash_work (&req->work,file_inode (req->file)); }else if (!req->file || !S_ISBLK (file_inode (req->file)->i_mode)) {if (def->unbound_nonreg_file)atomic_or (IO_WQ_WORK_UNBOUND , &req->work.flags); }
@@ -1417,11 +1416,11 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)if (ret ==IOU_ISSUE_SKIP_COMPLETE ) { ret =0 ;/* If the op doesn't have a file, we're not polling for it */ -if ((req->ctx->flags &IORING_SETUP_IOPOLL ) && def->iopoll_queue) +if ((req->flags &REQ_F_IOPOLL ) && def->iopoll_queue)io_iopoll_req_issued (req, issue_flags); }return ret; }
@@ -1433,11 +1432,11 @@ int io_poll_issue(struct io_kiocb *req, io_tw_token_t tw)int ret;io_tw_lock (req->ctx, tw);WARN_ON_ONCE (!req->file); -if (WARN_ON_ONCE (req->ctx->flags &IORING_SETUP_IOPOLL )) +if (WARN_ON_ONCE (req->flags &REQ_F_IOPOLL ))return -EFAULT ; ret =__io_issue_sqe (req, issue_flags, &io_issue_defs[req->opcode]);WARN_ON_ONCE (ret ==IOU_ISSUE_SKIP_COMPLETE );
@@ -1531,11 +1530,11 @@ void io_wq_submit_work(struct io_wq_work *work)* We can get EAGAIN for iopolled IO even though we're * forcing a sync submission from here, since we can't * wait for request slots on the block side. */ if (!needs_poll) { -if (!(req->ctx->flags &IORING_SETUP_IOPOLL )) +if (!(req->flags &REQ_F_IOPOLL ))break ;if (io_wq_worker_stopped ())break ;cond_resched ();continue ;
io_uring/rw.c +6 -5
@@ -502,11 +502,11 @@ static bool io_rw_should_reissue(struct io_kiocb *req)struct io_ring_ctx *ctx = req->ctx;if (!S_ISBLK (mode) && !S_ISREG (mode))return false;if ((req->flags &REQ_F_NOWAIT ) || (io_wq_current_is_worker () && - !(ctx->flags &IORING_SETUP_IOPOLL ))) + !(req->flags &REQ_F_IOPOLL )))return false;/* * If ref is dying, we might be running poll reap from the exit work. * Don't attempt to reissue from that path, just let it fail with * -EAGAIN.
@@ -638,11 +638,11 @@ static inline void io_rw_done(struct io_kiocb *req, ssize_t ret) ret = -EINTR ;break ; } } -if (req->ctx->flags &IORING_SETUP_IOPOLL ) +if (req->flags &REQ_F_IOPOLL )io_complete_rw_iopoll (&rw->kiocb, ret);else io_complete_rw (&rw->kiocb, ret); }
@@ -652,11 +652,11 @@ static int kiocb_done(struct io_kiocb *req, ssize_t ret, struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw);unsigned final_ret =io_fixup_rw_res (req, ret);if (ret >=0 && req->flags &REQ_F_CUR_POS ) req->file->f_pos = rw->kiocb.ki_pos; -if (ret >=0 && !(req->ctx->flags &IORING_SETUP_IOPOLL )) { +if (ret >=0 && !(req->flags &REQ_F_IOPOLL )) {u32 cflags =0 ;__io_complete_rw_common (req, ret);/* * Safe to call io_end from here as we're inline
@@ -874,10 +874,11 @@ static int io_rw_init_file(struct io_kiocb *req, fmode_t mode, int rw_type) req->flags |=REQ_F_NOWAIT ;if (ctx->flags &IORING_SETUP_IOPOLL ) {if (!(kiocb->ki_flags &IOCB_DIRECT ) || !file->f_op->iopoll)return -EOPNOTSUPP ; + req->flags |=REQ_F_IOPOLL ; kiocb->private =NULL ; kiocb->ki_flags |=IOCB_HIPRI ; req->iopoll_completed =0 ;if (ctx->flags &IORING_SETUP_HYBRID_IOPOLL ) {/* make sure every req only blocks once*/
@@ -961,11 +962,11 @@ static int __io_read(struct io_kiocb *req, struct io_br_sel *sel,if (ret == -EAGAIN ) {/* If we can poll, just do that. */ if (io_file_can_poll (req))return -EAGAIN ;/* IOPOLL retry should happen for io-wq threads */ -if (!force_nonblock && !(req->ctx->flags &IORING_SETUP_IOPOLL )) +if (!force_nonblock && !(req->flags &REQ_F_IOPOLL )) goto done;/* no retry on NONBLOCK nor RWF_NOWAIT */ if (req->flags &REQ_F_NOWAIT ) goto done; ret =0 ;
@@ -1186,11 +1187,11 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags)/* no retry on NONBLOCK nor RWF_NOWAIT */ if (ret2 == -EAGAIN && (req->flags &REQ_F_NOWAIT )) goto done;if (!force_nonblock || ret2 != -EAGAIN ) {/* IOPOLL retry should happen for io-wq threads */ -if (ret2 == -EAGAIN && (req->ctx->flags &IORING_SETUP_IOPOLL )) +if (ret2 == -EAGAIN && (req->flags &REQ_F_IOPOLL )) goto ret_eagain;if (ret2 != req->cqe.res && ret2 >=0 &&need_complete_io (req)) {trace_io_uring_short_write (req->ctx, kiocb->ki_pos - ret2, req->cqe.res, ret2);
io_uring/uring_cmd.c +3 -2
@@ -108,11 +108,11 @@ void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd,* Doing cancelations on IOPOLL requests are not supported. Both * because they can't get canceled in the block stack, but also * because iopoll completion data overlaps with the hash_node used * for tracking. */ -if (ctx->flags &IORING_SETUP_IOPOLL ) +if (req->flags &REQ_F_IOPOLL )return ;if (!(cmd->flags &IORING_URING_CMD_CANCELABLE )) { cmd->flags |=IORING_URING_CMD_CANCELABLE ;io_ring_submit_lock (ctx, issue_flags);
@@ -165,11 +165,11 @@ void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2,if (req->ctx->flags &IORING_SETUP_CQE_MIXED ) req->cqe.flags |=IORING_CQE_F_32 ;io_req_set_cqe32_extra (req, res2,0 ); }io_req_uring_cleanup (req, issue_flags); -if (req->ctx->flags &IORING_SETUP_IOPOLL ) { +if (req->flags &REQ_F_IOPOLL ) {/* order with io_iopoll_req_issued() checking ->iopoll_complete */ smp_store_release (&req->iopoll_completed,1 ); }else if (issue_flags &IO_URING_F_COMPLETE_DEFER ) {if (WARN_ON_ONCE (issue_flags &IO_URING_F_UNLOCKED ))return ;
@@ -258,10 +258,11 @@ int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags)if (io_is_compat (ctx)) issue_flags |=IO_URING_F_COMPAT ;if (ctx->flags &IORING_SETUP_IOPOLL ) {if (!file->f_op->uring_cmd_iopoll)return -EOPNOTSUPP ; + req->flags |=REQ_F_IOPOLL ; issue_flags |=IO_URING_F_IOPOLL ; req->iopoll_completed =0 ;if (ctx->flags &IORING_SETUP_HYBRID_IOPOLL ) {/* make sure every req only blocks once */ req->flags &= ~REQ_F_IOPOLL_STATE ;
Signature · 1 line
2.45.2
PATCH v5 2/5 io_uring: remove iopoll_queue from struct io_issue_def
In reply to Caleb Sander MateosThe opcode iopoll_queue flag is now redundant with REQ_F_IOPOLL. Only io_{read,write}{,_fixed}() and io_uring_cmd() set the REQ_F_IOPOLL flag, and the opcodes with these ->issue() implementations are precisely the ones that set iopoll_queue. So don't bother checking the iopoll_queue flag in io_issue_sqe(). Remove the unused flag from struct io_issue_def.
--- io_uring/io_uring.c | 3 +-- io_uring/opdef.c | 10 ---------- io_uring/opdef.h | 2 -- 3 files changed, 1 insertion(+), 14 deletions(-)
io_uring/io_uring.c +1 -2
@@ -1415,12 +1415,11 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) }if (ret ==IOU_ISSUE_SKIP_COMPLETE ) { ret =0 ; -/* If the op doesn't have a file, we're not polling for it */ -if ((req->flags &REQ_F_IOPOLL ) && def->iopoll_queue) +if (req->flags &REQ_F_IOPOLL )io_iopoll_req_issued (req, issue_flags); }return ret; }
io_uring/opdef.c +0 -10
@@ -65,11 +65,10 @@ const struct io_issue_def io_issue_defs[] = { .buffer_select = 1, .plug = 1, .audit_skip = 1, .ioprio = 1, .iopoll = 1, - .iopoll_queue = 1, .vectored = 1, .async_size = sizeof(struct io_async_rw), .prep = io_prep_readv, .issue = io_read, },
@@ -80,11 +79,10 @@ const struct io_issue_def io_issue_defs[] = { .pollout = 1, .plug = 1, .audit_skip = 1, .ioprio = 1, .iopoll = 1, - .iopoll_queue = 1, .vectored = 1, .async_size = sizeof(struct io_async_rw), .prep = io_prep_writev, .issue = io_write, },
@@ -100,11 +98,10 @@ const struct io_issue_def io_issue_defs[] = { .pollin = 1, .plug = 1, .audit_skip = 1, .ioprio = 1, .iopoll = 1, - .iopoll_queue = 1, .async_size = sizeof(struct io_async_rw), .prep = io_prep_read_fixed, .issue = io_read_fixed, }, [IORING_OP_WRITE_FIXED] = {
@@ -114,11 +111,10 @@ const struct io_issue_def io_issue_defs[] = { .pollout = 1, .plug = 1, .audit_skip = 1, .ioprio = 1, .iopoll = 1, - .iopoll_queue = 1, .async_size = sizeof(struct io_async_rw), .prep = io_prep_write_fixed, .issue = io_write_fixed, }, [IORING_OP_POLL_ADD] = {
@@ -248,11 +244,10 @@ const struct io_issue_def io_issue_defs[] = { .buffer_select = 1, .plug = 1, .audit_skip = 1, .ioprio = 1, .iopoll = 1, - .iopoll_queue = 1, .async_size = sizeof(struct io_async_rw), .prep = io_prep_read, .issue = io_read, }, [IORING_OP_WRITE] = {
@@ -262,11 +257,10 @@ const struct io_issue_def io_issue_defs[] = { .pollout = 1, .plug = 1, .audit_skip = 1, .ioprio = 1, .iopoll = 1, - .iopoll_queue = 1, .async_size = sizeof(struct io_async_rw), .prep = io_prep_write, .issue = io_write, }, [IORING_OP_FADVISE] = {
@@ -421,11 +415,10 @@ const struct io_issue_def io_issue_defs[] = { [IORING_OP_URING_CMD] = { .buffer_select = 1, .needs_file = 1, .plug = 1, .iopoll = 1, - .iopoll_queue = 1, .async_size = sizeof(struct io_async_cmd), .prep = io_uring_cmd_prep, .issue = io_uring_cmd, }, [IORING_OP_SEND_ZC] = {
@@ -554,11 +547,10 @@ const struct io_issue_def io_issue_defs[] = { .pollin = 1, .plug = 1, .audit_skip = 1, .ioprio = 1, .iopoll = 1, - .iopoll_queue = 1, .vectored = 1, .async_size = sizeof(struct io_async_rw), .prep = io_prep_readv_fixed, .issue = io_read, },
@@ -569,11 +561,10 @@ const struct io_issue_def io_issue_defs[] = { .pollout = 1, .plug = 1, .audit_skip = 1, .ioprio = 1, .iopoll = 1, - .iopoll_queue = 1, .vectored = 1, .async_size = sizeof(struct io_async_rw), .prep = io_prep_writev_fixed, .issue = io_write, },
@@ -591,11 +582,10 @@ const struct io_issue_def io_issue_defs[] = { [IORING_OP_URING_CMD128] = { .buffer_select = 1, .needs_file = 1, .plug = 1, .iopoll = 1, - .iopoll_queue = 1, .is_128 = 1, .async_size = sizeof(struct io_async_cmd), .prep = io_uring_cmd_prep, .issue = io_uring_cmd, },
io_uring/opdef.h +0 -2
@@ -23,12 +23,10 @@ struct io_issue_def { unsigned pollin : 1; unsigned pollout : 1; unsigned poll_exclusive : 1;/* skip auditing */ unsigned audit_skip : 1; -/* have to be put into the iopoll list */ - unsigned iopoll_queue : 1;/* vectored opcode, set if 1) vectored, and 2) handler needs to know */ unsigned vectored : 1;/* set to 1 if this opcode uses 128b sqes in a mixed sq */ unsigned is_128 : 1;
Signature · 1 line
2.45.2
PATCH v5 3/5 io_uring: count CQEs in io_iopoll_check()
In reply to Caleb Sander MateosA subsequent commit will allow uring_cmds that don't use iopoll on IORING_SETUP_IOPOLL io_urings. As a result, CQEs can be posted without setting the iopoll_completed flag for a request in iopoll_list or going through task work. For example, a UBLK_U_IO_FETCH_IO_CMDS command could call io_uring_mshot_cmd_post_cqe() to directly post a CQE. The io_iopoll_check() loop currently only counts completions posted in io_do_iopoll() when determining whether the min_events threshold has been met. It also exits early if there are any existing CQEs before polling, or if any CQEs are posted while running task work. CQEs posted via io_uring_mshot_cmd_post_cqe() or other mechanisms won't be counted against min_events.
Explicitly check the available CQEs in each io_iopoll_check() loop iteration to account for CQEs posted in any fashion.
--- io_uring/io_uring.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
io_uring/io_uring.c +2 -7
@@ -1184,11 +1184,10 @@ __cold void io_iopoll_try_reap_events(struct io_ring_ctx *ctx)io_move_task_work_from_local (ctx); }static int io_iopoll_check (struct io_ring_ctx *ctx,unsigned int min_events) { -unsigned int nr_events =0 ;unsigned long check_cq; min_events =min (min_events, ctx->cq_entries);lockdep_assert_held (&ctx->uring_lock);
@@ -1227,34 +1226,30 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events)* the poll to the issued list. Otherwise we can spin here * forever, while the workqueue is stuck trying to acquire the * very same mutex. */ if (list_empty (&ctx->iopoll_list) ||io_task_work_pending (ctx)) { -u32 tail = ctx->cached_cq_tail; - (void )io_run_local_work_locked (ctx, min_events);if (task_work_pending (current) ||list_empty (&ctx->iopoll_list)) {mutex_unlock (&ctx->uring_lock);io_run_task_work ();mutex_lock (&ctx->uring_lock); }/* some requests don't go through iopoll_list */ -if (tail != ctx->cached_cq_tail ||list_empty (&ctx->iopoll_list)) +if (list_empty (&ctx->iopoll_list))break ; } ret =io_do_iopoll (ctx, !min_events);if (unlikely (ret <0 ))return ret;if (task_sigpending (current))return -EINTR ;if (need_resched ())break ; - - nr_events += ret; - }while (nr_events < min_events); + }while (io_cqring_events (ctx) < min_events);return 0 ; }void io_req_task_complete (struct io_tw_req tw_req,io_tw_token_t tw)
Signature · 1 line
2.45.2
Re: [PATCH v5 3/5] io_uring: count CQEs in io_iopoll_check()
Quoted 58 lines · Caleb Sander Mateos
A subsequent commit will allow
uring_cmdsthat don't use iopoll onIORING_SETUP_IOPOLLio_urings. As a result, CQEs can be posted without setting theiopoll_completedflag for a request iniopoll_listor going through task work. For example, aUBLK_U_IO_FETCH_IO_CMDScommand could callio_uring_mshot_cmd_post_cqe()to directly post a CQE. Theio_iopoll_check()loop currently only counts completions posted inio_do_iopoll()when determining whether themin_eventsthreshold has been met. It also exits early if there are any existing CQEs before polling, or if any CQEs are posted while running task work. CQEs posted viaio_uring_mshot_cmd_post_cqe()or other mechanisms won't be counted againstmin_events.Explicitly check the available CQEs in each
io_iopoll_check()loop iteration to account for CQEs posted in any fashion.Signed-off-by Caleb Sander Mateos <[email protected]>--- io_uring/io_uring.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)io_uring/io_uring.c +1
-4@@ -1184,11 +1184,10 @@ __cold void io_iopoll_try_reap_events(struct io_ring_ctx *ctx)io_move_task_work_from_local (ctx); }static int io_iopoll_check (struct io_ring_ctx *ctx,unsigned int min_events) { -unsigned int nr_events =0 ;unsigned long check_cq; min_events =min (min_events, ctx->cq_entries);lockdep_assert_held (&ctx->uring_lock);@@ -1227,34 +1226,30 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events)* the poll to the issued list. Otherwise we can spin here * forever, while the workqueue is stuck trying to acquire the * very same mutex. */ if (list_empty (&ctx->iopoll_list) ||io_task_work_pending (ctx)) { -u32 tail = ctx->cached_cq_tail; - (void )io_run_local_work_locked (ctx, min_events);if (task_work_pending (current) ||list_empty (&ctx->iopoll_list)) {mutex_unlock (&ctx->uring_lock);io_run_task_work ();mutex_lock (&ctx->uring_lock); }/* some requests don't go through iopoll_list */ -if (tail != ctx->cached_cq_tail ||list_empty (&ctx->iopoll_list)) +if (list_empty (&ctx->iopoll_list))break ; } ret =io_do_iopoll (ctx, !min_events);if (unlikely (ret <0 ))
io_uring/io_uring.c static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) · line 1250
Quoted 10 lines
io_uring/io_uring.c +1
-3return ret;if (task_sigpending (current))return -EINTR ;if (need_resched ())break ; - - nr_events += ret; - }while (nr_events < min_events); + }while (io_cqring_events (ctx) < min_events);
Before entering the loop, if io_cqring_events() finds any queued CQE, io_iopoll_check() returns immediately without polling.
If the queued CQE is originated from non-iopoll uring_cmd, iopoll request will not be polled, may this be one issue?
Thanks,
Ming
Re: [PATCH v5 3/5] io_uring: count CQEs in io_iopoll_check()
Quoted 74 lines · Ming Lei
Quoted 58 lines · Caleb Sander Mateos
A subsequent commit will allow
uring_cmdsthat don't use iopoll onIORING_SETUP_IOPOLLio_urings. As a result, CQEs can be posted without setting theiopoll_completedflag for a request iniopoll_listor going through task work. For example, aUBLK_U_IO_FETCH_IO_CMDScommand could callio_uring_mshot_cmd_post_cqe()to directly post a CQE. Theio_iopoll_check()loop currently only counts completions posted inio_do_iopoll()when determining whether themin_eventsthreshold has been met. It also exits early if there are any existing CQEs before polling, or if any CQEs are posted while running task work. CQEs posted viaio_uring_mshot_cmd_post_cqe()or other mechanisms won't be counted againstmin_events.Explicitly check the available CQEs in each
io_iopoll_check()loop iteration to account for CQEs posted in any fashion.Signed-off-by Caleb Sander Mateos <[email protected]>--- io_uring/io_uring.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)io_uring/io_uring.c +1
-4@@ -1184,11 +1184,10 @@ __cold void io_iopoll_try_reap_events(struct io_ring_ctx *ctx)io_move_task_work_from_local (ctx); }static int io_iopoll_check (struct io_ring_ctx *ctx,unsigned int min_events) { -unsigned int nr_events =0 ;unsigned long check_cq; min_events =min (min_events, ctx->cq_entries);lockdep_assert_held (&ctx->uring_lock);@@ -1227,34 +1226,30 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events)* the poll to the issued list. Otherwise we can spin here * forever, while the workqueue is stuck trying to acquire the * very same mutex. */ if (list_empty (&ctx->iopoll_list) ||io_task_work_pending (ctx)) { -u32 tail = ctx->cached_cq_tail; - (void )io_run_local_work_locked (ctx, min_events);if (task_work_pending (current) ||list_empty (&ctx->iopoll_list)) {mutex_unlock (&ctx->uring_lock);io_run_task_work ();mutex_lock (&ctx->uring_lock); }/* some requests don't go through iopoll_list */ -if (tail != ctx->cached_cq_tail ||list_empty (&ctx->iopoll_list)) +if (list_empty (&ctx->iopoll_list))break ; } ret =io_do_iopoll (ctx, !min_events);if (unlikely (ret <0 ))io_uring/io_uring.c static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) · line 1250
Quoted 10 lines
io_uring/io_uring.c +1
-3return ret;if (task_sigpending (current))return -EINTR ;if (need_resched ())break ; - - nr_events += ret; - }while (nr_events < min_events); + }while (io_cqring_events (ctx) < min_events);Before entering the loop, if
io_cqring_events()finds any queued CQE,io_iopoll_check()returns immediately without polling.
Quoted 2 lines
If the queued CQE is originated from non-iopoll
uring_cmd, iopoll request will not be polled, may this be one issue?
I also noticed that logic and thought it seemed odd. I would think we'd always want to wait for min_events CQEs (and iopoll once even if min_events is 0). Looks like Jens added the early return in commit a3a0e43fd770 ("io_uring: don't enter poll loop if we have CQEs pending"), perhaps he can shed some light on it?
Thanks,
Caleb
Jens Axboe
maintainerRe: [PATCH v5 3/5] io_uring: count CQEs in io_iopoll_check()
Quoted 75 lines · Caleb Sander Mateos
Quoted 74 lines · On Wed, Mar 4, 2026 at 2:33?AM Ming Lei <[email protected]> wrote:
Quoted 58 lines · Caleb Sander Mateos
A subsequent commit will allow
uring_cmdsthat don't use iopoll onIORING_SETUP_IOPOLLio_urings. As a result, CQEs can be posted without setting theiopoll_completedflag for a request iniopoll_listor going through task work. For example, aUBLK_U_IO_FETCH_IO_CMDScommand could callio_uring_mshot_cmd_post_cqe()to directly post a CQE. Theio_iopoll_check()loop currently only counts completions posted inio_do_iopoll()when determining whether themin_eventsthreshold has been met. It also exits early if there are any existing CQEs before polling, or if any CQEs are posted while running task work. CQEs posted viaio_uring_mshot_cmd_post_cqe()or other mechanisms won't be counted againstmin_events.Explicitly check the available CQEs in each
io_iopoll_check()loop iteration to account for CQEs posted in any fashion.Signed-off-by Caleb Sander Mateos <[email protected]>--- io_uring/io_uring.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)io_uring/io_uring.c +1
-4@@ -1184,11 +1184,10 @@ __cold void io_iopoll_try_reap_events(struct io_ring_ctx *ctx)io_move_task_work_from_local (ctx); }static int io_iopoll_check (struct io_ring_ctx *ctx,unsigned int min_events) { -unsigned int nr_events =0 ;unsigned long check_cq; min_events =min (min_events, ctx->cq_entries);lockdep_assert_held (&ctx->uring_lock);@@ -1227,34 +1226,30 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events)* the poll to the issued list. Otherwise we can spin here * forever, while the workqueue is stuck trying to acquire the * very same mutex. */ if (list_empty (&ctx->iopoll_list) ||io_task_work_pending (ctx)) { -u32 tail = ctx->cached_cq_tail; - (void )io_run_local_work_locked (ctx, min_events);if (task_work_pending (current) ||list_empty (&ctx->iopoll_list)) {mutex_unlock (&ctx->uring_lock);io_run_task_work ();mutex_lock (&ctx->uring_lock); }/* some requests don't go through iopoll_list */ -if (tail != ctx->cached_cq_tail ||list_empty (&ctx->iopoll_list)) +if (list_empty (&ctx->iopoll_list))break ; } ret =io_do_iopoll (ctx, !min_events);if (unlikely (ret <0 ))io_uring/io_uring.c static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) · line 1250
Quoted 10 lines
io_uring/io_uring.c +1
-3return ret;if (task_sigpending (current))return -EINTR ;if (need_resched ())break ; - - nr_events += ret; - }while (nr_events < min_events); + }while (io_cqring_events (ctx) < min_events);Before entering the loop, if
io_cqring_events()finds any queued CQE,io_iopoll_check()returns immediately without polling.
Quoted 8 lines
Quoted 2 lines
If the queued CQE is originated from non-iopoll
uring_cmd, iopoll request will not be polled, may this be one issue?I also noticed that logic and thought it seemed odd. I would think we'd always want to wait for
min_eventsCQEs (and iopoll once even ifmin_eventsis 0). Looks like Jens added the early return in commita3a0e43fd770("io_uring: don't enter poll loop if we have CQEs pending"), perhaps he can shed some light on it?
I don't recall the bug in question, it's been a while... But it always makes sense to return events that are ready, and skip polling. It should only be done if there are no ready events to reap.
Signature · 1 line
Jens Axboe
Re: [PATCH v5 3/5] io_uring: count CQEs in io_iopoll_check()
Quoted 86 lines · Jens Axboe
Quoted 75 lines · Caleb Sander Mateos
Quoted 74 lines · On Wed, Mar 4, 2026 at 2:33?AM Ming Lei <[email protected]> wrote:
Quoted 58 lines · Caleb Sander Mateos
A subsequent commit will allow
uring_cmdsthat don't use iopoll onIORING_SETUP_IOPOLLio_urings. As a result, CQEs can be posted without setting theiopoll_completedflag for a request iniopoll_listor going through task work. For example, aUBLK_U_IO_FETCH_IO_CMDScommand could callio_uring_mshot_cmd_post_cqe()to directly post a CQE. Theio_iopoll_check()loop currently only counts completions posted inio_do_iopoll()when determining whether themin_eventsthreshold has been met. It also exits early if there are any existing CQEs before polling, or if any CQEs are posted while running task work. CQEs posted viaio_uring_mshot_cmd_post_cqe()or other mechanisms won't be counted againstmin_events.Explicitly check the available CQEs in each
io_iopoll_check()loop iteration to account for CQEs posted in any fashion.Signed-off-by Caleb Sander Mateos <[email protected]>--- io_uring/io_uring.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)io_uring/io_uring.c +1
-4@@ -1184,11 +1184,10 @@ __cold void io_iopoll_try_reap_events(struct io_ring_ctx *ctx)io_move_task_work_from_local (ctx); }static int io_iopoll_check (struct io_ring_ctx *ctx,unsigned int min_events) { -unsigned int nr_events =0 ;unsigned long check_cq; min_events =min (min_events, ctx->cq_entries);lockdep_assert_held (&ctx->uring_lock);@@ -1227,34 +1226,30 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events)* the poll to the issued list. Otherwise we can spin here * forever, while the workqueue is stuck trying to acquire the * very same mutex. */ if (list_empty (&ctx->iopoll_list) ||io_task_work_pending (ctx)) { -u32 tail = ctx->cached_cq_tail; - (void )io_run_local_work_locked (ctx, min_events);if (task_work_pending (current) ||list_empty (&ctx->iopoll_list)) {mutex_unlock (&ctx->uring_lock);io_run_task_work ();mutex_lock (&ctx->uring_lock); }/* some requests don't go through iopoll_list */ -if (tail != ctx->cached_cq_tail ||list_empty (&ctx->iopoll_list)) +if (list_empty (&ctx->iopoll_list))break ; } ret =io_do_iopoll (ctx, !min_events);if (unlikely (ret <0 ))io_uring/io_uring.c static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) · line 1250
Quoted 10 lines
io_uring/io_uring.c +1
-3return ret;if (task_sigpending (current))return -EINTR ;if (need_resched ())break ; - - nr_events += ret; - }while (nr_events < min_events); + }while (io_cqring_events (ctx) < min_events);Before entering the loop, if
io_cqring_events()finds any queued CQE,io_iopoll_check()returns immediately without polling.Quoted 8 lines
Quoted 2 lines
If the queued CQE is originated from non-iopoll
uring_cmd, iopoll request will not be polled, may this be one issue?I also noticed that logic and thought it seemed odd. I would think we'd always want to wait for
min_eventsCQEs (and iopoll once even ifmin_eventsis 0). Looks like Jens added the early return in commita3a0e43fd770("io_uring: don't enter poll loop if we have CQEs pending"), perhaps he can shed some light on it?
Quoted 3 lines
I don't recall the bug in question, it's been a while... But it always makes sense to return events that are ready, and skip polling. It should only be done if there are no ready events to reap.
Ming, are you okay with preserving that behavior in this patch then? I guess there's a potential fairness concern where REQ_F_IOPOLL requests may not be polled for some time if non-REQ_F_IOPOLL requests continue to frequently post CQEs.
Jens, any thoughts on this series? Is it ready to merge?
Thanks,
Caleb
Re: [PATCH v5 3/5] io_uring: count CQEs in io_iopoll_check()
Quoted 87 lines · Caleb Sander Mateos
Quoted 86 lines · Jens Axboe
Quoted 83 lines · Caleb Sander Mateos
Quoted 74 lines · On Wed, Mar 4, 2026 at 2:33?AM Ming Lei <[email protected]> wrote:
Quoted 58 lines · Caleb Sander Mateos
A subsequent commit will allow
uring_cmdsthat don't use iopoll onIORING_SETUP_IOPOLLio_urings. As a result, CQEs can be posted without setting theiopoll_completedflag for a request iniopoll_listor going through task work. For example, aUBLK_U_IO_FETCH_IO_CMDScommand could callio_uring_mshot_cmd_post_cqe()to directly post a CQE. Theio_iopoll_check()loop currently only counts completions posted inio_do_iopoll()when determining whether themin_eventsthreshold has been met. It also exits early if there are any existing CQEs before polling, or if any CQEs are posted while running task work. CQEs posted viaio_uring_mshot_cmd_post_cqe()or other mechanisms won't be counted againstmin_events.Explicitly check the available CQEs in each
io_iopoll_check()loop iteration to account for CQEs posted in any fashion.Signed-off-by Caleb Sander Mateos <[email protected]>--- io_uring/io_uring.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)io_uring/io_uring.c +1
-4@@ -1184,11 +1184,10 @@ __cold void io_iopoll_try_reap_events(struct io_ring_ctx *ctx)io_move_task_work_from_local (ctx); }static int io_iopoll_check (struct io_ring_ctx *ctx,unsigned int min_events) { -unsigned int nr_events =0 ;unsigned long check_cq; min_events =min (min_events, ctx->cq_entries);lockdep_assert_held (&ctx->uring_lock);@@ -1227,34 +1226,30 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events)* the poll to the issued list. Otherwise we can spin here * forever, while the workqueue is stuck trying to acquire the * very same mutex. */ if (list_empty (&ctx->iopoll_list) ||io_task_work_pending (ctx)) { -u32 tail = ctx->cached_cq_tail; - (void )io_run_local_work_locked (ctx, min_events);if (task_work_pending (current) ||list_empty (&ctx->iopoll_list)) {mutex_unlock (&ctx->uring_lock);io_run_task_work ();mutex_lock (&ctx->uring_lock); }/* some requests don't go through iopoll_list */ -if (tail != ctx->cached_cq_tail ||list_empty (&ctx->iopoll_list)) +if (list_empty (&ctx->iopoll_list))break ; } ret =io_do_iopoll (ctx, !min_events);if (unlikely (ret <0 ))io_uring/io_uring.c static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) · line 1250
Quoted 10 lines
io_uring/io_uring.c +1
-3return ret;if (task_sigpending (current))return -EINTR ;if (need_resched ())break ; - - nr_events += ret; - }while (nr_events < min_events); + }while (io_cqring_events (ctx) < min_events);Before entering the loop, if
io_cqring_events()finds any queued CQE,io_iopoll_check()returns immediately without polling.Quoted 2 lines
If the queued CQE is originated from non-iopoll
uring_cmd, iopoll request will not be polled, may this be one issue?I also noticed that logic and thought it seemed odd. I would think we'd always want to wait for
min_eventsCQEs (and iopoll once even ifmin_eventsis 0). Looks like Jens added the early return in commita3a0e43fd770("io_uring: don't enter poll loop if we have CQEs pending"), perhaps he can shed some light on it?
Quoted 8 lines
Quoted 3 lines
I don't recall the bug in question, it's been a while... But it always makes sense to return events that are ready, and skip polling. It should only be done if there are no ready events to reap.
Ming, are you okay with preserving that behavior in this patch then? I guess there's a potential fairness concern where
REQ_F_IOPOLLrequests may not be polled for some time if non-REQ_F_IOPOLL requests continue to frequently post CQEs.
IMO, the fairness may not a big deal given userspace should keep polling if the iopoll IO isn't done.
But forget to mention, if non-iopoll CQE is posted and ->cq_flush becomes true, io_submit_flush_completions() may not get chance to run in case of the early return.
Maybe something like below is needed:
@@ -1556,8 +1556,10 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events)
* If we do, we can potentially be spinning for commands that
* already triggered a CQE (eg in error).
*/
- if (io_cqring_events(ctx))
+ if (io_cqring_events(ctx)) {
+ io_submit_flush_completions(ctx);
return 0;+ }
Thanks,
Ming
Jens Axboe
maintainerRe: [PATCH v5 3/5] io_uring: count CQEs in io_iopoll_check()
Quoted 92 lines · Ming Lei
Quoted 91 lines · Caleb Sander Mateos
Quoted 89 lines · On Wed, Mar 4, 2026 at 8:29?AM Jens Axboe <[email protected]> wrote:
Quoted 75 lines · Caleb Sander Mateos
Quoted 74 lines · On Wed, Mar 4, 2026 at 2:33?AM Ming Lei <[email protected]> wrote:
Quoted 58 lines · Caleb Sander Mateos
A subsequent commit will allow
uring_cmdsthat don't use iopoll onIORING_SETUP_IOPOLLio_urings. As a result, CQEs can be posted without setting theiopoll_completedflag for a request iniopoll_listor going through task work. For example, aUBLK_U_IO_FETCH_IO_CMDScommand could callio_uring_mshot_cmd_post_cqe()to directly post a CQE. Theio_iopoll_check()loop currently only counts completions posted inio_do_iopoll()when determining whether themin_eventsthreshold has been met. It also exits early if there are any existing CQEs before polling, or if any CQEs are posted while running task work. CQEs posted viaio_uring_mshot_cmd_post_cqe()or other mechanisms won't be counted againstmin_events.Explicitly check the available CQEs in each
io_iopoll_check()loop iteration to account for CQEs posted in any fashion.Signed-off-by Caleb Sander Mateos <[email protected]>--- io_uring/io_uring.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)io_uring/io_uring.c +1
-4@@ -1184,11 +1184,10 @@ __cold void io_iopoll_try_reap_events(struct io_ring_ctx *ctx)io_move_task_work_from_local (ctx); }static int io_iopoll_check (struct io_ring_ctx *ctx,unsigned int min_events) { -unsigned int nr_events =0 ;unsigned long check_cq; min_events =min (min_events, ctx->cq_entries);lockdep_assert_held (&ctx->uring_lock);@@ -1227,34 +1226,30 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events)* the poll to the issued list. Otherwise we can spin here * forever, while the workqueue is stuck trying to acquire the * very same mutex. */ if (list_empty (&ctx->iopoll_list) ||io_task_work_pending (ctx)) { -u32 tail = ctx->cached_cq_tail; - (void )io_run_local_work_locked (ctx, min_events);if (task_work_pending (current) ||list_empty (&ctx->iopoll_list)) {mutex_unlock (&ctx->uring_lock);io_run_task_work ();mutex_lock (&ctx->uring_lock); }/* some requests don't go through iopoll_list */ -if (tail != ctx->cached_cq_tail ||list_empty (&ctx->iopoll_list)) +if (list_empty (&ctx->iopoll_list))break ; } ret =io_do_iopoll (ctx, !min_events);if (unlikely (ret <0 ))io_uring/io_uring.c static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) · line 1250
Quoted 10 lines
io_uring/io_uring.c +1
-3return ret;if (task_sigpending (current))return -EINTR ;if (need_resched ())break ; - - nr_events += ret; - }while (nr_events < min_events); + }while (io_cqring_events (ctx) < min_events);Before entering the loop, if
io_cqring_events()finds any queued CQE,io_iopoll_check()returns immediately without polling.Quoted 8 lines
Quoted 2 lines
If the queued CQE is originated from non-iopoll
uring_cmd, iopoll request will not be polled, may this be one issue?I also noticed that logic and thought it seemed odd. I would think we'd always want to wait for
min_eventsCQEs (and iopoll once even ifmin_eventsis 0). Looks like Jens added the early return in commita3a0e43fd770("io_uring: don't enter poll loop if we have CQEs pending"), perhaps he can shed some light on it?I don't recall the bug in question, it's been a while... But it always makes sense to return events that are ready, and skip polling. It should only be done if there are no ready events to reap.
Quoted 7 lines
Quoted 4 lines
Ming, are you okay with preserving that behavior in this patch then? I guess there's a potential fairness concern where
REQ_F_IOPOLLrequests may not be polled for some time if non-REQ_F_IOPOLL requests continue to frequently post CQEs.IMO, the fairness may not a big deal given userspace should keep polling if the iopoll IO isn't done.
I think so.
Quoted 7 lines
But forget to mention, if non-iopoll CQE is posted and ->cq_flush becomes true,
io_submit_flush_completions()may not get chance to run in case of the early return.Maybe something like below is needed:
Review static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) · line 1562
Quoted 8 lines
(quoted diff) +3
-1@@ -1556,8 +1556,10 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) * If we do, we can potentially be spinning for commands that * already triggered a CQE (eg in error). */ - if (io_cqring_events(ctx)) + if (io_cqring_events(ctx)) { + io_submit_flush_completions(ctx); return 0; + }
We should probably experiment with this a bit, I don't think it's a showstopper for merging this.
Caleb, I think we can stage this for 7.1 and see how it goes.
Signature · 1 line
Jens Axboe
Re: [PATCH v5 3/5] io_uring: count CQEs in io_iopoll_check()
Quoted 110 lines · Jens Axboe
Quoted 92 lines · Ming Lei
Quoted 91 lines · Caleb Sander Mateos
Quoted 89 lines · On Wed, Mar 4, 2026 at 8:29?AM Jens Axboe <[email protected]> wrote:
Quoted 75 lines · Caleb Sander Mateos
Quoted 74 lines · On Wed, Mar 4, 2026 at 2:33?AM Ming Lei <[email protected]> wrote:
Quoted 58 lines · Caleb Sander Mateos
A subsequent commit will allow
uring_cmdsthat don't use iopoll onIORING_SETUP_IOPOLLio_urings. As a result, CQEs can be posted without setting theiopoll_completedflag for a request iniopoll_listor going through task work. For example, aUBLK_U_IO_FETCH_IO_CMDScommand could callio_uring_mshot_cmd_post_cqe()to directly post a CQE. Theio_iopoll_check()loop currently only counts completions posted inio_do_iopoll()when determining whether themin_eventsthreshold has been met. It also exits early if there are any existing CQEs before polling, or if any CQEs are posted while running task work. CQEs posted viaio_uring_mshot_cmd_post_cqe()or other mechanisms won't be counted againstmin_events.Explicitly check the available CQEs in each
io_iopoll_check()loop iteration to account for CQEs posted in any fashion.Signed-off-by Caleb Sander Mateos <[email protected]>--- io_uring/io_uring.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)io_uring/io_uring.c +1
-4@@ -1184,11 +1184,10 @@ __cold void io_iopoll_try_reap_events(struct io_ring_ctx *ctx)io_move_task_work_from_local (ctx); }static int io_iopoll_check (struct io_ring_ctx *ctx,unsigned int min_events) { -unsigned int nr_events =0 ;unsigned long check_cq; min_events =min (min_events, ctx->cq_entries);lockdep_assert_held (&ctx->uring_lock);@@ -1227,34 +1226,30 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events)* the poll to the issued list. Otherwise we can spin here * forever, while the workqueue is stuck trying to acquire the * very same mutex. */ if (list_empty (&ctx->iopoll_list) ||io_task_work_pending (ctx)) { -u32 tail = ctx->cached_cq_tail; - (void )io_run_local_work_locked (ctx, min_events);if (task_work_pending (current) ||list_empty (&ctx->iopoll_list)) {mutex_unlock (&ctx->uring_lock);io_run_task_work ();mutex_lock (&ctx->uring_lock); }/* some requests don't go through iopoll_list */ -if (tail != ctx->cached_cq_tail ||list_empty (&ctx->iopoll_list)) +if (list_empty (&ctx->iopoll_list))break ; } ret =io_do_iopoll (ctx, !min_events);if (unlikely (ret <0 ))io_uring/io_uring.c static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) · line 1250
Quoted 10 lines
io_uring/io_uring.c +1
-3return ret;if (task_sigpending (current))return -EINTR ;if (need_resched ())break ; - - nr_events += ret; - }while (nr_events < min_events); + }while (io_cqring_events (ctx) < min_events);Before entering the loop, if
io_cqring_events()finds any queued CQE,io_iopoll_check()returns immediately without polling.Quoted 8 lines
Quoted 2 lines
If the queued CQE is originated from non-iopoll
uring_cmd, iopoll request will not be polled, may this be one issue?I also noticed that logic and thought it seemed odd. I would think we'd always want to wait for
min_eventsCQEs (and iopoll once even ifmin_eventsis 0). Looks like Jens added the early return in commita3a0e43fd770("io_uring: don't enter poll loop if we have CQEs pending"), perhaps he can shed some light on it?I don't recall the bug in question, it's been a while... But it always makes sense to return events that are ready, and skip polling. It should only be done if there are no ready events to reap.
Quoted 7 lines
Quoted 4 lines
Ming, are you okay with preserving that behavior in this patch then? I guess there's a potential fairness concern where
REQ_F_IOPOLLrequests may not be polled for some time if non-REQ_F_IOPOLL requests continue to frequently post CQEs.IMO, the fairness may not a big deal given userspace should keep polling if the iopoll IO isn't done.
I think so.
Quoted 6 lines
But forget to mention, if non-iopoll CQE is posted and ->cq_flush becomes true,
io_submit_flush_completions()may not get chance to run in case of the early return.Maybe something like below is needed:
Quoted 12 lines
Review static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) · line 1562
Quoted 9 lines
(quoted diff) +3
-1@@ -1556,8 +1556,10 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) * If we do, we can potentially be spinning for commands that * already triggered a CQE (eg in error). */ - if (io_cqring_events(ctx)) + if (io_cqring_events(ctx)) { + io_submit_flush_completions(ctx); return 0; + }We should probably experiment with this a bit, I don't think it's a showstopper for merging this.
I don't think it's an issue currently. Any path that posts CQEs (synchronously during submit, from task work, or iopoll) already calls io_submit_flush_completions()/__io_submit_flush_completions(). In other words, there's no way to reach this io_cqring_events() branch without the CQEs having already been flushed. If a new path were added in the future that could post CQEs outside of those contexts, it would of course need to ensure io_submit_flush_completions() is called after posting the CQEs.
Quoted 2 lines
Caleb, I think we can stage this for 7.1 and see how it goes.
Sounds good, thank you.
--Caleb
PATCH v5 4/5 io_uring/uring_cmd: allow non-iopoll cmds with IORING_SETUP_IOPOLL
In reply to Caleb Sander MateosCurrently, creating an io_uring with IORING_SETUP_IOPOLL requires all requests issued to it to support iopoll. This prevents, for example, using ublk zero-copy together with IORING_SETUP_IOPOLL, as ublk zero-copy buffer registrations are performed using a uring_cmd. There's no technical reason why these non-iopoll uring_cmds can't be supported. They will either complete synchronously or via an external mechanism that calls io_uring_cmd_done(), io_uring_cmd_post_mshot_cqe32(), or io_uring_mshot_cmd_post_cqe(), so they don't need to be polled.
Allow uring_cmd requests to be issued to IORING_SETUP_IOPOLL io_urings even if their files don't implement ->uring_cmd_iopoll(). For these uring_cmd requests, skip initializing struct io_kiocb's iopoll fields, don't set REQ_F_IOPOLL, and don't set IO_URING_F_IOPOLL in issue_flags.
--- io_uring/uring_cmd.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
io_uring/uring_cmd.c +1 -3
@@ -255,13 +255,11 @@ int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags) issue_flags |=IO_URING_F_SQE128 ;if (ctx->flags & (IORING_SETUP_CQE32 |IORING_SETUP_CQE_MIXED )) issue_flags |=IO_URING_F_CQE32 ;if (io_is_compat (ctx)) issue_flags |=IO_URING_F_COMPAT ; -if (ctx->flags &IORING_SETUP_IOPOLL ) { -if (!file->f_op->uring_cmd_iopoll) -return -EOPNOTSUPP ; +if (ctx->flags &IORING_SETUP_IOPOLL && file->f_op->uring_cmd_iopoll) { req->flags |=REQ_F_IOPOLL ; issue_flags |=IO_URING_F_IOPOLL ; req->iopoll_completed =0 ;if (ctx->flags &IORING_SETUP_HYBRID_IOPOLL ) {/* make sure every req only blocks once */
Signature · 1 line
2.45.2
PATCH v5 5/5 nvme: remove nvme_dev_uring_cmd() IO_URING_F_IOPOLL check
In reply to Caleb Sander Mateosnvme_dev_uring_cmd() is part of struct file_operations nvme_dev_fops, which doesn't implement ->uring_cmd_iopoll(). So it won't be called with issue_flags that include IO_URING_F_IOPOLL. Drop the unnecessary IO_URING_F_IOPOLL check in nvme_dev_uring_cmd().
--- drivers/nvme/host/ioctl.c | 4 ---- 1 file changed, 4 deletions(-)
drivers/nvme/host/ioctl.c +0 -4
@@ -784,14 +784,10 @@ int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd,int nvme_dev_uring_cmd (struct io_uring_cmd *ioucmd,unsigned int issue_flags) {struct nvme_ctrl *ctrl = ioucmd->file->private_data;int ret; -/* IOPOLL not supported yet */ -if (issue_flags &IO_URING_F_IOPOLL ) -return -EOPNOTSUPP ; - ret =nvme_uring_cmd_checks (issue_flags);if (ret)return ret;switch (ioucmd->cmd_op) {
Signature · 1 line
2.45.2
Quoted 10 lines · Caleb Sander Mateos
Currently, creating an
io_uringwithIORING_SETUP_IOPOLLrequires all requests issued to it to support iopoll. This prevents, for example, using ublk zero-copy together withIORING_SETUP_IOPOLL, as ublk zero-copy buffer registrations are performed using auring_cmd. There's no technical reason why these non-iopolluring_cmdscan't be supported. They will either complete synchronously or via an external mechanism that callsio_uring_cmd_done(),io_uring_cmd_post_mshot_cqe32(), orio_uring_mshot_cmd_post_cqe(), so they don't need to be polled.[...]
Applied, thanks!
[1/5] io_uring: add REQ_F_IOPOLL
commit: 9165dc4fa969b64c2d4396ee4e1546a719978dd1
[2/5] io_uring: remove iopoll_queue from struct io_issue_def
commit: 7995be40deb3ab8b5df7bdf0621f33aa546aefa7
[3/5] io_uring: count CQEs in io_iopoll_check()
commit: 3a5e96d47f7ea37fb6adf37882eec1521f8ca75e
[4/5] io_uring/uring_cmd: allow non-iopoll cmds with IORING_SETUP_IOPOLL
commit: 23475637b0c47e5028817c9fd4dabe8f7409ca6c
[5/5] nvme: remove nvme_dev_uring_cmd() IO_URING_F_IOPOLL check
commit: f144dbac4b177cfd026e417ab98da518ff3372cbBest regards,
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 16 lines · Jens Axboe
Quoted 10 lines · Caleb Sander Mateos
Currently, creating an
io_uringwithIORING_SETUP_IOPOLLrequires all requests issued to it to support iopoll. This prevents, for example, using ublk zero-copy together withIORING_SETUP_IOPOLL, as ublk zero-copy buffer registrations are performed using auring_cmd. There's no technical reason why these non-iopolluring_cmdscan't be supported. They will either complete synchronously or via an external mechanism that callsio_uring_cmd_done(),io_uring_cmd_post_mshot_cqe32(), orio_uring_mshot_cmd_post_cqe(), so they don't need to be polled.[...]
Applied, thanks!
[1/5]
io_uring: addREQ_F_IOPOLL
Quoted 9 lines
commit: 9165dc4fa969b64c2d4396ee4e1546a719978dd1
[2/5] io_uring: remove iopoll_queue from struct io_issue_def
commit: 7995be40deb3ab8b5df7bdf0621f33aa546aefa7
[3/5] io_uring: count CQEs in io_iopoll_check()
commit: 3a5e96d47f7ea37fb6adf37882eec1521f8ca75e
[4/5] io_uring/uring_cmd: allow non-iopoll cmds with IORING_SETUP_IOPOLL
commit: 23475637b0c47e5028817c9fd4dabe8f7409ca6c
[5/5] nvme: remove nvme_dev_uring_cmd() IO_URING_F_IOPOLL check
commit: f144dbac4b177cfd026e417ab98da518ff3372cbCaleb, want to send the liburing tests and documentation updates too?
Signature · 1 line
Jens Axboe
Quoted 28 lines · Jens Axboe
Quoted 16 lines · Jens Axboe
Quoted 10 lines · Caleb Sander Mateos
Currently, creating an
io_uringwithIORING_SETUP_IOPOLLrequires all requests issued to it to support iopoll. This prevents, for example, using ublk zero-copy together withIORING_SETUP_IOPOLL, as ublk zero-copy buffer registrations are performed using auring_cmd. There's no technical reason why these non-iopolluring_cmdscan't be supported. They will either complete synchronously or via an external mechanism that callsio_uring_cmd_done(),io_uring_cmd_post_mshot_cqe32(), orio_uring_mshot_cmd_post_cqe(), so they don't need to be polled.[...]
Applied, thanks!
[1/5]
io_uring: addREQ_F_IOPOLLQuoted 9 lines
commit: 9165dc4fa969b64c2d4396ee4e1546a719978dd1 [2/5] io_uring: remove iopoll_queue from struct io_issue_def commit: 7995be40deb3ab8b5df7bdf0621f33aa546aefa7 [3/5] io_uring: count CQEs in io_iopoll_check() commit: 3a5e96d47f7ea37fb6adf37882eec1521f8ca75e [4/5] io_uring/uring_cmd: allow non-iopoll cmds with IORING_SETUP_IOPOLL commit: 23475637b0c47e5028817c9fd4dabe8f7409ca6c [5/5] nvme: remove nvme_dev_uring_cmd() IO_URING_F_IOPOLL check commit: f144dbac4b177cfd026e417ab98da518ff3372cb
Quoted 1 line
Caleb, want to send the liburing tests and documentation updates too?
Sure. What type of file do you recommend using for non-iopoll uring_cmds? Most of them seem to have relatively specific hardware (e.g. blkdev_uring_cmd, nvme_dev_uring_cmd) or kernel configuration (e.g. ublk_ch_uring_cmd, io_mock_cmd) requirements, as well as requiring elevated permissions. Maybe io_uring_cmd_sock would be the most general?
Best,
Caleb
Jens Axboe
maintainerQuoted 29 lines · Caleb Sander Mateos
Quoted 28 lines · Jens Axboe
Quoted 25 lines · Jens Axboe
Quoted 10 lines · Caleb Sander Mateos
Currently, creating an
io_uringwithIORING_SETUP_IOPOLLrequires all requests issued to it to support iopoll. This prevents, for example, using ublk zero-copy together withIORING_SETUP_IOPOLL, as ublk zero-copy buffer registrations are performed using auring_cmd. There's no technical reason why these non-iopolluring_cmdscan't be supported. They will either complete synchronously or via an external mechanism that callsio_uring_cmd_done(),io_uring_cmd_post_mshot_cqe32(), orio_uring_mshot_cmd_post_cqe(), so they don't need to be polled.[...]
Applied, thanks!
[1/5]
io_uring: addREQ_F_IOPOLLcommit: 9165dc4fa969b64c2d4396ee4e1546a719978dd1 [2/5] io_uring: remove iopoll_queue from struct io_issue_def commit: 7995be40deb3ab8b5df7bdf0621f33aa546aefa7 [3/5] io_uring: count CQEs in io_iopoll_check() commit: 3a5e96d47f7ea37fb6adf37882eec1521f8ca75e [4/5] io_uring/uring_cmd: allow non-iopoll cmds with IORING_SETUP_IOPOLL commit: 23475637b0c47e5028817c9fd4dabe8f7409ca6c [5/5] nvme: remove nvme_dev_uring_cmd() IO_URING_F_IOPOLL check commit: f144dbac4b177cfd026e417ab98da518ff3372cb
Quoted 8 lines
Quoted 1 line
Caleb, want to send the liburing tests and documentation updates too?
Sure. What type of file do you recommend using for non-iopoll
uring_cmds? Most of them seem to have relatively specific hardware (e.g.blkdev_uring_cmd,nvme_dev_uring_cmd) or kernel configuration (e.g.ublk_ch_uring_cmd,io_mock_cmd) requirements, as well as requiring elevated permissions. Maybeio_uring_cmd_sockwould be the most general?
Yep I think uring_cmd sock commands would be a good choice. Bonus points if you write the test as such that we can easily plug in future commands we allow with IOPOLL as well, as I would imagine we'd expand which we allow going forward once vetted.
Signature · 1 line
Jens Axboe