io_uring/uring_cmd: allow non-iopoll cmds with IORING_SETUP_IOPOLL
cover io_uring/uring_cmd: allow non-iopoll cmds with IORING_SETUP_IOPOLL
Currently, creating an io_uring with IORING_SETUP_IOPOLL requires all requests issued to it to support iopoll. This prevents, for example, using ublk zero-copy together with IORING_SETUP_IOPOLL, as ublk zero-copy buffer registrations are performed using a uring_cmd. There's no technical reason why these non-iopoll uring_cmds can't be supported. They will either complete synchronously or via an external mechanism that calls io_uring_cmd_done(), so they don't need to be polled.
Allow uring_cmd requests to be issued to IORING_SETUP_IOPOLL io_urings even if their files don't implement ->uring_cmd_iopoll().
The first commit fixes a few bugs where IORING_OP_URING_CMD128 isn't treated as IORING_OP_URING_CMD with iopoll and provided buffers.
The last commit removes an unnecessary IO_URING_F_IOPOLL check in nvme_dev_uring_cmd() as NVMe admin passthru commands can be issued to IORING_SETUP_IOPOLL io_urings now.
Caleb Sander Mateos (3):
io_uring: add IORING_OP_URING_CMD128 to opcode checks io_uring/uring_cmd: allow non-iopoll cmds with IORING_SETUP_IOPOLL
nvme: remove nvme_dev_uring_cmd() IO_URING_F_IOPOLL check
drivers/nvme/host/ioctl.c | 4 ---- io_uring/io_uring.c | 4 +++- io_uring/io_uring.h | 6 ++++++ io_uring/kbuf.c | 2 +- io_uring/rw.c | 4 ++-- io_uring/uring_cmd.c | 11 +++++------ 6 files changed, 17 insertions(+), 14 deletions(-)
Signature · 1 line
2.45.2
PATCH 1/3 io_uring: add IORING_OP_URING_CMD128 to opcode checks
io_should_commit(), io_uring_classic_poll(), and io_do_iopoll() compare struct io_kiocb's opcode against IORING_OP_URING_CMD to implement special treatment for uring_cmds. The recently added opcode IORING_OP_URING_CMD128 is meant to be equivalent to IORING_OP_URING_CMD, so treat it the same way in these functions.
--- io_uring/io_uring.h | 6 ++++++ io_uring/kbuf.c | 2 +- io_uring/rw.c | 4 ++-- 3 files changed, 9 insertions(+), 3 deletions(-)
io_uring/io_uring.h +6 -0
@@ -528,10 +528,16 @@ static inline bool io_file_can_poll(struct io_kiocb *req)return true; }return false; } +static inline bool io_is_uring_cmd (const struct io_kiocb *req) +{ +return req->opcode ==IORING_OP_URING_CMD || + req->opcode ==IORING_OP_URING_CMD128 ; +} +static inline ktime_t io_get_time (struct io_ring_ctx *ctx) {if (ctx->clockid ==CLOCK_MONOTONIC )return ktime_get ();
io_uring/kbuf.c +1 -1
@@ -169,11 +169,11 @@ static bool io_should_commit(struct io_kiocb *req, unsigned int issue_flags)*/ if (issue_flags &IO_URING_F_UNLOCKED )return true;/* uring_cmd commits kbuf upfront, no need to auto-commit */ -if (!io_file_can_poll (req) && req->opcode !=IORING_OP_URING_CMD ) +if (!io_file_can_poll (req) && !io_is_uring_cmd (req))return true;return false; } static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len,
io_uring/rw.c +2 -2
@@ -1252,11 +1252,11 @@ void io_rw_fail(struct io_kiocb *req)static int io_uring_classic_poll (struct io_kiocb *req,struct io_comp_batch *iob,unsigned int poll_flags) {struct file *file = req->file; -if (req->opcode ==IORING_OP_URING_CMD ) { +if (io_is_uring_cmd (req)) {struct io_uring_cmd *ioucmd; ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd);return file->f_op->uring_cmd_iopoll (ioucmd, iob, poll_flags); }else {
@@ -1378,11 +1378,11 @@ int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin)continue ;list_del (&req->iopoll_node);wq_list_add_tail (&req->comp_list, &ctx->submit_state.compl_reqs); nr_events++; req->cqe.flags =io_put_kbuf (req, req->cqe.res,NULL ); -if (req->opcode !=IORING_OP_URING_CMD ) +if (!io_is_uring_cmd (req))io_req_rw_cleanup (req,0 ); }if (nr_events)__io_submit_flush_completions (ctx);return nr_events;
Signature · 1 line
2.45.2
PATCH 2/3 io_uring/uring_cmd: allow non-iopoll cmds with IORING_SETUP_IOPOLL
In reply to Caleb Sander MateosCurrently, creating an io_uring with IORING_SETUP_IOPOLL requires all requests issued to it to support iopoll. This prevents, for example, using ublk zero-copy together with IORING_SETUP_IOPOLL, as ublk zero-copy buffer registrations are performed using a uring_cmd. There's no technical reason why these non-iopoll uring_cmds can't be supported. They will either complete synchronously or via an external mechanism that calls io_uring_cmd_done(), so they don't need to be polled.
Allow uring_cmd requests to be issued to IORING_SETUP_IOPOLL io_urings even if their files don't implement ->uring_cmd_iopoll(). For these uring_cmd requests, skip initializing struct io_kiocb's iopoll fields, don't insert the request into iopoll_list, and take the io_req_complete_defer() or io_req_task_work_add() path in __io_uring_cmd_done() instead of setting the iopoll_completed flag. Also allow io_uring_cmd_mark_cancelable() to be called on these uring_cmds. Assert that io_uring_cmd_mark_cancelable() is only called on non-IORING_SETUP_IOPOLL io_urings or uring_cmds to files that don't implement ->uring_cmd_iopoll().
--- io_uring/io_uring.c | 4 +++- io_uring/uring_cmd.c | 11 +++++------ 2 files changed, 8 insertions(+), 7 deletions(-)
io_uring/io_uring.c +3 -1
@@ -1417,11 +1417,13 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)if (ret ==IOU_ISSUE_SKIP_COMPLETE ) { ret =0 ;/* If the op doesn't have a file, we're not polling for it */ -if ((req->ctx->flags &IORING_SETUP_IOPOLL ) && def->iopoll_queue) +if ((req->ctx->flags &IORING_SETUP_IOPOLL ) && + def->iopoll_queue && (!io_is_uring_cmd (req) || + req->file->f_op->uring_cmd_iopoll))io_iopoll_req_issued (req, issue_flags); }return ret; }
io_uring/uring_cmd.c +5 -6
@@ -108,12 +108,12 @@ void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd,* Doing cancelations on IOPOLL requests are not supported. Both * because they can't get canceled in the block stack, but also * because iopoll completion data overlaps with the hash_node used * for tracking. */ -if (ctx->flags &IORING_SETUP_IOPOLL ) -return ; +WARN_ON_ONCE (ctx->flags &IORING_SETUP_IOPOLL && + req->file->f_op->uring_cmd_iopoll);if (!(cmd->flags &IORING_URING_CMD_CANCELABLE )) { cmd->flags |=IORING_URING_CMD_CANCELABLE ;io_ring_submit_lock (ctx, issue_flags);hlist_add_head (&req->hash_node, &ctx->cancelable_uring_cmd);
@@ -165,11 +165,12 @@ void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2,if (req->ctx->flags &IORING_SETUP_CQE_MIXED ) req->cqe.flags |=IORING_CQE_F_32 ;io_req_set_cqe32_extra (req, res2,0 ); }io_req_uring_cleanup (req, issue_flags); -if (req->ctx->flags &IORING_SETUP_IOPOLL ) { +if (req->ctx->flags &IORING_SETUP_IOPOLL && + req->file->f_op->uring_cmd_iopoll) {/* order with io_iopoll_req_issued() checking ->iopoll_complete */ smp_store_release (&req->iopoll_completed,1 ); }else if (issue_flags &IO_URING_F_COMPLETE_DEFER ) {if (WARN_ON_ONCE (issue_flags &IO_URING_F_UNLOCKED ))return ;
@@ -255,13 +256,11 @@ int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags) issue_flags |=IO_URING_F_SQE128 ;if (ctx->flags & (IORING_SETUP_CQE32 |IORING_SETUP_CQE_MIXED )) issue_flags |=IO_URING_F_CQE32 ;if (io_is_compat (ctx)) issue_flags |=IO_URING_F_COMPAT ; -if (ctx->flags &IORING_SETUP_IOPOLL ) { -if (!file->f_op->uring_cmd_iopoll) -return -EOPNOTSUPP ; +if (ctx->flags &IORING_SETUP_IOPOLL && file->f_op->uring_cmd_iopoll) { issue_flags |=IO_URING_F_IOPOLL ; req->iopoll_completed =0 ;if (ctx->flags &IORING_SETUP_HYBRID_IOPOLL ) {/* make sure every req only blocks once */ req->flags &= ~REQ_F_IOPOLL_STATE ;
Signature · 1 line
2.45.2
Quoted 65 lines · Caleb Sander Mateos
Currently, creating an
io_uringwithIORING_SETUP_IOPOLLrequires all requests issued to it to support iopoll. This prevents, for example, using ublk zero-copy together withIORING_SETUP_IOPOLL, as ublk zero-copy buffer registrations are performed using auring_cmd. There's no technical reason why these non-iopolluring_cmdscan't be supported. They will either complete synchronously or via an external mechanism that callsio_uring_cmd_done(), so they don't need to be polled.Allow
uring_cmdrequests to be issued toIORING_SETUP_IOPOLLio_uringseven if their files don't implement ->uring_cmd_iopoll(). For theseuring_cmdrequests, skip initializing struct io_kiocb's iopoll fields, don't insert the request intoiopoll_list, and take theio_req_complete_defer()orio_req_task_work_add()path in__io_uring_cmd_done()instead of setting theiopoll_completedflag. Also allowio_uring_cmd_mark_cancelable()to be called on theseuring_cmds. Assert thatio_uring_cmd_mark_cancelable()is only called on non-IORING_SETUP_IOPOLLio_uringsoruring_cmdsto files that don't implement ->uring_cmd_iopoll().Signed-off-by Caleb Sander Mateos <[email protected]>--- io_uring/io_uring.c | 4 +++- io_uring/uring_cmd.c | 11 +++++------ 2 files changed, 8 insertions(+), 7 deletions(-)io_uring/io_uring.c +3
-1@@ -1417,11 +1417,13 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)if (ret ==IOU_ISSUE_SKIP_COMPLETE ) { ret =0 ;/* If the op doesn't have a file, we're not polling for it */ -if ((req->ctx->flags &IORING_SETUP_IOPOLL ) && def->iopoll_queue) +if ((req->ctx->flags &IORING_SETUP_IOPOLL ) && + def->iopoll_queue && (!io_is_uring_cmd (req) || + req->file->f_op->uring_cmd_iopoll))io_iopoll_req_issued (req, issue_flags); }return ret; }io_uring/uring_cmd.c +2
-2@@ -108,12 +108,12 @@ void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd,* Doing cancelations on IOPOLL requests are not supported. Both * because they can't get canceled in the block stack, but also * because iopoll completion data overlaps with the hash_node used * for tracking. */ -if (ctx->flags &IORING_SETUP_IOPOLL ) -return ; +WARN_ON_ONCE (ctx->flags &IORING_SETUP_IOPOLL && + req->file->f_op->uring_cmd_iopoll);if (!(cmd->flags &IORING_URING_CMD_CANCELABLE )) { cmd->flags |=IORING_URING_CMD_CANCELABLE ;io_ring_submit_lock (ctx, issue_flags);hlist_add_head (&req->hash_node, &ctx->cancelable_uring_cmd);
io_uring/uring_cmd.c void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2, · line 171
Quoted 8 lines
io_uring/uring_cmd.c +2
-1@@ -165,11 +165,12 @@ void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2,if (req->ctx->flags &IORING_SETUP_CQE_MIXED ) req->cqe.flags |=IORING_CQE_F_32 ;io_req_set_cqe32_extra (req, res2,0 ); }io_req_uring_cleanup (req, issue_flags); -if (req->ctx->flags &IORING_SETUP_IOPOLL ) { +if (req->ctx->flags &IORING_SETUP_IOPOLL && + req->file->f_op->uring_cmd_iopoll) {
I do worry that the pointer chasing here may be expensive, ->file and ->f_op could both be uncached. Would it make sense to add a flag to req->flags to indicate whether a request should actually be IOPOLLed?
Thanks,
Caleb
Quoted 23 lines
io_uring/uring_cmd.c +1
-4/* order with io_iopoll_req_issued() checking ->iopoll_complete */ smp_store_release (&req->iopoll_completed,1 ); }else if (issue_flags &IO_URING_F_COMPLETE_DEFER ) {if (WARN_ON_ONCE (issue_flags &IO_URING_F_UNLOCKED ))return ;@@ -255,13 +256,11 @@ int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags) issue_flags |=IO_URING_F_SQE128 ;if (ctx->flags & (IORING_SETUP_CQE32 |IORING_SETUP_CQE_MIXED )) issue_flags |=IO_URING_F_CQE32 ;if (io_is_compat (ctx)) issue_flags |=IO_URING_F_COMPAT ; -if (ctx->flags &IORING_SETUP_IOPOLL ) { -if (!file->f_op->uring_cmd_iopoll) -return -EOPNOTSUPP ; +if (ctx->flags &IORING_SETUP_IOPOLL && file->f_op->uring_cmd_iopoll) { issue_flags |=IO_URING_F_IOPOLL ; req->iopoll_completed =0 ;if (ctx->flags &IORING_SETUP_HYBRID_IOPOLL ) {/* make sure every req only blocks once */ req->flags &= ~REQ_F_IOPOLL_STATE ; --2.45.2
Jens Axboe
maintainerQuoted 71 lines · Caleb Sander Mateos
Quoted 69 lines · On Thu, Feb 12, 2026 at 7:21?PM Caleb Sander Mateos <[email protected]>…
Currently, creating an
io_uringwithIORING_SETUP_IOPOLLrequires all requests issued to it to support iopoll. This prevents, for example, using ublk zero-copy together withIORING_SETUP_IOPOLL, as ublk zero-copy buffer registrations are performed using auring_cmd. There's no technical reason why these non-iopolluring_cmdscan't be supported. They will either complete synchronously or via an external mechanism that callsio_uring_cmd_done(), so they don't need to be polled.Allow
uring_cmdrequests to be issued toIORING_SETUP_IOPOLLio_uringseven if their files don't implement ->uring_cmd_iopoll(). For theseuring_cmdrequests, skip initializing struct io_kiocb's iopoll fields, don't insert the request intoiopoll_list, and take theio_req_complete_defer()orio_req_task_work_add()path in__io_uring_cmd_done()instead of setting theiopoll_completedflag. Also allowio_uring_cmd_mark_cancelable()to be called on theseuring_cmds. Assert thatio_uring_cmd_mark_cancelable()is only called on non-IORING_SETUP_IOPOLLio_uringsoruring_cmdsto files that don't implement ->uring_cmd_iopoll().Signed-off-by Caleb Sander Mateos <[email protected]>--- io_uring/io_uring.c | 4 +++- io_uring/uring_cmd.c | 11 +++++------ 2 files changed, 8 insertions(+), 7 deletions(-)io_uring/io_uring.c +3
-1@@ -1417,11 +1417,13 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)if (ret ==IOU_ISSUE_SKIP_COMPLETE ) { ret =0 ;/* If the op doesn't have a file, we're not polling for it */ -if ((req->ctx->flags &IORING_SETUP_IOPOLL ) && def->iopoll_queue) +if ((req->ctx->flags &IORING_SETUP_IOPOLL ) && + def->iopoll_queue && (!io_is_uring_cmd (req) || + req->file->f_op->uring_cmd_iopoll))io_iopoll_req_issued (req, issue_flags); }return ret; }io_uring/uring_cmd.c +2
-2@@ -108,12 +108,12 @@ void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd,* Doing cancelations on IOPOLL requests are not supported. Both * because they can't get canceled in the block stack, but also * because iopoll completion data overlaps with the hash_node used * for tracking. */ -if (ctx->flags &IORING_SETUP_IOPOLL ) -return ; +WARN_ON_ONCE (ctx->flags &IORING_SETUP_IOPOLL && + req->file->f_op->uring_cmd_iopoll);if (!(cmd->flags &IORING_URING_CMD_CANCELABLE )) { cmd->flags |=IORING_URING_CMD_CANCELABLE ;io_ring_submit_lock (ctx, issue_flags);hlist_add_head (&req->hash_node, &ctx->cancelable_uring_cmd);@@ -165,11 +165,12 @@ void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2,if (req->ctx->flags &IORING_SETUP_CQE_MIXED ) req->cqe.flags |=IORING_CQE_F_32 ;io_req_set_cqe32_extra (req, res2,0 ); }
Quoted 8 lines
io_uring/uring_cmd.c void __io_uring_cmd_done(struct io_uring_cmd *ioucmd, s32 ret, u64 res2, · line 171
Quoted 4 lines
io_uring/uring_cmd.c +2
-1io_req_uring_cleanup (req, issue_flags); -if (req->ctx->flags &IORING_SETUP_IOPOLL ) { +if (req->ctx->flags &IORING_SETUP_IOPOLL && + req->file->f_op->uring_cmd_iopoll) {I do worry that the pointer chasing here may be expensive, ->file and ->f_op could both be uncached. Would it make sense to add a flag to
req->flagsto indicate whether a request should actually be IOPOLLed?
I think adding a REQ_F flag for that similar to what is done for NOWAIT etc would be a good idea.
Signature · 1 line
Jens Axboe
PATCH 3/3 nvme: remove nvme_dev_uring_cmd() IO_URING_F_IOPOLL check
In reply to Caleb Sander Mateosnvme_dev_uring_cmd() is part of struct file_operations nvme_dev_fops, which doesn't implement ->uring_cmd_iopoll(). So it won't be called with issue_flags that include IO_URING_F_IOPOLL. Drop the unnecessary IO_URING_F_IOPOLL check in nvme_dev_uring_cmd().
--- drivers/nvme/host/ioctl.c | 4 ---- 1 file changed, 4 deletions(-)
drivers/nvme/host/ioctl.c +0 -4
@@ -783,14 +783,10 @@ int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd,int nvme_dev_uring_cmd (struct io_uring_cmd *ioucmd,unsigned int issue_flags) {struct nvme_ctrl *ctrl = ioucmd->file->private_data;int ret; -/* IOPOLL not supported yet */ -if (issue_flags &IO_URING_F_IOPOLL ) -return -EOPNOTSUPP ; - ret =nvme_uring_cmd_checks (issue_flags);if (ret)return ret;switch (ioucmd->cmd_op) {
Signature · 1 line
2.45.2