io_uring: fix IOPOLL with passthrough I/O
Jens Axboe
maintainerPATCH 1/1 io_uring: fix IOPOLL with passthrough I/O
A previous commit improving IOPOLL made an incorrect assumption that task_work isn't used with IOPOLL. This can cause crashes when doing passthrough I/O on nvme, where queueing the completion task_work will trample on the same memory that holds the completed list of requests.
Fix it up by shuffling the members around, so we're not sharing any parts that end up getting used in this path.
---
include/linux/io_uring_types.h +4 -7
@@ -713,13 +713,10 @@ struct io_kiocb {atomic_t refs;bool cancel_seq_set; -/* -* IOPOLL doesn't use task_work, so use the ->iopoll_node list -* entry to manage pending iopoll requests. -*/ union {struct io_task_work io_task_work; -struct list_head iopoll_node; +/* For IOPOLL setup queues, with hybrid polling */ +u64 iopoll_start; };union {
@@ -728,8 +725,8 @@ struct io_kiocb {* poll */ struct hlist_node hash_node; -/* For IOPOLL setup queues, with hybrid polling */ -u64 iopoll_start; +/* IOPOLL completion handling */ +struct list_head iopoll_node;/* for private io_kiocb freeing */ struct rcu_head rcu_head; };
Signature · 1 line
Jens Axboe
Quoted 36 lines · Jens Axboe
A previous commit improving IOPOLL made an incorrect assumption that
task_workisn't used with IOPOLL. This can cause crashes when doing passthrough I/O on nvme, where queueing the completiontask_workwill trample on the same memory that holds the completed list of requests.Fix it up by shuffling the members around, so we're not sharing any parts that end up getting used in this path.
Fixes 3c7d76d6128a ("io_uring: IOPOLL polling improvements")Reported-by Yi Zhang <[email protected]>Link https://lore.kernel.org/linux-block/CAHj4cs_SLPj9v9w5MgfzHKy+983enPx3ZQY2kMuMJ1202DBefw@mail.gmail.com/ open hereSigned-off-by Jens Axboe <[email protected]>---
include/linux/io_uring_types.h +2
-5@@ -713,13 +713,10 @@ struct io_kiocb {atomic_t refs;bool cancel_seq_set; -/* -* IOPOLL doesn't use task_work, so use the ->iopoll_node list -* entry to manage pending iopoll requests. -*/ union {struct io_task_work io_task_work; -struct list_head iopoll_node; +/* For IOPOLL setup queues, with hybrid polling */ +u64 iopoll_start; };union {
include/linux/io_uring_types.h struct io_kiocb { · line 731
Quoted 9 lines
include/linux/io_uring_types.h +2
-2@@ -728,8 +725,8 @@ struct io_kiocb {* poll */ struct hlist_node hash_node; -/* For IOPOLL setup queues, with hybrid polling */ -u64 iopoll_start; +/* IOPOLL completion handling */ +struct list_head iopoll_node;/* for private io_kiocb freeing */ struct rcu_head rcu_head;
->hash_node is used by uring_cmd in io_uring_cmd_mark_cancelable()/io_uring_cmd_del_cancelable(), so this way may break uring_cmd if supporting iopoll and cancelable in future.
Thanks,
Ming
Jens Axboe
maintainerQuoted 35 lines · Ming Lei
Quoted 34 lines · Jens Axboe
A previous commit improving IOPOLL made an incorrect assumption that
task_workisn't used with IOPOLL. This can cause crashes when doing passthrough I/O on nvme, where queueing the completiontask_workwill trample on the same memory that holds the completed list of requests.Fix it up by shuffling the members around, so we're not sharing any parts that end up getting used in this path.
Fixes 3c7d76d6128a ("io_uring: IOPOLL polling improvements")Reported-by Yi Zhang <[email protected]>Link https://lore.kernel.org/linux-block/CAHj4cs_SLPj9v9w5MgfzHKy+983enPx3ZQY2kMuMJ1202DBefw@mail.gmail.com/ open hereSigned-off-by Jens Axboe <[email protected]>---
include/linux/io_uring_types.h +2
-5@@ -713,13 +713,10 @@ struct io_kiocb {atomic_t refs;bool cancel_seq_set; -/* -* IOPOLL doesn't use task_work, so use the ->iopoll_node list -* entry to manage pending iopoll requests. -*/ union {struct io_task_work io_task_work; -struct list_head iopoll_node; +/* For IOPOLL setup queues, with hybrid polling */ +u64 iopoll_start; };
Quoted 15 lines
include/linux/io_uring_types.h struct io_kiocb { · line 731
Quoted 11 lines
include/linux/io_uring_types.h +2
-2union {@@ -728,8 +725,8 @@ struct io_kiocb {* poll */ struct hlist_node hash_node; -/* For IOPOLL setup queues, with hybrid polling */ -u64 iopoll_start; +/* IOPOLL completion handling */ +struct list_head iopoll_node;/* for private io_kiocb freeing */ struct rcu_head rcu_head;->hash_node is used by
uring_cmdin io_uring_cmd_mark_cancelable()/io_uring_cmd_del_cancelable(), so this way may breakuring_cmdif supporting iopoll and cancelable in future.
We don't support cancelation on requests that go via the block stack, never have and probably never will. But I should make a comment about that, just in case...
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 52 lines · Jens Axboe
Quoted 35 lines · Ming Lei
Quoted 34 lines · Jens Axboe
A previous commit improving IOPOLL made an incorrect assumption that
task_workisn't used with IOPOLL. This can cause crashes when doing passthrough I/O on nvme, where queueing the completiontask_workwill trample on the same memory that holds the completed list of requests.Fix it up by shuffling the members around, so we're not sharing any parts that end up getting used in this path.
Fixes 3c7d76d6128a ("io_uring: IOPOLL polling improvements")Reported-by Yi Zhang <[email protected]>Link https://lore.kernel.org/linux-block/CAHj4cs_SLPj9v9w5MgfzHKy+983enPx3ZQY2kMuMJ1202DBefw@mail.gmail.com/ open hereSigned-off-by Jens Axboe <[email protected]>---
include/linux/io_uring_types.h +2
-5@@ -713,13 +713,10 @@ struct io_kiocb {atomic_t refs;bool cancel_seq_set; -/* -* IOPOLL doesn't use task_work, so use the ->iopoll_node list -* entry to manage pending iopoll requests. -*/ union {struct io_task_work io_task_work; -struct list_head iopoll_node; +/* For IOPOLL setup queues, with hybrid polling */ +u64 iopoll_start; };Quoted 15 lines
include/linux/io_uring_types.h struct io_kiocb { · line 731
Quoted 11 lines
include/linux/io_uring_types.h +2
-2union {@@ -728,8 +725,8 @@ struct io_kiocb {* poll */ struct hlist_node hash_node; -/* For IOPOLL setup queues, with hybrid polling */ -u64 iopoll_start; +/* IOPOLL completion handling */ +struct list_head iopoll_node;/* for private io_kiocb freeing */ struct rcu_head rcu_head;->hash_node is used by
uring_cmdin io_uring_cmd_mark_cancelable()/io_uring_cmd_del_cancelable(), so this way may breakuring_cmdif supporting iopoll and cancelable in future.
Quoted 3 lines
We don't support cancelation on requests that go via the block stack, never have and probably never will. But I should make a comment about that, just in case...
Should be trivial enough to just explicitly disallow it.
io_uring/uring_cmd.c +9 -0
@@ -104,6 +104,15 @@ void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd,struct io_kiocb *req =cmd_to_io_kiocb (cmd);struct io_ring_ctx *ctx = req->ctx; +/* +* Doing cancelations on IOPOLL requests are not supported. Both +* because they can't get canceled in the block stack, but also +* because iopoll completion data overlaps with the hash_node used +* for tracking. +*/ +if (ctx->flags &IORING_SETUP_IOPOLL ) +return ; +if (!(cmd->flags &IORING_URING_CMD_CANCELABLE )) { cmd->flags |=IORING_URING_CMD_CANCELABLE ;io_ring_submit_lock (ctx, issue_flags);
Signature · 1 line
Jens Axboe