io_uring: fix IOPOLL with passthrough I/O
Jens Axboe
maintainerPATCH v2 1/1 io_uring: fix IOPOLL with passthrough I/O
A previous commit improving IOPOLL made an incorrect assumption that task_work isn't used with IOPOLL. This can cause crashes when doing passthrough I/O on nvme, where queueing the completion task_work will trample on the same memory that holds the completed list of requests.
Fix it up by shuffling the members around, so we're not sharing any parts that end up getting used in this path.
---
v2: ensure ->iopoll_start is read before doing actual polling
include/linux/io_uring_types.h +4 -7
@@ -713,13 +713,10 @@ struct io_kiocb {atomic_t refs;bool cancel_seq_set; -/* -* IOPOLL doesn't use task_work, so use the ->iopoll_node list -* entry to manage pending iopoll requests. -*/ union {struct io_task_work io_task_work; -struct list_head iopoll_node; +/* For IOPOLL setup queues, with hybrid polling */ +u64 iopoll_start; };union {
@@ -728,8 +725,8 @@ struct io_kiocb {* poll */ struct hlist_node hash_node; -/* For IOPOLL setup queues, with hybrid polling */ -u64 iopoll_start; +/* IOPOLL completion handling */ +struct list_head iopoll_node;/* for private io_kiocb freeing */ struct rcu_head rcu_head; };
io_uring/rw.c +3 -2
@@ -1296,12 +1296,13 @@ static int io_uring_hybrid_poll(struct io_kiocb *req, struct io_comp_batch *iob, unsigned int poll_flags) { struct io_ring_ctx *ctx = req->ctx; -u64 runtime, sleep_time; +u64 runtime, sleep_time, iopoll_start;int ret; + iopoll_start =READ_ONCE (req->iopoll_start); sleep_time =io_hybrid_iopoll_delay (ctx, req); ret =io_uring_classic_poll (req, iob, poll_flags); - runtime =ktime_get_ns () - req->iopoll_start - sleep_time; + runtime =ktime_get_ns () - iopoll_start - sleep_time;/* * Use minimum sleep time if we're polling devices with different
Signature · 1 line
Jens Axboe
Quoted 8 lines · Jens Axboe
A previous commit improving IOPOLL made an incorrect assumption that
task_workisn't used with IOPOLL. This can cause crashes when doing passthrough I/O on nvme, where queueing the completiontask_workwill trample on the same memory that holds the completed list of requests.Fix it up by shuffling the members around, so we're not sharing any parts that end up getting used in this path.
I tried the v2 and confirmed the issue was fixed:
# ./check nvme/049
nvme/049 => nvme0n1 (basic test for uring-passthrough I/O on /dev/ngX) [passed]
runtime ... 7.991s
nvme/049 => nvme1n1 (basic test for uring-passthrough I/O on /dev/ngX) [passed]
runtime ... 7.970s
nvme/049 => nvme2n1 (basic test for uring-passthrough I/O on /dev/ngX) [passed]
runtime ... 7.965s
nvme/049 => nvme3n1 (basic test for uring-passthrough I/O on /dev/ngX) [passed]
runtime ... 7.975s
nvme/049 => nvme4n1 (basic test for uring-passthrough I/O on /dev/ngX) [passed]
runtime ... 8.003s
nvme/049 => nvme5n1 (basic test for uring-passthrough I/O on /dev/ngX) [passed]
runtime ... 7.999sQuoted 65 lines
Fixes 3c7d76d6128a ("io_uring: IOPOLL polling improvements")Reported-by Yi Zhang <[email protected]>Link https://lore.kernel.org/linux-block/CAHj4cs_SLPj9v9w5MgfzHKy+983enPx3ZQY2kMuMJ1202DBefw@mail.gmail.com/ open hereCc Ming Lei <[email protected]>Signed-off-by Jens Axboe <[email protected]>---
v2: ensure ->iopoll_start is read before doing actual polling
include/linux/io_uring_types.h +4
-7@@ -713,13 +713,10 @@ struct io_kiocb {atomic_t refs;bool cancel_seq_set; -/* -* IOPOLL doesn't use task_work, so use the ->iopoll_node list -* entry to manage pending iopoll requests. -*/ union {struct io_task_work io_task_work; -struct list_head iopoll_node; +/* For IOPOLL setup queues, with hybrid polling */ +u64 iopoll_start; };union {@@ -728,8 +725,8 @@ struct io_kiocb {* poll */ struct hlist_node hash_node; -/* For IOPOLL setup queues, with hybrid polling */ -u64 iopoll_start; +/* IOPOLL completion handling */ +struct list_head iopoll_node;/* for private io_kiocb freeing */ struct rcu_head rcu_head; };io_uring/rw.c +3
-3@@ -1296,12 +1296,13 @@ static int io_uring_hybrid_poll(struct io_kiocb *req, struct io_comp_batch *iob, unsigned int poll_flags) { struct io_ring_ctx *ctx = req->ctx; -u64 runtime, sleep_time; +u64 runtime, sleep_time, iopoll_start;int ret; + iopoll_start =READ_ONCE (req->iopoll_start); sleep_time =io_hybrid_iopoll_delay (ctx, req); ret =io_uring_classic_poll (req, iob, poll_flags); - runtime =ktime_get_ns () - req->iopoll_start - sleep_time; + runtime =ktime_get_ns () - iopoll_start - sleep_time;/* * Use minimum sleep time if we're polling devices with different -- Jens Axboe
Signature · 2 lines
Best Regards, Yi Zhang
Quoted 14 lines · Jens Axboe
A previous commit improving IOPOLL made an incorrect assumption that
task_workisn't used with IOPOLL. This can cause crashes when doing passthrough I/O on nvme, where queueing the completiontask_workwill trample on the same memory that holds the completed list of requests.Fix it up by shuffling the members around, so we're not sharing any parts that end up getting used in this path.
Fixes 3c7d76d6128a ("io_uring: IOPOLL polling improvements")Reported-by Yi Zhang <[email protected]>Link https://lore.kernel.org/linux-block/CAHj4cs_SLPj9v9w5MgfzHKy+983enPx3ZQY2kMuMJ1202DBefw@mail.gmail.com/ open hereCc Ming Lei <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Quoted 3 lines
---
v2: ensure ->iopoll_start is read before doing actual polling
Looks fine, also not see regression in ublk selftest:
Thanks,
Ming