io_uring: hold uring_lock to complete faild polled io in io_wq_submit_work()
PATCH 1/1 io_uring: hold uring_lock to complete faild polled io in io_wq_submit_work()
io_iopoll_complete() does not hold completion_lock to complete polled io, so in io_wq_submit_work(), we can not call io_req_complete() directly, to complete polled io, otherwise there maybe concurrent access to cqring, defer_list, etc, which is not safe. Commit dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unreg requests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.
Given that io_iopoll_complete() is always called under uring_lock, so here for polled io, we can also get uring_lock to fix this issue.
--- fs/io_uring.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-)
fs/io_uring.c +15 -10
@@ -6354,19 +6354,24 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work) }if (ret) { +bool iopoll_enabled = req->ctx->flags &IORING_SETUP_IOPOLL ; +/* -* io_iopoll_complete() does not hold completion_lock to complete -* polled io, so here for polled io, just mark it done and still let -* io_iopoll_complete() complete it. +* io_iopoll_complete() does not hold completion_lock to complete polled +* io, so here for polled io, we can not call io_req_complete() directly, +* otherwise there maybe concurrent access to cqring, defer_list, etc, +* which is not safe. Given that io_iopoll_complete() is always called +* under uring_lock, so here for polled io, we also get uring_lock to +* complete it. */ -if (req->ctx->flags &IORING_SETUP_IOPOLL ) { -struct kiocb *kiocb = &req->rw.kiocb; +if (iopoll_enabled) +mutex_lock (&req->ctx->uring_lock); -kiocb_done (kiocb, ret,NULL ); - }else { -req_set_fail_links (req); -io_req_complete (req, ret); - } +req_set_fail_links (req); +io_req_complete (req, ret); + +if (iopoll_enabled) +mutex_unlock (&req->ctx->uring_lock); }return io_steal_work (req);
Signature · 1 line
2.17.2
Quoted 11 lines · Xiaoguang Wang
io_iopoll_complete()does not holdcompletion_lockto complete polled io, so inio_wq_submit_work(), we can not callio_req_complete()directly, to complete polled io, otherwise there maybe concurrent access to cqring,defer_list, etc, which is not safe. Commitdad1b1242fd5("io_uring: always letio_iopoll_complete()complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unregrequests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.Given that
io_iopoll_complete()is always called underuring_lock, so here for polled io, we can also geturing_lockto fix this issue.
One thing I don't like is that io_wq_submit_work() won't be able to publish an event while someone polling io_uring_enter(ENTER_GETEVENTS), that's because both take the lock. The problem is when the poller waits for an event that is currently in io-wq (i.e. io_wq_submit_work()). The polling loop will eventually exit, so that's not a deadlock, but latency,etc. would be huge.
Quoted 47 lines
Fixes dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io")Signed-off-by Xiaoguang Wang <[email protected]>--- fs/io_uring.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-)fs/io_uring.c +15
-10@@ -6354,19 +6354,24 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work) }if (ret) { +bool iopoll_enabled = req->ctx->flags &IORING_SETUP_IOPOLL ; +/* -* io_iopoll_complete() does not hold completion_lock to complete -* polled io, so here for polled io, just mark it done and still let -* io_iopoll_complete() complete it. +* io_iopoll_complete() does not hold completion_lock to complete polled +* io, so here for polled io, we can not call io_req_complete() directly, +* otherwise there maybe concurrent access to cqring, defer_list, etc, +* which is not safe. Given that io_iopoll_complete() is always called +* under uring_lock, so here for polled io, we also get uring_lock to +* complete it. */ -if (req->ctx->flags &IORING_SETUP_IOPOLL ) { -struct kiocb *kiocb = &req->rw.kiocb; +if (iopoll_enabled) +mutex_lock (&req->ctx->uring_lock); -kiocb_done (kiocb, ret,NULL ); - }else { -req_set_fail_links (req); -io_req_complete (req, ret); - } +req_set_fail_links (req); +io_req_complete (req, ret); + +if (iopoll_enabled) +mutex_unlock (&req->ctx->uring_lock); }return io_steal_work (req);
Signature · 1 line
Pavel Begunkov
hi,
Quoted 13 lines · Xiaoguang Wang
Quoted 11 lines
io_iopoll_complete()does not holdcompletion_lockto complete polled io, so inio_wq_submit_work(), we can not callio_req_complete()directly, to complete polled io, otherwise there maybe concurrent access to cqring,defer_list, etc, which is not safe. Commitdad1b1242fd5("io_uring: always letio_iopoll_complete()complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unregrequests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.Given that
io_iopoll_complete()is always called underuring_lock, so here for polled io, we can also geturing_lockto fix this issue.
Quoted 6 lines
One thing I don't like is that
io_wq_submit_work()won't be able to publish an event while someone pollingio_uring_enter(ENTER_GETEVENTS), that's because both take the lock. The problem is when the poller waits for an event that is currently in io-wq (i.e.io_wq_submit_work()). The polling loop will eventually exit, so that's not a deadlock, but latency,etc. would be huge.
In this patch, we just hold uring_lock for polled io in error path, so I think normally it maybe not an issue, and seems that the critical section is not that big, so it also may not result in huge latecy.
I also noticed that current codes also hold uring_lock in io_wq_submit_work() call chain:
==> io_wq_submit_work()
====> io_issue_sqe()
======> io_provide_buffers()
========> io_ring_submit_lock(ctx, !force_nonblock);
Regards,
Xiaoguang Wang
Quoted 49 lines
Quoted 48 lines
Fixes dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io")Signed-off-by Xiaoguang Wang <[email protected]>--- fs/io_uring.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-)fs/io_uring.c +15
-10@@ -6354,19 +6354,24 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work) }if (ret) { +bool iopoll_enabled = req->ctx->flags &IORING_SETUP_IOPOLL ; +/* -* io_iopoll_complete() does not hold completion_lock to complete -* polled io, so here for polled io, just mark it done and still let -* io_iopoll_complete() complete it. +* io_iopoll_complete() does not hold completion_lock to complete polled +* io, so here for polled io, we can not call io_req_complete() directly, +* otherwise there maybe concurrent access to cqring, defer_list, etc, +* which is not safe. Given that io_iopoll_complete() is always called +* under uring_lock, so here for polled io, we also get uring_lock to +* complete it. */ -if (req->ctx->flags &IORING_SETUP_IOPOLL ) { -struct kiocb *kiocb = &req->rw.kiocb; +if (iopoll_enabled) +mutex_lock (&req->ctx->uring_lock); -kiocb_done (kiocb, ret,NULL ); - }else { -req_set_fail_links (req); -io_req_complete (req, ret); - } +req_set_fail_links (req); +io_req_complete (req, ret); + +if (iopoll_enabled) +mutex_unlock (&req->ctx->uring_lock); }return io_steal_work (req);
Quoted 21 lines · Xiaoguang Wang
hi,
Quoted 13 lines · Xiaoguang Wang
Quoted 11 lines
io_iopoll_complete()does not holdcompletion_lockto complete polled io, so inio_wq_submit_work(), we can not callio_req_complete()directly, to complete polled io, otherwise there maybe concurrent access to cqring,defer_list, etc, which is not safe. Commitdad1b1242fd5("io_uring: always letio_iopoll_complete()complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unregrequests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.Given that
io_iopoll_complete()is always called underuring_lock, so here for polled io, we can also geturing_lockto fix this issue.Quoted 6 lines
One thing I don't like is that
io_wq_submit_work()won't be able to publish an event while someone pollingio_uring_enter(ENTER_GETEVENTS), that's because both take the lock. The problem is when the poller waits for an event that is currently in io-wq (i.e.io_wq_submit_work()). The polling loop will eventually exit, so that's not a deadlock, but latency,etc. would be huge.
Quoted 3 lines
In this patch, we just hold
uring_lockfor polled io in error path, so I think normally it maybe not an issue, and seems that the critical section is not that big, so it also may not result in huge latecy.
To give a bit of a context, it breaks out of the io_iopoll_check()'s loop with need_resched(). Anyway, fair enough, probably should be dealt separately.
Quoted 6 lines
I also noticed that current codes also hold
uring_lockinio_wq_submit_work()call chain:
==>io_wq_submit_work()
====>io_issue_sqe()
======>io_provide_buffers()
========>io_ring_submit_lock(ctx, !force_nonblock);
Yep, that's not good either, though I care more about rw.
Quoted 49 lines
Regards,
Xiaoguang WangQuoted 46 lines
Quoted 45 lines
Fixes dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io")Signed-off-by Xiaoguang Wang <[email protected]>---
fs/io_uring.c| 25 +++++++++++++++----------1 file changed, 15 insertions(+), 10 deletions(-)fs/io_uring.c +15
-9@@ -6354,19 +6354,24 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work) } if (ret) { + bool iopoll_enabled = req->ctx->flags & IORING_SETUP_IOPOLL; + /* - * io_iopoll_complete() does not hold completion_lock to complete - * polled io, so here for polled io, just mark it done and still let - * io_iopoll_complete() complete it. + * io_iopoll_complete() does not hold completion_lock to complete polled + * io, so here for polled io, we can not call io_req_complete() directly, + * otherwise there maybe concurrent access to cqring, defer_list, etc, + * which is not safe. Given that io_iopoll_complete() is always called + * under uring_lock, so here for polled io, we also get uring_lock to + * complete it. */ - if (req->ctx->flags & IORING_SETUP_IOPOLL) { - struct kiocb *kiocb = &req->rw.kiocb; + if (iopoll_enabled) + mutex_lock(&req->ctx->uring_lock); - kiocb_done(kiocb, ret, NULL); - } else { - req_set_fail_links(req); - io_req_complete(req, ret); - } + req_set_fail_links(req); + io_req_complete(req, ret); + + if (iopoll_enabled) + mutex_unlock(&req->ctx->uring_lock); } return io_steal_work(req);
Signature · 1 line
Pavel Begunkov
Quoted 11 lines · Xiaoguang Wang
io_iopoll_complete()does not holdcompletion_lockto complete polled io, so inio_wq_submit_work(), we can not callio_req_complete()directly, to complete polled io, otherwise there maybe concurrent access to cqring,defer_list, etc, which is not safe. Commitdad1b1242fd5("io_uring: always letio_iopoll_complete()complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unregrequests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.Given that
io_iopoll_complete()is always called underuring_lock, so here for polled io, we can also geturing_lockto fix this issue.
This returns it to the state it was before fixing + mutex locking for IOPOLL, and it's much better than having it half-broken as it is now.
Quoted 47 lines
Fixes dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io")Signed-off-by Xiaoguang Wang <[email protected]>--- fs/io_uring.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-)fs/io_uring.c +15
-10@@ -6354,19 +6354,24 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work) }if (ret) { +bool iopoll_enabled = req->ctx->flags &IORING_SETUP_IOPOLL ; +/* -* io_iopoll_complete() does not hold completion_lock to complete -* polled io, so here for polled io, just mark it done and still let -* io_iopoll_complete() complete it. +* io_iopoll_complete() does not hold completion_lock to complete polled +* io, so here for polled io, we can not call io_req_complete() directly, +* otherwise there maybe concurrent access to cqring, defer_list, etc, +* which is not safe. Given that io_iopoll_complete() is always called +* under uring_lock, so here for polled io, we also get uring_lock to +* complete it. */ -if (req->ctx->flags &IORING_SETUP_IOPOLL ) { -struct kiocb *kiocb = &req->rw.kiocb; +if (iopoll_enabled) +mutex_lock (&req->ctx->uring_lock); -kiocb_done (kiocb, ret,NULL ); - }else { -req_set_fail_links (req); -io_req_complete (req, ret); - } +req_set_fail_links (req); +io_req_complete (req, ret); + +if (iopoll_enabled) +mutex_unlock (&req->ctx->uring_lock); }return io_steal_work (req);
Signature · 1 line
Pavel Begunkov
Quoted 10 lines · Pavel Begunkov
Quoted 9 lines · Xiaoguang Wang
io_iopoll_complete()does not holdcompletion_lockto complete polled io, so inio_wq_submit_work(), we can not callio_req_complete()directly, to complete polled io, otherwise there maybe concurrent access to cqring,defer_list, etc, which is not safe. Commitdad1b1242fd5("io_uring: always letio_iopoll_complete()complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unregrequests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.
Quoted 5 lines
Quoted 2 lines
Given that
io_iopoll_complete()is always called underuring_lock, so here for polled io, we can also geturing_lockto fix this issue.This returns it to the state it was before fixing + mutex locking for IOPOLL, and it's much better than having it half-broken as it is now.
btw, comments are over 80, but that's minor.
Quoted 52 lines
Cc <[email protected]> # 5.5+Reviewed-by Pavel Begunkov <[email protected]>Quoted 47 lines
Fixes dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io")Signed-off-by Xiaoguang Wang <[email protected]>--- fs/io_uring.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-)fs/io_uring.c +15
-10@@ -6354,19 +6354,24 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work) }if (ret) { +bool iopoll_enabled = req->ctx->flags &IORING_SETUP_IOPOLL ; +/* -* io_iopoll_complete() does not hold completion_lock to complete -* polled io, so here for polled io, just mark it done and still let -* io_iopoll_complete() complete it. +* io_iopoll_complete() does not hold completion_lock to complete polled +* io, so here for polled io, we can not call io_req_complete() directly, +* otherwise there maybe concurrent access to cqring, defer_list, etc, +* which is not safe. Given that io_iopoll_complete() is always called +* under uring_lock, so here for polled io, we also get uring_lock to +* complete it. */ -if (req->ctx->flags &IORING_SETUP_IOPOLL ) { -struct kiocb *kiocb = &req->rw.kiocb; +if (iopoll_enabled) +mutex_lock (&req->ctx->uring_lock); -kiocb_done (kiocb, ret,NULL ); - }else { -req_set_fail_links (req); -io_req_complete (req, ret); - } +req_set_fail_links (req); +io_req_complete (req, ret); + +if (iopoll_enabled) +mutex_unlock (&req->ctx->uring_lock); }return io_steal_work (req);
Signature · 1 line
Pavel Begunkov
Quoted 11 lines · Pavel Begunkov
Quoted 10 lines · Pavel Begunkov
Quoted 9 lines · Xiaoguang Wang
io_iopoll_complete()does not holdcompletion_lockto complete polled io, so inio_wq_submit_work(), we can not callio_req_complete()directly, to complete polled io, otherwise there maybe concurrent access to cqring,defer_list, etc, which is not safe. Commitdad1b1242fd5("io_uring: always letio_iopoll_complete()complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unregrequests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Given that
io_iopoll_complete()is always called underuring_lock, so here for polled io, we can also geturing_lockto fix this issue.This returns it to the state it was before fixing + mutex locking for IOPOLL, and it's much better than having it half-broken as it is now.
btw, comments are over 80, but that's minor.
I fixed that up, but I don't particularly like how 'req' is used after calling complete. How about the below variant - same as before, just using the ctx instead to determine if we need to lock it or not.
commit 253b60e7d8adcb980be91f77e64968a58d836b5e
Author: Xiaoguang Wang <[email protected]> Date: Mon Dec 14 23:49:41 2020 +0800
io_uring: hold uring_lock while completing failed polled io in io_wq_submit_work()
io_iopoll_complete() does not hold completion_lock to complete polled io, so in io_wq_submit_work(), we can not call io_req_complete() directly, to complete polled io, otherwise there maybe concurrent access to cqring, defer_list, etc, which is not safe. Commit dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unreg requests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.
Given that io_iopoll_complete() is always called under uring_lock, so here for polled io, we can also get uring_lock to fix this issue.
fs/io_uring.c +19 -10
@@ -6332,19 +6332,28 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work) }if (ret) { +struct io_ring_ctx *lock_ctx =NULL ; + +if (req->ctx->flags &IORING_SETUP_IOPOLL ) + lock_ctx = req->ctx; +/* -* io_iopoll_complete() does not hold completion_lock to complete -* polled io, so here for polled io, just mark it done and still let -* io_iopoll_complete() complete it. +* io_iopoll_complete() does not hold completion_lock to +* complete polled io, so here for polled io, we can not call +* io_req_complete() directly, otherwise there maybe concurrent +* access to cqring, defer_list, etc, which is not safe. Given +* that io_iopoll_complete() is always called under uring_lock, +* so here for polled io, we also get uring_lock to complete +* it. */ -if (req->ctx->flags &IORING_SETUP_IOPOLL ) { -struct kiocb *kiocb = &req->rw.kiocb; +if (lock_ctx) +mutex_lock (&lock_ctx->uring_lock); -kiocb_done (kiocb, ret,NULL ); - }else { -req_set_fail_links (req); -io_req_complete (req, ret); - } +req_set_fail_links (req); +io_req_complete (req, ret); + +if (lock_ctx) +mutex_unlock (&lock_ctx->uring_lock); }return io_steal_work (req);
Signature · 1 line
Jens Axboe
hi,
Quoted 15 lines · Pavel Begunkov
Quoted 14 lines · Pavel Begunkov
Quoted 13 lines · Xiaoguang Wang
Quoted 11 lines
io_iopoll_complete()does not holdcompletion_lockto complete polled io, so inio_wq_submit_work(), we can not callio_req_complete()directly, to complete polled io, otherwise there maybe concurrent access to cqring,defer_list, etc, which is not safe. Commitdad1b1242fd5("io_uring: always letio_iopoll_complete()complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unregrequests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.Given that
io_iopoll_complete()is always called underuring_lock, so here for polled io, we can also geturing_lockto fix this issue.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
This returns it to the state it was before fixing + mutex locking for IOPOLL, and it's much better than having it half-broken as it is now.
btw, comments are over 80, but that's minor.
I fixed that up, but I don't particularly like how 'req' is used after calling complete. How about the below variant - same as before, just using the ctx instead to determine if we need to lock it or not.
It looks better, thanks.
Regards,
Xiaoguang Wang
Quoted 71 lines
commit
253b60e7d8adcb980be91f77e64968a58d836b5eAuthor: Xiaoguang Wang <[email protected]> Date: Mon Dec 14 23:49:41 2020 +0800
io_uring: holduring_lockwhile completing failed polled io inio_wq_submit_work()
io_iopoll_complete()does not holdcompletion_lockto complete polled io, so inio_wq_submit_work(), we can not callio_req_complete()directly, to complete polled io, otherwise there maybe concurrent access to cqring,defer_list, etc, which is not safe. Commitdad1b1242fd5("io_uring: always letio_iopoll_complete()complete polled io") has fixed this issue, but Pavel reported that IOPOLL apart from rw can do buf reg/unregrequests( IORING_OP_PROVIDE_BUFFERS or IORING_OP_REMOVE_BUFFERS), so the fix is not good.Given that
io_iopoll_complete()is always called underuring_lock, so here for polled io, we can also geturing_lockto fix this issue.Fixes dad1b1242fd5 ("io_uring: always let io_iopoll_complete() complete polled io")Cc <[email protected]> # 5.5+Signed-off-by Xiaoguang Wang <[email protected]>Reviewed-by Pavel Begunkov <[email protected]>[axboe: don't deref 'req' after completing it']Signed-off-by Jens Axboe <[email protected]>fs/io_uring.c +19
-10@@ -6332,19 +6332,28 @@ static struct io_wq_work *io_wq_submit_work(struct io_wq_work *work) }if (ret) { +struct io_ring_ctx *lock_ctx =NULL ; + +if (req->ctx->flags &IORING_SETUP_IOPOLL ) + lock_ctx = req->ctx; +/* -* io_iopoll_complete() does not hold completion_lock to complete -* polled io, so here for polled io, just mark it done and still let -* io_iopoll_complete() complete it. +* io_iopoll_complete() does not hold completion_lock to +* complete polled io, so here for polled io, we can not call +* io_req_complete() directly, otherwise there maybe concurrent +* access to cqring, defer_list, etc, which is not safe. Given +* that io_iopoll_complete() is always called under uring_lock, +* so here for polled io, we also get uring_lock to complete +* it. */ -if (req->ctx->flags &IORING_SETUP_IOPOLL ) { -struct kiocb *kiocb = &req->rw.kiocb; +if (lock_ctx) +mutex_lock (&lock_ctx->uring_lock); -kiocb_done (kiocb, ret,NULL ); - }else { -req_set_fail_links (req); -io_req_complete (req, ret); - } +req_set_fail_links (req); +io_req_complete (req, ret); + +if (lock_ctx) +mutex_unlock (&lock_ctx->uring_lock); }return io_steal_work (req);