selected buffers recycling fixes
cover selected buffers recycling fixes
Fix two locking problems with new buffer recycling.
Jens, could you help to test it properly and with lockdep enabled?
Pavel Begunkov (2):
io_uring: fix invalid flags for io_put_kbuf()
io_uring: fix put_kbuf without proper locking
fs/io_uring.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
Signature · 1 line
2.35.1
PATCH 1/2 io_uring: fix invalid flags for io_put_kbuf()
io_req_complete_failed() doesn't require callers to hold ->uring_lock, use IO_URING_F_UNLOCKED version of io_put_kbuf(). The only affected place is the fail path of io_apoll_task_func(). Also add a lockdep annotation to catch such bugs in the future.
--- fs/io_uring.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
fs/io_uring.c +3 -1
@@ -1388,6 +1388,8 @@ static inline unsigned int io_put_kbuf(struct io_kiocb *req, cflags =__io_put_kbuf (req, &ctx->io_buffers_comp);spin_unlock (&ctx->completion_lock); }else { +lockdep_assert_held (&req->ctx->uring_lock); + cflags =__io_put_kbuf (req, &req->ctx->io_buffers_cache); }
@@ -2182,7 +2184,7 @@ static inline void io_req_complete(struct io_kiocb *req, s32 res)static void io_req_complete_failed (struct io_kiocb *req,s32 res) {req_set_fail (req); -io_req_complete_post (req, res,io_put_kbuf (req,0 )); +io_req_complete_post (req, res,io_put_kbuf (req,IO_URING_F_UNLOCKED )); }static void io_req_complete_fail_submit (struct io_kiocb *req)
Signature · 1 line
2.35.1
PATCH 2/2 io_uring: fix put_kbuf without proper locking
In reply to Pavel Begunkovio_put_kbuf_comp() should only be called while holding ->completion_lock, however there is no such assumption in io_clean_op() and thus it can corrupt ->io_buffer_comp. Take the lock there, and workaround the only user of io_clean_op() calling it with locks. Not the prettiest solution, but it's easier to refactor it for-next.
--- fs/io_uring.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
fs/io_uring.c +12 -1
@@ -1356,6 +1356,8 @@ static unsigned int __io_put_kbuf(struct io_kiocb *req, struct list_head *list)static inline unsigned int io_put_kbuf_comp (struct io_kiocb *req) { +lockdep_assert_held (&req->ctx->completion_lock); +if (likely (!(req->flags &REQ_F_BUFFER_SELECTED )))return 0 ;return __io_put_kbuf (req, &req->ctx->io_buffers_comp);
@@ -2140,6 +2142,12 @@ static void __io_req_complete_post(struct io_kiocb *req, s32 res, } }io_req_put_rsrc (req, ctx); +/* +* Selected buffer deallocation in io_clean_op() assumes that +* we don't hold ->completion_lock. Clean them here to avoid +* deadlocks. +*/ +io_put_kbuf_comp (req);io_dismantle_req (req);io_put_task (req->task,1 );wq_list_add_head (&req->comp_list, &ctx->locked_free_list);
@@ -7115,8 +7123,11 @@ static __cold void io_drain_req(struct io_kiocb *req)static void io_clean_op (struct io_kiocb *req) { -if (req->flags &REQ_F_BUFFER_SELECTED ) +if (req->flags &REQ_F_BUFFER_SELECTED ) { +spin_lock (&req->ctx->completion_lock);io_put_kbuf_comp (req); +spin_unlock (&req->ctx->completion_lock); + }if (req->flags &REQ_F_NEED_CLEANUP ) {switch (req->opcode) {
Signature · 1 line
2.35.1
Jens Axboe
maintainerRe: [PATCH 1/2] io_uring: fix invalid flags for io_put_kbuf()
In reply to Pavel BegunkovQuoted 25 lines · Pavel Begunkov
io_req_complete_failed()doesn't require callers to hold ->uring_lock, useIO_URING_F_UNLOCKEDversion ofio_put_kbuf(). The only affected place is the fail path ofio_apoll_task_func(). Also add a lockdep annotation to catch such bugs in the future.Fixes 3b2b78a8eb7cc ("io_uring: extend provided buf return to fails")Signed-off-by Pavel Begunkov <[email protected]>--- fs/io_uring.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)fs/io_uring.c +2
-0@@ -1388,6 +1388,8 @@ static inline unsigned int io_put_kbuf(struct io_kiocb *req, cflags =__io_put_kbuf (req, &ctx->io_buffers_comp);spin_unlock (&ctx->completion_lock); }else { +lockdep_assert_held (&req->ctx->uring_lock); + cflags =__io_put_kbuf (req, &req->ctx->io_buffers_cache); }
fs/io_uring.c static inline void io_req_complete(struct io_kiocb *req, s32 res) · line 2188
Quoted 6 lines
fs/io_uring.c +1
-1@@ -2182,7 +2184,7 @@ static inline void io_req_complete(struct io_kiocb *req, s32 res)static void io_req_complete_failed (struct io_kiocb *req,s32 res) {req_set_fail (req); -io_req_complete_post (req, res,io_put_kbuf (req,0 )); +io_req_complete_post (req, res,io_put_kbuf (req,IO_URING_F_UNLOCKED )); }
That took me a bit to grok, but it's because we don't use the flag here as "ok we need to grab that same lock, rather we use it for "should we use something else" which makes it safe. We should probably use a bool for this case instead rather than abuse issue_flags, for later.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 3 lines · Pavel Begunkov
Fix two locking problems with new buffer recycling.
Jens, could you help to test it properly and with lockdep enabled?
Yep, I'll run them through the testing, thanks!
Signature · 1 line
Jens Axboe
Quoted 9 lines · Pavel Begunkov
Fix two locking problems with new buffer recycling.
Jens, could you help to test it properly and with lockdep enabled?
Pavel Begunkov (2):
io_uring: fix invalid flags forio_put_kbuf()
io_uring: fixput_kbufwithout proper locking[...]
Applied, thanks!
[1/2] io_uring: fix invalid flags for io_put_kbuf()
commit: ab0ac0959b028779ea43002db81daa12203cb57d
[2/2] io_uring: fix put_kbuf without proper locking
commit: 8197b053a83335dd1b7eb7581a933924e25c1025Best regards,
Signature · 1 line
Jens Axboe